%\VignetteIndexEntry{micompr: An R Package for Multivariate Independent Comparison of Observations} %\VignetteEngine{knitr::knitr} \documentclass{article} \usepackage[utf8]{inputenc} \usepackage[margin=10pt,font=small,labelfont=bf,labelsep=endash]{caption} \usepackage[T1]{fontenc} \usepackage{booktabs} \usepackage{multirow} \usepackage{amsmath} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{color} \usepackage{algorithm} \usepackage{url} \usepackage[normalem]{ulem} \usepackage{siunitx} \usepackage[htt]{hyphenat} \usepackage{tikz} \usepackage{pgfplots} \usetikzlibrary{plotmarks} \usepackage{authblk} \usepackage{xspace} \usepackage[top=3cm, bottom=3cm, inner=2.5cm, outer=2.5cm]{geometry} \usepackage[absolute]{textpos} \RequirePackage[sectionbib,round]{natbib} \bibliographystyle{abbrvnat} \RequirePackage{fancyvrb} \RequirePackage{alltt} \DefineVerbatimEnvironment{example}{Verbatim}{} \renewenvironment{example*}{\begin{alltt}}{\end{alltt}} \providecommand{\href}{\url} \providecommand{\address}{} \providecommand{\email}{} \renewcommand\Affilfont{\itshape\small} \providecommand{\nameref}{\ref} % defs \newcommand{\statevar}{\texttt} \newcommand{\algfunc}{\texttt} \newcommand{\modparam}{\texttt} \newcommand{\progfunc}{\textnhtt} \newcommand{\swpackage}{\texttt} \providecommand{\pkg}{\swpackage} \providecommand{\code}{\progfunc} \providecommand{\CRANpkg}{\progfunc} \floatstyle{ruled} \newfloat{proc}{thp}{lop} \floatname{proc}{Procedure} \makeatletter \@ifpackageloaded{amsmath}{ \DeclareMathOperator*{\argmax}{arg\,max} \DeclareMathOperator*{\argmin}{arg\,min}} {} \makeatother \newcommand*\mean[1]{\overline{#1}} \newcommand{\TikZ}{Ti\textit{k}Z\xspace} \begin{document} \date{} \begin{textblock*}{202mm}(3mm,3mm) \noindent The peer-reviewed version of this vignette is published in The R Journal (\url{https://doi.org/10.32614/RJ-2016-055}). This version is typeset by the authors and differs only in pagination and typographical detail. \end{textblock*} \title{\pkg{micompr}: An R Package for Multivariate Independent Comparison of Observations} \author[1]{Nuno Fachada} \author[2]{Jo\~{a}o Rodrigues} \author[3]{Vitor V. Lopes} \author[4]{Rui C. Martins} \author[1]{Agostinho C. Rosa} \affil[1]{Institute for Systems and Robotics, LARSyS, Instituto Superior T\'{e}cnico, Universidade de Lisboa, Lisboa, Portugal} \affil[2]{\'{E}cole Polytechnique F\'{e}d\'{e}rale de Lausanne, Lausanne, Switzerland} \affil[3]{UTEC - Universidad de Ingenier\'{i}a \& Tecnolog\'{i}a, Lima, Per\'{u}} \affil[4]{Life and Health Sciences Research Institute, School of Health Sciences, University of Minho, Braga, Portugal} \maketitle \abstract{ The R package \pkg{micompr} implements a procedure for assessing if two or more multivariate samples are drawn from the same distribution. The procedure uses principal component analysis to convert multivariate observations into a set of linearly uncorrelated statistical measures, which are then compared using a number of statistical methods. This technique is independent of the distributional properties of samples and automatically selects features that best explain their differences. The procedure is appropriate for comparing samples of time series, images, spectrometric measures or similar high-dimension multivariate observations. } \section{Introduction}\label{micompr:sec:intro} The aim of this paper is to present the \CRANpkg{micompr} package for R \citep{r2015stats}, which implements a procedure for comparing multivariate samples associated with different factor levels or groups. The research goal is to differentiate among pre-specified, well-defined classes or groups of sampling entities generating highly multivariate observations in which the dimensions or dependent variables are correlated, and to test for significant differences among groups. The procedure uses principal component analysis (PCA) \citep{jolliffe2002principal} to convert multivariate observations into a set of linearly uncorrelated statistical measures, which are then compared using a number of statistical methods, such as hypothesis tests and score plots. This technique has several desirable attributes: a) it automatically selects observation features that best explain sample differences; b) it does not depend on the distributional properties of samples; and, c) it simplifies the researchers' work, as it can be used directly on multivariate observations. The procedure is appropriate for comparing samples of multivariate observations with highly correlated and similar scale dimensions, such as time series, images or spectrometric measures. However, the \pkg{micompr} package goes one step further by also accommodating the simultaneous comparison of multiple observation types, i.e., multiple \emph{outputs} from a given ``system''. In this context, a ``system'' can be defined as an abstract entity capable of generating one or more stochastic data streams, i.e., \emph{outputs}. Thus, \pkg{micompr} can determine if two or more instances of such a ``system'' display the same behavior by comparing observations of their \emph{outputs}. The remainder of this paper is organized as follows. First, in Section \nameref{micompr:sec:finding}, commonly used techniques for differentiating samples of multivariate observations are discussed. The methodology employed by \pkg{micompr} is described in Section \nameref{micompr:sec:theory}. Section \nameref{micompr:sec:micompr} introduces the software and its architecture, namely the available objects and functions. Several concrete application examples, and how the ``system''-\emph{output} terminology fits each one, are presented in Section \nameref{micompr:sec:examples}. The paper closes with Section \nameref{micompr:sec:summary}, in which the overall approach and the R package are summarized. \section{Testing for significant differences in multivariate samples}\label{micompr:sec:finding} Two-sample or multi-sample hypothesis tests are commonly used for assessing statistically dissimilarity in univariate samples, i.e., samples composed of scalar observations. If samples are drawn from normally distributed populations, the $t$ (two samples) and ANOVA ($n$-samples) tests are adequate \citep{montgomery2010applied}. Non-parametric tests are more appropriate if population normality cannot be assumed. The Mann-Whitney $U$ test \citep{gibbons2011nonparametric} and the Kolmogorov-Smirnov test \citep{massey1951kolmogorov} are typically employed for comparing two samples. The Kruskal-Wallis test \citep{kruskal1952use} extends the former for the $n$-sample case. Multivariate analysis of variance (MANOVA) \citep{krzanowski1998,tabachnick2013using} can be used as a statistical test for comparing multivariate samples. In this context, samples are composed of multi-dimensional observations, for which each dimension is a dependent variable (DV). However, MANOVA is not appropriate for cases with highly correlated DVs and when the number of DVs or dimensions is higher than the number of observations. Additionally, MANOVA is a parametric method which makes a series of assumptions on the underlying data which are not always met in practice. Analogous non-parametric tests exist, but they are not as widespread and are commonly oriented towards specific research topics. Multiple Response Permutation Procedures (MRPP) \citep{mielke1976multi} and associated permutation-based methods, such as ANOSIM \citep{clarke1993nonparametric} or permutational MANOVA \citep{anderson2001new}, test for differences in distances between observations from each group. These tests are implemented in the \CRANpkg{vegan} package \citep{vegan2016}, typically used in Ecology studies. The \CRANpkg{Blossom} package \citep{talbert2016blossom} also provides MRPP and other distance-function based permutation tests. In a similar note, \cite{szekely2004testing} proposed a multi-sample test for equality of multivariate distributions based on the Euclidean distance between sample elements. The test statistic belongs to a class of multivariate statistics (energy statistics) proposed by the same authors. The \CRANpkg{energy} package \citep{rizzo2016energy} implements this test and other energy statistics-related functionality. The cross-match test is another distance-based test \citep{rosenbaum2005exact}, with the particularity of not requiring permutation techniques. It is available for the R environment via the \CRANpkg{crossmatch} package \citep{heller2012crossmatch}. In turn, the \CRANpkg{cramer} package \citep{franz2014cramer} provides a multivariate implementation of the non-parametric two-sample Cram\'{e}r test, originally proposed by \cite{baringhaus2004new}. The critical value of the test can be determined with bootstrap (ordinary or permutation-based) or eigenvalue-based methods. Another test which avoids permutation was proposed by \cite{duong2012closed}. It is a kernel-based test, originally developed to assess the statistical differences between two cellular topologies. The test is implemented in the \CRANpkg{ks} package \citep{duong2016ks}, although limited to six-dimensional data. An alternative to multivariate tests is to extract a number of statistical summaries (e.g., averages or extreme values) or specific points from individual multivariate observations, and then perform a univariate test for each summary measure. This approach also has its issues: a) it does not provide a single answer, i.e., it will yield as many $p$-values as there are summary measures; b) the choice of summary is problem-dependent and empirically driven, and consequently, error-prone, in the sense that the chosen summaries may not be representative of the original multivariate observations. While only careful analysis can minimize the latter issue, the former problem can be addressed with a multiple comparison adjustment procedure, such as the Bonferroni correction \citep{shaffer1995multiple}. \section{Independent comparison of multivariate observations}\label{micompr:sec:theory} Given a matrix $\mathbf{X}_{(n \times m)}$ of $n$ observations and $m$ variables or dimensions, PCA can be used to obtain matrix $\mathbf{T}_{(n \times r)}$, which is the representation of $\mathbf{X}_{(n \times m)}$ in the principal components (PCs) space, and vector $\boldsymbol{\lambda}_{(1 \times r)}$, containing the eigenvalues of the covariance matrix of the original mean-centered data. Rows of $\mathbf{T}$ directly correspond to the observations of the original samples, while columns correspond to PCs. Columns are ordered by decreasing variance, i.e., the first column corresponds to the first PC, and so on. Variance is given by the eigenvalues in vector $\boldsymbol{\lambda}$, which are likewise ordered, each eigenvalue corresponding to the variance of the columns of $\mathbf{T}$. The percentage of variance explained by each PC can be obtained by dividing the respective eigenvalue with the sum of all eigenvalues. At this stage, PCA-reshaped observations associated with different groups can be compared using statistical methods. More specifically, hypothesis tests can be used to check if the sample projections on the PC space are drawn from populations with the same distribution. There are two possible lines of action: \begin{enumerate} \item Apply a MANOVA test to the samples, where each observation has $q$-dimensions, corresponding to the first $q$ PCs (dimensions) such that these explain a user\hyp{}defined minimum percentage of variance. \item Apply a univariate test to observations in individual PCs. Possible tests include the $t$-test and the Mann-Whitney U test for comparing two samples, or ANOVA and Kruskal-Wallis test, which are the respective parametric and non-parametric versions for comparing more than two samples. \end{enumerate} The MANOVA test yields a single $p$-value from the simultaneous comparison of observations along multiple PCs. An equally succinct answer can be obtained with a univariate test using the Bonferroni correction or a similar method for handling $p$-values from multiple comparisons. However, both approaches will not prioritize dimensions, even though the first PCs can be more important for characterizing an output, as they explain more variance. In the univariate case one can prioritize PCs according to the explained variance using the weighted Bonferroni procedure \citep{rosenthal1983ensemble}. Conclusions concerning whether samples are statistically similar can be drawn by analyzing the $p$-values produced by the employed statistical tests, which should be below the typical 1\% or 5\% when samples are significantly different. In such case, less PCs should be required to explain the same percentage of variance than when, in the same context, no significant differences are found. The scatter plot of the first two PC dimensions can also provide visual, although subjective feedback on sample similarity. While the procedure is most appropriate for comparing multivariate observations with highly correlated and similar scale dimensions, assessing the similarity of ``systems'' with multiple outputs of different scales is also possible. The simplest approach would be to apply the proposed method to samples of individual outputs, and analyze the results in a multiple comparison context. An alternative approach consists in concatenating, observation-wise, all outputs, centered and scaled to the same order of magnitude, thus reducing a ``system'' with $k$ outputs to a ``system'' with one output. The proposed method would then be applied to samples composed of concatenated observations encompassing the existing outputs. This technique is described in detail by \citet{fachada2015model} in the context of comparing simulation outputs. \section{The \pkg{micompr} package}\label{micompr:sec:micompr} \subsection{Overview} The \pkg{micompr} package for the R statistical computing environment implements the methodology proposed in Section \nameref{micompr:sec:theory}. Here we describe version 1.1.2 of the package, which is available at \url{https://cran.r-project.org/package=micompr}. The development version is hosted at \url{https://github.com/nunofachada/micompr}. The package is covered by the MIT license. The \pkg{micompr} package is built upon two functions, \code{cmpoutput} and \code{micomp}. The former compares two or more samples of multivariate observations collected from one output. The latter is used for comparing multiple outputs and/or comparing outputs in multiple contexts. \code{grpoutputs} is a helper function for loading data from two or more set of files and preparing the data to be processed by the \code{cmpoutput} and/or \code{micomp} functions. \code{assumptions} is a generic function for assessing the assumptions of the parametric tests used in sample comparisons. \subsection{Architecture} \pkg{micompr} is structured according to the S3 object-oriented system. The \code{cmpoutput}, \code{micomp} and \code{grpoutputs} functions produce S3 objects with the same name. The package also provides the generic function \code{assumptions}, and two concrete implementations of methods for ''cmpoutput'' and ''micomp'' objects, which return objects of class ''assumptions\_cmpoutput'' and ''assumptions\_micomp'', respectively. All classes have method implementations of the common S3 generic functions \code{print}, \code{summary} and \code{plot}. Additionally, method implementations of the \code{toLatex} function, for producing user-configurable \LaTeX\ tables with information about the performed comparisons, are provided for ''cmpoutput'' and ''micomp'' objects. \subsubsection{\code{grpoutputs}} This function groups outputs from sets of files containing multiple observations into samples. It returns a list of output matrices, ready to be processed by \code{micomp}. Alternatively, individual output matrices can be handled by \code{cmpoutput}. Separate files contain one multivariate observation of one or more outputs, one column per output, one row per dimension or variable. Each specified set of files is associated with a different sample. The function is also able to create an additional concatenated output, composed from the centered and scaled original outputs. The plot method for ``grpoutputs'' objects shows $k$ plots, one per output. Output observations are plotted on top of each other, with different samples colored distinctively. The \code{summary} method for ``grpoutputs'' objects returns a list containing two elements: a) the $n \times m$ dimensions of each output matrix; and, b) the sizes of individual samples. The \code{print} method for ``grpoutputs'' objects simply outputs the summary in a more adequate presentation format. \subsubsection{\code{cmpoutput}} The \code{cmpoutput} function is at the core of \pkg{micompr}. It compares two or more samples of multivariate observations using the technique described in Section \nameref{micompr:sec:theory}. It accepts an output matrix, $\mathbf{X}_{(n \times m)}$, with $n$ observations and $m$ variables or dimensions, a factor vector of length $n$, specifying the group associated with each observation, and a vector of explained variances with which to determine the number of PCs to use in the MANOVA test (alternatively, the number of PCs can also be directly specified). The function returns matrix $\mathbf{T}_{(n \times r)}$ of PCA scores and the $p$-values for the performed statistical tests, namely: a) a MANOVA test for each explained variance (or number of PCs); and, b) parametric ($t$-test or ANOVA) and non-parametric (Mann-Whitney or Kruskal-Wallis) univariate tests for each PC. Regarding the latter, the function also returns $p$-values adjusted with the weighted Bonferroni correction, using the percentages of explained variance by PC as weights. The \code{plot} implementation for ``cmpoutput'' objects shows six sub-plots, namely a scatter plot with the PC1 vs. PC2 scores and five bar plots. The horizontal scale of the latter consists of the $r$ PCs, and the vertical bars represent the explained variance (one plot) or univariate parametric and non-parametric $p$-values, before and after weighted Bonferroni correction (four plots). The \code{summary} method for ``cmpoutput'' objects returns a list with the following items: a) percentage of variance explained by each PC; b) $p$-values of the MANOVA test or tests; c) $p$-values of the parametric test, per PC, before and after weighted Bonferroni correction; d) $p$-values of the non-parametric test, per PC, before and after weighted Bonferroni correction; and, e) name of the parametric and non-parametric univariate tests employed (either $t$-test and Mann-Whitney $U$ test for comparing two samples, or ANOVA and Kruskal-Wallis for more than two samples). The \code{print} method for ``cmpoutput'' objects shows the information provided by the \code{summary} implementation, but the $p$-values of the univariate tests are only shown for the first PC. \subsubsection{\code{micomp}} The \code{micomp} function performs one or more comparisons of multiple outputs, invoking \code{cmpoutput} for each comparison/output combination. It accepts a list of comparisons, where individual comparisons can have one of two configurations: a) a vector of folders and a vector of file sets containing data in the format required by \code{grpoutputs}, where each file set corresponds to a different sample; and, b) a ``grpoutputs'' object, passed directly. The returned objects, of class ``micomp'', are basically two-dimensional lists of ``cmpoutput'' instances, with rows associated with individual outputs, and columns with separate comparisons. The \code{plot} method for ``micomp'' objects shows the PC1 vs. PC2 score plots for each comparison/output combination. The \code{summary} implementation for ``micomp'' objects returns a list of comparisons, each one containing a $a \times k$ matrix of $p$-values or number of PCs, associated with $a \geq 6$ measures and $k$ outputs. Four rows represent the $p$-values of the parametric and non-parametric univariate tests for the first PC, before and after weighted Bonferroni correction. The remaining pairs of rows are associated with the MANOVA test for a given percentage of variance to explain. One row shows the $p$-values, and the other displays the number of PCs required to explain the specified percentage of variance for the given output. As with other \pkg{micompr} objects, the \code{print} method for ``micomp'' objects also shows the summary with a better presentation. \subsubsection{\code{assumptions}} \code{assumptions} is a generic function which performs a number of statistical tests concerning the assumptions of the parametric tests performed by the package functions. Implementations of this generic function exist for ``cmpoutput'' and ``micomp'' objects. The former method returns objects of class ``assumptions\_cmpoutput'' containing results of the assumptions tests for a single output comparison. The latter returns a two-dimensional list of ``assumptions\_cmpoutput'' objects, with rows associated to individual outputs, and columns to separate comparisons. These objects are tagged with the ``assumptions\_micomp'' class attribute. The following assumptions are checked: a) observations are normally distributed within each sample along individual PCs (Shapiro-Wilk test); b) observations follow a multivariate normal distribution within each sample for all PCs used in MANOVA (Royston test); c) samples have homogeneous variance along individual PCs (Bartlett test); and, d) samples have homogeneous covariance matrices for all PCs used in MANOVA (Box's $M$ test). Assumptions a) and c) should be verified for the parametric test applied to each PC, while assumptions b) and d) should be verified for individual MANOVA tests performed for each variance to explain (or, alternatively, for each specified number of PCs). The \code{plot} implementations for classes ``assumptions\allowbreak{}\_\allowbreak{}cmpoutput'' and ``assumptions\allowbreak{}\_\allowbreak{}micomp'' display a number of bar plots for the $p$-values of the performed tests. These are more detailed for ``assumptions\allowbreak{}\_\allowbreak{}cmpoutput'' objects, showing the $p$-values of the univariate test for all PCs. For ``assumptions\_micomp'' objects, one bar plot is shown per output/comparison combination, but in the case of the univariate tests only the $p$-values of the first PC are shown. Implementations of \code{summary} return a list of tabular data containing the $p$-values of the assumption tests. The \code{summary} method for ``assumptions\allowbreak{}\_\allowbreak{}cmpoutput'' objects returns a list with two matrices of $p$-values, one for the MANOVA tests, another for the univariate tests. The \code{summary} method for ``assumptions\allowbreak{}\_\allowbreak{}micomp'' objects follows the approach taken by the \code{summary} method for ``micomp'' objects, returning a list of $p$-value matrices, one matrix per comparison. Rows of individual matrices correspond to the assumptions tests, and columns to outputs. The \code{print} methods for ``assumptions\allowbreak{}\_\allowbreak{}cmpoutput'' objects and for ``assumptions\allowbreak{}\_\allowbreak{}micomp'' objects again show the summary information in a printable format. \subsubsection{\code{toLatex} methods for ``cmpoutput'' and ``micomp'' objects} These methods are implementations of the \code{toLatex} generic function, and convert ``cmpoutput'' and ``micomp'' objects to character vectors representing \LaTeX\ tables. The generated tables are configurable via function arguments, with sensible defaults. Tables can present the following data for each output/comparison combination: a) number of principal components required to explain a user-specified percentage of variance; b) MANOVA $p$-value for a user-specified percentage of variance to explain or number of PCs; c) parametric test $p$-value for a given PC, before and/or after weighted Bonferroni correction; d) non-parametric test $p$-value for a given PC, before and/or after weighted Bonferroni correction; e) variance explained by a specific PC; and, f) a score plot with the output projection on the first two PCs. \subsubsection{Other functions} The \pkg{micompr} package is bundled with additional functions whose purpose is to aid the main package methods do their job. However, some of these may be useful in other contexts. The \code{concat\_outputs} function concatenates outputs collected from multiple observations. It accepts two arguments, namely a list of output matrices, and the centering and scaling method. Several centering and scaling methods, such as ``range'', ``iqrange'', ``vast'' or ``pareto'' \citep{berg2006centering}, are recognized in the second argument. The function returns an $n \times p$ matrix of $n$ observations with length $p$, which is the sum of individual output lengths. Lower-level centering and scaling of individual outputs is performed by the \code{centerscale} function, which accepts a numeric vector and returns a new vector, centered and scaled with the specified method. The \code{pvalf} generic function formats $p$-values for \LaTeX . A concrete default implementation is used by the \pkg{micompr} \code{toLatex} implementations. This implementation underlines and double-underlines $p$-values lower than $0.05$ and $0.01$, respectively, although these limits are configurable, and underlining can be turned off by setting both limits to zero. It is also possible to specify a limit below which $p$-values are capped. For example, if this limit is set to $1 \times 10^{-5}$, a $p$-value equal to $1 \times 10^{-6}$ would be displayed as ``$< 1\text{e}^{-5}$''. The default method of \code{pvalf} will format $p$-values lower than $5 \times 10^{-4}$ using scientific E notation, which is more compact and thus a better fit for tables. $p$-values between $5 \times 10^{-4}$ and $1$ are formatted using regular decimal notation with three decimal places. This aspect is not configurable. However, another implementation of \code{pvalf} can be passed to the \pkg{micompr} \code{toLatex} implementations if different formatting is desired. Simple \TikZ 2D scatter plots, as the ones produced by the \pkg{micompr} \code{toLatex} implementations, can be generated with the \code{tikzscat} function. The function accepts the data to plot, an $n$ x 2 numeric matrix, of $n$ observations and $2$ dimensions, and a factor vector specifying the levels or groups associated with each observation. Several plot characteristics, such as mark types, scale and axes color, are configurable via function arguments. \code{tikzscat} returns a string containing the \TikZ figure code for plotting the specified data. \subsection{Included data} The package includes test data produced by several implementations of the Predator-Prey for High Performance Computing (PPHPC) simulation model \citep{fachada2015template}. The data is provided in \code{rdata} format, and is readily available on loading the package. The same data is also provided in TSV format. This is a limited subset of the complete data, and is included for package testing and exemplification purposes. The example discussed in Section \nameref{micompr:sec:examples:pphpc} uses a superset of this data, which is available for public download, but could not be included with the package due to its large size. \subsection{Dependencies} \pkg{micompr} has a number of optional dependencies, not required for package installation and for using most of its functionality. The \CRANpkg{biotools} \citep{silva2015biotools} and \CRANpkg{MVN} \citep{korkmaz2014mvn} packages are required by the \code{assumptions} functions, providing the statistical tests for assessing MANOVA and $t$-test assumptions. If these functions are invoked without the presence of the specified packages, they will inform the user of that fact, and terminate cleanly. The \CRANpkg{testthat} \citep{wickham2011testthat}, \CRANpkg{knitr} \citep{xie2015dynamic} and \CRANpkg{roxygen2} \citep{wickham2015roxygen2} packages are required for package development. The \pkg{deseasonalize} package \citep{mcleod2013optimal} is required for building one of the vignettes\footnote{The \pkg{deseasonalize} package \citep{mcleod2013optimal} has since been removed from CRAN, therefore the respective examples have been commented out of this vignette.}. \section{Examples}\label{micompr:sec:examples} In this section we discuss four concrete application examples for the \pkg{micompr} package. The complete scripts used in these examples are available at \url{https://github.com/nunofachada/micompr-examples}. \subsection{Simulation model with multiple outputs}\label{micompr:sec:examples:pphpc} The replication of a simulation model in a new context highlights differences between the conceptual and implemented models, as well as inconsistencies in the conceptual model specification \citep{edmonds2003replication}, promoting model verification, model validation \citep{wilensky2007making}, and model credibility \citep{thiele2015replicating}. Some argue that a simulation model is untrustworthy until it has been successfully replicated \citep{edmonds2003replication,david2013validating}. Model parallelization is a an illustrative example of the importance of replication, as it is often required for simulating large models in practical time frames \citep{fachada2015parallelization}. By definition, model parallelization implies a number of changes, or even full reimplementation, of the original model, such that a robust comparison methodology, as provided by the \pkg{micompr} package, is required in order to make sure a parallelized model faithfully reproduces the behavior of the original serial model. PPHPC is a reference model for studying and evaluating implementation strategies for spatial agent-based models, capturing important characteristics such as agent movement and local agent interactions \citep{fachada2015template}. The model describes a prototypical predator-prey system, and has six outputs, namely prey population, $P^s$, predator population, $P^w$, cell-bound food quantity, $P^c$, mean prey energy, $\mean{E}^s$, mean predator energy, $\mean{E}^w$, and mean cell-bound food levels, $\mean{C}$. Since outputs are collected once per iteration, each simulation run yields six time series, associated with the individual outputs. With several open source implementations publicly available, the model provides a good test case for multivariate comparison purposes. Here we show the main comparison cases discussed in a previous article \citep{fachada2015model}, in which the model implementations are parameterized with size 400 and parameter set 1 \citep{fachada2015template}. A canonical PPHPC realization, implemented in NetLogo \citep{wilensky1999compat}, is compared with three configurations of a parallel Java implementation \citep{fachada2015parallelization}. The NetLogo implementation and the first Java configuration follow the PPHPC conceptual model and the specified parameters. The second Java configuration disables agent shuffling prior to agent actions, which is explicitly mandated in the conceptual model description. The third Java configuration performs a minimal change in one of the parameters specified by parameter set 1. As such, we define three comparison cases: \begin{description} \item[Case I] Compare the NetLogo implementation with the first Java configuration. These should yield distributionally equivalent results. \item[Case II] Compare the NetLogo implementation with the second Java configuration. A small misalignment is to be expected. \item[Case III] Compare the NetLogo implementation with the third Java configuration. There should be a mismatch in the outputs. \end{description} Independent samples of the six model outputs were obtained from $n=30$ replications for each implementation or configuration, in a total of $4n=120$ runs. Each replication $r=1,\ldots,4n$ was performed with a PRNG seed obtained by taking the MD5 checksum of $r$, guaranteeing independence between seeds, and consequently, between replications. The data generated by this computational experiment, as well as the scripts used to set up the experiment, are made available to other researchers at \url{https://zenodo.org/record/46848}. The following script performs these comparisons. Note that the \code{concat = TRUE} option of the \code{micomp} function specifies that an additional concatenated output, $\widetilde{A}$, should be generated from the original outputs and analyzed in a similar fashion. The \code{dir\_data} variable specifies the location of the dataset. <>= # Load library library(micompr) # Output names outputs <- c("$P^s$", "$P^w$", "$P^c$", "$\\mean{E}^s$", "$\\overline{E}^w$", "$\\overline{C}$", "$\\widetilde{A}$") # Outputs from the NetLogo implementation dir_nl_ok <- paste0(dir_data, "nl_ok") # Outputs from the Java implementation, first configuration dir_jex_ok <- paste0(dir_data, "j_ex_ok") # Outputs from the Java implementation, second configuration dir_jex_noshuff <- paste0(dir_data, "j_ex_noshuff") # Outputs from the Java implementation, third configuration dir_jex_diff <- paste0(dir_data, "j_ex_diff") # Files for model size 400, parameter set 1 filez <- glob2rx("stats400v1*.txt") # Perform the three comparison cases mic <- micomp(outputs, ve_npcs = 0.75, list(list(name = "I", folders = c(dir_nl_ok, dir_jex_ok), files = c(filez, filez), lvls = c("NLOK", "JEXOK")), list(name = "II", folders = c(dir_nl_ok, dir_jex_noshuff), files = c(filez, filez), lvls = c("NLOK", "JEXNS")), list(name = "III", folders = c(dir_nl_ok, dir_jex_diff), files = c(filez, filez), lvls = c("NLOK","JEXDIF"))), concat = T) @ \noindent The \code{mic} object can be inspected at the R prompt using the common S3 generic functions \code{print}, \code{summary} and \code{plot}. For publication purposes, the \code{toLatex} method for ``micomp'' objects produces \LaTeX\ tables with user\hyp{}specified information. For example, to generate a table similar to Table~4 of our previous work \citep{fachada2015model}, \code{toLatex} is invoked as follows:: <>= toLatex(mic, booktabs = T, data_show = c("npcs-1", "mnvp-1", "parp-1", "scoreplot"), data_labels = c("$\\#$PCs", "MNV", "$t$-test", "PCS"), col_width = T, pvalf_params = list(minval = 1e-8, na_str = "*"), label = "tab:pphpc", caption = paste("Comparison of a NetLogo implementation of", "the PPHPC model against three configurations", "of a parallel Java implementation.")) @ \noindent This call produces Table~\ref{tab:pphpc} with booktabs \citep{fear2005booktabsmanual} table style (\code{booktabs = TRUE}) and width set to document column width (\code{col\_width = TRUE}), since the table is somewhat large. The \code{label} and \code{caption} parameters set the label and caption of the \LaTeX\ table, respectively, while the \code{pvalf\_params} argument accepts a list of options for formatting $p$-values. The \code{data\_show} parameter specifies what data to show, which in this case is: 1) \code{npcs-1}, the number of PCs for the first specified variance (the \code{micomp} function accepts and performs output comparison with one or more specified variances); 2) \code{mnvp-1}, the MANOVA $p$-value for the first specified variance; 3) \code{parp-1}, $t$-test $p$-value for the first PC; and, 4) score plot for the first two PCs. \begin{table}[ht] \begin{center} \resizebox{\columnwidth}{!}{% \begin{tabular}{clrrrrrrr} \toprule \multirow{2}{*}{Comp.} & \multirow{2}{*}{Data} & \multicolumn{7}{c}{Outputs} \\ \cmidrule(l){3-9} & & $P^s$ & $P^w$ & $P^c$ & $\mean{E}^s$ & $\mean{E}^w$ & $\mean{C}$ & $\widetilde{A}$\\ \midrule \multirow{4}{*}{I} & $\#$PCs & 13 & 18 & 14 & 22 & 28 & 14 & 22\\ & MNV & 0.325 & 0.640 & 0.462 & 0.052 & 0.796 & 0.463 & 0.523\\ & $t$-test & 0.530 & 0.836 & 0.804 & 0.784 & 0.378 & 0.805 & 0.879\\ & PCS & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.664,0.010) (-0.801,0.094) (0.313,0.051) (-0.144,-0.250) (0.223,0.241) (0.211,-0.246) (-0.333,-0.005) (0.956,0.258) (0.280,0.114) (-0.150,-0.331) (0.515,0.091) (0.030,-0.002) (-0.151,-0.034) (-0.379,0.703) (0.237,-0.165) (-0.224,0.151) (0.338,0.258) (0.594,0.703) (-0.275,0.125) (0.241,-0.208) (0.506,-0.037) (0.921,-0.306) (-0.053,-0.061) (0.359,0.074) (-1.000,0.352) (-0.057,0.068) (-0.153,-0.554) (-0.287,0.239) (0.076,0.330) (-0.130,0.750)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.146,0.204) (0.156,-0.377) (-0.670,-0.205) (0.516,0.020) (-0.095,-0.147) (0.461,0.112) (-0.533,0.207) (-0.521,0.078) (0.060,-0.341) (0.434,-0.115) (0.252,-0.108) (-0.108,-0.193) (-0.782,-0.153) (0.086,-0.250) (0.193,-0.338) (-0.137,-0.026) (-0.176,-0.141) (0.210,0.023) (0.416,0.476) (-0.326,0.142) (-0.188,-0.274) (-0.366,-0.086) (-0.227,-0.059) (-0.017,-0.345) (0.587,-0.370) (-0.279,-0.204) (0.198,-0.052) (-0.313,0.138) (0.313,0.292) (-0.288,-0.326)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.632,0.161) (-0.778,0.155) (0.223,-0.213) (-0.072,0.179) (0.150,-0.120) (0.230,0.143) (-0.315,-0.049) (0.869,-0.226) (0.216,-0.019) (-0.082,0.559) (0.459,-0.179) (0.085,-0.062) (-0.093,0.099) (-0.546,-0.591) (0.328,0.052) (-0.321,-0.209) (0.170,-0.394) (0.404,-0.664) (-0.374,-0.194) (0.282,0.149) (0.354,-0.090) (1.000,0.167) (0.151,0.212) (0.183,-0.269) (-0.943,-0.221) (-0.130,0.092) (0.116,0.666) (-0.262,-0.253) (-0.053,-0.311) (-0.310,-0.698)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.075,-0.182) (0.220,0.285) (-0.529,0.355) (0.492,-0.039) (-0.095,0.322) (0.307,-0.195) (-0.528,-0.128) (-0.492,-0.068) (0.077,0.268) (0.406,-0.042) (0.324,0.032) (-0.091,0.209) (-0.719,0.205) (0.126,0.347) (0.276,0.344) (0.025,0.135) (-0.084,0.198) (0.209,-0.169) (0.233,-0.535) (-0.327,-0.319) (-0.057,0.242) (-0.338,0.202) (-0.112,0.016) (-0.003,0.311) (0.589,0.187) (-0.198,0.207) (0.288,0.144) (-0.378,-0.162) (0.149,-0.478) (-0.154,0.437)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.613,-0.071) (0.723,0.021) (-0.219,0.042) (0.073,-0.256) (-0.138,0.273) (-0.217,-0.239) (0.300,-0.069) (-0.831,0.432) (-0.248,0.171) (0.017,-0.289) (-0.462,0.182) (-0.034,-0.006) (0.082,-0.053) (0.501,0.700) (-0.241,-0.144) (0.235,0.098) (-0.226,0.258) (-0.374,0.814) (0.333,0.060) (-0.270,-0.168) (-0.464,0.025) (-0.895,-0.201) (-0.004,-0.058) (-0.282,0.108) (1.000,0.225) (0.066,0.129) (-0.047,-0.537) (0.308,0.262) (0.015,0.298) (0.321,0.673)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.101,0.242) (-0.222,-0.356) (0.546,-0.271) (-0.501,0.088) (0.054,-0.107) (-0.423,0.218) (0.545,0.178) (0.536,-0.048) (-0.115,-0.387) (-0.446,-0.090) (-0.222,-0.121) (0.087,-0.271) (0.697,-0.250) (-0.152,-0.244) (-0.269,-0.305) (0.116,-0.045) (0.122,-0.174) (-0.174,-0.001) (-0.260,0.533) (0.333,0.067) (0.083,-0.275) (0.271,-0.070) (0.186,-0.015) (-0.075,-0.357) (-0.615,-0.325) (0.228,-0.265) (-0.223,0.022) (0.363,0.069) (-0.189,0.255) (0.188,-0.375)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.760,-0.049) (0.625,0.290) (-0.263,-0.272) (0.187,-0.380) (-0.314,0.409) (-0.069,-0.474) (0.459,-0.095) (-0.596,0.306) (-0.322,0.214) (0.055,-0.035) (-0.302,0.062) (-0.053,-0.004) (0.322,-0.038) (0.458,1.000) (-0.085,-0.455) (0.086,0.150) (-0.325,0.110) (-0.327,0.476) (0.336,-0.119) (-0.160,-0.281) (-0.611,0.045) (-0.882,-0.050) (0.174,-0.055) (-0.250,-0.404) (0.954,0.215) (0.059,0.003) (0.220,-0.172) (0.189,0.332) (-0.017,0.322) (0.101,0.662)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.101,0.215) (-0.167,-0.236) (0.630,-0.236) (-0.618,0.345) (-0.120,0.067) (-0.318,-0.014) (0.409,0.041) (0.497,-0.177) (-0.106,-0.312) (-0.269,0.039) (-0.199,-0.160) (0.078,-0.335) (0.532,-0.066) (-0.242,-0.061) (-0.493,-0.220) (0.125,0.085) (0.001,-0.101) (-0.117,-0.046) (-0.541,0.489) (0.352,0.020) (0.062,-0.121) (0.335,0.151) (0.278,-0.028) (-0.023,-0.207) (-0.821,-0.352) (0.243,-0.330) (0.043,0.093) (0.241,-0.233) (-0.290,0.190) (0.193,-0.215)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.239,-0.084) (-0.329,0.400) (-0.040,0.354) (0.252,0.019) (0.166,-0.123) (0.322,0.185) (0.422,-0.103) (0.186,-0.133) (0.275,0.279) (0.367,-0.412) (-0.236,0.427) (-0.042,-0.484) (0.129,-0.160) (-0.145,-0.732) (0.279,0.199) (-0.128,0.225) (-0.199,0.168) (-0.121,0.295) (-0.427,-0.098) (-0.092,-0.254) (0.194,-0.154) (0.414,0.588) (0.532,0.075) (0.331,0.297) (-0.026,-0.045) (-0.212,0.045) (-0.373,-0.050) (-0.734,-0.474) (-0.073,-0.045) (0.222,-0.081)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.511,0.358) (0.764,0.162) (0.216,-0.155) (0.403,-0.573) (0.121,-0.225) (-0.250,0.026) (-0.335,-0.047) (1.000,-0.022) (-0.278,0.190) (0.155,-0.025) (0.048,0.022) (-0.052,0.455) (-0.073,0.218) (-0.403,-0.195) (0.099,-0.104) (0.302,0.090) (-0.386,-0.207) (-0.194,0.025) (-0.062,0.045) (-0.366,-0.046) (-0.251,0.132) (-0.670,-0.333) (-0.140,-0.158) (-0.532,0.486) (-0.206,0.606) (-0.077,-0.793) (0.073,-0.075) (0.404,-0.188) (0.165,0.118) (-0.115,0.093)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.613,-0.071) (0.723,0.021) (-0.219,0.042) (0.072,-0.256) (-0.138,0.273) (-0.217,-0.239) (0.300,-0.070) (-0.830,0.433) (-0.248,0.172) (0.017,-0.289) (-0.462,0.183) (-0.035,-0.007) (0.082,-0.053) (0.502,0.700) (-0.242,-0.144) (0.235,0.097) (-0.226,0.258) (-0.374,0.815) (0.333,0.059) (-0.271,-0.168) (-0.464,0.025) (-0.895,-0.200) (-0.004,-0.059) (-0.283,0.108) (1.000,0.225) (0.066,0.130) (-0.047,-0.538) (0.309,0.263) (0.015,0.298) (0.322,0.673)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.101,0.242) (-0.222,-0.356) (0.546,-0.272) (-0.501,0.088) (0.055,-0.106) (-0.423,0.218) (0.546,0.178) (0.536,-0.048) (-0.115,-0.388) (-0.446,-0.091) (-0.222,-0.121) (0.087,-0.271) (0.697,-0.251) (-0.152,-0.244) (-0.268,-0.305) (0.116,-0.045) (0.122,-0.174) (-0.174,-0.001) (-0.259,0.533) (0.333,0.067) (0.083,-0.275) (0.271,-0.070) (0.185,-0.015) (-0.075,-0.357) (-0.615,-0.325) (0.227,-0.265) (-0.224,0.022) (0.363,0.068) (-0.189,0.255) (0.188,-0.375)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.609,-0.155) (0.718,-0.115) (-0.195,0.125) (0.083,-0.274) (-0.138,0.188) (-0.213,-0.228) (0.319,-0.025) (-0.792,0.417) (-0.217,0.127) (0.004,-0.460) (-0.455,0.245) (-0.041,0.036) (0.073,-0.082) (0.654,0.628) (-0.302,-0.096) (0.267,0.185) (-0.129,0.368) (-0.230,0.646) (0.401,0.117) (-0.336,-0.115) (-0.402,0.138) (-0.901,-0.176) (-0.074,-0.198) (-0.176,0.181) (1.000,0.127) (0.083,-0.032) (-0.203,-0.588) (0.322,0.288) (0.091,0.313) (0.404,0.691)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.078,0.211) (-0.247,-0.320) (0.498,-0.389) (-0.482,0.164) (0.039,-0.262) (-0.319,0.229) (0.519,0.112) (0.542,-0.122) (-0.131,-0.295) (-0.427,0.097) (-0.310,-0.021) (0.074,-0.321) (0.643,-0.287) (-0.189,-0.246) (-0.357,-0.295) (0.025,-0.172) (0.068,-0.215) (-0.193,0.150) (-0.169,0.580) (0.342,0.225) (-0.026,-0.229) (0.246,-0.090) (0.139,-0.072) (-0.111,-0.293) (-0.675,-0.176) (0.185,-0.269) (-0.235,-0.057) (0.391,0.038) (-0.107,0.472) (0.119,-0.423)}; \end{tikzpicture} }}\\ \midrule \multirow{4}{*}{II} & $\#$PCs & 13 & 15 & 13 & 1 & 28 & 14 & 21\\ & MNV & \uuline{4e-05} & \uuline{<1e-08} & \uuline{0.002} & * & 0.089 & \uuline{0.003} & \uuline{2e-07}\\ & $t$-test & \uline{0.042} & \uuline{<1e-08} & 0.108 & \uuline{<1e-08} & \uline{0.017} & 0.109 & 0.390\\ & PCS & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.522,0.202) (-0.668,0.179) (0.418,0.063) (-0.061,-0.241) (0.266,0.131) (0.333,-0.202) (-0.276,-0.078) (1.000,0.245) (0.265,0.086) (-0.126,-0.121) (0.570,-0.025) (0.100,-0.048) (-0.144,-0.099) (-0.228,0.653) (0.245,-0.243) (-0.164,0.091) (0.492,0.286) (0.656,0.253) (-0.112,0.214) (0.278,-0.190) (0.571,-0.093) (0.935,-0.366) (-0.087,-0.180) (0.517,0.037) (-0.834,0.377) (0.038,0.063) (-0.171,-0.519) (-0.222,0.288) (0.169,0.207) (0.054,0.846)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.098,0.111) (-0.030,-0.590) (0.006,0.333) (0.494,0.509) (-0.423,0.314) (-0.610,-0.485) (-0.187,0.114) (-0.840,-0.384) (-0.019,-0.225) (-0.125,-0.276) (-0.231,0.591) (-0.876,0.131) (-0.256,-0.061) (-0.384,-0.310) (0.647,-0.150) (-0.220,0.330) (-0.356,-0.103) (0.244,-0.227) (-0.073,-0.179) (-0.709,-0.407) (0.097,-0.018) (-0.096,-0.332) (-0.368,0.340) (0.561,-0.334) (-0.295,-0.140) (0.057,-0.432) (-0.078,-0.104) (0.054,-0.194) (0.535,-0.159) (0.095,0.526)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.066,0.635) (-0.033,0.650) (-0.634,-0.113) (-0.417,0.085) (-0.598,0.057) (-0.473,-0.276) (-0.241,0.309) (-1.000,-0.412) (-0.577,0.105) (-0.182,0.100) (-0.732,-0.251) (-0.508,0.033) (-0.375,0.220) (-0.356,0.750) (-0.545,-0.137) (-0.336,0.452) (-0.705,-0.004) (-0.804,-0.119) (-0.343,0.404) (-0.494,-0.174) (-0.610,-0.316) (-0.959,-0.764) (-0.450,0.020) (-0.675,-0.168) (-0.127,0.999) (-0.309,0.098) (-0.273,-0.141) (-0.478,0.535) (-0.493,0.165) (-0.507,0.534)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.424,-0.108) (0.476,-0.633) (0.293,0.064) (0.042,-0.044) (0.603,0.451) (0.918,-0.097) (0.560,0.062) (0.815,0.270) (0.510,-0.297) (0.516,-0.272) (0.576,0.415) (0.888,0.561) (0.496,0.052) (0.619,-0.075) (0.149,-0.741) (0.542,0.289) (0.588,0.104) (0.445,-0.633) (0.520,-0.213) (0.797,0.043) (0.355,-0.263) (0.441,-0.337) (0.401,0.432) (0.207,-0.731) (0.576,-0.032) (0.335,-0.512) (0.454,-0.194) (0.461,-0.367) (0.107,-0.688) (0.188,0.218)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.530,-0.165) (0.657,-0.087) (-0.352,-0.131) (-0.000,0.243) (-0.201,-0.162) (-0.368,0.174) (0.278,0.152) (-0.924,-0.409) (-0.204,-0.083) (0.033,0.154) (-0.567,-0.047) (-0.118,0.037) (0.125,0.171) (0.389,-0.660) (-0.261,0.242) (0.220,-0.031) (-0.424,-0.374) (-0.496,-0.334) (0.180,-0.245) (-0.334,0.160) (-0.605,0.010) (-1.000,0.290) (0.075,0.246) (-0.502,-0.126) (0.929,-0.285) (-0.039,-0.116) (0.036,0.655) (0.300,-0.291) (-0.089,-0.220) (0.150,-0.905)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.103,-0.167) (-0.088,0.639) (0.074,-0.392) (-0.423,-0.653) (0.503,-0.279) (0.452,0.595) (0.209,-0.112) (0.722,0.530) (-0.085,0.198) (0.048,0.318) (0.310,-0.648) (0.866,-0.032) (0.241,0.104) (0.263,0.375) (-0.667,0.056) (0.295,-0.363) (0.273,0.158) (-0.346,0.182) (0.033,0.171) (0.624,0.557) (-0.131,-0.005) (0.079,0.434) (0.424,-0.303) (-0.615,0.260) (0.298,0.183) (-0.145,0.479) (0.090,0.137) (-0.121,0.190) (-0.574,0.099) (0.077,-0.571)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.877,0.287) (-0.960,0.248) (-0.815,-0.141) (-0.824,0.011) (-0.763,-0.079) (-0.784,-0.112) (-0.885,0.154) (-0.722,-0.229) (-0.861,-0.121) (-0.850,0.063) (-0.791,-0.134) (-0.871,-0.066) (-0.914,0.100) (-0.834,0.222) (-0.811,-0.073) (-0.841,0.039) (-0.833,-0.150) (-0.769,-0.145) (-0.893,0.071) (-0.809,-0.102) (-0.794,-0.230) (-0.700,-0.337) (-0.796,0.084) (-0.758,-0.160) (-0.883,0.353) (-0.853,0.012) (-1.000,0.035) (-0.839,0.094) (-0.784,0.005) (-0.828,0.086)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.783,-0.079) (0.815,-0.132) (0.858,0.054) (0.913,-0.129) (0.851,0.179) (0.832,0.122) (0.861,0.019) (0.764,0.287) (0.928,-0.055) (0.924,0.011) (0.766,0.154) (0.725,0.346) (0.754,-0.017) (0.766,0.091) (0.887,-0.192) (0.866,0.095) (0.763,0.118) (0.867,-0.104) (0.842,-0.063) (0.678,0.177) (0.868,-0.095) (0.760,-0.032) (0.819,0.149) (0.876,-0.298) (0.834,0.046) (0.854,-0.092) (0.872,0.026) (0.834,-0.063) (0.897,-0.280) (0.886,-0.024)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.177,-0.178) (-0.523,0.086) (-0.287,0.380) (0.090,0.092) (0.054,0.121) (0.099,0.143) (0.205,-0.088) (0.049,0.029) (0.034,0.227) (0.395,-0.021) (-0.429,0.368) (0.050,-0.307) (-0.060,-0.003) (-0.054,-0.570) (0.076,0.395) (-0.333,0.095) (-0.274,-0.003) (-0.311,0.121) (-0.464,-0.286) (-0.127,0.097) (0.052,0.021) (0.089,0.774) (0.274,0.266) (0.104,0.220) (-0.204,-0.098) (-0.267,0.013) (-0.264,0.227) (-0.650,-0.435) (-0.263,-0.069) (0.051,-0.252)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.269,-0.025) (0.413,0.376) (0.489,-0.280) (-0.119,-0.487) (0.299,-0.262) (1.000,-0.221) (0.004,-0.048) (-0.273,0.059) (0.147,0.199) (0.140,0.101) (-0.501,-0.360) (-0.095,-0.120) (0.095,0.101) (0.263,0.037) (0.738,-0.160) (-0.247,-0.150) (0.230,0.079) (-0.029,0.366) (0.191,-0.314) (-0.324,0.045) (-0.042,-0.192) (0.128,-0.029) (0.301,-0.496) (-0.070,0.323) (0.250,0.131) (0.062,0.072) (-0.102,0.169) (0.053,0.218) (0.166,-0.015) (-0.187,-0.481)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.530,-0.164) (0.657,-0.086) (-0.352,-0.132) (-0.001,0.244) (-0.201,-0.163) (-0.369,0.173) (0.277,0.152) (-0.923,-0.411) (-0.205,-0.082) (0.034,0.153) (-0.567,-0.047) (-0.118,0.037) (0.125,0.172) (0.390,-0.660) (-0.262,0.243) (0.220,-0.029) (-0.424,-0.374) (-0.496,-0.334) (0.181,-0.245) (-0.335,0.160) (-0.605,0.009) (-1.000,0.288) (0.075,0.247) (-0.502,-0.126) (0.928,-0.283) (-0.038,-0.117) (0.034,0.655) (0.301,-0.291) (-0.089,-0.220) (0.152,-0.905)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.102,-0.169) (-0.089,0.639) (0.074,-0.392) (-0.421,-0.654) (0.504,-0.280) (0.452,0.595) (0.210,-0.112) (0.720,0.532) (-0.085,0.197) (0.047,0.318) (0.312,-0.649) (0.866,-0.032) (0.241,0.105) (0.262,0.376) (-0.667,0.055) (0.297,-0.364) (0.272,0.159) (-0.346,0.180) (0.033,0.171) (0.623,0.559) (-0.131,-0.005) (0.077,0.435) (0.424,-0.303) (-0.615,0.259) (0.298,0.183) (-0.146,0.479) (0.090,0.138) (-0.121,0.190) (-0.574,0.098) (0.077,-0.570)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (0.601,-0.084) (0.668,0.035) (-0.331,-0.136) (-0.006,0.243) (-0.167,-0.135) (-0.389,0.054) (0.266,0.173) (-0.807,-0.606) (-0.125,-0.125) (0.045,0.114) (-0.564,-0.224) (-0.127,-0.006) (0.100,0.214) (0.646,-0.542) (-0.322,0.251) (0.256,0.018) (-0.279,-0.493) (-0.317,-0.261) (0.299,-0.263) (-0.395,0.056) (-0.566,-0.180) (-1.000,0.115) (-0.002,0.443) (-0.416,-0.245) (0.991,-0.013) (-0.019,-0.109) (-0.188,0.655) (0.389,-0.272) (0.002,-0.222) (0.336,-0.919)}; \path plot[mark=*,mark size=0.6pt] coordinates { (-0.064,-0.297) (-0.361,0.678) (0.165,-0.384) (-0.192,-0.878) (0.648,-0.035) (0.293,0.699) (0.222,-0.046) (0.541,0.729) (-0.143,0.120) (-0.068,0.340) (0.535,-0.604) (0.851,0.143) (0.154,0.172) (0.102,0.410) (-0.631,-0.087) (0.428,-0.306) (0.244,0.191) (-0.444,0.022) (-0.027,0.144) (0.395,0.743) (-0.183,-0.134) (-0.094,0.452) (0.507,-0.245) (-0.685,0.154) (0.228,0.294) (-0.323,0.545) (0.012,0.163) (-0.251,0.136) (-0.648,-0.057) (0.210,-0.597)}; \end{tikzpicture} }}\\ \midrule \multirow{4}{*}{III} & $\#$PCs & 7 & 1 & 2 & 2 & 20 & 1 & 14\\ & MNV & \uuline{<1e-08} & * & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{<1e-08} & * & \uuline{<1e-08}\\ & $t$-test & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{<1e-08}\\ & PCS & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.419,0.468) (-0.489,0.668) (-0.690,-0.147) (-0.630,0.139) (-0.704,-0.092) (-0.701,-0.180) (-0.568,0.334) (-1.000,-0.646) (-0.708,-0.084) (-0.462,0.058) (-0.821,-0.268) (-0.655,0.052) (-0.656,0.253) (-0.646,0.355) (-0.652,-0.075) (-0.570,0.371) (-0.815,-0.092) (-0.993,-0.257) (-0.606,0.258) (-0.685,-0.121) (-0.839,-0.310) (-0.770,-0.716) (-0.490,0.186) (-0.843,-0.211) (-0.377,0.880) (-0.733,0.048) (-0.609,0.234) (-0.558,0.397) (-0.794,0.078) (-0.699,0.202)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.615,-0.498) (0.812,-0.124) (0.690,0.050) (0.778,0.265) (0.339,-0.590) (0.562,-0.365) (0.706,0.222) (0.535,-0.504) (0.668,0.229) (0.458,-0.111) (0.765,-0.143) (0.699,-0.230) (0.741,0.393) (0.872,0.346) (0.648,0.307) (0.828,0.451) (0.598,-0.395) (0.726,0.331) (0.828,0.285) (0.428,-0.506) (0.263,-0.722) (0.862,-0.152) (0.418,-0.191) (0.928,0.061) (0.770,0.206) (0.564,-0.497) (0.889,0.180) (0.777,0.125) (0.723,-0.165) (0.689,-0.038)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.751,0.071) (-0.751,-0.042) (-0.903,0.066) (-0.873,0.013) (-0.917,-0.049) (-0.883,0.094) (-0.816,-0.033) (-1.000,0.007) (-0.878,-0.040) (-0.775,0.073) (-0.924,-0.039) (-0.876,0.101) (-0.831,-0.042) (-0.862,-0.142) (-0.865,0.084) (-0.797,-0.072) (-0.922,0.048) (-0.982,-0.007) (-0.842,0.021) (-0.855,-0.016) (-0.916,-0.040) (-0.990,-0.025) (-0.828,-0.027) (-0.958,-0.008) (-0.748,0.012) (-0.853,0.054) (-0.772,-0.102) (-0.842,-0.124) (-0.892,-0.029) (-0.870,0.220)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.893,0.001) (0.870,-0.008) (0.867,0.027) (0.925,0.069) (0.762,0.139) (0.851,-0.379) (0.926,-0.171) (0.788,0.036) (0.892,-0.301) (0.804,-0.015) (0.867,0.111) (0.828,-0.272) (0.893,-0.374) (0.940,0.014) (0.940,0.247) (0.886,-0.400) (0.837,0.371) (0.927,0.154) (0.949,-0.002) (0.827,0.383) (0.676,-0.053) (0.930,0.270) (0.782,-0.244) (0.933,0.218) (0.907,0.172) (0.781,-0.012) (0.894,0.243) (0.879,-0.024) (0.829,-0.170) (0.889,-0.052)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.711,0.104) (-0.652,0.228) (-0.850,-0.120) (-0.788,-0.059) (-0.860,0.017) (-0.806,-0.193) (-0.767,0.088) (-0.987,-0.175) (-0.805,-0.010) (-0.699,-0.104) (-0.864,-0.092) (-0.797,-0.113) (-0.767,0.037) (-0.795,0.390) (-0.798,-0.176) (-0.761,0.128) (-0.912,-0.021) (-0.963,0.042) (-0.762,0.101) (-0.816,-0.143) (-0.881,-0.098) (-1.000,-0.338) (-0.787,0.003) (-0.901,-0.064) (-0.634,0.342) (-0.807,-0.001) (-0.691,-0.099) (-0.747,0.267) (-0.877,0.091) (-0.805,0.150)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.811,-0.267) (0.797,-0.108) (0.806,0.045) (0.884,0.059) (0.639,-0.103) (0.772,0.005) (0.847,0.175) (0.737,-0.077) (0.827,0.318) (0.761,0.203) (0.844,-0.153) (0.764,0.012) (0.888,0.280) (0.889,0.100) (0.876,-0.029) (0.859,0.502) (0.777,-0.336) (0.883,0.020) (0.941,-0.125) (0.699,-0.173) (0.569,-0.108) (0.876,-0.402) (0.735,0.041) (0.909,-0.183) (0.843,0.044) (0.700,-0.074) (0.860,0.088) (0.861,0.020) (0.759,0.153) (0.879,-0.110)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.741,-0.270) (-0.688,-0.216) (-0.870,0.079) (-0.853,-0.002) (-0.924,0.048) (-0.886,0.084) (-0.768,-0.156) (-0.994,0.218) (-0.830,0.134) (-0.792,0.008) (-0.907,0.112) (-0.819,0.026) (-0.753,-0.132) (-0.837,-0.234) (-0.856,0.035) (-0.841,-0.111) (-0.882,0.099) (-0.937,0.076) (-0.757,-0.065) (-0.874,0.084) (-0.889,0.242) (-1.000,0.323) (-0.863,-0.118) (-0.931,0.146) (-0.733,-0.408) (-0.811,0.025) (-0.667,-0.015) (-0.810,-0.123) (-0.911,-0.055) (-0.857,-0.115)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.873,0.308) (0.785,-0.013) (0.858,-0.051) (0.905,-0.104) (0.688,0.181) (0.870,0.317) (0.878,-0.080) (0.838,0.128) (0.831,-0.114) (0.900,0.072) (0.886,0.150) (0.781,0.114) (0.991,-0.205) (0.958,-0.204) (0.927,-0.170) (0.838,-0.245) (0.768,0.165) (0.837,-0.165) (0.945,-0.051) (0.794,0.153) (0.628,0.318) (0.856,0.120) (0.815,0.133) (0.849,-0.031) (0.867,-0.164) (0.763,0.130) (0.905,-0.312) (0.813,-0.128) (0.775,0.007) (0.859,0.026)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.544,-0.107) (-0.870,-0.080) (-0.871,-0.213) (-0.694,0.018) (-0.605,0.055) (-0.597,-0.232) (-0.610,0.175) (-0.617,0.007) (-0.515,-0.152) (-0.332,-0.147) (-0.852,-0.082) (-0.708,0.030) (-0.743,0.097) (-0.773,0.571) (-0.665,-0.178) (-0.871,0.069) (-0.818,-0.084) (-0.830,-0.098) (-0.933,0.218) (-0.674,0.156) (-0.615,0.139) (-0.385,-0.345) (-0.568,-0.107) (-0.648,0.032) (-0.779,0.044) (-0.878,-0.116) (-0.820,0.056) (-0.958,0.405) (-0.835,0.156) (-0.605,-0.262)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.730,-0.193) (0.622,-0.026) (0.619,0.048) (0.673,0.016) (0.743,-0.105) (0.884,0.509) (0.582,0.219) (0.510,-0.359) (0.824,0.419) (0.648,0.345) (0.637,-0.068) (0.781,0.145) (0.663,0.677) (0.537,0.014) (0.634,-0.297) (1.000,0.573) (0.583,-0.834) (0.674,-0.157) (0.717,0.005) (0.734,-0.423) (0.937,0.199) (0.825,-0.478) (0.695,0.412) (0.830,-0.210) (0.797,0.031) (0.657,-0.121) (0.816,-0.183) (0.622,-0.078) (0.730,0.079) (0.508,-0.186)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.874,0.094) (-0.841,0.132) (-0.929,-0.046) (-0.901,-0.022) (-0.933,-0.009) (-0.904,-0.087) (-0.897,0.052) (-0.976,-0.133) (-0.904,-0.028) (-0.868,-0.039) (-0.928,-0.077) (-0.903,-0.034) (-0.889,0.015) (-0.902,0.167) (-0.904,-0.070) (-0.893,0.070) (-0.951,-0.017) (-0.964,-0.014) (-0.889,0.064) (-0.912,-0.080) (-0.935,-0.087) (-1.000,-0.202) (-0.911,0.014) (-0.943,-0.053) (-0.840,0.225) (-0.903,-0.000) (-0.854,-0.053) (-0.885,0.122) (-0.935,0.034) (-0.905,0.112)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.912,-0.148) (0.894,-0.044) (0.906,0.032) (0.935,0.061) (0.850,-0.056) (0.897,-0.078) (0.922,0.063) (0.884,-0.056) (0.918,0.106) (0.901,0.070) (0.918,-0.047) (0.887,-0.042) (0.942,0.093) (0.934,0.072) (0.943,0.038) (0.923,0.186) (0.898,-0.104) (0.939,0.052) (0.959,-0.024) (0.875,-0.051) (0.824,-0.091) (0.927,-0.127) (0.890,-0.032) (0.938,-0.029) (0.919,0.061) (0.867,-0.062) (0.920,0.097) (0.928,0.016) (0.884,0.037) (0.940,-0.045)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.237,0.056) (-0.167,0.294) (-0.597,-0.219) (-0.375,-0.103) (-0.572,0.060) (-0.583,-0.282) (-0.316,0.123) (-1.000,-0.138) (-0.533,0.021) (-0.222,-0.200) (-0.734,-0.038) (-0.497,-0.246) (-0.408,0.039) (-0.480,0.690) (-0.522,-0.327) (-0.366,0.180) (-0.699,-0.038) (-0.874,0.098) (-0.419,0.143) (-0.550,-0.167) (-0.740,-0.011) (-0.763,-0.410) (-0.263,-0.116) (-0.748,-0.001) (-0.022,0.361) (-0.551,-0.015) (-0.333,-0.146) (-0.365,0.433) (-0.645,0.172) (-0.554,0.085)}; \path plot[mark=*,mark size=0.6pt] coordinates { (0.432,-0.313) (0.612,-0.211) (0.500,0.057) (0.743,-0.006) (0.108,-0.078) (0.419,0.226) (0.666,0.281) (0.195,-0.058) (0.534,0.560) (0.221,0.362) (0.633,-0.274) (0.472,0.118) (0.648,0.523) (0.751,0.068) (0.579,-0.177) (0.684,0.851) (0.400,-0.647) (0.701,-0.120) (0.792,-0.303) (0.161,-0.277) (-0.064,0.015) (0.789,-0.737) (0.253,0.194) (0.806,-0.405) (0.611,0.002) (0.235,-0.020) (0.666,-0.038) (0.628,-0.000) (0.429,0.300) (0.528,-0.191)}; \end{tikzpicture} }}\\ \bottomrule \end{tabular} } \caption{Comparison of a NetLogo implementation of the PPHPC model against three configurations of a parallel Java implementation.} \label{tab:pphpc} \end{center} \end{table} In terms of comparison, the method does not find significant differences in case I. However, it successfully differentiates the configurations compared in cases II and III. This is in line with what would be expected, and is discussed in further detail by \cite{fachada2015model}. While not shown here, the \code{assumptions(mic)} command reveals that most assumptions of the MANOVA and $t$-tests are verified. \subsection{Monthly sunspots}\label{micompr:sec:examples:sunspots} This example uses the monthly sunspot data \citep{sunspots}, included with R, which contains the monthly numbers of sunspots from 1749 to the present day. The solar cycle is an approximate 11-year period of changes in the number of sunspots and other associated phenomena. Thus, we divide the data into 11-year (132-month) periods, and consider each period to be an observation. In practice this is an oversimplification, since the cycles can be a bit longer or shorter than 11 years. Given the data, we define two samples of 10 observations each, over a period of 110 years or 1320 months. The first sample includes solar cycles from 1749 to 1859, while the second encompasses cycles from 1902 to 2012. We can now ask the following question: were the solar cycles during the 1749--1859 interval significantly different from the more recent observations? The following code compares observations from the two time intervals, and attempts to provide an answer: <>= # Load library library(micompr) # Months in the 1749-1859 interval (110 years) # Months in the 1902-2012 interval (110 years) m <- sunspot.month[c(1:1320, 1837:3156)] m <- matrix(m, nrow = 20) # Factor vector, two levels: # a) ten 11-year cycles from 1749 to 1859 # b) ten 11-year cycles from 1902 to 2012 groups <- factor(c(rep("A", 10), rep("B", 10))) # Compare the two groups, use 9 PCs for MANOVA cmp <- cmpoutput("SunSpots", 9, m, groups) @ \noindent The \code{cmp} object can now be analyzed: <>= cmp @ \noindent The MANOVA $p$-value is significant, as well as the $t$-test and Mann-Whitney PC1 $p$-values, before and after weighted Bonferroni correction. As such, it is possible to conclude that solar cycles from 1749 to 1859 were significantly different from cycles between 1902 and 2012. However, is the data in accordance with the assumptions for the MANOVA and $t$-test? This can be checked with the \code{assumptions} function: <>= assumptions(cmp) @ \noindent Only Box's M test, which checks for homogeneity of variance-covariance matrices, is significant. However, this test is prone to false positives, and this assumption is not critical when samples are of the same size \citep{tabachnick2013using}. Given this information, it seems plausible to consider the results provided by the parametric tests in our final decision, i.e., that there is in fact a significant difference between samples. A good way to visualize the overall results is to plot the ``cmpoutput'' object: <>= plot(cmp) @ \noindent This command generates the plots shown in Figure \ref{micompr:fig:sunspots}. The score plot shows the samples to be distinctly separated, and the variance explained by PC decreases abruptly from the first PC to the second. Univariate $p$-values for PC1 are visibly significant, though not very much for the remaining PCs. \subsection{Saugeen river flow}\label{micompr:sec:examples:saugeen} This example uses the Saugeen River daily flow data \citep{hipel1994time}, included with the \pkg{deseasonalize} package. This data consists of a time series of the rivers' daily flow (m\textsuperscript{3}/s) from 1915 to 1979. Considering one year as an observation, there are a total of 65 observations. We can, for example, define two samples of 30 observations each, with the first and last 30 years of records, and ask the following question: is there any statistical difference between the flow dynamics during the 1915--1944 and 1950--1979 periods (perhaps due to climate change or some other factor)? \textbf{Update 23-05-2022}: The \pkg{deseasonalize} package has been removed from CRAN, therefore this example no longer runs. However, the example is still available in \pkg{micompr}'s original paper at \url{https://doi.org/10.32614/RJ-2016-055}. % %The following code compares observations from the two periods: % %<>= %# Load libraries %library(micompr) %library(deseasonalize) % %# Unique years % years <- unique(sapply(rownames(SaugeenDay), substr, 1, 4)) % % # Number of days in each year % ndays <- sapply(years, function(x) % sum(substr(rownames(SaugeenDay), 1, 4) == x)) % % # Indexes of last day in each year % lastdays <- cumsum(ndays) % % # Prepare data for PCA % saugdata <- t(mapply( % function(nd, ld) { % rflows <- rep(NA, 366) % rflows[1:nd] <- SaugeenDay[(ld - nd + 1):ld] % # Discard last day in leap years % rflows[1:365] % }, % ndays, lastdays)) % % # Consider first 30 years and last 30 years (discard 5 years in between) % saugdata <- saugdata[c(1:30, 36:65), ] % % # Factor vector, two levels: first 30 years and last 30 years % groups <- factor(c(rep("A", 30), rep("B", 30))) % % # Compare % cmp <- cmpoutput("SaugeenFlow", 0.9, saugdata, groups) % @ % % \noindent The \code{cmp} object can now be analyzed: % % <>= % cmp % @ % % \noindent There are no significant $p$-values. As such, it is not possible to conclude that the river flow dynamic during the first 30 years of measurements is statistically different from the last 30 years. The MANOVA assumptions are not verified, as shown by be \code{assumptions} function: % % <>= % assumptions(cmp) % @ % % \noindent The $t$-test assumptions mostly hold, in spite of the PC1 normality test for sample A (first 30 years) being significant at the $\alpha=0.05$ level. Nonetheless, the $U$ test $p$-value provides a similar conclusion. Plotting the \code{cmp} object provides another perspective: % % <>= % plot(cmp) % @ % % \noindent The previous command will produce plots in Figure~\ref{micompr:fig:saugeen}. The PC1 vs. PC2 score plot does not show any clear sample separation and the decrease in explained variance from the first PC to the second is considerably less abrupt than what was observed for the sunspots example. Additionally, no $t$ and $U$ test $p$-values are significant for the 60 PCs after weighted Bonferroni correction, further reinforcing the conclusion that the yearly Saugeen river flow dynamic was similar during the compared periods of time. \subsection{PH\textsuperscript{2} database of dermoscopic images}\label{micompr:sec:examples:ph2} In this example we use the tools provided by the \pkg{micompr} package to study the PH\textsuperscript{2} database of dermoscopic images \citep{mendonca2013ph2}. This image database contains a total of 200 dermoscopic images of melanocytic lesions, including, from benign to more serious, 80 common nevi, 80 atypical nevi, and 40 melanomas. These are 8-bit RGB color images, with a resolution of purportedly $768 \times 560$ pixels. We have found, however, that resolutions vary between $761 \times 570$ and $769 \times 577$. As such, we resized all images to $760 \times 570$ prior to our analysis. The goal is to verify if images of the three types of lesions form statistically distinguishable samples, i.e., this is not a classification exercise such as performed by \cite{barata2014two}. Each image is considered an observation of three outputs, \textbf{r}ed, \textbf{g}reen and \textbf{b}lue, corresponding to the respective color channels. The concatenation of all outputs, i.e., channels, provides a 4\textsuperscript{th} output. The three lesion samples are compared pairwise, as follows: \begin{description} \item[1v2] Common nevi and atypical nevi. \item[1v3] Common nevi and melanomas. \item[2v3] Atypical nevi and melanomas. \end{description} The following code reads the image dataset from disk and compares images grouped by lesion type. The \code{imgfolder} variable specifies the path containing the images (resized to $760 \times 570$), while the \code{grpsfile} variable specifies the path to the file containing the sample to which each image belongs. <>= # Load libraries library(bmp) library(micompr) # Image definitions imgs <- dir(imgfolder) nimgs <- length(imgs) npixels <- 760 * 570 # Specify image groups (Common nevi, atypical nevi, # melanomas). f <- read.table(grpsfile, row.names = 1) grps <- f[order(row.names(f)), ] # Read images from disk # Use different color channels as outputs, and also # use a concatenated output rimgs <- matrix(nrow = nimgs, ncol = npixels) gimgs <- matrix(nrow = nimgs, ncol = npixels) bimgs <- matrix(nrow = nimgs, ncol = npixels) rgbimgs <- matrix(nrow = nimgs, ncol = npixels * 3) for (i in 1:nimgs) { cimg <- read.bmp(paste0(imgfolder, imgs[i])) rimgs[i, ] <- c(cimg[ , , 1]) gimgs[i, ] <- c(cimg[ , , 2]) bimgs[i, ] <- c(cimg[ , , 3]) rgbimgs[i, ] <- c(cimg[ , , 1], cimg[ , , 2], cimg[ , , 3]) } # Perform multivariate independent comparison of images mic <- micomp(outputs = c("R", "G", "B", "RGB"), ve_npcs = 0.9, comps = list( list(name = "1v2", grpout = list( data = list(R = rimgs[grps != 3, ], G = gimgs[grps != 3, ], B = bimgs[grps != 3, ], RGB = rgbimgs[grps != 3, ]), obs_lvls = factor(grps[grps != 3]))), list(name = "1v3", grpout = list( data = list(R = rimgs[grps != 2, ], G = gimgs[grps != 2, ], B = bimgs[grps != 2, ], RGB = rgbimgs[grps != 2, ]), obs_lvls = factor(grps[grps != 2]))), list(name = "2v3", grpout = list( data = list(R = rimgs[grps != 1, ], G = gimgs[grps != 1, ], B = bimgs[grps != 1, ], RGB = rgbimgs[grps != 1, ]), obs_lvls = factor(grps[grps != 1]))))) @ \noindent As in the \nameref{micompr:sec:examples:pphpc} example, the \code{mic} object can be inspected at the R prompt using the common S3 generic functions. Likewise, the \code{toLatex} function produces \LaTeX\ tables summarizing the object. The following code generates Table~\ref{tab:ph2}: <>= toLatex(mic, booktabs = T, data_show = c("parp-1", "nparp-1", "scoreplot"), data_labels = c("$t$-test", "$U$ test", "PCS"), pvalf_params = list(minval = 1e-8, na_str = "*"), label = "tab:ph2", caption = paste("Comparison of PH$^2$ dataset images", "grouped by lesion type.")) @ \noindent Note that we did not request the MANOVA $p$-values in the \code{data\_show} parameter, as in this case the required assumptions do not appear to be verified. However, assumptions for the $t$-test on the first PC seem to hold. In any case, and to complement the information provided by the $t$-test, we specified the \code{"nparp-1"} option to the \code{data\_show} argument, such that the table shows the $p$-value of the Mann-Whitney $U$ test on the first PC. \begin{table}[!t] \begin{center} \begin{tabular}{clrrrr} \toprule \multirow{2}{*}{Comp.} & \multirow{2}{*}{Data} & \multicolumn{4}{c}{Outputs} \\ \cmidrule(l){3-6} & & R & G & B & RGB\\ \midrule \multirow{3}{*}{1v2} & $t$-test & \uuline{2e-04} & \uuline{0.001} & \uline{0.033} & \uuline{0.002}\\ & $U$ test & \uuline{8e-05} & \uuline{5e-04} & \uline{0.024} & \uuline{0.001}\\ & PCS & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.226,0.203) (0.123,0.419) (0.026,-0.177) (0.235,-0.115) (-0.074,-0.254) (-0.238,-0.048) (-0.712,0.244) (0.439,0.098) (0.474,0.038) (-0.327,0.017) (0.066,-0.143) (0.197,0.193) (-0.401,0.013) (-1.000,0.372) (-0.166,0.145) (-0.099,0.498) (-0.102,-0.101) (0.691,0.032) (0.185,-0.005) (-0.444,-0.235) (-0.575,-0.226) (-0.517,-0.343) (-0.457,-0.031) (-0.838,0.362) (-0.222,0.115) (-0.296,-0.138) (-0.025,0.434) (-0.155,-0.041) (0.021,-0.093) (0.407,-0.128) (0.191,-0.010) (-0.189,-0.094) (-0.032,-0.065) (-0.510,0.191) (-0.454,0.007) (-0.511,-0.053) (-0.339,0.133) (-0.133,0.157) (0.056,-0.015) (0.051,0.015) (-0.595,0.127) (-0.350,0.016) (-0.436,0.022) (-0.405,0.303) (-0.353,-0.241) (-0.313,0.279) (-0.589,-0.181) (-0.357,0.002) (-0.615,-0.122) (0.353,-0.113) (0.211,-0.126) (0.704,0.070) (0.013,-0.123) (-0.095,-0.158) (-0.092,-0.115) (0.326,0.243) (-0.290,-0.267) (-0.246,-0.105) (-0.194,-0.205) (-0.892,-0.088) (0.039,-0.074) (0.493,0.257) (0.740,0.047) (0.046,0.260) (0.151,0.044) (-0.133,-0.098) (0.417,0.238) (-0.355,-0.064) (0.125,0.094) (0.576,0.089) (-0.440,-0.126) (-0.004,0.287) (-0.194,0.260) (-0.273,-0.092) (-0.033,0.216) (0.051,-0.013) (0.630,0.153) (-0.213,0.078) (0.097,0.282) (-0.292,0.543)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (-0.205,-0.235) (-0.004,-0.184) (-0.050,-0.307) (0.169,-0.004) (-0.516,0.295) (0.034,0.031) (-0.046,-0.130) (0.128,-0.114) (-0.669,-0.265) (0.539,-0.101) (0.188,0.114) (0.507,0.095) (0.024,0.376) (-0.349,0.270) (-0.907,-0.178) (-0.140,-0.320) (-0.364,-0.331) (-0.066,-0.328) (-0.158,-0.224) (0.388,-0.166) (0.326,-0.223) (0.380,-0.143) (0.497,-0.133) (0.373,-0.141) (-0.137,-0.102) (0.216,-0.139) (0.325,-0.190) (0.134,-0.239) (0.018,-0.167) (-0.219,0.023) (0.383,0.038) (0.526,-0.061) (0.247,-0.112) (0.104,-0.159) (-0.273,-0.134) (-0.022,0.558) (-0.676,0.130) (0.040,-0.120) (-0.091,-0.423) (-0.356,-0.185) (-0.263,-0.040) (0.412,-0.087) (0.014,0.011) (0.295,0.001) (0.156,0.025) (0.199,0.004) (0.643,0.073) (0.287,-0.210) (0.273,0.269) (0.399,-0.183) (0.342,-0.144) (0.267,-0.108) (0.343,-0.115) (0.501,-0.118) (0.152,-0.043) (0.227,-0.139) (-0.121,0.022) (0.179,-0.200) (0.424,-0.143) (0.360,-0.095) (-0.592,0.207) (0.070,-0.154) (0.224,-0.133) (0.561,0.019) (0.495,0.277) (-0.022,0.057) (0.315,0.075) (0.618,0.076) (0.297,-0.218) (0.453,0.100) (0.545,0.272) (0.105,0.267) (0.402,-0.109) (0.690,0.120) (0.395,0.327) (-0.125,0.305) (-0.189,0.096) (-0.618,-0.219) (-0.153,0.238) (-0.189,0.278)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.255,0.255) (0.346,0.351) (-0.079,-0.197) (0.455,-0.097) (-0.045,-0.262) (-0.293,-0.035) (0.014,0.290) (0.537,-0.073) (0.687,-0.154) (-0.403,-0.016) (-0.317,-0.120) (0.183,0.205) (-0.406,0.171) (-0.846,0.330) (-0.214,0.123) (0.040,0.461) (-0.493,-0.027) (0.795,-0.107) (0.111,-0.009) (-0.473,-0.339) (-0.486,-0.111) (-0.454,-0.221) (-0.248,-0.058) (-0.717,0.201) (-0.462,-0.104) (-0.203,-0.120) (0.319,0.371) (-0.203,0.041) (0.181,-0.024) (0.356,-0.163) (-0.060,-0.072) (-0.407,-0.239) (0.133,-0.130) (-0.466,0.256) (-0.550,0.064) (-0.728,0.026) (-0.687,0.067) (-0.461,0.046) (-0.241,-0.030) (-0.193,0.087) (-0.345,0.085) (-0.391,0.092) (-0.489,-0.087) (-0.205,0.234) (-0.414,-0.134) (-0.064,0.349) (-0.698,-0.263) (-0.446,-0.005) (-0.477,-0.170) (-0.005,-0.144) (-0.322,-0.209) (0.678,-0.057) (0.024,-0.108) (-0.105,-0.123) (-0.003,-0.127) (0.352,0.157) (-0.221,-0.283) (-0.030,-0.139) (-0.171,-0.194) (-0.652,-0.007) (0.211,0.003) (0.646,0.169) (0.780,-0.028) (0.293,0.280) (0.246,0.143) (0.040,-0.132) (0.566,0.216) (-0.210,-0.090) (-0.327,0.062) (0.158,0.363) (-0.339,-0.122) (-0.157,0.238) (-0.211,0.235) (-0.116,-0.014) (-0.134,0.362) (0.193,0.018) (0.694,0.061) (-0.178,-0.004) (0.007,0.311) (-0.284,0.532)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (-0.049,-0.245) (0.088,-0.192) (-0.238,-0.238) (0.129,-0.074) (-0.543,0.451) (0.111,-0.054) (-0.342,-0.157) (0.331,-0.098) (-0.488,-0.143) (0.410,-0.159) (0.184,0.027) (0.753,-0.062) (0.171,0.270) (-0.333,0.221) (-0.800,-0.065) (0.036,-0.266) (-0.313,-0.286) (-0.020,-0.298) (-0.420,-0.065) (0.329,-0.240) (0.214,-0.282) (0.351,-0.158) (0.455,-0.142) (0.291,-0.079) (-0.171,-0.124) (0.034,-0.184) (0.583,-0.171) (0.322,-0.197) (0.188,-0.145) (-0.433,0.148) (0.467,0.028) (0.755,-0.119) (0.383,-0.227) (0.380,-0.102) (-0.307,0.069) (0.034,0.774) (-0.730,0.067) (0.035,-0.218) (-0.440,-0.261) (-0.422,-0.006) (-0.202,0.128) (0.715,-0.052) (-0.074,0.064) (0.355,0.089) (0.045,0.177) (0.217,-0.029) (0.605,0.236) (0.039,-0.258) (0.323,0.089) (0.464,-0.234) (-0.063,-0.129) (-0.191,-0.125) (0.377,-0.095) (0.694,-0.178) (0.151,-0.146) (0.081,-0.201) (-0.188,0.034) (0.072,-0.246) (0.271,-0.159) (0.562,-0.073) (-0.667,0.139) (0.062,-0.114) (-0.182,-0.225) (0.675,-0.076) (0.492,0.431) (-0.146,0.022) (0.749,0.109) (0.479,0.236) (0.151,-0.243) (0.192,-0.022) (0.797,0.355) (-0.435,0.293) (0.416,-0.057) (1.000,0.098) (0.669,0.292) (-0.045,0.316) (-0.126,0.131) (-0.681,-0.156) (-0.120,0.301) (-0.185,0.173)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.220,0.151) (0.543,0.183) (-0.004,-0.119) (0.543,-0.063) (0.097,-0.108) (-0.304,0.101) (0.245,0.094) (0.540,-0.161) (0.811,-0.113) (-0.248,-0.037) (-0.373,0.002) (0.089,0.062) (-0.241,0.204) (-0.464,0.353) (-0.066,0.206) (0.223,0.314) (-0.517,0.057) (0.585,-0.170) (0.004,-0.051) (-0.331,-0.222) (-0.415,-0.099) (-0.400,-0.124) (0.070,-0.025) (-0.285,0.207) (-0.448,-0.052) (-0.268,-0.148) (0.470,0.136) (-0.145,0.049) (0.198,-0.045) (0.126,-0.138) (-0.385,-0.078) (-0.665,-0.217) (0.033,-0.188) (-0.210,0.279) (-0.425,0.231) (-0.651,0.262) (-0.553,0.131) (-0.391,0.053) (-0.469,-0.077) (-0.225,0.159) (-0.215,-0.025) (-0.550,0.080) (-0.599,-0.162) (0.158,0.263) (-0.283,-0.053) (0.055,0.194) (-0.501,-0.101) (-0.230,0.138) (-0.376,-0.181) (-0.251,-0.117) (-0.539,-0.174) (0.181,-0.128) (0.023,-0.085) (-0.130,-0.088) (0.065,-0.111) (0.323,0.018) (-0.047,-0.205) (0.072,-0.125) (-0.159,-0.120) (-0.274,0.100) (0.352,0.085) (0.682,-0.009) (0.499,-0.089) (0.374,0.158) (0.243,0.099) (0.006,-0.134) (0.611,0.033) (-0.276,-0.145) (-0.647,-0.000) (0.082,0.250) (-0.610,-0.166) (0.081,0.325) (0.157,0.221) (-0.024,0.039) (-0.023,0.371) (-0.012,-0.055) (0.312,-0.068) (0.313,0.063) (-0.015,0.343) (0.244,0.576)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (0.231,-0.178) (0.015,-0.167) (-0.287,-0.161) (0.202,-0.031) (-0.268,0.516) (0.263,-0.115) (-0.679,-0.092) (0.346,-0.102) (-0.274,-0.095) (0.240,-0.115) (0.185,-0.031) (0.777,-0.144) (0.364,0.127) (-0.350,0.100) (-0.642,-0.009) (0.269,-0.163) (-0.197,-0.061) (0.055,-0.222) (-0.487,0.073) (0.267,-0.137) (-0.069,-0.228) (0.258,-0.140) (0.227,-0.116) (0.177,0.078) (-0.104,-0.122) (-0.254,-0.147) (0.351,-0.179) (0.031,-0.181) (0.053,-0.155) (-0.429,0.158) (0.260,-0.034) (0.570,-0.168) (0.189,-0.248) (0.574,-0.063) (-0.165,0.250) (0.202,0.584) (-0.745,0.035) (0.179,-0.200) (-0.498,-0.123) (-0.454,0.115) (-0.319,0.161) (0.380,-0.114) (0.004,0.134) (0.215,0.031) (-0.039,0.156) (-0.003,-0.100) (0.615,0.131) (-0.253,-0.212) (0.422,-0.026) (0.377,-0.186) (-0.479,-0.099) (-0.635,-0.127) (0.485,-0.074) (0.709,-0.179) (0.155,-0.162) (0.095,-0.115) (0.181,0.109) (-0.262,-0.243) (-0.087,-0.155) (0.396,-0.096) (-0.681,0.140) (0.092,-0.078) (-0.336,-0.174) (0.257,-0.156) (0.644,0.242) (-0.357,-0.051) (0.567,-0.053) (0.215,0.200) (-0.083,-0.149) (-0.183,-0.120) (1.000,0.167) (-0.217,0.246) (0.270,-0.042) (0.788,-0.020) (0.697,0.055) (-0.137,0.125) (0.240,0.227) (-0.428,-0.092) (0.223,0.408) (0.145,0.166)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.272,0.198) (0.467,0.397) (-0.027,-0.173) (0.524,-0.026) (0.001,-0.142) (-0.355,0.046) (-0.063,0.401) (0.627,-0.063) (0.828,0.007) (-0.375,-0.008) (-0.307,-0.154) (0.181,0.117) (-0.403,0.195) (-0.866,0.566) (-0.174,0.238) (0.111,0.523) (-0.497,-0.079) (0.816,-0.134) (0.101,-0.080) (-0.485,-0.242) (-0.572,-0.127) (-0.540,-0.209) (-0.189,0.084) (-0.661,0.395) (-0.472,-0.059) (-0.305,-0.154) (0.380,0.382) (-0.199,0.024) (0.181,-0.039) (0.318,-0.206) (-0.170,-0.179) (-0.556,-0.325) (0.065,-0.160) (-0.436,0.343) (-0.580,0.163) (-0.789,0.132) (-0.657,0.111) (-0.419,0.056) (-0.321,-0.161) (-0.180,0.048) (-0.408,0.123) (-0.541,0.010) (-0.620,-0.164) (-0.113,0.450) (-0.413,-0.128) (-0.077,0.337) (-0.713,-0.121) (-0.405,0.125) (-0.554,-0.135) (-0.034,-0.267) (-0.346,-0.345) (0.561,-0.188) (0.024,-0.106) (-0.139,-0.142) (0.008,-0.101) (0.409,0.114) (-0.196,-0.204) (-0.046,-0.072) (-0.210,-0.160) (-0.660,0.172) (0.270,0.082) (0.760,0.146) (0.776,-0.103) (0.332,0.296) (0.267,0.117) (-0.017,-0.113) (0.673,0.180) (-0.321,-0.131) (-0.426,-0.124) (0.272,0.134) (-0.577,-0.221) (-0.022,0.364) (-0.048,0.346) (-0.140,0.048) (-0.076,0.354) (0.086,-0.063) (0.617,-0.058) (0.035,0.204) (0.018,0.336) (-0.066,0.756)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (0.042,-0.124) (0.039,-0.198) (-0.261,-0.288) (0.203,-0.001) (-0.502,0.530) (0.199,-0.029) (-0.505,-0.262) (0.346,-0.083) (-0.527,-0.093) (0.436,-0.196) (0.228,0.018) (0.852,-0.006) (0.278,0.325) (-0.403,0.186) (-0.911,-0.019) (0.109,-0.157) (-0.345,-0.154) (-0.003,-0.269) (-0.474,-0.117) (0.374,-0.207) (0.137,-0.349) (0.380,-0.184) (0.436,-0.202) (0.302,-0.070) (-0.156,-0.132) (-0.059,-0.271) (0.499,-0.213) (0.175,-0.267) (0.105,-0.183) (-0.461,0.101) (0.429,-0.030) (0.743,-0.137) (0.321,-0.252) (0.474,-0.004) (-0.296,0.135) (0.118,0.734) (-0.868,0.080) (0.117,-0.143) (-0.466,-0.346) (-0.516,-0.009) (-0.327,0.074) (0.594,-0.137) (-0.026,0.093) (0.337,0.009) (0.041,0.083) (0.143,-0.119) (0.743,0.143) (-0.036,-0.370) (0.436,0.131) (0.488,-0.211) (-0.180,-0.308) (-0.331,-0.329) (0.499,-0.071) (0.784,-0.133) (0.188,-0.136) (0.143,-0.165) (-0.012,0.156) (-0.058,-0.366) (0.181,-0.289) (0.526,-0.098) (-0.787,0.174) (0.092,-0.122) (-0.182,-0.313) (0.559,-0.179) (0.676,0.362) (-0.246,-0.089) (0.684,0.043) (0.473,0.092) (0.094,-0.294) (0.115,-0.186) (0.996,0.367) (-0.249,0.241) (0.416,-0.112) (1.000,0.065) (0.751,0.250) (-0.114,0.221) (0.020,0.294) (-0.666,-0.114) (0.022,0.470) (-0.044,0.309)}; \end{tikzpicture} }}\\ \midrule \multirow{3}{*}{1v3} & $t$-test & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{7e-07} & \uuline{<1e-08}\\ & $U$ test & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{7e-06} & \uuline{<1e-08}\\ & PCS & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.040,0.133) (0.088,0.111) (0.048,0.152) (0.215,-0.005) (-0.198,-0.150) (0.127,0.024) (0.069,0.078) (0.173,0.089) (-0.325,0.132) (0.418,0.058) (0.231,-0.058) (0.416,0.026) (0.146,-0.128) (-0.095,-0.092) (-0.472,0.016) (-0.003,0.134) (-0.133,0.075) (0.036,0.165) (-0.014,0.117) (0.326,0.076) (0.282,0.153) (0.322,0.104) (0.393,0.085) (0.317,0.075) (0.009,0.085) (0.225,0.088) (0.284,0.121) (0.159,0.172) (0.098,0.136) (-0.034,-0.022) (0.338,-0.008) (0.413,0.048) (0.241,0.100) (0.148,0.119) (-0.080,0.018) (0.116,-0.267) (-0.295,-0.140) (0.117,0.023) (0.027,0.236) (-0.129,0.045) (-0.075,0.000) (0.341,0.106) (0.119,-0.048) (0.273,0.027) (0.190,0.002) (0.218,0.023) (0.491,-0.021) (0.259,0.140) (0.293,-0.131) (0.333,0.113) (0.296,0.078) (0.254,0.059) (0.306,0.081) (0.399,0.089) (0.198,0.038) (0.237,0.070) (0.022,-0.073) (0.196,0.141) (0.346,0.115) (0.312,0.088) (-0.236,-0.202) (0.137,0.082) (0.227,0.084) (0.440,0.005) (0.415,-0.111) (0.094,-0.014) (0.289,0.046) (0.477,-0.020) (0.271,0.110) (0.386,-0.027) (0.447,-0.124) (0.168,-0.127) (0.333,0.068) (0.529,-0.065) (0.344,-0.103) (0.039,-0.111) (-0.007,-0.128) (-0.293,0.084) (0.023,-0.218) (0.013,-0.228)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (-0.509,0.009) (-0.220,0.185) (-0.611,-0.075) (-0.826,-0.021) (-0.966,0.013) (-0.785,0.101) (-0.383,0.117) (-0.481,0.237) (-0.547,0.148) (-0.580,-0.261) (-0.248,-0.236) (-0.500,-0.014) (-0.608,0.119) (-0.697,0.080) (-0.236,0.027) (0.001,-0.226) (-0.039,-0.040) (-0.949,0.115) (-0.338,0.028) (-0.184,0.044) (-0.165,0.111) (-0.007,-0.174) (-0.299,-0.080) (-0.168,-0.188) (-0.027,-0.304) (-0.115,-0.018) (0.064,-0.171) (-1.000,0.180) (-0.265,0.039) (-0.479,0.142) (-0.448,0.050) (-0.582,-0.104) (-0.049,-0.345) (0.154,-0.137) (-0.109,-0.425) (-0.005,-0.412) (-0.002,-0.170) (0.040,-0.129) (-0.014,-0.013) (0.112,-0.224)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.067,0.122) (-0.167,0.105) (0.070,0.161) (-0.214,-0.032) (0.252,-0.348) (-0.191,0.048) (0.130,0.038) (-0.342,0.077) (0.244,0.102) (-0.390,0.102) (-0.250,-0.048) (-0.633,0.081) (-0.247,-0.192) (0.122,-0.143) (0.459,0.007) (-0.128,0.150) (0.097,0.080) (-0.080,0.202) (0.193,0.028) (-0.344,0.087) (-0.244,0.191) (-0.347,0.114) (-0.424,0.102) (-0.323,-0.001) (0.018,0.078) (-0.130,0.089) (-0.514,0.123) (-0.314,0.187) (-0.229,0.127) (0.183,-0.166) (-0.448,-0.045) (-0.635,0.095) (-0.365,0.161) (-0.364,0.099) (0.098,-0.102) (-0.163,-0.532) (0.391,-0.084) (-0.131,0.124) (0.210,0.158) (0.177,-0.068) (0.031,-0.105) (-0.599,0.104) (-0.086,-0.151) (-0.348,-0.012) (-0.136,-0.093) (-0.251,0.052) (-0.541,-0.132) (-0.124,0.180) (-0.351,-0.114) (-0.431,0.142) (-0.061,0.060) (0.028,0.057) (-0.377,0.057) (-0.587,0.153) (-0.216,0.068) (-0.170,0.074) (0.024,-0.023) (-0.145,0.177) (-0.289,0.134) (-0.500,0.058) (0.337,-0.157) (-0.152,0.086) (0.030,0.141) (-0.574,0.078) (-0.461,-0.239) (-0.010,-0.031) (-0.626,0.012) (-0.451,-0.135) (-0.218,0.124) (-0.243,0.021) (-0.678,-0.201) (0.193,-0.223) (-0.391,0.069) (-0.825,-0.057) (-0.562,-0.074) (-0.077,-0.171) (-0.035,-0.118) (0.379,0.095) (-0.051,-0.268) (-0.003,-0.181)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (0.552,0.075) (0.166,0.072) (0.508,-0.160) (0.896,0.079) (0.736,-0.057) (0.663,0.007) (0.419,0.191) (0.414,0.158) (0.543,0.177) (0.508,-0.119) (0.223,-0.248) (0.369,-0.046) (0.552,0.033) (0.624,0.157) (0.344,0.058) (0.080,-0.113) (0.043,-0.113) (0.874,0.066) (0.441,-0.043) (0.542,0.071) (0.417,0.111) (0.009,-0.069) (0.177,-0.163) (0.162,-0.182) (0.141,-0.135) (0.219,0.112) (0.023,-0.107) (1.000,0.206) (0.620,0.068) (0.523,0.157) (0.496,0.075) (0.783,0.022) (0.137,-0.203) (0.140,0.003) (0.064,-0.382) (0.009,-0.212) (0.074,-0.109) (0.078,-0.048) (-0.056,-0.005) (0.075,-0.120)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.236,0.115) (-0.066,0.107) (0.177,0.124) (-0.219,-0.013) (0.143,-0.411) (-0.269,0.091) (0.485,0.021) (-0.331,0.089) (0.163,0.074) (-0.243,0.083) (-0.208,0.017) (-0.668,0.145) (-0.351,-0.083) (0.221,-0.078) (0.453,-0.027) (-0.269,0.108) (0.089,-0.038) (-0.094,0.162) (0.333,-0.069) (-0.269,0.049) (0.010,0.171) (-0.254,0.122) (-0.232,0.107) (-0.206,-0.097) (0.028,0.088) (0.149,0.085) (-0.333,0.146) (-0.071,0.168) (-0.094,0.140) (0.281,-0.156) (-0.266,0.023) (-0.504,0.152) (-0.198,0.189) (-0.504,0.085) (0.061,-0.229) (-0.235,-0.426) (0.530,-0.055) (-0.199,0.141) (0.347,0.074) (0.295,-0.133) (0.190,-0.147) (-0.350,0.146) (-0.075,-0.159) (-0.221,0.024) (-0.026,-0.098) (-0.048,0.095) (-0.551,-0.073) (0.150,0.154) (-0.393,0.012) (-0.345,0.146) (0.324,0.046) (0.449,0.065) (-0.445,0.068) (-0.610,0.178) (-0.177,0.112) (-0.131,0.053) (-0.209,-0.066) (0.159,0.182) (0.019,0.137) (-0.365,0.092) (0.475,-0.136) (-0.133,0.070) (0.213,0.113) (-0.255,0.134) (-0.575,-0.133) (0.226,0.023) (-0.499,0.115) (-0.230,-0.123) (0.008,0.079) (0.091,0.086) (-0.858,-0.083) (0.112,-0.202) (-0.266,0.068) (-0.686,0.048) (-0.603,0.036) (0.053,-0.074) (-0.266,-0.170) (0.284,0.054) (-0.256,-0.331) (-0.192,-0.154)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (0.459,0.102) (0.063,0.036) (0.348,-0.209) (0.768,0.003) (0.459,-0.084) (0.516,-0.043) (0.290,0.159) (0.272,0.090) (0.401,0.115) (0.343,-0.062) (0.269,-0.112) (0.066,-0.087) (0.436,-0.022) (0.462,0.112) (0.339,0.071) (-0.142,-0.102) (0.021,-0.101) (0.696,0.033) (0.485,-0.062) (0.435,0.035) (0.444,0.102) (-0.285,-0.063) (0.113,-0.158) (0.186,-0.118) (-0.122,-0.091) (-0.054,0.053) (-0.170,-0.217) (1.000,0.155) (0.684,0.036) (0.444,0.076) (0.366,0.061) (0.605,-0.047) (-0.239,-0.192) (-0.154,-0.088) (-0.342,-0.430) (-0.224,-0.035) (-0.222,-0.100) (-0.131,-0.123) (-0.044,0.038) (-0.275,-0.180)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.087,-0.049) (-0.106,-0.113) (0.067,-0.159) (-0.217,-0.009) (0.199,0.229) (-0.198,-0.012) (0.184,-0.172) (-0.281,-0.056) (0.243,-0.074) (-0.349,-0.107) (-0.232,-0.004) (-0.571,-0.024) (-0.251,0.107) (0.145,0.026) (0.462,0.000) (-0.132,-0.052) (0.109,-0.018) (-0.068,-0.119) (0.183,-0.094) (-0.312,-0.073) (-0.169,-0.205) (-0.307,-0.113) (-0.348,-0.117) (-0.280,-0.031) (0.011,-0.094) (-0.068,-0.161) (-0.373,-0.109) (-0.176,-0.166) (-0.139,-0.132) (0.167,0.015) (-0.350,-0.032) (-0.515,-0.061) (-0.266,-0.140) (-0.336,-0.012) (0.082,0.094) (-0.172,0.322) (0.405,0.033) (-0.149,-0.022) (0.177,-0.219) (0.203,0.007) (0.102,0.033) (-0.425,-0.117) (-0.094,0.056) (-0.280,-0.040) (-0.117,0.005) (-0.172,-0.095) (-0.527,0.072) (-0.076,-0.207) (-0.348,0.055) (-0.368,-0.117) (-0.009,-0.176) (0.076,-0.189) (-0.377,-0.041) (-0.530,-0.064) (-0.198,-0.082) (-0.180,-0.086) (-0.071,0.134) (-0.059,-0.212) (-0.203,-0.171) (-0.390,-0.082) (0.348,0.088) (-0.141,-0.064) (0.005,-0.163) (-0.420,-0.110) (-0.484,0.165) (0.040,-0.083) (-0.469,-0.041) (-0.384,0.006) (-0.158,-0.144) (-0.179,-0.127) (-0.662,0.185) (0.043,0.104) (-0.328,-0.067) (-0.679,0.031) (-0.503,0.080) (-0.021,0.054) (-0.099,0.205) (0.316,-0.056) (-0.111,0.295) (-0.073,0.211)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (0.505,-0.030) (0.151,-0.123) (0.489,0.150) (0.827,0.027) (0.720,0.104) (0.656,-0.017) (0.362,-0.096) (0.389,-0.094) (0.497,-0.095) (0.475,0.196) (0.246,0.114) (0.310,0.104) (0.533,-0.057) (0.594,-0.045) (0.304,-0.072) (-0.022,0.224) (0.035,0.068) (0.840,-0.045) (0.422,-0.029) (0.385,-0.048) (0.341,-0.127) (-0.094,0.187) (0.197,0.120) (0.170,0.111) (0.013,0.279) (0.091,0.085) (-0.071,0.210) (1.000,-0.174) (0.522,-0.117) (0.482,-0.073) (0.435,-0.021) (0.654,0.096) (-0.022,0.310) (-0.058,0.171) (-0.057,0.555) (-0.076,0.280) (-0.053,0.166) (-0.033,0.136) (-0.029,-0.007) (-0.106,0.247)}; \end{tikzpicture} }}\\ \midrule \multirow{3}{*}{2v3} & $t$-test & \uuline{<1e-08} & \uuline{<1e-08} & \uuline{3e-04} & \uuline{<1e-08}\\ & $U$ test & \uuline{6e-08} & \uuline{<1e-08} & \uuline{0.001} & \uuline{8e-07}\\ & PCS & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.065,-0.035) (-0.331,-0.021) (-0.187,-0.101) (-0.338,-0.090) (-0.129,-0.073) (-0.050,0.180) (0.233,0.030) (-0.492,-0.129) (-0.514,-0.017) (0.012,-0.073) (-0.217,-0.065) (-0.341,-0.051) (0.069,-0.002) (0.396,0.485) (-0.117,0.178) (-0.199,0.227) (-0.115,-0.074) (-0.650,-0.040) (-0.315,-0.078) (0.108,-0.124) (0.215,-0.154) (0.183,-0.104) (0.099,-0.062) (0.321,0.058) (-0.072,-0.072) (0.030,-0.132) (-0.218,-0.031) (-0.084,-0.104) (-0.184,-0.171) (-0.444,-0.096) (-0.317,-0.072) (-0.065,-0.174) (-0.152,-0.192) (0.115,-0.027) (0.086,0.159) (0.136,0.254) (0.002,0.191) (-0.130,0.144) (-0.224,-0.103) (-0.229,-0.093) (0.178,-0.092) (0.042,-0.065) (0.090,-0.147) (0.024,0.155) (0.068,-0.135) (-0.013,-0.033) (0.194,0.145) (0.024,0.134) (0.220,-0.187) (-0.408,-0.084) (-0.317,-0.125) (-0.663,-0.013) (-0.191,-0.100) (-0.115,-0.127) (-0.120,-0.114) (-0.435,-0.044) (0.017,-0.116) (-0.035,-0.069) (-0.054,-0.082) (0.394,0.047) (-0.216,-0.038) (-0.549,-0.009) (-0.678,0.011) (-0.249,-0.067) (-0.292,-0.087) (-0.098,-0.176) (-0.482,-0.068) (0.048,-0.164) (-0.281,-0.074) (-0.570,0.010) (0.105,-0.163) (-0.227,0.168) (-0.094,0.092) (-0.008,-0.073) (-0.195,0.070) (-0.224,-0.146) (-0.623,0.016) (-0.083,0.082) (-0.302,0.154) (-0.072,0.314)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (0.461,-0.000) (0.170,-0.235) (0.563,0.043) (0.801,0.127) (0.950,0.010) (0.767,-0.107) (0.331,-0.125) (0.445,-0.205) (0.511,-0.100) (0.512,0.331) (0.164,0.125) (0.449,-0.036) (0.574,-0.167) (0.667,-0.042) (0.170,-0.106) (-0.108,0.238) (-0.054,-0.021) (0.941,-0.092) (0.277,-0.093) (0.113,-0.020) (0.097,-0.122) (-0.096,0.124) (0.231,0.003) (0.082,0.095) (-0.088,0.385) (0.031,0.124) (-0.168,0.175) (1.000,-0.142) (0.197,-0.092) (0.434,-0.080) (0.394,-0.022) (0.527,0.203) (-0.058,0.315) (-0.266,0.194) (-0.005,0.514) (-0.117,0.351) (-0.096,0.081) (-0.136,0.126) (-0.079,-0.100) (-0.229,0.203)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.041,-0.007) (-0.542,0.022) (-0.130,-0.065) (-0.564,-0.033) (-0.180,-0.041) (-0.014,0.238) (-0.280,0.057) (-0.623,-0.151) (-0.753,-0.021) (0.087,-0.064) (0.034,0.040) (-0.371,-0.019) (0.072,0.073) (0.355,0.537) (-0.093,0.233) (-0.306,0.288) (0.149,-0.031) (-0.812,-0.135) (-0.297,-0.096) (0.148,-0.155) (0.181,-0.158) (0.153,-0.083) (-0.039,-0.057) (0.303,0.023) (0.123,-0.072) (-0.040,-0.124) (-0.491,0.002) (-0.072,-0.065) (-0.342,-0.158) (-0.474,-0.122) (-0.170,-0.098) (0.107,-0.173) (-0.290,-0.198) (0.095,0.014) (0.173,0.163) (0.318,0.327) (0.290,0.145) (0.131,0.076) (-0.030,-0.114) (-0.082,-0.007) (0.030,-0.083) (0.073,0.029) (0.166,-0.160) (-0.112,0.180) (0.118,-0.126) (-0.196,-0.022) (0.324,0.063) (0.104,0.113) (0.159,-0.229) (-0.196,-0.092) (0.049,-0.119) (-0.734,-0.077) (-0.233,-0.101) (-0.133,-0.138) (-0.205,-0.118) (-0.501,-0.050) (-0.031,-0.115) (-0.199,-0.065) (-0.084,-0.077) (0.267,0.073) (-0.398,0.039) (-0.730,-0.048) (-0.792,-0.097) (-0.463,-0.045) (-0.415,-0.078) (-0.242,-0.157) (-0.648,-0.153) (-0.038,-0.209) (0.024,-0.042) (-0.344,0.054) (0.048,-0.155) (-0.134,0.231) (-0.076,0.098) (-0.151,-0.055) (-0.162,0.183) (-0.367,-0.118) (-0.750,-0.022) (-0.106,0.088) (-0.291,0.256) (-0.075,0.372)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (0.508,-0.046) (0.082,-0.226) (0.434,0.039) (0.883,0.035) (0.693,-0.056) (0.616,-0.145) (0.369,-0.149) (0.356,-0.169) (0.499,-0.149) (0.446,0.216) (0.120,0.071) (0.293,-0.047) (0.498,-0.183) (0.590,-0.118) (0.275,-0.126) (-0.015,0.253) (-0.068,0.009) (0.851,-0.171) (0.371,-0.071) (0.495,-0.008) (0.360,-0.105) (-0.095,0.106) (0.075,0.055) (0.062,0.069) (0.049,0.368) (0.147,0.039) (-0.077,0.213) (1.000,-0.247) (0.576,-0.103) (0.481,-0.075) (0.441,-0.056) (0.755,0.124) (0.038,0.269) (0.054,0.200) (-0.053,0.642) (-0.107,0.268) (-0.027,0.054) (-0.014,0.117) (-0.169,-0.107) (-0.026,0.190)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.052,-0.019) (0.604,-0.070) (0.108,-0.057) (0.579,-0.078) (0.211,-0.026) (-0.102,0.211) (0.348,-0.034) (0.560,-0.195) (0.805,-0.093) (-0.082,-0.058) (-0.182,0.101) (0.197,-0.074) (-0.057,0.094) (-0.204,0.461) (0.109,0.209) (0.348,0.165) (-0.298,0.004) (0.592,-0.201) (0.120,-0.111) (-0.156,-0.120) (-0.240,-0.111) (-0.222,-0.062) (0.191,-0.059) (-0.091,0.043) (-0.241,-0.031) (-0.117,-0.112) (0.522,-0.087) (0.009,-0.062) (0.277,-0.172) (0.209,-0.190) (-0.203,-0.101) (-0.444,-0.113) (0.126,-0.184) (-0.020,0.042) (-0.191,0.196) (-0.375,0.393) (-0.315,0.142) (-0.197,0.031) (-0.273,-0.085) (-0.052,0.024) (-0.054,-0.097) (-0.322,0.063) (-0.391,-0.124) (0.299,0.160) (-0.121,-0.097) (0.179,-0.056) (-0.288,0.073) (-0.040,0.121) (-0.205,-0.198) (-0.095,-0.087) (-0.336,-0.085) (0.269,-0.121) (0.140,-0.101) (0.015,-0.111) (0.171,-0.122) (0.394,-0.103) (0.072,-0.117) (0.184,-0.090) (-0.010,-0.088) (-0.085,0.125) (0.445,0.014) (0.694,-0.134) (0.522,-0.142) (0.440,-0.083) (0.328,-0.116) (0.121,-0.139) (0.615,-0.217) (-0.128,-0.189) (-0.414,-0.015) (0.200,0.007) (-0.396,-0.117) (0.236,0.254) (0.274,0.064) (0.122,-0.084) (0.148,0.190) (0.111,-0.139) (0.377,-0.107) (0.410,0.051) (0.164,0.231) (0.397,0.364)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (-0.427,-0.045) (0.007,-0.132) (-0.282,0.099) (-0.745,0.063) (-0.409,0.047) (-0.467,-0.013) (-0.254,-0.111) (-0.229,-0.112) (-0.362,-0.079) (-0.289,0.144) (-0.202,0.039) (0.004,-0.001) (-0.389,-0.056) (-0.431,-0.057) (-0.296,-0.094) (0.222,0.213) (0.051,0.023) (-0.669,-0.060) (-0.438,0.014) (-0.397,0.022) (-0.410,-0.079) (0.368,0.063) (-0.038,0.071) (-0.120,0.010) (0.198,0.245) (0.109,-0.005) (0.264,0.228) (-1.000,-0.099) (-0.654,0.002) (-0.406,0.000) (-0.325,-0.052) (-0.568,0.145) (0.337,0.212) (0.236,0.195) (0.459,0.541) (0.307,0.085) (0.308,0.044) (0.218,0.130) (0.107,-0.100) (0.370,0.217)}; \end{tikzpicture} }} & \raisebox{-.5\height}{\resizebox {1.2cm} {1.2cm} { \begin{tikzpicture}[scale=6] \path (-1.2,-1.2) (1.2,1.2);\draw[very thin,color=gray] (0,1.1)--(0,-1.1); \draw[very thin,color=gray] (1.1,0)--(-1.1,0); \path plot[mark=square*,mark options={color=red},mark size=0.8pt] coordinates { (-0.018,-0.043) (-0.495,0.016) (-0.139,-0.086) (-0.489,-0.040) (-0.169,-0.029) (0.018,0.163) (-0.133,0.081) (-0.556,-0.141) (-0.688,0.003) (0.058,-0.072) (0.005,-0.031) (-0.299,-0.077) (0.068,0.049) (0.312,0.485) (-0.105,0.190) (-0.288,0.224) (0.115,-0.082) (-0.676,-0.138) (-0.239,-0.125) (0.137,-0.120) (0.215,-0.144) (0.189,-0.096) (-0.047,-0.007) (0.231,0.087) (0.096,-0.082) (0.042,-0.137) (-0.413,-0.009) (-0.053,-0.084) (-0.262,-0.153) (-0.367,-0.166) (-0.087,-0.156) (0.167,-0.215) (-0.183,-0.195) (0.075,0.030) (0.153,0.143) (0.279,0.262) (0.200,0.113) (0.064,0.037) (0.012,-0.161) (-0.084,-0.064) (0.086,-0.068) (0.151,-0.047) (0.218,-0.185) (-0.134,0.217) (0.103,-0.122) (-0.130,-0.021) (0.268,0.079) (0.056,0.115) (0.195,-0.186) (-0.163,-0.157) (0.027,-0.190) (-0.543,-0.141) (-0.184,-0.105) (-0.083,-0.135) (-0.163,-0.108) (-0.441,-0.074) (-0.028,-0.096) (-0.138,-0.057) (-0.039,-0.087) (0.246,0.121) (-0.350,0.026) (-0.657,-0.047) (-0.654,-0.108) (-0.383,-0.038) (-0.341,-0.088) (-0.149,-0.148) (-0.580,-0.119) (0.050,-0.198) (0.058,-0.139) (-0.368,-0.037) (0.191,-0.192) (-0.202,0.212) (-0.155,0.112) (-0.091,-0.052) (-0.168,0.123) (-0.226,-0.157) (-0.573,-0.094) (-0.208,0.133) (-0.249,0.174) (-0.191,0.389)}; \path plot[mark=diamond*,mark size=1pt] coordinates { (0.461,-0.025) (0.085,-0.148) (0.423,0.094) (0.803,0.076) (0.681,0.076) (0.617,-0.037) (0.315,-0.105) (0.343,-0.129) (0.456,-0.076) (0.408,0.246) (0.160,0.077) (0.240,0.046) (0.487,-0.096) (0.559,-0.034) (0.245,-0.108) (-0.122,0.247) (-0.056,-0.001) (0.818,-0.058) (0.363,-0.066) (0.330,-0.024) (0.290,-0.132) (-0.195,0.137) (0.113,0.066) (0.085,0.043) (-0.088,0.334) (0.017,0.064) (-0.174,0.219) (1.000,-0.153) (0.475,-0.107) (0.442,-0.048) (0.383,-0.037) (0.609,0.148) (-0.132,0.314) (-0.158,0.205) (-0.184,0.596) (-0.190,0.264) (-0.152,0.111) (-0.128,0.149) (-0.118,-0.095) (-0.218,0.236)}; \end{tikzpicture} }}\\ \bottomrule \end{tabular} \caption{Comparison of PH$^2$ dataset images grouped by lesion type.} \label{tab:ph2} \end{center} \end{table} Results in Table~\ref{tab:ph2} show that images of different lesions have statistically significant differences, when compared either by individual color channels or with the three channels concatenated. The latter seems to provide better differentiation, with the common nevi and melanoma samples (1v3 comparison) appearing to be the most dissimilar. \section{Summary}\label{micompr:sec:summary} In this vignette we presented the R package \pkg{micompr}, which implements a procedure for comparing multivariate samples associated with different factor levels or groups. The package architecture and its core components were discussed and four examples were examined. \begin{thebibliography}{29} \providecommand{\natexlab}[1]{#1} \providecommand{\url}[1]{\texttt{#1}} \expandafter\ifx\csname urlstyle\endcsname\relax \providecommand{\doi}[1]{doi: #1}\else \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi \bibitem[Anderson(2001)]{anderson2001new} M.~J. Anderson. \newblock A new method for non-parametric multivariate analysis of variance. \newblock \emph{Austral Ecology}, 26\penalty0 (1):\penalty0 32--46, Feb. 2001. \newblock \doi{10.1111/j.1442-9993.2001.01070.pp.x}. \bibitem[Barata et~al.(2014)Barata, Ruela, Francisco, Mendon{\c{c}}a, and Marques]{barata2014two} C.~Barata, M.~Ruela, M.~Francisco, T.~Mendon{\c{c}}a, and J.~S. Marques. \newblock Two systems for the detection of melanomas in dermoscopy images using texture and color features. \newblock \emph{IEEE Systems Journal}, 8\penalty0 (3):\penalty0 965--979, Sept. 2014. \newblock \doi{10.1109/JSYST.2013.2271540}. \bibitem[Baringhaus and Franz(2004)]{baringhaus2004new} L.~Baringhaus and C.~Franz. \newblock On a new multivariate two-sample test. \newblock \emph{Journal of Multivariate Analysis}, 88\penalty0 (1):\penalty0 190--206, jan 2004. \newblock \doi{10.1016/s0047-259x(03)00079-4}. \bibitem[Berg et~al.(2006)Berg, Hoefsloot, Westerhuis, Smilde, and Werf]{berg2006centering} R.~A. Berg, H.~C. Hoefsloot, J.~A. Westerhuis, A.~K. Smilde, and M.~J. Werf. \newblock Centering, scaling, and transformations: improving the biological information content of metabolomics data. \newblock \emph{BMC Genomics}, 7\penalty0 (1):\penalty0 142, 2006. \newblock \doi{10.1186/1471-2164-7-142}. \bibitem[Clarke(1993)]{clarke1993nonparametric} K.~R. Clarke. \newblock Non-parametric multivariate analyses of changes in community structure. \newblock \emph{Australian Journal of Ecology}, 18\penalty0 (1):\penalty0 117--143, mar 1993. \newblock \doi{10.1111/j.1442-9993.1993.tb00438.x}. \newblock URL \url{http://dx.doi.org/10.1111/j.1442-9993.1993.tb00438.x}. \bibitem[da~Silva(2016)]{silva2015biotools} A.~R. da~Silva. \newblock \emph{biotools: Tools for Biometry and Applied Statistics in Agricultural Science}, 2016. \newblock URL \url{https://CRAN.R-project.org/package=biotools}. \newblock R package version 3.0. \bibitem[David(2013)]{david2013validating} N.~David. \newblock Validating simulations. \newblock In \emph{Simulating Social Complexity}, Understanding Complex Systems, pages 135--171. Springer Berlin Heidelberg, 2013. \newblock ISBN 978-3-540-93812-5. \newblock \doi{10.1007/978-3-540-93813-2\_8}. \bibitem[Duong(2016)]{duong2016ks} T.~Duong. \newblock \emph{ks: Kernel Smoothing}, 2016. \newblock URL \url{https://CRAN.R-project.org/package=ks}. \newblock R package version 1.10.4. \bibitem[Duong et~al.(2012)Duong, Goud, and Schauer]{duong2012closed} T.~Duong, B.~Goud, and K.~Schauer. \newblock Closed-form density-based framework for automatic detection of cellular morphology changes. \newblock \emph{Proceedings of the National Academy of Sciences}, 109\penalty0 (22):\penalty0 8382--8387, may 2012. \newblock \doi{10.1073/pnas.1117796109}. \bibitem[Edmonds and Hales(2003)]{edmonds2003replication} B.~Edmonds and D.~Hales. \newblock Replication, replication and replication: Some hard lessons from model alignment. \newblock \emph{Journal of Artificial Societies and Social Simulation}, 6\penalty0 (4):\penalty0 11, 2003. \newblock URL \url{http://jasss.soc.surrey.ac.uk/6/4/11.html}. \bibitem[Fachada et~al.(2015)Fachada, Lopes, Martins, and Rosa]{fachada2015template} N.~Fachada, V.~V. Lopes, R.~C. Martins, and A.~C. Rosa. \newblock Towards a standard model for research in agent-based modeling and simulation. \newblock \emph{PeerJ Computer Science}, 1:\penalty0 e36, Nov. 2015. \newblock \doi{10.7717/peerj-cs.36}. \bibitem[Fachada et~al.(2017b)Fachada, Lopes, Martins, and Rosa]{fachada2015parallelization} N.~Fachada, V.~V. Lopes, R.~C. Martins, and A.~C. Rosa. (2017b) \newblock Parallelization strategies for spatial agent-based models. \newblock \emph{International Journal of Parallel Programming}, 45\penalty0 (3):\penalty0 449--481, Jun. 2017. \newblock \doi{10.1007/s10766-015-0399-9}. \bibitem[Fachada et~al.(2017a)Fachada, Lopes, Martins, and Rosa]{fachada2015model} N.~Fachada, V.~V. Lopes, R.~C. Martins, and A.~C. Rosa. (2017a) \newblock Model-independent comparison of simulation output. \newblock \emph{Simulation Modelling Practice and Theory}, 72:\penalty0 131--149, Mar. 2017. \newblock ISSN 1569-190X. \newblock \doi{10.1016/j.simpat.2016.12.013}. \newblock URL \url{http://www.sciencedirect.com/science/article/pii/S1569190X16302854}. \bibitem[Fear(2005)]{fear2005booktabsmanual} S.~Fear. \newblock \emph{Publication quality tables in {LATEX}}, Apr. 2005. \newblock URL \url{https://www.ctan.org/pkg/booktabs}. \bibitem[Franz(2014)]{franz2014cramer} C.~Franz. \newblock \emph{cramer: Multivariate nonparametric {C}ramer-Test for the two-sample-problem}, 2014. \newblock URL \url{https://CRAN.R-project.org/package=cramer}. \newblock R package version 0.9-1. \bibitem[Gibbons and Chakraborti(2010)]{gibbons2011nonparametric} J.~D. Gibbons and S.~Chakraborti. \newblock \emph{Nonparametric statistical inference}. \newblock Statistics: Textbooks \& Monographs. Chapman and Hall/CRC, Boca Raton, FL, USA, fifth edition, July 2010. \bibitem[Heller et~al.(2012)Heller, Small, and Rosenbaum]{heller2012crossmatch} R.~Heller, D.~Small, and P.~Rosenbaum. \newblock \emph{crossmatch: The Cross-match Test}, 2012. \newblock URL \url{https://CRAN.R-project.org/package=crossmatch}. \newblock R package version 1.3-1. \bibitem[Hipel and McLeod(1994)]{hipel1994time} K.~W. Hipel and A.~I. McLeod. \newblock \emph{Time series modelling of water resources and environmental systems}. \newblock Elsevier, 1994. \bibitem[Jolliffe(2002)]{jolliffe2002principal} I.~Jolliffe. \newblock \emph{Principal component analysis}. \newblock Springer Series in Statistics. Springer, second edition, 2002. \newblock \doi{10.1007/b98835}. \bibitem[Korkmaz et~al.(2014)Korkmaz, Goksuluk, and Zararsiz]{korkmaz2014mvn} S.~Korkmaz, D.~Goksuluk, and G.~Zararsiz. \newblock {MVN}: An {R} package for assessing multivariate normality. \newblock \emph{The R Journal}, 6\penalty0 (2):\penalty0 151--162, 2014. \newblock URL \url{http://journal.r-project.org/archive/2014-2/korkmaz-goksuluk-zararsiz.pdf}. \bibitem[Kruskal and Wallis(1952)]{kruskal1952use} W.~H. Kruskal and W.~A. Wallis. \newblock Use of ranks in one-criterion variance analysis. \newblock \emph{Journal of the American Statistical Association}, 47\penalty0 (260):\penalty0 583--621, 1952. \newblock \doi{10.1080/01621459.1952.10483441}. \bibitem[Krzanowski(1988)]{krzanowski1998} W.~J. Krzanowski. \newblock \emph{Principles of Multivariate Analysis: A User's Perspective}. \newblock Oxford University Press, New York, USA, 1988. \bibitem[Massey~Jr.(1951)]{massey1951kolmogorov} F.~J. Massey~Jr. \newblock The {Kolmogorov-Smirnov} test for goodness of fit. \newblock \emph{Journal of the American Statistical Association}, 46\penalty0 (253):\penalty0 68--78, 1951. \newblock \doi{10.1080/01621459.1951.10500769}. \bibitem[McLeod and Gweon(2013)]{mcleod2013optimal} A.~I. McLeod and H.~Gweon. \newblock Optimal deseasonalization for monthly and daily geophysical time series. \newblock \emph{Journal of Environmental Statistics}, 4\penalty0 (11), 2013. \newblock URL \url{http://www.jenvstat.org/v04/i11}. \bibitem[Mendon{\c{c}}a et~al.(2013)Mendon{\c{c}}a, Ferreira, Marques, Marcal, and Rozeira]{mendonca2013ph2} T.~Mendon{\c{c}}a, P.~M. Ferreira, J.~Marques, A.~R.~S. Marcal, and J.~Rozeira. \newblock {PH}\textsuperscript{2} - a dermoscopic image database for research and benchmarking. \newblock In \emph{35th International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)}, pages 5437--5440. IEEE, July 2013. \newblock \doi{10.1109/EMBC.2013.6610779}. \bibitem[Mielke~Jr et~al.(1976)Mielke~Jr, Berry, and Johnson]{mielke1976multi} P.~W. Mielke~Jr, K.~J. Berry, and E.~S. Johnson. \newblock Multi-response permutation procedures for a priori classifications. \newblock \emph{Communications in Statistics - Theory and Methods}, 5\penalty0 (14):\penalty0 1409--1424, jan 1976. \newblock \doi{10.1080/03610927608827451}. \bibitem[Montgomery and Runger(2014)]{montgomery2010applied} D.~C. Montgomery and G.~C. Runger. \newblock \emph{Applied statistics and probability for engineers}. \newblock John Wiley \& Sons, sixth edition, 2014. \bibitem[Oksanen et~al.(2016)Oksanen, Blanchet, Friendly, Kindt, Legendre, McGlinn, Minchin, O'Hara, Simpson, Solymos, Stevens, Szoecs, and Wagner]{vegan2016} J.~Oksanen, F.~G. Blanchet, M.~Friendly, R.~Kindt, P.~Legendre, D.~McGlinn, P.~R. Minchin, R.~B. O'Hara, G.~L. Simpson, P.~Solymos, M.~H.~H. Stevens, E.~Szoecs, and H.~Wagner. \newblock \emph{vegan: Community Ecology Package}, 2016. \newblock URL \url{https://CRAN.R-project.org/package=vegan}. \newblock R package version 2.4-1. \bibitem[{R Core Team}(2015)]{r2015stats} {R Core Team}. \newblock \emph{R: A Language and Environment for Statistical Computing}. \newblock R Foundation for Statistical Computing, Vienna, Austria, 2015. \newblock URL \url{https://www.R-project.org/}. \bibitem[Rizzo and Szekely(2016)]{rizzo2016energy} M.~L. Rizzo and G.~J. Szekely. \newblock \emph{energy: E-Statistics: Multivariate Inference via the Energy of Data}, 2016. \newblock URL \url{https://CRAN.R-project.org/package=energy}. \newblock R package version 1.7-0. \bibitem[Rosenbaum(2005)]{rosenbaum2005exact} P.~R. Rosenbaum. \newblock An exact distribution-free test comparing two multivariate distributions based on adjacency. \newblock \emph{Journal of the Royal Statistical Society: Series B (Statistical Methodology)}, 67\penalty0 (4):\penalty0 515--530, sep 2005. \newblock \doi{10.1111/j.1467-9868.2005.00513.x}. \bibitem[Rosenthal and Rubin(1983)]{rosenthal1983ensemble} R.~Rosenthal and D.~B. Rubin. \newblock Ensemble-adjusted p values. \newblock \emph{Psychological Bulletin}, 94\penalty0 (3):\penalty0 540--541, 1983. \newblock \doi{10.1037/0033-2909.94.3.540}. \bibitem[Shaffer(1995)]{shaffer1995multiple} J.~P. Shaffer. \newblock Multiple hypothesis testing. \newblock \emph{Annual Review of Psychology}, 46:\penalty0 561--584, Feb. 1995. \newblock \doi{10.1146/annurev.ps.46.020195.003021}. \bibitem[Sz{\'e}kely and Rizzo(2004)]{szekely2004testing} G.~J. Sz{\'e}kely and M.~L. Rizzo. \newblock Testing for equal distributions in high dimension. \newblock \emph{InterStat}, 5:\penalty0 1--6, Nov. 2004. \bibitem[Tabachnick and Fidell(2013)]{tabachnick2013using} B.~G. Tabachnick and L.~S. Fidell. \newblock \emph{Using multivariate statistics}. \newblock Pearson, sixth edition, July 2013. \bibitem[Talbert et~al.(2016)Talbert, Richards, Mielke, and Cade]{talbert2016blossom} M.~Talbert, J.~Richards, P.~Mielke, and B.~Cade. \newblock \emph{Blossom: Statistical Comparisons with Distance-Function Based Permutation Tests}, 2016. \newblock URL \url{https://CRAN.R-project.org/package=Blossom}. \newblock R package version 1.4. \bibitem[Thiele and Grimm(2015)]{thiele2015replicating} J.~C. Thiele and V.~Grimm. \newblock Replicating and breaking models: good for you and good for ecology. \newblock \emph{Oikos}, 124\penalty0 (6):\penalty0 691--696, 2015. \newblock \doi{10.1111/oik.02170}. \bibitem[WDC(2016)]{sunspots} \emph{Sunspot Number}. \newblock WDC-SILSO, Solar Influences Data Analysis Center (SIDC), Royal Observatory of Belgium, Brussels, Belgium, Apr. 2016. \newblock URL \url{http://www.sidc.be/silso/datafiles}. \bibitem[Wickham(2011)]{wickham2011testthat} H.~Wickham. \newblock testthat: Get started with testing. \newblock \emph{The R Journal}, 3:\penalty0 5--10, 2011. \newblock URL \url{http://journal.r-project.org/archive/2011-1/RJournal_2011-1_Wickham.pdf}. \bibitem[Wickham et~al.(2015)Wickham, Danenberg, and Eugster]{wickham2015roxygen2} H.~Wickham, P.~Danenberg, and M.~Eugster. \newblock \emph{roxygen2: In-Source Documentation for {R}}, 2015. \newblock URL \url{https://CRAN.R-project.org/package=roxygen2}. \newblock R package version 5.0.1. \bibitem[Wilensky(1999)]{wilensky1999compat} U.~Wilensky. \newblock \emph{{NetLogo}}. \newblock Center for Connected Learning and Computer-Based Modeling, Northwestern University, Evanston, IL, USA, 1999. \newblock URL \url{http://ccl.northwestern.edu/netlogo/}. \bibitem[Wilensky and Rand(2007)]{wilensky2007making} U.~Wilensky and W.~Rand. \newblock Making models match: replicating an agent-based model. \newblock \emph{Journal of Artificial Societies and Social Simulation}, 10\penalty0 (4):\penalty0 2, 2007. \newblock URL \url{http://jasss.soc.surrey.ac.uk/10/4/2.html}. \bibitem[Xie(2015)]{xie2015dynamic} Y.~Xie. \newblock \emph{Dynamic Documents with {R} and knitr}. \newblock CRC Press, second edition, 2015. \end{thebibliography} \end{document}