Introduction

The evaluomeR package permits to evaluate the reliability of bioinformatic metrics by analysing the stability and goodness of the classifications of such metrics. The method takes the measurements of the metrics for the dataset and evaluates the reliability of the metrics according to the following analyses: Correlations, Stability and Goodness of classifications.

• Correlations: Calculation of Pearson correlation coefficient between every pair of metrics available in order to quantify their interrelationship degree. The score is in the range [-1,1].

  • Perfect correlations: -1 (inverse), and 1 (direct).

• Stability: This analysis permits to estimate whether the clustering is meaningfully affected by small variations in the sample (Milligan and Cheng 1996). First, a clustering using the k-means algorithm is carried out. The value of K can be provided by the user. Then, the stability index is the mean of the Jaccard coefficient (Jaccard 1901) values of a number of bootstrap replicates. The values are in the range [0,1], having the following meaning:

  • Unstable: [0, 0.60[.
  • Doubtful: [0.60, 0.75].
  • Stable: ]0.75, 0.85].
  • Highly Stable: ]0.85, 1].

• Goodness of classifications: The goodness of the classifications are assessed by validating the clusters generated. For this purpose, we use the Silhouette width as validity index. This index computes and compares the quality of the clustering outputs found by the different metrics, thus enabling to measure the goodness of the classification for both instances and metrics. More precisely, this goodness measurement provides an assessment of how similar an instance is to other instances from the same cluster and dissimilar to the rest of clusters. The average on all the instances quantifies how the instances appropriately are clustered. Kaufman and Rousseeuw (Kaufman and Rousseeuw 2009) suggested the interpretation of the global Silhouette width score as the effectiveness of the clustering structure. The values are in the range [0,1], having the following meaning:

  • There is no substantial clustering structure: [-1, 0.25].
  • The clustering structure is weak and could be artificial: ]0.25, 0.50].
  • There is a reasonable clustering structure: ]0.50, 0.70].
  • A strong clustering structure has been found: ]0.70, 1].

Installation

The installation of evaluomeR package is performed via Bioconductor:

if (!requireNamespace("BiocManager", quietly=TRUE))
    install.packages("BiocManager")
BiocManager::install("evaluomeR")

Using evaluomeR

library(evaluomeR)
data("ontMetrics")
data("rnaMetrics")
data("bioMetrics")

library(evaluomeR)
data("rnaMetrics")
correlationSE <- metricsCorrelations(rnaMetrics, margins =  c(4,4,12,10))

## 

## Data loaded.
## Number of rows: 16
## Number of columns: 3

# Access the correlation matrix via its first assay:
# assay(correlationSE,1)

stabilityData <- stability(rnaMetrics, k=2, bs = 100)

stabilityData

## ExperimentList class object of length 9:
##  [1] stability_mean: SummarizedExperiment with 2 rows and 2 columns
##  [2] cluster_partition: SummarizedExperiment with 2 rows and 2 columns
##  [3] cluster_mean: SummarizedExperiment with 2 rows and 2 columns
##  [4] cluster_centers: SummarizedExperiment with 2 rows and 2 columns
##  [5] cluster_size: SummarizedExperiment with 2 rows and 2 columns
##  [6] cluster_betweenss: SummarizedExperiment with 2 rows and 2 columns
##  [7] cluster_totss: SummarizedExperiment with 2 rows and 2 columns
##  [8] cluster_tot.withinss: SummarizedExperiment with 2 rows and 2 columns
##  [9] cluster_anova: SummarizedExperiment with 2 rows and 2 columns

assay(stabilityData, "stability_mean")

stabilityRangeData = stabilityRange(rnaMetrics, k.range=c(2,4), bs = 100)

qualityData = quality(rnaMetrics, k = 4)

assay(qualityData,1)

k.range = c(4,6)
qualityRangeData = qualityRange(rnaMetrics, k.range)

diff(k.range)+1

## [1] 3

length(qualityRangeData)

## [1] 3

k5Data = qualityRangeData$k_5
k5Data = qualityRangeData[["k_5"]]
k5Data = getDataQualityRange(qualityRangeData, 5)
assay(k5Data, 1)

stabilityData <- stability(rnaMetrics, k=5, bs = 50, getImages = FALSE)

Selecting the optimal value of k

evaluomeR analyzes the behavior of the metrics in terms of stability and goodness of the clusters for a range of values of \(k\). In case of wishing to select the optimal value for \(k\) for a metric in a given dataset we have implemented the getOptimalKValue function, which returns a table stating which is the optimal value of k for each metric.

The algorithm works as follows: The highest stability and the highest goodness are obtained for the same value of \(k\). In such case, that value would be the optimal one. On the other hand, the highest stability and the highest goodness are obtained for different values of \(k\). In this case, additional criteria are needed. does not currently aim at providing those criteria, but to provide the data that could permit the user to make decisions. In the use cases described in this paper, we will apply the following criteria for the latter case:

• If both values of \(k\) provide at least stable classifications (value >0.75), then we select the value of \(k\) that provides the largest Silhouette width. The same would happen if none provides stable classifications.

• If \(k_1\) provides stable classifications and \(k_2\) does not, we will select \(k_1\) if the width of the Silhouette is at least reasonable.

• If \(k_1\) provides stable classifications, \(k_2\) does not, and the width of the Silhouette of \(k_1\) is less than reasonable, then we will select the value of \(k\) with the largest Silhouette width.

stabilityData <- stabilityRange(data=ontMetrics, k.range=c(2,4), 
                                bs=20, getImages = FALSE, seed=100)
qualityData <- qualityRange(data=ontMetrics, k.range=c(2,4),
                            getImages = FALSE, seed=100)

kOptTable <- getOptimalKValue(stabilityData, qualityData)

Additionally, you can select another subset of k.range to delimit the range of the optimal k.

kOptTable <- getOptimalKValue(stabilityData, qualityData, k.range=c(3,4))

## Processing metric: ANOnto

##  Stability k '4' is stable but quality k '3' is not

##  Using '4' since it provides higher stability

## Processing metric: AROnto

##  Maximum stability and quality values matches the same K value: '4'

## Processing metric: CBOOnto

##  Both Ks have a stable classification: '4', '3'

##  Using '3' since it provides higher silhouette width

## Processing metric: CBOOnto2

##  Both Ks have a stable classification: '4', '3'

##  Using '3' since it provides higher silhouette width

## Processing metric: CROnto

##  Both Ks have a stable classification: '4', '3'

##  Using '3' since it provides higher silhouette width

## Processing metric: DITOnto

##  Maximum stability and quality values matches the same K value: '4'

## Processing metric: INROnto

##  Stability k '4' is stable but quality k '3' is not

##  Using '4' since it provides higher stability

## Processing metric: LCOMOnto

##  Maximum stability and quality values matches the same K value: '3'

## Processing metric: NACOnto

##  Both Ks do not have a stable classification: '4', '3'

##  Using '3' since it provides higher silhouette width

## Processing metric: NOCOnto

##  Maximum stability and quality values matches the same K value: '3'

## Processing metric: NOMOnto

##  Maximum stability and quality values matches the same K value: '3'

## Processing metric: POnto

##  Stability k '4' is stable but quality k '3' is not

##  Using '4' since it provides higher stability

## Processing metric: PROnto

##  Maximum stability and quality values matches the same K value: '3'

## Processing metric: RFCOnto

##  Maximum stability and quality values matches the same K value: '3'

## Processing metric: RROnto

##  Maximum stability and quality values matches the same K value: '3'

## Processing metric: TMOnto

##  Both Ks have a stable classification: '4', '3'

##  Using '3' since it provides higher silhouette width

## Processing metric: TMOnto2

##  Both Ks have a stable classification: '3', '4'

##  Using '4' since it provides higher silhouette width

## Processing metric: WMCOnto

##  Both Ks have a stable classification: '4', '3'

##  Using '3' since it provides higher silhouette width

## Processing metric: WMCOnto2

##  Both Ks have a stable classification: '4', '3'

##  Using '3' since it provides higher silhouette width

Metric analysis

We provide a series of methods for a further analysis on the metrics. These methods are: plotMetricsMinMax, plotMetricsBoxplot, plotMetricsCluster and plotMetricsViolin.

The plotMetricsMinMax function plots the minimum, maximum and standard deviation of min/max points of the values of each metric:

plotMetricsMinMax(ontMetrics)

## Warning: The `size` argument of `element_line()` is deprecated as of ggplot2 3.4.0.
## ℹ Please use the `linewidth` argument instead.
## ℹ The deprecated feature was likely used in the evaluomeR package.
##   Please report the issue at <https://github.com/neobernad/evaluomeR/issues>.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.

## Warning: Use of `dataStats.df.t$Min` is discouraged.
## ℹ Use `Min` instead.

## Warning: Use of `dataStats.df.t$Max` is discouraged.
## ℹ Use `Max` instead.

## Warning: Use of `dataStats.df.t$Metric` is discouraged.
## ℹ Use `Metric` instead.

## Warning: Use of `dataStats.df.t$Min` is discouraged.
## ℹ Use `Min` instead.

## Warning: Use of `dataStats.df.t$Metric` is discouraged.
## ℹ Use `Metric` instead.

## Warning: Use of `dataStats.df.t$Max` is discouraged.
## ℹ Use `Max` instead.

## Warning: Use of `dataStats.df.t$Metric` is discouraged.
## ℹ Use `Metric` instead.

## Warning: Use of `dataStats.df.t$Max` is discouraged.
## ℹ Use `Max` instead.

## Warning: Use of `dataStats.df.t$Sd` is discouraged.
## ℹ Use `Sd` instead.

## Warning: Use of `dataStats.df.t$Max` is discouraged.
## ℹ Use `Max` instead.

## Warning: Use of `dataStats.df.t$Sd` is discouraged.
## ℹ Use `Sd` instead.

## Warning: Use of `dataStats.df.t$Metric` is discouraged.
## ℹ Use `Metric` instead.

## Warning: Use of `dataStats.df.t$Min` is discouraged.
## ℹ Use `Min` instead.

## Warning: Use of `dataStats.df.t$Sd` is discouraged.
## ℹ Use `Sd` instead.

## Warning: Use of `dataStats.df.t$Min` is discouraged.
## ℹ Use `Min` instead.

## Warning: Use of `dataStats.df.t$Sd` is discouraged.
## ℹ Use `Sd` instead.

## Warning: Use of `dataStats.df.t$Metric` is discouraged.
## ℹ Use `Metric` instead.

The plotMetricsBoxplot method boxplots the value of each metric:

plotMetricsBoxplot(rnaMetrics)

## Warning: Use of `data.melt$variable` is discouraged.
## ℹ Use `variable` instead.

## Warning: Use of `data.melt$value` is discouraged.
## ℹ Use `value` instead.

## Ignoring unknown labels:
## • fill : "Metrics"

Next, the plotMetricsCluster function clusters the values of the metrics by using the euclidean distance and the method ward.D2 from hclust:

plotMetricsCluster(ontMetrics)

And finally the plotMetricsViolin function:

plotMetricsViolin(rnaMetrics)

## Warning: Use of `data.melt$variable` is discouraged.
## ℹ Use `variable` instead.

## Warning: Use of `data.melt$value` is discouraged.
## ℹ Use `value` instead.

## Warning: Use of `data.melt$variable` is discouraged.
## ℹ Use `variable` instead.

## Warning: Use of `data.melt$value` is discouraged.
## ℹ Use `value` instead.

## Ignoring unknown labels:
## • fill : "Metrics"

Information

sessionInfo()

## R version 4.5.2 (2025-10-31)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.3 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## time zone: Etc/UTC
## tzcode source: system (glibc)
## 
## attached base packages:
## [1] stats4    stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] evaluomeR_1.26.0            sparcl_1.0.4               
##  [3] RSKC_2.4.2                  flexclust_1.5.0            
##  [5] flexmix_2.3-20              lattice_0.22-7             
##  [7] randomForest_4.7-1.2        fpc_2.2-13                 
##  [9] cluster_2.1.8.1             MultiAssayExperiment_1.37.1
## [11] SummarizedExperiment_1.41.0 Biobase_2.70.0             
## [13] GenomicRanges_1.63.0        Seqinfo_1.1.0              
## [15] IRanges_2.45.0              S4Vectors_0.49.0           
## [17] BiocGenerics_0.56.0         generics_0.1.4             
## [19] MatrixGenerics_1.23.0       matrixStats_1.5.0          
## [21] magrittr_2.0.4              kableExtra_1.4.0           
## [23] BiocStyle_2.38.0           
## 
## loaded via a namespace (and not attached):
##  [1] tidyselect_1.2.1    viridisLite_0.4.2   dplyr_1.1.4        
##  [4] farver_2.1.2        S7_0.2.0            fastmap_1.2.0      
##  [7] digest_0.6.37       lifecycle_1.0.4     kernlab_0.9-33     
## [10] compiler_4.5.2      rlang_1.1.6         sass_0.4.10        
## [13] tools_4.5.2         plotrix_3.8-4       yaml_2.3.10        
## [16] corrplot_0.95       knitr_1.50          labeling_0.4.3     
## [19] S4Arrays_1.11.0     mclust_6.1.2        DelayedArray_0.37.0
## [22] plyr_1.8.9          xml2_1.4.1          RColorBrewer_1.1-3 
## [25] abind_1.4-8         withr_3.0.2         sys_3.4.3          
## [28] nnet_7.3-20         grid_4.5.2          ggplot2_4.0.0      
## [31] scales_1.4.0        MASS_7.3-65         prabclus_2.3-4     
## [34] cli_3.6.5           rmarkdown_2.30      rstudioapi_0.17.1  
## [37] robustbase_0.99-6   reshape2_1.4.4      cachem_1.1.0       
## [40] stringr_1.6.0       modeltools_0.2-24   parallel_4.5.2     
## [43] BiocManager_1.30.26 XVector_0.51.0      vctrs_0.6.5        
## [46] Matrix_1.7-4        jsonlite_2.0.0      systemfonts_1.3.1  
## [49] maketools_1.3.2     diptest_0.77-2      jquerylib_0.1.4    
## [52] ggdendro_0.2.0      glue_1.8.0          DEoptimR_1.1-4     
## [55] stringi_1.8.7       gtable_0.3.6        tibble_3.3.0       
## [58] pillar_1.11.1       htmltools_0.5.8.1   R6_2.6.1           
## [61] textshaping_1.0.4   Rdpack_2.6.4        evaluate_1.0.5     
## [64] rbibutils_2.4       bslib_0.9.0         class_7.3-23       
## [67] Rcpp_1.1.0          svglite_2.2.2       SparseArray_1.11.1 
## [70] xfun_0.54           buildtools_1.0.0    pkgconfig_2.0.3