Agreement measures, such as Cohen’s (Cohen 1960) and (Casagrande et al. 2020), gauge the agreement between two discrete classifiers mapping their confusion matrices into real values.
# create a confusion matrix
M <- matrix(1:9, nrow = 3, ncol = 3)
print(M)
#> [,1] [,2] [,3]
#> [1,] 1 4 7
#> [2,] 2 5 8
#> [3,] 3 6 9
# rSignificativity implements some agreement measures
rSignificativity::cohen_kappa(M)
#> [1] -0.04166667
rSignificativity::scott_pi(M)
#> [1] -0.05633803
rSignificativity::IA(M)
#> [1] 0.005631984When a golden standard classifier is available, the best among a set of classifiers can be selected by picking the one whose confusion matrix with respect to the golden standard has the maximal agreement value.
The agreement values alone lack of a meaningfulness indication. Understanding whether two classifiers significantly agree on the considered dataset is not possible by simply looking at the corresponding agreement value.
Let be the set of -confusion matrices built over a dataset having size , i.e., the elements of any matrix sum up to . For any agreement measure , the -significativity of in (Casagrande et al. 2025) is $$ \varrho_{\sigma, n, m}(c) \stackrel{\tiny\textrm{def}}{=} \frac{\left|\left\{M \in \mathcal{M}_{n,m} \, |\, \sigma(M) < c\right\}\right|}{\left|\mathcal{M}_{n,m}\right|}. $$ The -significativity is the [0,1]-normalised number of matrices in having a -value greater than . From a different point of view, it corresponds to the probability of choosing by chance a matrix whose -value greater than .
Since , computing takes time . Nevertheless, the Monte Carlo method (Metropolis and Ulam 1949) can estimate with samples in time with an error proportional to (e.g., see (Mackay 1998)).
The function significativity() lets us to both exactly
compute
and estimate it by using the Monte Carlo method.
library("rSignificativity")
# evaluate kappa-significativity of 0.5 in M_{2,100} with 10000 samples
significativity(cohen_kappa, c = 0.5, n = 2, m = 100)
#> [1] 0.888
# evaluate kappa-significativity of 0.5 in M_{2,100} with 1000 samples
significativity(cohen_kappa, c = 0.5, n = 2, m = 100, N = 1000)
#> [1] 0.897
# evaluate kappa-significativity of 0.5 in M_{2,5} with 1000 samples
significativity(cohen_kappa, c = 0.5, n = 2, m = 5, N = 1000)
#> [1] 0.835
# exactly compute kappa-significativity of 0.5 in M_{2,5}
significativity(cohen_kappa, c = 0.5, n = 2, m = 5, N = NULL)
#> [1] 0.8214286The -significativity of in (Casagrande et al. 2025), where is the set of all the -probability matrices, is $$ \rho_{\sigma, n}(c) \stackrel{\tiny\textrm{def}}{=} \frac{V\left(\left\{M \in \mathcal{P}_{n} \, |\, \sigma(M) < c\right\}\right)}{V(\Delta^{(n^2-1)})} $$ where is the -dimensional probability simplex and is the -dimensional Lebesgue measure (Casagrande et al. 2025).
If is definable in an o-minimal theory (van den Dries 1998), such as in the case of Cohen’s and , then
We can estimate
by using the Monte Carlo method with
samples in time
.
The function significativity() also implements this
algorithm.
# evaluate kappa-significativity of 0.5 in P_{2} with 1000 samples
significativity(cohen_kappa, 0.5, n = 2, N = 1000)
#> [1] 0.895
# evaluate kappa-significativity of 0.5 in P_{2} with 10000 samples
significativity(cohen_kappa, 0.5, n = 2)
#> [1] 0.8979
# successive calls to Monte Carlo methods may produce different results
significativity(cohen_kappa, 0.5, n = 2)
#> [1] 0.8961
# setting the random seed before the call guarantee repeatability
set.seed(1)
significativity(cohen_kappa, 0.5, n = 2)
#> [1] 0.9012
set.seed(1)
significativity(cohen_kappa, 0.5, n = 2)
#> [1] 0.9012Ground-truth and Agreement: Marginal Significativity
The -significativity does not distinguish between the two classifiers that produce a confusion matrix: both of them are assumed to be equally relevant and any matrix in is admissible. However, in many contexts one of the two classifiers is the gold standard, or the de facto standard, of the domain. Its classifications are correct by definition and they should not change: only the other classifier may vary.
For instance, if the gold standard labels 3 entries as and 17 entries as in a dataset of size 20, the only admissible confusion matrices are those whose rows associated with and sum up to 3 and 17, respectively. Not all the matrices in are admissible anymore and the significativity of an agreement value should take this into account. The marginal -significativity (Casagrande and Pagliarini 2025) addresses this issue.
Definition
An confusion matrix respects on rows when is the sum of the elements in the -th row of for all . The set of the confusion matrices respecting on rows is $$ \mathcal{M}_{(\mathbf{s},\cdot)} \stackrel{\tiny\textrm{def}}{=} \left\{M \in \mathbb{N}^{n\times n} \,\Big|\, \mathbf{s}(j) = \sum_{i=1}^n M(j,i) \textrm{ for all } j \in [1,n]\right\}. $$ The set of the confusion matrices respecting on columns is defined analogously. Whenever , both and are subsets of .
The marginal -significativity of respecting is $$ \varrho_{\sigma, (\mathbf{s},\cdot)}(c) \stackrel{\tiny\textrm{def}}{=} \frac{\left|\left\{M \in \mathcal{M}_{(\mathbf{s},\cdot)} \, |\, \sigma(M) < c\right\}\right|}{\left|\mathcal{M}_{(\mathbf{s},\cdot)}\right|}. $$ It is the probability of selecting by chance, with uniform distribution, a confusion matrix whose rows sum up to and whose -value is lower than . If the rows of the matrix represent the gold standard, is the probability that a classifier randomly labelling the dataset entries reaches an agreement value lower than with the gold standard.
The -th row of every matrix in is a weak composition of in parts, i.e., a tuple of natural numbers summing up to . The set of the weak compositions of in parts has cardinality (Bóna 2016). Since the rows of the matrices in are pairwise independent, Hence, enumerating takes time . Nevertheless, the Monte Carlo method (Metropolis and Ulam 1949) can estimate using samples with an error in in time .
This is the algorithm implemented by
marginal.significativity(), whose parameter N
sets the number of samples (default: 10000).
s <- as.integer(c(10, 20, 30))
set.seed(1)
marginal.significativity(cohen_kappa, 0.2, s, N = 1000)
#> [1] 0.799
set.seed(1)
marginal.significativity(cohen_kappa, 0.2, s, N = 100000)
#> [1] 0.8186