Skip to contents

Agreement measures, such as Cohen’s κ\kappa(Cohen 1960) and IA\textrm{IA}(Casagrande et al. 2020), gauge the agreement between two discrete classifiers mapping their confusion matrices into real values.

# create a confusion matrix
M <- matrix(1:9, nrow = 3, ncol = 3)
print(M)
#>      [,1] [,2] [,3]
#> [1,]    1    4    7
#> [2,]    2    5    8
#> [3,]    3    6    9

# rSignificativity implements some agreement measures
rSignificativity::cohen_kappa(M)
#> [1] -0.04166667
rSignificativity::scott_pi(M)
#> [1] -0.05633803
rSignificativity::IA(M)
#> [1] 0.005631984

When a golden standard classifier is available, the best among a set of classifiers can be selected by picking the one whose confusion matrix with respect to the golden standard has the maximal agreement value.

The agreement values alone lack of a meaningfulness indication. Understanding whether two classifiers significantly agree on the considered dataset is not possible by simply looking at the corresponding agreement value.

Let ℳn,m\mathcal{M}_{n,m} be the set of n×nn \times n-confusion matrices built over a dataset having size mm, i.e., the elements of any matrix ℳn,m\mathcal{M}_{n,m} sum up to mm. For any agreement measure σ\sigma, the σ\sigma-significativity of cc in ℳn,m\mathcal{M}_{n,m}(Casagrande et al. 2025) is $$ \varrho_{\sigma, n, m}(c) \stackrel{\tiny\textrm{def}}{=} \frac{\left|\left\{M \in \mathcal{M}_{n,m} \, |\, \sigma(M) < c\right\}\right|}{\left|\mathcal{M}_{n,m}\right|}. $$ The σ\sigma-significativity is the [0,1]-normalised number of matrices in ℳn,m\mathcal{M}_{n,m} having a σ\sigma-value greater than cc. From a different point of view, it corresponds to the probability of choosing by chance a matrix whose σ\sigma-value greater than cc.

Since |ℳn,m|=(n2+m−1m)|\mathcal{M}_{n,m}|=\binom{n^2+m-1}{m}, computing 𝜚σ,n,m(c)\varrho_{\sigma, n, m}(c) takes time O(n2(n2+m−1m))O\left(n^2\binom{n^2+m-1}{m}\right). Nevertheless, the Monte Carlo method (Metropolis and Ulam 1949) can estimate 𝜚σ,n,m(c)\varrho_{\sigma, n, m}(c) with NN samples in time Θ(N(n2+m))\Theta(N(n^2+m)) with an error proportional to 1/N1/\sqrt{N} (e.g., see (Mackay 1998)).

The function significativity() lets us to both exactly compute 𝜚σ,n,m(c)\varrho_{\sigma, n, m}(c) and estimate it by using the Monte Carlo method.

library("rSignificativity")

# evaluate kappa-significativity of 0.5 in M_{2,100} with 10000 samples
significativity(cohen_kappa, c = 0.5, n = 2, m = 100)
#> [1] 0.888

# evaluate kappa-significativity of 0.5 in M_{2,100} with 1000 samples
significativity(cohen_kappa, c = 0.5, n = 2, m = 100, N = 1000)
#> [1] 0.897

# evaluate kappa-significativity of 0.5 in M_{2,5} with 1000 samples
significativity(cohen_kappa, c = 0.5, n = 2, m = 5, N = 1000)
#> [1] 0.835

# exactly compute kappa-significativity of 0.5 in M_{2,5}
significativity(cohen_kappa, c = 0.5, n = 2, m = 5, N = NULL)
#> [1] 0.8214286

The σ\sigma-significativity of cc in 𝒫n\mathcal{P}_{n}(Casagrande et al. 2025), where 𝒫n\mathcal{P}_{n} is the set of all the n×nn \times n-probability matrices, is $$ \rho_{\sigma, n}(c) \stackrel{\tiny\textrm{def}}{=} \frac{V\left(\left\{M \in \mathcal{P}_{n} \, |\, \sigma(M) < c\right\}\right)}{V(\Delta^{(n^2-1)})} $$ where Δ(k−1)={⟨x1,…xk⟩∈ℝ≥0k|∑i=1kxi=1}\Delta^{(k-1)} =\left\{ \langle x_1, \ldots x_k \rangle \in \mathbb{R}_{\geq 0}^k\, |\, \sum_{i=1}^k x_i = 1\right\} is the kk-dimensional probability simplex and V(⋅)V(\cdot) is the n2−1n^2-1-dimensional Lebesgue measure (Casagrande et al. 2025).

If σ(M)<c\sigma(M) < c is definable in an o-minimal theory (van den Dries 1998), such as in the case of Cohen’s κ\kappa and IA*\textrm{IA}*, then limm→+∞𝜚σ,n,m(c)=ρσ,n(c). \lim_{m \rightarrow +\infty}\varrho_{\sigma, n, m}(c) = \rho_{\sigma, n}(c).

We can estimate ρσ,n(c)\rho_{\sigma, n}(c) by using the Monte Carlo method with NN samples in time Θ(n2N)\Theta(n^2 N). The function significativity() also implements this algorithm.

# evaluate kappa-significativity of 0.5 in P_{2} with 1000 samples
significativity(cohen_kappa, 0.5, n = 2, N = 1000)
#> [1] 0.895

# evaluate kappa-significativity of 0.5 in P_{2} with 10000 samples
significativity(cohen_kappa, 0.5, n = 2)
#> [1] 0.8979

# successive calls to Monte Carlo methods may produce different results
significativity(cohen_kappa, 0.5, n = 2)
#> [1] 0.8961

# setting the random seed before the call guarantee repeatability
set.seed(1)
significativity(cohen_kappa, 0.5, n = 2)
#> [1] 0.9012

set.seed(1)
significativity(cohen_kappa, 0.5, n = 2)
#> [1] 0.9012

Ground-truth and Agreement: Marginal Significativity

The σ\sigma-significativity does not distinguish between the two classifiers that produce a confusion matrix: both of them are assumed to be equally relevant and any matrix in ℳn,m\mathcal{M}_{n,m} is admissible. However, in many contexts one of the two classifiers is the gold standard, or the de facto standard, of the domain. Its classifications are correct by definition and they should not change: only the other classifier may vary.

For instance, if the gold standard labels 3 entries as AA and 17 entries as BB in a dataset of size 20, the only admissible confusion matrices are those whose rows associated with AA and BB sum up to 3 and 17, respectively. Not all the matrices in ℳn,m\mathcal{M}_{n,m} are admissible anymore and the significativity of an agreement value should take this into account. The marginal σ\sigma-significativity (Casagrande and Pagliarini 2025) addresses this issue.

Definition

An n×nn \times n confusion matrix MMrespects 𝐬∈ℕn\mathbf{s} \in \mathbb{N}^n on rows when 𝐬(i)\mathbf{s}(i) is the sum of the elements in the ii-th row of MM for all i∈[1,n]i \in [1, n]. The set of the confusion matrices respecting 𝐬\mathbf{s} on rows is $$ \mathcal{M}_{(\mathbf{s},\cdot)} \stackrel{\tiny\textrm{def}}{=} \left\{M \in \mathbb{N}^{n\times n} \,\Big|\, \mathbf{s}(j) = \sum_{i=1}^n M(j,i) \textrm{ for all } j \in [1,n]\right\}. $$ The set ℳ(⋅,𝐬)\mathcal{M}_{(\cdot,\mathbf{s})} of the confusion matrices respecting 𝐬\mathbf{s} on columns is defined analogously. Whenever m=∑i=1n𝐬(i)m = \sum_{i=1}^n \mathbf{s}(i), both ℳ(𝐬,⋅)\mathcal{M}_{(\mathbf{s},\cdot)} and ℳ(⋅,𝐬)\mathcal{M}_{(\cdot,\mathbf{s})} are subsets of ℳn,m\mathcal{M}_{n,m}.

The marginal σ\sigma-significativity of cc respecting 𝐬\mathbf{s} is $$ \varrho_{\sigma, (\mathbf{s},\cdot)}(c) \stackrel{\tiny\textrm{def}}{=} \frac{\left|\left\{M \in \mathcal{M}_{(\mathbf{s},\cdot)} \, |\, \sigma(M) < c\right\}\right|}{\left|\mathcal{M}_{(\mathbf{s},\cdot)}\right|}. $$ It is the probability of selecting by chance, with uniform distribution, a confusion matrix whose rows sum up to 𝐬\mathbf{s} and whose σ\sigma-value is lower than cc. If the rows of the matrix represent the gold standard, 𝜚σ,(𝐬,⋅)(c)\varrho_{\sigma, (\mathbf{s},\cdot)}(c) is the probability that a classifier randomly labelling the dataset entries reaches an agreement value lower than cc with the gold standard.

The jj-th row of every matrix in ℳ(𝐬,⋅)\mathcal{M}_{(\mathbf{s},\cdot)} is a weak composition of 𝐬(j)\mathbf{s}(j) in nn parts, i.e., a tuple of nn natural numbers summing up to 𝐬(j)\mathbf{s}(j). The set 𝒞m,k\mathcal{C}_{m,k} of the weak compositions of mm in kk parts has cardinality (m+k−1m)\binom{m+k-1}{m}(Bóna 2016). Since the rows of the matrices in ℳ(𝐬,⋅)\mathcal{M}_{(\mathbf{s},\cdot)} are pairwise independent, |ℳ(𝐬,⋅)|=∏j=1n(𝐬(j)+n−1𝐬(j)). \left|\mathcal{M}_{(\mathbf{s},\cdot)}\right| = \prod_{j=1}^n \binom{\mathbf{s}(j)+n-1}{\mathbf{s}(j)}. Hence, enumerating ℳ(𝐬,⋅)\mathcal{M}_{(\mathbf{s},\cdot)} takes time O(n2(n2+m−1m))O\left(n^2\binom{n^2+m-1}{m}\right). Nevertheless, the Monte Carlo method (Metropolis and Ulam 1949) can estimate 𝜚σ,(𝐬,⋅)(c)\varrho_{\sigma, (\mathbf{s},\cdot)}(c) using NN samples with an error in O(1/N)O(1/\sqrt{N}) in time O(N(n2+m))∩Ω(Nn2)O(N(n^2+m)) \cap \Omega(N n^2).

This is the algorithm implemented by marginal.significativity(), whose parameter N sets the number of samples (default: 10000).

s <- as.integer(c(10, 20, 30))

set.seed(1)
marginal.significativity(cohen_kappa, 0.2, s, N = 1000)
#> [1] 0.799

set.seed(1)
marginal.significativity(cohen_kappa, 0.2, s, N = 100000)
#> [1] 0.8186

References

Bóna, Miklós. 2016. A Walk Through Combinatorics: An Introduction to Enumeration and Graph Theory, Second Edition. 4th ed. World Scientific Publishing Company. https://doi.org/10.1142/10258.
Casagrande, Alberto, Francesco Fabris, and Rossano Girometti. 2020. “Extending Information Agreement by Continuity.” 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 1432–39. https://doi.org/10.1109/BIBM49941.2020.9313173.
Casagrande, Alberto, Francesco Fabris, Rossano Girometti, and Roberto Pagliarini. 2025. “Significativity Indices for Agreement Values.” Statistics and Computing 35 (6): 197. https://doi.org/10.1007/s11222-025-10728-1.
Casagrande, Alberto, and Roberto Pagliarini. 2025. “A Marginal Significativity Index for Agreement Values.” 2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 5392–99. https://doi.org/10.1109/BIBM66473.2025.11356859.
Cohen, Jacob. 1960. “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement 20: 37–46. https://doi.org/10.1177/001316446002000104.
Mackay, D. J. C. 1998. “Introduction to Monte Carlo Methods.” In Learning in Graphical Models, edited by Michael I. Jordan. Springer Netherlands. https://doi.org/10.1007/978-94-011-5014-9_7.
Metropolis, Nicholas, and Stanisław Ulam. 1949. “The Monte Carlo Method.” Journal of the American Statistical Association 44 (247): 335–41. https://doi.org/10.1080/01621459.1949.10483310.
van den Dries, Lou P. D. 1998. Tame Topology and o-Minimal Structures. London Mathematical Society Lecture Note Series. Cambridge University Press.