Agreement measures are useful metrics to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. While many of them have been proposed in the literature so far, Cohen’s n is still the de facto standard in gauging the agreement. Information Agreement (IA) is a novel two-observers information-theoretic-based metric introduced to overcome all the limitations and alleged pitfalls of Cohen’s n. It offers an operative meaning to the agreement since it measures – in both dichotomous and multi-value ordered-categorical cases-the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. Unfortunately, this measure is only able to deal with agreement matrices whose values are all strictly positive numbers. This work extends IA by admitting also 0 as a possible value for the entries of an agreement matrix. Moreover, a Python software library to compute the extended version of IA, together with some of the most used agreement measures, is presented and tested.

Extending Information Agreement by Continuity

Alberto Casagrande
;
Francesco Fabris;Rossano Girometti
2020-01-01

Abstract

Agreement measures are useful metrics to both compare different evaluations of the same diagnostic outcomes and validate new rating systems or devices. While many of them have been proposed in the literature so far, Cohen’s n is still the de facto standard in gauging the agreement. Information Agreement (IA) is a novel two-observers information-theoretic-based metric introduced to overcome all the limitations and alleged pitfalls of Cohen’s n. It offers an operative meaning to the agreement since it measures – in both dichotomous and multi-value ordered-categorical cases-the information shared between two raters through the virtual diagnostic channel connecting them: the more information exchanged between the raters, the higher their agreement. Unfortunately, this measure is only able to deal with agreement matrices whose values are all strictly positive numbers. This work extends IA by admitting also 0 as a possible value for the entries of an agreement matrix. Moreover, a Python software library to compute the extended version of IA, together with some of the most used agreement measures, is presented and tested.
2020
https://ieeexplore.ieee.org/document/9313173
File in questo prodotto:
File Dimensione Formato  
09313173.pdf

Accesso chiuso

Tipologia: Documento in Versione Editoriale
Licenza: Copyright Editore
Dimensione 178.83 kB
Formato Adobe PDF
178.83 kB Adobe PDF   Visualizza/Apri   Richiedi una copia
57316webtoc.pdf

Accesso chiuso

Descrizione: webtoc
Tipologia: Altro materiale allegato
Licenza: Copyright Editore
Dimensione 264.87 kB
Formato Adobe PDF
264.87 kB Adobe PDF   Visualizza/Apri   Richiedi una copia
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11368/2978213
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 2
  • ???jsp.display-item.citation.isi??? 2
social impact