Table 4 Agreement by Cohen’s and Fleiss’ Kappa statistics
Agreement | Kappa | SE/Zb | CI95% | Strength of agreementa |
|---|---|---|---|---|
a. Nominal binary Cohen’s kappa | ||||
Overall addenda vs appraisals added benefit | 0.183 | 0.088 | 0.010–0.357 | poor |
Overall addenda vs appraisals evidence level | 0.353 | 0.085 | 0.187–0.520 | fair |
Infectious diseases addenda vs appraisals added benefit | −0.029 | 0.108 | −0.241 – 0.183 | less than by chance |
Infectious diseases addenda vs appraisals evidence level | 0.492 | 0.152 | 0.194–0.791 | moderate |
Infectious diseases addenda vs appraisals benefit and evidence level | 0.186 | 0.170 | −0.146 – 0.518 | poor |
Metabolic diseases addenda vs appraisals added benefit | 0.308 | 0.301 | −0.283 – 0.898 | fair |
Metabolic diseases addenda vs appraisals evidence level | 0.308 | 0.301 | −0.283 – 0.898 | fair |
Metabolic diseases addenda vs appraisals benefit and evidence level | 0.308 | 0.301 | −0.283 – 0.898 | fair |
Oncological diseases addenda vs appraisals added benefit | 0.073 | 0.133 | −0.188 – 0.333 | poor |
Oncological diseases addenda vs appraisals evidence level | 0.176 | 0.132 | −0.081 – 0.434 | poor |
Oncological diseases addenda vs appraisals benefit and evidence level | 0.188 | 0.159 | − 0.123 – 0.499 | poor |
Neurological diseases addenda vs appraisals all three outcomes | 1.000 | 1.000 | 1.000–1.000 | perfect |
Respiratory diseases addenda vs appraisals added benefit | −0.154 | 0.415 | −0.967 – 0.660 | less than by chance |
Respiratory diseases addenda vs appraisals evidence level | −0.667 | 0.248 | −1.000 – −0.180 | less than by chance |
Respiratory diseases addenda vs appraisals benefit and evidence level | −0.500 | 0.375 | −1.000 – 0.235 | less than by chance |
Other diseases addenda vs appraisals added benefit | 0.613 | 0.199 | 0.222–1.000 | substantial |
Other diseases addenda vs appraisals evidence level | 0.625 | 0.181 | 0.270–0.980 | substantial |
Other diseases addenda vs appraisals benefit and evidence level | 0.800 | 0.188 | 0.432–1.000 | substantial |
b. Ordinal Fleiss’ kappa | ||||
Overall assessments vs addenda vs appraisals added benefit Cat1 “no added benefit” Cat2 “non-quantifiable added benefit” Cat3 “minor added benefit” Cat4 “considerable added benefit” Cat5 “major added benefit” | 0.474 0.622 0.384 0.338 0.386 0.375 | 0.034 Z 12.007 Z 7.399 Z 6.513 Z 7.453 Z 7.229 | 0.408–0.540 | moderate |
Overall assessments vs addenda vs appraisals evidence level Cat1 “not proven” Cat2 “hint” Cat3 “indication” Cat4 “proof” | 0.520 0.596 0.318 0.638 0.431 | 0.034 Z 11.505 Z 6.126 Z 12.312 Z 8.306 | 0.454–0.586 | moderate |
Overall assessments vs addenda vs appraisals combined categories Cat1 “added benefit not proven” Cat2 “hint of non-quantifiable added benefit” Cat3 “indication of non-quantifiable added benefit” Cat4 “proof of non-quantifiable added benefit” Cat5 “hint of minor added benefit” Cat6 “indication of minor added benefit” Cat7 “proof of minor added benefit” Cat8 “hint of considerable added benefit” Cat9 “indication of considerable added benefit” Cat10 “proof of considerable added benefit” Cat11 “hint of major added benefit” Cat12 “indication of major added benefit” Cat13 “proof of major added benefit” | 0.421 0.622 0.241 0.397 NaN 0.293 0.272 0.272 0.191 0.384 −0.005 0.242 0.344 NaN | 0.036 Z 12.001 Z 4.649 Z 7.661 NaN Z 5.660 Z 5.246 Z 5.246 Z 3.676 Z 7.401 Z − 0.104 Z 4.665 Z 6.640 NaN | 0.351–0.491 | moderate |
c. Ordinal Cohen’s kappa | ||||
Oncological diseases addenda vs appraisals added benefit Weighted | 0.434 0.565 | 0.091 0.082 | 0.256–0.611 0.403–0.727 | moderate moderate |
Without oncological diseases addenda vs appraisals added benefit Weighted | 0.456 0.517 | 0.102 0.095 | 0.255–0.656 0.331–0.703 | moderate moderate |
Oncological diseases addenda vs appraisals evidence level Weighted | 0.389 0.470 | 0.098 0.091 | 0.197–0.580 0.292–0.649 | fair moderate |
Without oncological diseases addenda vs appraisals evidence level Weighted | 0.602 0.733 | 0.091 0.063 | 0.423–0.779 0.610–0.857 | moderate substantial |
Infectious diseases addenda vs appraisals added benefit Weighted | 0.314 0.417 | 0.280 0.249 | −0.234 – 0.863 − 0.071 – 0.904 | fair moderate |
Without infectious diseases addenda vs appraisals added benefit Weighted | 0.475 0.582 | 0.068 0.060 | 0.342–0.608 0.464–0.700 | moderate moderate |
Infectious diseases addenda vs appraisals evidence level Weighted | 0.824 0.902 | 0.169 0.094 | 0.493–1.000 0.717–1.000 | almost perfect almost perfect |
Without infectious diseases addenda vs appraisals evidence level Weighted | 0.488 0.598 | 0.069 0.057 | 0.353–0.623 0.486–0.710 | moderate moderate |
Metabolic diseases addenda vs appraisals added benefit Weighted | 0.000 0.000 | 0.661 0.676 | −1.000 – 1.000 −1.000 – 1.000 | by chance by chance |
Without metabolic diseases addenda vs appraisals added benefit Weighted | 0.466 0.526 | 0.076 0.070 | 0.317–0.614 0.389–0.664 | moderate moderate |
Metabolic diseases addenda vs appraisals evidence level (ordinal) Weighted | −0.091 − 0.091 | 0.568 0.568 | −1.000 – 1.000 − 1.000 – 1.000 | less than by chance less than by chance |
Without metabolic diseases addenda vs appraisals evidence level Weighted | 0.598 0.726 | 0.069 0.049 | 0.463–0.732 0.631–0.822 | moderate substantial |