Since there is no standard software in which Fleiss` K and Krippendorffs Alpha are implemented with bootstrap confidence intervals (see preview of additional file 2), we provide with this article an R script called „K_alpha“.. The reference was the R kripp.alpha function of the irr package [31] and andrew Hayes` macro-kalpha SAS [30]. The K_alpha function calculates Fleiss` K (for nominal data) with asymptotic intervals and bootstrap and Krippendorff alpha with standard start interval. The program description as well as the program itself, the call of functions for a fictitious dataset and the corresponding output are shown in the additional file 3. Is it reasonable to calculate Krippendorff`s alpha for a single test? I have a project that is divided into several parts (because the options are different), one of which has only one theme. Advisors are asked to choose between options A and B. Bland and Altman[15] have expanded this idea by graphically representing the difference in each point, the average difference and the limits of vertical match with the average of the two horizontal evaluations. The resulting Bland-Altman plot shows not only the general degree of compliance, but also whether the agreement is related to the underlying value of the article. For example, two advisors could closely match the estimate of the size of small objects, but could disagree on larger objects.

The point estimates of Fleiss` K and Krippendorff Alpha did not differ in all scenarios. In the absence of data (completely random), the Krippendorff alpha provided stable estimates, while the comprehensive case analysis approach for Fleiss` K resulted in biased estimates. In the case of zero staggered hypothesis, the probability of coverage of the asymptomatic confidence interval for Fleiss` K was low, while bootstrap confidence intervals for both measures provided a probability of coverage close to the theoretical interval. For three specific scenarios, we removed (completely randomly) a predetermined proportion of the data (10, 25 and 50 percent) to assess the impact of the missing values in the completely random missing hypothesis (MCAR). Scenario selection criteria were an empirical coverage probability of nearly 95% for Fleiss` K and Krippendorffs Alpha, a sample size of 100, and variation in scenarios, categories and advice. One interpretation of Krippendorff`s alpha is: α – 1 – D inside the units – in units D inside and between The total units „Displaystyle“ (alpha – 1- „frac“ text „D_“ „inside the units“ – „Text“ in Error, D_, and between units. We need to eliminate all lines that do not contain or only a non-absent value.