Performance of intraclass correlation coefficient (ICC) as a reliability index under various distributions in scale reliability studies

<jats:p>Many published scale validation studies determine inter‐rater reliability using the intra‐class correlation coefficient <jats:styled-content>(<jats:italic>ICC</jats:italic>)</jats:styled-content>. However, the use of this statistic must consider its advantages, limitations, and applicability. This paper evaluates how interaction of subject distribution, sample size, and levels of rater disagreement affects <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content> and provides an approach for obtaining relevant <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content> estimates under suboptimal conditions. Simulation results suggest that for a fixed number of subjects, <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content> from the convex distribution is smaller than <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content> for the uniform distribution, which in turn is smaller than <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content> for the concave distribution. The variance component estimates also show that the dissimilarity of <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content> among distributions is attributed to the study design (ie, distribution of subjects) component of subject variability and not the scale quality component of rater error variability. The dependency of <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content> on the distribution of subjects makes it difficult to compare results across reliability studies. Hence, it is proposed that reliability studies should be designed using a uniform distribution of subjects because of the standardization it provides for representing objective disagreement. In the absence of uniform distribution, a sampling method is proposed to reduce the non‐uniformity. In addition, as expected, high levels of disagreement result in low <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content>, and when the type of distribution is fixed, any increase in the number of subjects beyond a moderately large specification such as <jats:styled-content><jats:italic>n</jats:italic> = 80</jats:styled-content> does not have a major impact on <jats:styled-content><jats:italic>ICC</jats:italic></jats:styled-content>.</jats:p>