As explained above, we found a significant amount of divergent ratings only with the more conservative approach to calculating THE ROI. We looked at factors that could influence the likelihood of diverging ratings. Neither the sex of the child, nor whether it was assessed by two parents or a parent and a teacher, systematically influenced this probability. The bilingualism of the child studied was the only factor studied that increased the likelihood that a child would have divergent values. It is possible that different assessments for the small group of bilingual children reflect systematic differences in vocabulary used in the two different environments: German unilingual daycares and bilingual family environments. Larger groups and more systematic variability in bilingual environmental characteristics are needed to determine whether bilingualism has a systematic effect on advisor compliance, as proposed in this report, and, if so, where this effect originates. We compared the average assessments for each of the different counsellors, i.e. parents and teachers for the 34 children who live in daycare, and for mothers and fathers for the 19 children in parental care with T-tests. In addition, the extent of individual differences was assessed in a descriptive manner. We showed the distribution of differences in relation to the standard deviation of T distribution using a dispersal diagram (see Figure 3). If we consider only children who received significantly different assessments, we also examined the magnitude of these differences by examining the difference between a couple`s assessments using a graphic approach: a Bland-Altman diagram (see Figure 4). A Bland-Altman plot, also known as the Tukey Medium Difference Plot, illustrates the dispersion of concordance by showing individual differences in T values compared to the average difference. This allows for the classing differences in the standard difference (Bland and Altman, 2003).

Another way to conduct reliability tests is the use of the intraclass correlation coefficient (CCI). [12] There are several types, and one is defined as „the percentage of variance of an observation because of the variability between subjects in actual values.“ [13] The ICC area can be between 0.0 and 1.0 (an early definition of CCI could be between 1 and 1). CCI will be high if there are few differences between the partitions that are given to each item by the advisors, z.B. if all advisors give values identical or similar to each of the elements. CCI is an improvement over Pearsons r`displaystyle r` and Spearmans `displaystyle `rho`, as it takes into account differences in evaluations for different segments, as well as the correlation between Denern. Mitchell, S. K. (1979).

Interobserver agreement, reliability and generalization of data collected in observational studies. Psychol. Bull. 86, 376-390. doi: 10.1037/0033-2909.86.2.376 We first assessed the reliability of Inter-Rater within and beyond the rating subgroups.