Chapter 4 considers how the equivalence of measures is determined across different cultural groups. Equivalence of measures in this instance refers to whether or not the assessment means the same thing across those different cultures. The types of bias that might exist as well as methods used to assess potential bias are described. These ideas are presented for assessments in general, and are not limited to visual assessments. Examples from the literature are provided to illustrate the methods discussed in this chapter. Application to visual assessment is also discussed at the conclusion of the chapter.
TopIntroduction
Assessments are often given across multiple cultural, ethnic or language groups. The reasons for cross-cultural assessment vary; in some cases, there is a desire to compare results of assessments across different cultural groups, while other times, there is efficiency to using a pre-existing instrument instead of developing a new one. While the specific issues might vary in these two contexts, great care is needed to understand what the assessments means in each culture, and whether or not the assessment means the same thing across those different cultures. When scores are being compared, it is essential that the scores mean the same thing, so that we aren’t comparing apples to oranges. When an instrument is adapted from one culture to another, without the goal of comparison, it may not be crucial that the sores are comparable, however, if the scores are not comparable, understanding what the scores mean and represent in the particular context is necessary. There is a wealth of research in the field of psychometrics that centers on evaluating the cross-cultural equivalence of assessments.
Van de Vijver and Tanzer (2004) present a framework for thinking about the equivalence of scores across cultures. They define equivalence as a lack of bias in the measure, and identify three sources of bias:
- •
Construct Bias occurs if the construct that is being measured is not identical across cultures.
- •
Method bias, which consists of sample bias, instrument bias, and administration bias. Sample bias occurs when the samples are not comparable on characteristics other than the target construct, which may influence the construct. Instrument bias occurs when one group is less familiar with the type of questions being administered than the other group, leading to score differences that are artifacts of the familiarity of the testing conditions. Lastly, administration bias occurs when communication differences occur, say between an interviewer and interviewee with different language or cultural backgrounds.
- •
Item bias, or differential item functioning: item bias occurs at the item level, when items have different meaning across the cultural groups.
Depending on the context, the types of bias that need to be evaluated should be determined. Van de Vijver and Tanzer (2004) present some strategies for dealing with each type of bias. An excerpt of that is provided in Table 1.
Table 1. Types of bias and strategies to address each
| Type of Bias | Strategies |
| Construct | 1. Develop measure simultaneously in multiple cultures.
2. Develop measure in each culture separately, and then administer them across the multiple cultures
3. Use experts that span all cultures in the development
4. Use samples of examinees from multiple cultures
5. Conduct think aloud protocols across cultures
6. Convergent/discriminant validity studies |
| Method | 1. Detailed training of administrators
2. Detailed instructions for administration
3. Assessment of response styles of examinees |
| Item | 1. Judgmental review of items
2. Differential item functioning analyses
3. Distractor analyses |