2.1 Research Designs

Measurement validity refers to the degree to which an instrument measures what it is intended to measure (Messick, 1989). There are several types of measurement validity, such as face validity, content validity, criterion validity, and construct validity. Validity is crucial in measurement development because it ensures that the instrument captures the intended construct. Face validity refers to whether the instrument measures what it is supposed to measure at the surface level. Content validity is the extent to which the content of the instrument represents all the facets of the construct that are being measured. Criterion validity refers to how much the new instrument correlates with the criterion of a valid pre-existing instrument that measures the same construct. Construct validity is the extent to which an instrument measures the theoretical construct that it is supposed to measure.

The principle of measurement validity becomes more important and challenging when researching diverse samples. For example, when researching cross-cultural participants, if the measure requires translation, the translated version of the instrument should have the same meaning as the original instrument to ensure validity. Direct translation can be problematic, and research typically requires a back-translation strategy (i.e., translating back to the original language to check accuracy) to ensure validity (Zhao et al., 2024).

Furthermore, the response styles across different cultures are also different. Cheung and Rensvold (2000) discussed that the use of extreme response styles (i.e., the tendency to use extreme categories on scales) and acquiescence response styles (i.e., the tendency to agree with attitude statements) differ across cultures. For example, on a 5-point Likert scale, the response 3 means “no opinion” for Americans but “mildly agree” for Koreans (Riordan & Vandenberg, 1994).

When researching diverse samples, it also raises questions about the internal validity, particularly regarding the equivalence of samples from different groups and the methodologies and measures used (Sue, 1999). Internal validity is the ability to establish causal relationships in research. For instance, a measure of psychological well-being may have a strong causal relationship with another variable for one ethnic or cultural group but not another. Researching diverse samples may also raise issues with external validity. This includes the issue of how well each group of participants reflects its overall population, as well as whether each underlying construct and the relationships between constructs are consistent across all groups in the population (Sue, 1999). Moreover, Sue discussed how overemphasizing internal validity often results in findings that lack external validity. It limits the application to real-world settings, particularly for diverse populations. This could result in findings that are methodologically appropriate for an ethnic majority group but not generalizable to ethnic minority groups. In addition, Chen (2008) also emphasized the importance of measurement equivalence in cross-cultural research. Measurement equivalence means that questions asked within a measure have similar correlations with each other in one ethnic group as in another ethnic group. Failure to establish measurement equivalence can lead to misleading conclusions about cultural differences.