In Canada, many large-scale assessments are administered in English and French. The validity of decisions made from using these assessments critically depends on the meaningfulness and comparability of scores from using different versions of assessments. This research study focused on examining (1) the degree of construct comparability and (2) possible sources of incomparability of the Canadian English and French versions of the programme for international student assessment (PISA), 2003 problem-solving measure (PSM). In this study, statistical and qualitative linguistic reviews were used to examine construct comparability and potential sources of incomparability. These procedures sought to (1) determine the degree of comparability of the measure (2) identify if there are items that function differentially and (3) identify the potential sources of differential item functioning in the two language versions of the measure. Evidence from these procedures was used to determine the comparability of the inferences based on test scores from PISA 2003, PSM. A comparative analysis of the two language versions of the measure indicated that there were some psychometric differences at the scale and item level between the two languages which may jeopardize the comparability of assessment results.