Translation quality assessment remains pertinent in both translation theory and in the industry. Specifically, the process of assessing a target document's quality or a person's translation competence involves a lot of time and money on the part of various governments, organizations and individuals. In response to this issue, this project builds on the ongoing research of Hague et al. (2012), who seek to determine the capabilities of a computerized translation test for the French-to-English and Spanish-to-English language pairs. Specifically, Hague et al. (2012) question whether a good score on a detect-and-correct style computerized translation test that is calculated by a computer also indicates a good score on a traditional full translation test that is calculated by hand. This project seeks to further this research by seeking to answer the same question using an Arabic-to-English language pair. The methods used in this research involve testing individuals using two different style translation tests and then comparing the results. The first style translation test involves a detect-and-correct format where a subject is given a list of project specifications in the form of a translation brief, a source text passage and a corresponding target text passage that has errors introduced throughout. The subject is expected to detect and fix the errors while leaving the rest of the text alone. A score is given for this test using an automated algorithm. The second style test is a traditional translation test where a subject is given the same translation brief and a source text. The subject is expected to produce an acceptable target text, which is subsequently scored by hand. Thereafter, various forms of analysis are used to determine the relationship between the scores of the two types of tests. The results of this research do not strongly suggest that a high score on the detect-and-correct portion of the test indicates a high score on a hand-graded full translation test for the subject population used. However, this research still provides insight, especially concerning whether the detect-and-correct portion of the test actually measures translation competence and concerning second language acquisition (SLA) programs and their intentions. In addition, this research provides insight into logistical issues in testing such as the impact text difficulty and length may have on a detect-and-correct style test as well as the negative impact the American Translators Association (ATA) grading practices of weighting errors and capping errors can have on an experiment such as the one described in this research.

No description available.

This document is a supplement to the article entitled “The Multidimensional Quality Metric (MQM) Framework: A New Framework for Translation Quality Assessment”, which has been acepted for publication in the upcoming January volume of JoSTrans, the Journal of Specialized Translation. The article is a coauthored project between Dr. Alan K. Melby, Dr. Troy Cox and myself. In this document you will find a preface describing the process of writing the article, an annotated bibliography of sources consulted in my research, a summary of what I learned, and a conclusion that considers the future avenues opened up by this research. Our article examines a new method for assessing the quality of a translation known as the Multidimensional Quality Metric, MQM. In our experiment we set the MQM framework to mirror, as closely as possible, the American Translators Association's (ATA) translator certification exam. To do this we mapped the ATA error categories to corresponding MQM error categories. We acquired a set of 29 student translations and had a group of student raters use the MQM framework to rate these translations. We measured the practicality of the MQM framework by comparing the time required for ratings to the average time required to rate translations in the industry. In addition, we had 2 ATA certified translators rate the anchor translation (a translation that was scored by every rater in order to have a point of comparison). The certified translators' ratings were used to verify that the scores given by the student raters were valid. Reliability was also measured, which found that the student raters were not interchangeable, but that the measurement estimate of reliability was adequate. The article's goal was to determine the extent to which the Multidimensional Quality Metric framework for translation evaluation is viable (practical, reliable and valid) when designed to mirror the ATA certification exam. Overall, the results of the experiment showed that MQM could be a viable way to rate translation quality when operationalized based on the ATA's translator certification exam. This is an important discovery in the field of translation quality, because it shows that MQM could be a viable tool for future researchers. Our experiment suggests that researchers ought to take advantage of the MQM framework because, not only is it free, but any studies completed using the MQM framework would have a common base, making these studies more easily comparable.

Cette thèse analyse les pratiques actuelles de l’enseignement de la traduction dans les universités et les écoles professionnelles en Corée, par comparaison avec la situation en France, à partir d’entretiens, de QCM et de l’observation de cours de traduction. L’objectif est de reconnaître les particularités de la situation en Corée et d’en repérer les principales difficultés. Il apparaît que les facteurs socioculturels coréens, tels que la société autoritaire et collectiviste, induisent des cours centrés sur l’enseignant et la passivité des élèves. Nous découvrons aussi le faible niveau de créativité des étudiants coréens du fait d’un recours systématique à la mémorisation et aux QCM dans le système éducatif coréen. L’évaluation orientée processus et l’évaluation diagnostique font largement défaut tant à l’université qu’à l’école professionnelle de traduction, où l’évaluation est sommative plus que formative. Sous forme d’évaluation microstructurelle, les enseignants coréens pratiquent une évaluation globale et variable qui leur est personnelle, par manque de critères précis et concrets d’évaluation et de barème commun ayant force de directive. L’accent est mis sur les compétences linguistiques et textuelles plus que sur les autres compétences, au constat établi que ce sont celles qui font le plus défaut aux étudiants, aussi bien à l’université qu’à l’école professionnelle. Il y a peu de différences entre les étudiants coréens de l’université et de l’école professionnelle en matière de recherche d’informations. Malgré les difficultés particulières affectant les cours de traduction à l’université, la possibilité existe d’y former à la traduction professionnelle. / This thesis analyzes the translation education currently conducted in Korean universities and other highereducational institutions in comparison with those conducted in France. The data is gathered through three methods: interviews, MCQ (Multiple-Choice Questions), and class observation. The purpose is to pinpoint the characteristics of translation education in Korea and to outline the key issues deriving from these particularities. According to the study, socio-cultural factors in Korean society including hierarchy, collectivism, and the cultural tendency to refrain from classroom discussion in favor of professor-centered teaching methods, have led to students’ passive attitude toward learning. In addition, we found that cramming for exams and permanent MCQ assessment in the Korean education system weaken their students’ creativity. Korean teachers place more emphasis on result-oriented, summative assessment than on process-oriented, diagnostic and formative assessment. In microstructural assessment, teachers rely on subjective criteria due to a lack of concrete and precise assessment rules to serve as grading guidelines. In most cases, translation education is dedicated to enhancing linguistic and textual competencies found most lacking among Korean students. Additionally, in terms of information acquisition skills, there are few differences between Korean students in undergraduate programs and those in higher-education institutions. Despite the difficulties of teaching translation, there are still opportunities for professional translation courses at the university.

[pt] O presente trabalho discute práticas de avaliação de tradução, notadamente no contexto de sistemas de tradução automática (TA). É apresentada uma descrição de trabalhos relevantes de avaliação de TA, que abrangem a análise de sistemas e também a criação de taxonomias para avaliação de erros de TA. Em seguida, é realizada uma série de três jogos de avaliação de qualidade de tradução automática, baseados nos jogos de linguagem descritos nas Investigações fiilosóficas de Ludwig Wittgenstein. O jogo 1 abrangeu a pós-edição da tradução automática de pedidos de patentes por um tradutor profissional em dois sistemas distintos, com posterior avaliação de produtividade e análise das edições efetuadas. Os resultados quantitativos do jogo 1 revelam o impacto positivo do uso de TA na produtividade. Para uma avaliação qualitativa dos resultados, foi desenvolvida uma taxonomia das edições feitas durante a tarefa de pós-edição, e os resultados indicaram que o aumento de produtividade da TA não se deu às expensas da qualidade final do texto. O jogo 2 envolveu a comparação de duas traduções do mesmo texto, feitas pelo mesmo tradutor em um intervalo de vários anos, a primeira com memória de tradução, a segunda com pós-edição de TA, e serviu de base para uma discussão sobre a qualidade de tradução produzida em diferentes sistemas e condições de trabalhos. O jogo 3 compara a versão final dos dois textos-alvo editados no jogo 1 aos pedidos de patentes correspondentes, depositados do Instituto Nacional de Propriedade Industrial (INPI), para avaliar se o uso de TA teve algum impacto na qualidade final dos textos-alvo. Os resultados dos jogos 2 e 3 apontaram para um nível de qualidade superior das traduções produzidas por pós-edição de TA para este estudo, quando comparadas a traduções feitas em contextos profissionais. / [en] This work discusses translation assessment practices, most importantly machine translation (MT) assessment. Relevant literature for MT assessment is discussed, from MT systems assessment to the design of error typologies for MT assessment. The study encompasses a series of three MT quality assessment experiments based on the language games as described by Wittgenstein in his work Philosophical Investigations. Game 1 encompassed post-editing two patent documents using different MT systems, and further evaluating translator s productivity and analyzing the edits made. The quantitative results showed the positive impact of using MT in productivity. The qualitative results included a typology of the edits made during the PE task in both MT systems, and the results achieved showed that increased productivity did not affect translation quality level. Game 2 compares two translations of the same text produced several years apart by the same translator. The former was produced using a CAT tool; the latter, a TA system. Both served as the basis for discussing the quality of translations produced in different systems and under different work conditions. Game 3 compares the post-edited documents produced in the PE task of Game 1 to the actual patent documents as filed in Brazil s Patent Office (INPI) in order to evaluate the impact of MT over the quality of post-edited documents. The results of both games 2 and 3 pointed to a superior quality of the post-edited translations of the present study over translations produced in professional contexts.

[pt] Esta dissertação discute a utilização de sistemas de tradução automática em ambientes de memória de tradução (CAT), uma modalidade de trabalho cada vez mais presente no mercado de tradução especializada atual. Foi realizado um estudo experimental envolvendo quatro tradutores profissionais especializados na área de informática. Cada um dos profissionais traduziu dois textos, um deles de marketing de tecnologia e o outro altamente técnico, utilizando diferentes modalidades de trabalho. O objetivo do estudo foi verificar a existência de diferenças entre o uso de tradução automática com pós-edição no nível de segmento e o uso de tradução automática como sugestão no nível de subsegmento. As traduções foram analisadas utilizando recursos de linguística computacional por meio das seguintes métricas: variedade lexical, densidade lexical, distância de edição, considerando sequências de classes gramaticais, e produtividade. Para efeitos comparativos, foram incluídas no estudo experimental traduções 100 por cento humanas e traduções automáticas sem pós-edição. As métricas utilizadas permitiram observar diferenças nos resultados atribuíveis às modalidades de trabalho, bem como comparar os efeitos nos diferentes tipos de textos traduzidos. Finalmente, as diversas traduções de um dos textos foram submetidas à avaliação de leitores para determinar as preferências. / [en] This dissertation addresses the use of automatic translation in translation memory systems (CAT), a fast-growing modality of work in today s specialized translation market. An experimental study was conducted with four professional translators specializing in the field of computing. Each professional translated two texts, one about technology marketing and the other, a highly technical document, using different modalities of work. The purpose of the study was to identify any differences resulting from the use of automatic translation, with segment-based post-editing, and the use of automatic translation as sub-segment translation suggestions. The resources of computational linguistics were employed to analyze the translations, considering the following metrics: lexical diversity, lexical density, edit distance, taking into account grammatical sequences, and productivity. For comparative purposes, the experimental study included 100 percent human translations and automatic translations that were not submitted to post-editing. The metrics employed turned out differing results attributable to the modalities of work, and allowed for the comparison of the effects on the different types of texts translated. Finally, the multiple translations of one of the texts were submitted to the evaluation of the readers, to determine their preferences.