Translation and Cultural Adaptation of COAs: Is Machine Translation Viable?

Exploring Accuracy and Use in Clinical Outcome Assessments

Get in touch
data healthcare

Objectives

As technology continues to advance and the need to expedite drug development increases, the applications of Machine Translation (MT) and Artificial Intelligence (AI) in the translation process should be further explored.

The capabilities of iterative learning and feedback can serve as a tool for translation, but the limitations should be considered. This poster will compare human translation (HT) and machine translation (MT) in the back translation step for Clinical Outcome Assessments (COAs) as well as supporting study materials.

Harnessing AI Blog

Methods

DATA SPECIFICATIONS: 

  • 71 items from various COAs and patient diaries 
  • 10 languages (Arabic, Chinese, Czech, French, Hebrew, Hungarian, Korean, Polish, Russian, and Spanish) 
  • 710 total items analyzed - for both Human Translation & Machine Translation

Patient-facing material was assessed to identify individual items with a diversity in text length as well as level of registry. Language selection was aimed at ensuring a global representation and included character-based languages, right-to-left scripts, as well as various language families. 

Seventy-one items were selected from various COAs and patient diaries and compiled into an individual report. Each item selected previously underwent dual forward translation and reconciliation. The reconciled forward translation was then prepared for human and machine back translation. 

A side-by-side comparison of two translation methods, human versus machine, was performed by COA experts with 30+ years of relevant combined experience. The comparison was performed specifically on un-edited, single-step, back translations (prior to resolution). The analysis categorized the findings based on similarities, differences, and any errors found.

Contact us ›
Using transcreation and localization to resonate with multilingual audiences.

Results

The review of each method of back translation demonstrated that machine translation is comparable to human translation. Conceptual equivalency is found in machine as well as human translation. Errors were also found in both methods. 

SPECIAL FINDINGS: 

  • HT was found to use more words than MT in many instances 
  • MT had a higher degree of literal equivalency than HT 
  • HT had a higher degree of conceptual equivalency than MT 
  • There was less than a 2% difference between MT & HT in conceptual equivalency 
  • MT was exactly the same as HT ~30% of the time 
  • Back translations of simple text (“date”, for example) resulted in the same BT 
  • Table 1 demonstrates more accuracy in HT in conceptual equivalency for longer, more complex text 
  • Table 2 demonstrates more accuracy in MT in literal equivalency for shorter, simpler text

Download the PDF

Get My Free Printable PDF ›


TABLE 1: Representative Excerpt

ENGLISH ITEM LANGUAGE (COUNTRY) FRONT TRANSLATION HUMAN BACK TRANSLATION MACHINE BACK TRANSLATION
What was the severity of your nausea (feeling like you wanted to throw up) at its worst over the last 24 hours? Arabic (Israel) كيف كنت تقيم شدة غثيانك (الشعور وكأنك على وشك القيء) في أسوأ حالاته خلال الـ 24 ساعة الماضية؟ How would you rate the severity of your nausea (feeling like you are about to vomit) at its worst within the past 24 hours? How would you rate the severity of your nausea (feeling like you want to vomit) at its worst in the past 24 hours?
Czech (Czech Republic) Jak silná byla Vaše nejhorší nevolnost (pocit na zvracení) během posledních 24 hodin? How strong was your worst nausea (urge to vomit) during the last 24 hours? How severe was your worst nausea (feeling sick) in the past 24 hours?
Spanish (Spain) ¿Cuál fue la intensidad de sus náuseas (sensación de ganas de vomitar) en su peor momento durante las últimas 24 horas? What was the severity of your nausea (feeling of wanting to vomit) in the past 24 hours? How severe was your nausea (feeling sick) in the past 24 hours?
French (Canada) Quelle a été la gravité de vos pires nausées (sensation d’avoir envie de vomir) au cours des dernières 24 heures? What was the severity of your worst nausea (feeling of wanting to vomit) over the past 24 hours? How severe was your worst nausea (feeling like you want to vomit) in the past 24 hours?
Hebrew (Israel) ?מה היה המצב החמור ביותר של בחילתך (התחושה שאתה עומד להקיא) ב־24 השעות האחרונות What was the level of severity of your nausea (feeling you need to vomit) at the worst state in the last 24 hours? How severe was your worst nausea (feeling sick) in the past 24 hours?
Hungarian (Hungary) Milyen súlyos volt a hányingere (olyan érzés, mintha hánynia kellene), amikor a legrosszabb volt az elmúlt 24 órában? How serious was your nausea (a sensation that you want to vomit), when you were the worst over the last 24 hours? How severe was your nausea (feeling sick) in the past 24 hours?
Korean (South Korea) 지난 24시간 동안 귀하의 가장 심한 메스꺼움 (구토 충동)을 평가한다면 어떻습니까? How severe was your nausea (feeling like vomiting) at its worst over the last 24 hours? In the past 24 hours, how severe was your nausea (feeling like you want to vomit)?
Polish (Poland) Jak duże było nasilenie występujących u Pana/Pani nudności (uczucia, że chce się zwymiotować) w najgorszym momencie w ciągu ostatnich 24 godzin? How severe was your nausea (feeling like you want to vomit) at the worst point in the last 24 hours? How severe was your nausea (feeling sick) in the past 24 hours?
Simplified Chinese (China) 在过去 24 小时内, 您的恶心 (想要呕吐) 的严重程度是?在过去 24 小时内, 您的恶心 (想要呕吐) 的严重程度是? What was the severity of your nausea (feeling of wanting to vomit) at its worst in the past 24 hours? How severe was your nausea (feeling like you want to vomit) in the past 24 hours?


TABLE 2: Representative Excerpt

ENGLISH ITEM LANGUAGE (COUNTRY) FRONT TRANSLATION HUMAN BACK TRANSLATION MACHINE BACK TRANSLATION
Stool Frequency: Arabic (Israel) وتيرة التبرز Bowel movement frequency. Frequency of bowel movements.
Czech (Czech Republic) Jak často jste byla/na stolici How often did you have a stool? How often did you have a bowel movement?
Spanish (Spain) Frecuencia de las deposiciones Frequency of bowel movements. Stool Frequency.
French (Canada) Fréquence des selles Frequency of bowel movements. Stool Frequency.
Hebrew (Israel) תדירות פעולת מעיים Bowel movement frequency. Bowel movement frequency.
Hungarian (Hungary) Székletgyakoriság Frequency of bowel movements. Stool Frequency.
Korean (South Korea) 대변 빈도 Stool Frequency: Stool Frequency.
Polish (Poland) Częstotliwość oddawania stolców The frequency of passing stool. Stool Frequency.
Simplified Chinese (China) 大便频率 Bowel movement frequency. Stool Frequency.


Conclusion

The effectiveness and quality of machine translation relies heavily on its underlying engine. In this examination, the engine was trained and underwent post-editing, the process where the machine translated text is reviewed and corrected, in order to achieve the required quality level. It is crucial to emphasize that human interaction is needed in both traditional, purely human translation and in machine translation. Without proper set-up, guidance, parameter establishment and continuous real-time language oversight, the machine translation component cannot operate independently at this time for COAs specifically. 


Although machine translation can continually improve through training of the engines as well as evolve to an iterative learning system (artificial intelligence, AI), its use and process must still involve humans for this material type. The use of machine translation should be considered as a future option in the translation process for COAs, and the advancement of this technology should continue to be a high priority. Although there are cautionary points to consider, iterative learning can provide improvement in quality and reduce time friction that is often seen in the traditional translation< process. Further analysis is necessary to determine the time savings associated with the use of MT and AI in this and other linguistic steps.

Applying machine translation in life sciences content workstreams.

Get support now from the #1 language solutions provider in the world.

How did you hear about us?