COA Translation & Cultural Adaptation: Can Machine Translation Deliver Accuracy?
Evaluating Machine Translation for Clinical Outcome Assessments: Accuracy, Compliance, and Practical Use
Get in touch
Objectives
As technology continues to advance and the need to expedite drug development increases, the applications of Machine Translation (MT) and Artificial Intelligence (AI) in the translation process should be further explored.
The capabilities of iterative learning and feedback can serve as a tool for translation, but the limitations should be considered. This poster will compare human translation (HT) and machine translation (MT) in the back translation step for Clinical Outcome Assessments (COAs) as well as supporting study materials.
Methods
A side-by-side comparison of two translation methods, human versus machine, was performed by COA experts with 30+ years of relevant combined experience. The comparison was performed specifically on un-edited, single-step, back translations (prior to resolution). The analysis categorized the findings based on similarities, differences, and any errors found.
DATA SPECIFICATIONS:
-71 items from various COAs and patient diaries
-10 languages (Arabic, Chinese, Czech, French, Hebrew, Hungarian, Korean, Polish, Russian, and Spanish)
-710 total items analyzed - for both Human Translation & Machine Translation
EVALUATION:
Seventy-one items were selected from various COAs and patient diaries and compiled into an individual report. Each item selected previously underwent dual forward translation and reconciliation. The reconciled forward translation was then prepared for human and machine back translation.
Results
The review of each method of back translation demonstrated that machine translation is comparable to human translation. Conceptual equivalency is found in machine as well as human translation. Errors were also found in both methods.
HT was found to use more words than MT in many instances
MT had a higher degree of literal equivalency than HT
HT had a higher degree of conceptual equivalency than MT
There was less than a 2% difference between MT & HT in conceptual equivalency
MT was exactly the same as HT ~30% of the time
Back translations of simple text (“date”, for example) resulted in the same BT
Table 1 demonstrates more accuracy in HT in conceptual equivalency for longer, more complex text
Table 2 demonstrates more accuracy in MT in literal equivalency for shorter, simpler text
Download the Full PDF
TABLE 1: Representative Excerpt
| ENGLISH ITEM | LANGUAGE (COUNTRY) | FRONT TRANSLATION | HUMAN BACK TRANSLATION | MACHINE BACK TRANSLATION |
|---|---|---|---|---|
| What was the severity of your nausea (feeling like you wanted to throw up) at its worst over the last 24 hours? | Arabic (Israel) | كيف كنت تقيم شدة غثيانك (الشعور وكأنك على وشك القيء) في أسوأ حالاته خلال الـ 24 ساعة الماضية؟ | How would you rate the severity of your nausea (feeling like you are about to vomit) at its worst within the past 24 hours? | How would you rate the severity of your nausea (feeling like you want to vomit) at its worst in the past 24 hours? |
| Czech (Czech Republic) | Jak silná byla Vaše nejhorší nevolnost (pocit na zvracení) během posledních 24 hodin? | How strong was your worst nausea (urge to vomit) during the last 24 hours? | How severe was your worst nausea (feeling sick) in the past 24 hours? | |
| Spanish (Spain) | ¿Cuál fue la intensidad de sus náuseas (sensación de ganas de vomitar) en su peor momento durante las últimas 24 horas? | What was the severity of your nausea (feeling of wanting to vomit) in the past 24 hours? | How severe was your nausea (feeling sick) in the past 24 hours? | |
| French (Canada) | Quelle a été la gravité de vos pires nausées (sensation d’avoir envie de vomir) au cours des dernières 24 heures? | What was the severity of your worst nausea (feeling of wanting to vomit) over the past 24 hours? | How severe was your worst nausea (feeling like you want to vomit) in the past 24 hours? | |
| Hebrew (Israel) | ?מה היה המצב החמור ביותר של בחילתך (התחושה שאתה עומד להקיא) ב־24 השעות האחרונות | What was the level of severity of your nausea (feeling you need to vomit) at the worst state in the last 24 hours? | How severe was your worst nausea (feeling sick) in the past 24 hours? | |
| Hungarian (Hungary) | Milyen súlyos volt a hányingere (olyan érzés, mintha hánynia kellene), amikor a legrosszabb volt az elmúlt 24 órában? | How serious was your nausea (a sensation that you want to vomit), when you were the worst over the last 24 hours? | How severe was your nausea (feeling sick) in the past 24 hours? | |
| Korean (South Korea) | 지난 24시간 동안 귀하의 가장 심한 메스꺼움 (구토 충동)을 평가한다면 어떻습니까? | How severe was your nausea (feeling like vomiting) at its worst over the last 24 hours? | In the past 24 hours, how severe was your nausea (feeling like you want to vomit)? | |
| Polish (Poland) | Jak duże było nasilenie występujących u Pana/Pani nudności (uczucia, że chce się zwymiotować) w najgorszym momencie w ciągu ostatnich 24 godzin? | How severe was your nausea (feeling like you want to vomit) at the worst point in the last 24 hours? | How severe was your nausea (feeling sick) in the past 24 hours? | |
| Simplified Chinese (China) | 在过去 24 小时内, 您的恶心 (想要呕吐) 的严重程度是?在过去 24 小时内, 您的恶心 (想要呕吐) 的严重程度是? | What was the severity of your nausea (feeling of wanting to vomit) at its worst in the past 24 hours? | How severe was your nausea (feeling like you want to vomit) in the past 24 hours? |
TABLE 2: Representative Excerpt
| ENGLISH ITEM | LANGUAGE (COUNTRY) | FRONT TRANSLATION | HUMAN BACK TRANSLATION | MACHINE BACK TRANSLATION |
|---|---|---|---|---|
| Stool Frequency: | Arabic (Israel) | وتيرة التبرز | Bowel movement frequency. | Frequency of bowel movements. |
| Czech (Czech Republic) | Jak často jste byla/na stolici | How often did you have a stool? | How often did you have a bowel movement? | |
| Spanish (Spain) | Frecuencia de las deposiciones | Frequency of bowel movements. | Stool Frequency. | |
| French (Canada) | Fréquence des selles | Frequency of bowel movements. | Stool Frequency. | |
| Hebrew (Israel) | תדירות פעולת מעיים | Bowel movement frequency. | Bowel movement frequency. | |
| Hungarian (Hungary) | Székletgyakoriság | Frequency of bowel movements. | Stool Frequency. | |
| Korean (South Korea) | 대변 빈도 | Stool Frequency: | Stool Frequency. | |
| Polish (Poland) | Częstotliwość oddawania stolców | The frequency of passing stool. | Stool Frequency. | |
| Simplified Chinese (China) | 大便频率 | Bowel movement frequency. | Stool Frequency. |
Conclusion
Machine translation is only as good as the engine behind it, and for COAs, human expertise is still essential. In this evaluation, the engine was trained and carefully post-edited to ensure accuracy. But even with advanced AI, machine translation can’t function independently in this space just yet. Success depends on proper setup, clear parameters, and continuous linguistic oversight.
That said, machine translation has real potential to streamline the COA translation process. With ongoing advancements, it could help improve efficiency and reduce turnaround times. While there are still challenges to overcome, continued investment in AI-driven translation is key to making this a viable option in the future.
Curious about how AI can support your translation needs? Get in touch to explore the possibilities.
