Automatic post-editing with MQM - TQE

With automatic post-editing, translate5 corrects the errors that MQM-based translation quality estimation (MQM - TQE) finds, without anyone opening the task. The AI already returns a suggested correction for every error it marks. translate5 applies these suggestions to the translation, lets the AI assess the corrected translation again and keeps the version with the best result. With the TrackChanges plug-in active, every correction is visible as a tracked change, together with the MQM marker that explains why the text was changed.

Automatic post-editing builds on MQM - TQE. Read AI Translation quality estimation with MQM tags: MQM - TQE first if you do not know it yet.

Requirements

• TQE runs in MQM mode: the configuration TQE: mode (runtimeOptions.plugins.OpenAI.tqeMode) is set to mqm for the system or for the client of the task.
• Manual QA (inside segment) (runtimeOptions.autoQA.enableMqmTags) is active, so the task has MQM issue types.
• An AI language resource that supports TQE (translate5 AI, OpenAI or Azure OpenAI) is assigned to the task as TQE resource. Post-editing uses the same resource as the quality estimation, there is no separate assignment.
• Recommended: the TrackChanges plug-in is active, so that you can see and review every automatic correction.

If one of these requirements is missing, the post-editing settings stay visible but disabled, and a short note next to them names the requirements.

Where to find it

The settings are part of the TQE settings, which are now called Quality estimation & auto post-editing. You find them in two places:

• In the import wizard, on the Quality estimation & auto post-editing card.
• In the project overview, in the Quality estimation & auto post-editing tab of a task. Here you can also change the settings of existing tasks and start a new run.

Settings

The settings are shown below the list of TQE resources.

The defaults come from the system or client configuration (see Configuration). In a task you can overwrite them, also after the import. The threshold and the correction passes can only be changed while Automatic post-editing is switched on.

The help icon next to the settings shows a short summary of how the batches are processed:

How to use it

During the import

1. Create a project as usual.
2. On the Quality estimation & auto post-editing card of the import wizard, check that an AI resource is assigned.
3. Switch on Automatic post-editing (if it is not already on by default) and adjust TQE threshold and Correction passes if needed.
4. Continue the wizard and import the task.

The quality estimation and the post-editing run as part of the import. The task opens when both are finished.

For an existing task

1. In the project overview, select the task and open the Quality estimation & auto post-editing tab.
2. Make sure an AI resource is assigned and switch on Automatic post-editing.
3. Adjust TQE threshold and Correction passes if needed. The settings are saved automatically.
4. Click Start TQE and confirm the dialog with Yes.

The dialog reminds you that the AI marks the translation errors again: all existing MQM markers of the task, including manually added or corrected ones, are replaced (with track changes they stay visible as deleted markers), and the translation of the batches below the threshold is corrected by the system user. To recalculate the scores without changing markers or text, unassign the AI resource before you click Start TQE.

You can start another run later at any time. Each run has its own limit of correction passes.

How it works

1. The AI assesses all segments once, exactly like MQM - TQE without post-editing. This gives the first scores, MQM markers and correction suggestions.
2. The segments are grouped in task order into batches of at most 10 segments.
3. For each batch, the lowest segment score decides: when it is below the TQE threshold, the suggestions for all segments of the batch are applied, also for the segments of the batch that are already above the threshold. The segments of a batch are assessed together and serve each other as context, so the batch is corrected and compared as a whole.
4. The AI assesses the corrected batch again. One correction followed by a new assessment is one correction pass.
5. This is repeated while the lowest score of the batch is still below the threshold and the number of Correction passes is not reached. A batch also stops early when no suggestion can be applied anymore or when the new assessment is incomplete.
6. Of all assessed versions of the batch, translate5 keeps the one with the highest lowest score. The unchanged translation from step 1 takes part in this comparison, so a correction that makes the result worse is never kept. When two versions have the same lowest score, the one with the higher average score wins; if that is equal too, the earlier one is kept.

The kept version is stored as a whole: the translations, scores, MQM markers and reasons of all segments in the batch belong together. The task's quality score and the TQE analysis are calculated from the kept version.

Which segments are changed

• Only editable segments with a translation are corrected. Locked, blocked, non-editable and empty segments are only assessed.
• Editable 100% and higher matches are corrected as well. If you want to protect them, leave Edit unchanged 100% TM matches disabled on import, or lock them.
• Segments that a person has already edited are corrected as well, if they are editable.
• Repetitions: only the first occurrence of a repeated source text goes through post-editing. Its corrected translation is copied to the repetitions that had the same translation. Repetitions with a different translation keep their own translation and assessment. Each repetition keeps its own history, match rate and segment status. The editor setting for repetitions does not apply to automatic post-editing.
• A suggestion is only applied when it can be placed safely: suggestions that are missing or contain tags, quotes that cannot be found unambiguously in the segment, unchanged text and overlapping suggestions are skipped. Internal tags are never changed.
• A correction that breaks a length restriction of the segment (maximum length, pixel width, number of lines) is not saved. The segment keeps its previous translation.
• AutoQA runs on every corrected segment.

What you see in the editor

With the TrackChanges plug-in active, every automatic correction is a tracked change of the system user: the replaced text is shown as deleted, the new text as inserted. Only the words that really changed are marked.

The MQM marker that led to a correction stays in the segment as a deleted marker (shown crossed out). It tells you why the text was changed. The MQM markers of the kept assessment are shown as normal, active markers. Deleted markers are history only: they do not count as errors and do not affect the score.

Move the mouse over a deleted MQM marker to see the error category, severity and reason, and who deleted it and when:

To see the text changes without the deleted MQM markers, open the Settings menu of the segment grid and uncheck Display deleted MQM markers. The option is only available while Display tracked changes is checked. translate5 remembers your choice.

Further details:

• The system user becomes the last editor of a corrected segment. A later edit by a person replaces it as last editor.
• Match rate, match type, the segment status and earlier tracked changes of other users stay as they are. Earlier changes are not accepted or rejected automatically.
• The changes are tracked for the workflow step the task is currently in. The workflow settings that decide whether track changes are recorded in the editor do not apply to these automatic changes.
• When the post-editing runs during the import, the original translation of the file stays available as the segment's original target. The correction is an edit of the system user.
• Without the TrackChanges plug-in, the text is replaced directly, and the previous translation is available in the segment history.

The task's event log

After every run, the event log of the task contains one summary entry, for example "Quality estimation finished: 14 segments assessed, 0 without AI answer, post-editing: 8 segments corrected in 2 of 2 batches". You open it with Task event log in the action menu of the task in the project overview. The entry tells you how many segments were assessed and corrected, and, when something went wrong, which segments were affected (for example segments without AI answer or corrections that could not be saved). It never contains the text of the segments.

When automatic post-editing is switched on but not possible for a task (for example because the task is not in MQM mode), the same entry explains why.

InstantTranslate file translations

Automatic post-editing also works for file translations in InstantTranslate. It is used when, for the client of the user:

• TQE for InstantTranslate file translations is switched on (runtimeOptions.plugins.OpenAI.tqeForInstantTranslateFileTranslation),
• TQE runs in MQM mode and automatic post-editing is switched on,
• an AI resource is set as TQE default for InstantTranslate for the language combination.

The downloaded file then contains the post-edited translation.

Configuration

The defaults for new tasks are set in the configuration, for the whole system or per client (in the client overview, tab System configuration overwrites, filter for "post-editing"). They can be overwritten per task in the import wizard and in the task's Quality estimation & auto post-editing tab.

The TQE resource that is assigned by default is defined as before in the settings of the language resource (Use as TQE by default and Use as TQE for InstantTranslate by default). Automatic post-editing has no own resource defaults.

Good to know

• Automatic post-editing only works in MQM mode. In classic TQE mode the settings are disabled.
• Recalculating the MQM scores without an assigned AI resource does not change any text or MQM marker.
• Running MQM - TQE again (also without post-editing) replaces the MQM markers of the task. With track changes, the replaced markers stay visible as deleted markers.
• The segments show the result of the kept version only. The single correction passes of a run are not shown in the segment.