MQM-based Translation Quality Estimation (TQE)

What it is

Translation Quality Estimation (TQE, part of the OpenAI plugin) can now run in two modes:

• Classic mode (default): the AI receives a batch of segments and directly returns a numeric quality score plus a free-text reasoning per segment. The AI alone decides what is wrong and how much it costs.
• MQM mode (new): the AI only marks translation errors - it returns the MQM issues it finds inside each segment, and translate5 places them as normal sub-segment MQM tags over the affected target text. The score is then calculated by translate5 from those tags, using weights and severity multipliers the project manager configures. The AI never decides the score.

Why MQM mode: the result is transparent (every deduction is a visible MQM tag in the editor, with type, severity and reason), correctable (a reviewer can delete tags, add tags or mark false positives - the score follows), and configurable (the PM decides what a terminology error costs versus a style error). The scoring follows the official MQM scorecard model of the MQM Council (https://themqm.org/).

Switching the mode on

The mode is one config value:

The mode is deliberately not overridable at task level: it is fixed for a task at import, so a finished task's score can never change because someone later switched the config.

Everything else stays as in classic TQE: the same TQE resource panel in the task properties, the same resource assignment, the same worker and batching. Only the job given to the AI changes.

How an MQM-mode run works

1. The prompt sends the task's MQM issue catalog (the issue types of the task's MQM XML) and the configured severities to the assigned AI resource and asks it to return, per segment, the errors it finds: issue type + severity + the exact quoted target text that is wrong + a probability (confidence) + a short reason.
2. translate5 locates each quoted text in the target and places a standard sub-segment MQM tag over it - the same paired qmflag tags a human linguist places, stored the same way. The reviewer sees them in the editor and can edit, delete or mark them as false positives, exactly like manual MQM tags. If a quote cannot be located reliably in the segment, that issue is discarded and logged - positions are never guessed.
3. translate5 calculates the score per segment (see next section) and stores it where classic TQE stores it: the segment's quality score + reasoning (own columns in the segment grid, with the reasoning also shown as tooltip on the score column) and the TQE analysis results. All existing segment filters, KPI values and statistics work unchanged, whichever mode produced the score.

The score calculation

The calculation implements the Raw Quality Score of the official MQM scorecard (https://themqm.org/ - see the attached Excel files):

score = max(0, (1 - penalty points / word count) * 100)

• Every counted issue costs severity multiplier x error type weight penalty points. Example with the default multipliers: a major (multiplier 5) terminology issue with weight 2 costs 10 penalty points.
• The penalty points of all counted issues are summed and divided by the segment's source word count (minimum 1), giving the penalty per word.
• A flawless segment scores 100; once the penalty points reach the word count, the score is 0. Segments with an empty target are not evaluated and score 0.
• An issue is not counted when it is marked as false positive, or when the AI's probability is below the threshold configured for its issue type (see below).
• The AI's probability never scales the penalty - a counted issue always costs its full penalty, so the same set of issues always produces the same score. Manual tags (and tags edited by a reviewer) count with full confidence.
• The task-level score views (KPI) stay consistent with the MQM scorecard's document-level score, because the word-weighted average of the segment scores equals the scorecard's overall score.

The reasoning text of each segment lists every counted issue with its penalty, e.g. MQM: 2 issues, 6.0 penalty points / 25 words - Terminology/major (5.0), Style/minor (1.0, 72%) - the percentage is the AI's reported probability when it was below 100%.

Configuring the weights: the MQM XML

The scoring weights live directly in the MQM criteria XML - the same XML that has always defined the task's MQM issue tree. Three attributes were added:

Example:

<issues severity-critical="25" severity-major="5" severity-minor="1">
    <issue type="Accuracy" level="0" display="yes" weight="2" probability="60">
        <!-- children inherit weight 2 and the 60% threshold unless they override -->
        <issue type="Terminology" level="1" display="yes" weight="3"/>
    </issue>
</issues>

Where the XML comes from (resolution order at task import)

1. Config runtimeOptions.editor.qmFlagXmlContent - if this config value is not empty, its XML is used and the file hierarchy below is skipped. The value is edited in a validating XML editor window directly in the translate5 configuration UI and is overridable at system, client and import level. This is the recommended way to maintain customer-specific scorecards.
2. A task-specific file in the import zip (file name per runtimeOptions.editor.qmFlagXmlFileName, default QM_Subsegment_Issues.xml).
3. A client-specific override on the server: client-specific/public/modules/editor/QM_Subsegment_Issues.xml.
4. The shipped default: public/modules/editor/QM_Subsegment_Issues.xml (in the translate5 installation directory).

translate5 ships three catalog variants in public/modules/editor/, selectable via qmFlagXmlFileName; all three carry the default scorecard attributes and an explanatory comment:

• QM_Subsegment_Issues.xml - the active default, a compact MQM subset (20 issue types)
• QM_Subsegment_Issues_core.xml - the official MQM Core typology with the official descriptions
• QM_Subsegment_Issues-EN.xml - a large English full-typology variant (72 issue types)

Note: the parsed XML is frozen into the task at import (LEK_task.qmSubsegmentFlags). Changing the XML later affects only newly imported tasks.

The probability threshold

The AI reports a probability (confidence) for every issue it marks. This probability is used as a gate only: an issue whose probability is below the probability threshold of its issue type is thrown away during the run - it does not become an MQM tag and does not cost penalty points. Issues at or above the threshold count fully. The reported probability of a counted issue stays visible in the segment's reasoning text (as a percentage).

Starting a TQE run

The same run can be started three ways:

1. Automatically at import time - as today, when a TQE resource is assigned during the import wizard.
2. Manually with the "Start TQE" button in the task properties. In MQM mode the button is mode-aware - see the next section.
3. Via a workflow action at a configured workflow event (e.g. when the import completes, or when a workflow step finishes). The action ships inactive; an administrator activates it per installation by inserting a row into LEK_workflow_action for the action class MittagQI\Translate5\Plugins\OpenAI\Workflow\Actions\QueueTqeAction (method queueTqe), choosing workflow and trigger. Deactivation = deleting the row. A user-facing "Automation Manager" UI for this is a separate, future feature.

Re-running: the AI resource decides what happens

In MQM mode, the presence of the assigned AI resource controls what "Start TQE" (and the workflow action) does:

• AI resource assigned -> full AI re-run. ALL existing MQM tags of the task are removed - including manually placed ones - the AI re-tags every segment and the scores are recalculated. The button asks for confirmation before doing this (except during import, where no tags exist yet).
• No AI resource assigned -> calculation only. The scores of all segments are recalculated from the current MQM tags; nothing is deleted and the AI is not called. This is how you refresh the scores after reviewers corrected tags or marked false positives: unassign the TQE resource and press "Start TQE". The run is synchronous - a success message appears when it is done.
• In classic mode, no resource means the button does nothing (as today).

Score refresh on segment save

With runtimeOptions.plugins.OpenAI.qualityScoreSegmentCheck enabled (default: enabled), saving a segment in the editor recalculates that segment's MQM score immediately from its current MQM tags - purely a calculation, the AI is never called on save, regardless of the assigned resource. So a reviewer who deletes a wrong tag sees the score follow with the save.

Note on TrackChanges: an MQM tag deleted as a tracked change (the deletion is still pending in the text as strike-through) still counts for the score until the tracked deletion is actually applied.

Where the results are visible

• Segment grid: the quality score column and the quality score reasoning column (the reasoning is also the tooltip of the score column). Segment filters on the score work as in classic mode.
• Editor: the MQM tags themselves, with type, severity and the AI's reason as tag comment - editable like any manual MQM tag; false-positive handling in the qualities panel.
• TQE analysis panel (task properties): the per-run analysis results, as in classic mode.
• KPI window / statistics: unchanged, fed by the same stored score.

Attachments