Date: Tue, 19 Mar 2024 03:17:23 +0000 (UTC)
Message-ID: <1859966857.1130.1710818243805@node2041-mqi-confluence.web.inetsolutions.cloud>
Subject: Exported From Confluence
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="----=_Part_1129_672346868.1710818243804"
------=_Part_1129_672346868.1710818243804
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-Location: file:///C:/exported.html
For this setup we recommend using a server, that has at leas=
t 4 CPUs available for usage by translate5 and its dependencies.
This setup is the default setup, if you do not change anything to a =
default translate5 installation.
Why =
this setup is recommended
This setup sets up 3 TermTagger instances.
2 of these instances are reserved for imports (tagging terms during the =
import in translate5) and 1 is reserved for tagging terms while saving segm=
ents through the GUI by users.
This setup ensures, that
- the import is faster, because 2 TermTagger do the work instead of one=
li>
- saving of segments in the GUI is not slowed down by running imports
Understand the setup by explaining ow the O=
penTMSTermTagger is working
For the recommended setup there are multiple reasons.
CPU usage and amount of TermTagger instances
- While a TermTagger instance is tagging terms, it does not respond to any other requests. Each new request is waiting for the Term=
Tagger until it responds.
- This led to several TermTagger DOWN messages in the past, now the timeo=
uts are adjusted and a timeouted termtagger is not considered as DOWN anymo=
re.
- One TermTagger instance is using one CPU core when tagging, so the coun=
t of the parallel running TermTagger instances should be oriented along to =
the available CPU cores
- Since for the GUI tagger only the one saved segment is processed by the=
tagger, it responds fast and the latency is low, so one tagger for the GUI=
should be enough for most setups
- The word count of the segment is increasing the duration in an exponent=
ial way. Tags are not considered as words.
- Depending on how much imports are running and how big the imported task=
s are, multiple import taggers can improve the throughput. Recommendation: =
2-3 import term taggers.
Memory usage
- The memory usage of the TermTaggers increases over the time, since the =
used TBX files are kept in memory.
- Therefore the maximum memory of the import TermTaggers could be decreas=
ed (to about 1GB) while the GUI TermTagger should be slightly bigger to kee=
p more different TBX files at the same time.
- If the TermTaggers are consuming to much memory (3 TermTaggers with max=
mem 2,5GB =3D 7,5GB RAM just for the TermTaggers) the linux kernel may kil=
l a TermTagger instance.
- if a instance was killed by the kernel, check the dmesg output, it may =
look like:
[Mo M=C3=A4r 9 12:=
00:46 2020] Out of memory: Kil=
l process 29388 (java) score 209 or sacrifice child
[Mo M=C3=A4r 9 12=
:00:46 2020] Killed process 29388 (java) total-vm:45973548kB, anon-rss:2246=
996kB, file-rss:0kB, shmem-rss:0kB
[Mo M=C3=A4r 9 12:00:46 2020] oom_=
reaper: reaped process 29388 (java), now anon-rss:0kB, file-rss:0kB, shmem-=
rss:0kB
- When using supervisord with our example config the TermTagger instance =
is automatically restarted then
How to set=
up this set up
Edit your termTagger configuration, which you find in
/appli=
cation/modules/editor/ThirdParty/XliffTermTagger/termtagger-server.conf
Ensure, that in this file the following line exists and is NOT commented=
out (this is not needed if setup with supervisord):
SERVERS=
=3D("http://localhost:9001" "http://localhost:9002" "http://localhost:9003"=
)
transla=
te5.sh config runtimeOptions.termTagger.url.default '["http://localhost:900=
1", "http://localhost:9002"]'
translate5.sh config runtimeOptions.termTagger.url.import '["http://localho=
st:9001", "http://localhost:9002"]'
translate5.sh config runtimeOptions.termTagger.url.gui '["http://localhost:=
9003"]'
Restart your termTagger, if you run termTagger as a service. With superv=
isord this would look like:
supervis=
orctl restart termtagger:*
A nightly cronjob (for example when translate5 is in maintenance due you=
r daily backup) should restart the TermTagger instances to clean memory.
------=_Part_1129_672346868.1710818243804--