Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: updated to match with v0.4.48 and 0.5.16

...

  • import new openTM2-TMs
  • delete openTM2-TMs
  • create new empty openTM2-TM
  • import TMX
  • open TM and close TM: not possible see extra section in this document. Maybe we need trigger to flush tm to the disk, but also it could be done in some specific cases...
  • query TM for Matches: one query per TM, not quering multiple TMs at once.
  • query TM for concordance search
  • save new entry to TM
  • delete entry from TM
  • localy clone TM
  • reorganize TM
  • get some statistics about service
  • also you can use tagreplacement endpoint to test tag replacement mechanism

This can be achieved by the following specification of a RESTful HTTP Serive, the specification is given in the following form:

  1. URL of the HTTP Resource, where servername and an optional path prefix is configurable.
  2. HTTP Method with affected functionality
  3. Brief Description
  4. Sent and returned Body.

Request Data Format:

The transferred data in the requests is JSON and is directly done in the request body. It's should be pretty json and ends with '\n}" symbol, because of bug in proxygen that caused garbage after valid data. 

URL Format:

In this document, the OpenTM2 is always assumed under http://opentm2/.

...

Values

%service%Name of service(default - t5memory, could be changed in t5m3mory.conf file
%tm_name%

Name of Translation Memory

Examplehttp://localhost:4040/t5memory/examle_tm/fuzzysearch/?


Endpoints overview

default endpoint/example

Is async?

1Get the list of TMsReturns JSON list of TMsGET/
%service%
%service%//t5memory/
2Create TM

Creates TM with the provided name

POST/%service%//
%service%
t5memory/
3Create/Import TM in internal formatImport and unpack base64 encoded archive of .TMD, .TMI, .MEM files. Rename it to provided namePOST/%service%//
%service%
t5memory/
4Clone TM LocalyMakes clone of existing tmPOST/
%service%/%tm_name%/5
%service%/%tm_name%/clone/t5memory/my+TM/clone
(+is placeholder for whitespace in tm name, so there should be 'my TM.TMD' and 'my TM.TMI'(and in pre 0.5.x 'my TM.MEM' also) files on the disk )
tm name IS case sensetive in url

5Reorganize TMReorganizing tm(replacing tm with new one and reimporting segments from tmd) - asyncGET/
%service%/%tm_name%
%service%/%tm_name%/reorganize/t5memory/my+other_tm/reorganize+ in 0.5.x and up
5Delete TMDeletes .TMD, .TMI files DELETE/
%service%/%tm_name%/6
%service%/%tm_name%//t5memory/%tm_name%/
6Import TMX into TMImport provided base64 encoded TMX file into TM - asyncPOST/%service%/%tm_name%/import/
%service%
t5memory/
%tm
%tm_
name%
name%/import+
7Export TMX from TMCreates TMX from tm. Encoded in base64GET/
%service%/%tm_name%
%service%/%tm_name%//t5memory/%tm_name%/
8Export in Internal formatCreates and exports archive with .TMD, .TMI files of TMGET/
%service%/%tm_name%/
%service%/%tm_name%//t5memory/%tm_name%/status
9

Status of TM 

Returns status\import status of TMGET/%service%/%tm_name%/status/
%service%
t5memory/
%tm
%tm_
name%
name%/status
10Fuzzy searchReturns entries\translations with small differences from requestedPOST/%service%/%tm_name%/fuzzysearch/
%service%
t5memory/
%tm
%tm_
name%
name%/fuzzysearch
11Concordance searchReturns entries\translations that contain requested segmentPOST/
%service%/%tm_name%
%service%/%tm_name%/concordancesearch/t5memory/%tm_name%/concordancesearch
12Entry updateUpdates entry\translation POST/
%service%/%tm_name%
%service%/%tm_name%/entry/t5memory/%tm_name%/entry
13Entry deleteDeletes entry\translationPOST/
%service%/%tm_name%
%service%/%tm_name%/entrydelete/t5memory/%tm_name%/entrydelete
14Save all TMsFlushes all filebuffers(TMD, TMI files) into the filesystemGET/
%service%
%service%_service/savetms/t5memory_service/saveatms
15Shutdown serviceFlushes all filebuffers into the filesystem and shutting down the serviceGET/
%service%
%service%_service/shutdown/t5memory_service/shutdown
16Test tag replacement callFor testing tag replacementPOST/
%service%
%service%_service/tagreplacement/t5memory_service/tagreplacement
17ResourcesReturns resources and service dataGET

/%service%_service/resources

/

%service%

t5memory_service/resources



Available end points

List of TMs

PurposeReturns JSON list of TMs
RequestGET /%service%/
Params

-

Returns list of open TMs and then list of available(excluding open) in the app.

Code Block
languagejs
titleResponse
collapsetrue
Response example:
{
    "Open": [
        {
            "name": "mem2"
        }
    ],
    "Available on disk": [
        {
            "name": "mem_internal_format"
        },
        {
            "name": "mem1"
        },
        {
            "name": "newBtree3"
        },
        {
            "name": "newBtree3_cloned"
        }
    ]
}open - TM is in RAM, Available on disk - TM is not yet loaded from disk




Create TM

PurposeCreates TM with the provided name(tmd and tmi files in/MEM/ folder)
RequestPost /%service%/%tm_name%/
Params

Required: name, sourceLang


Code Block
languagejs
titleResponse
collapsetrue
Request example 
{    "name": "examle_tm",    
     "sourceLang": "bg-BG" 
   ["data": "base64_encoded_archive_see_import_in_internal_format"]
   ["loggingThreshold": 0]
}

Response example:Success:{
"name": "examle_tm",
}
TM already exists: 
{
     "ReturnValue": 17272,
     "ErrorMsg": "::ERROR_MEM_NAME_EXISTS:: TM with this name already exists: examle_tm1; res = 0"
}



Create/Import TM in internal format

PurposeImport and unpack base64 encoded archive of .TMD, .TMI, .MEM(in pre 0.5.x versions) files. Rename it to provided name
RequestPOST /%service%/%tm_name%/
Params

{    "name": "examle_tm",    "sourceLang": "bg-BG" , "data":"base64EncodedArchive" }

Do not import tms created in other version of t5memory. Starting from 0.5.x tmd and tmi files has t5memory version where they were created in the header of the file, and different middle version(0.5.x) or global version(0.5.x) would be represented as 
version mismatch. Instead export tmx in corresponding version and create new empty tm and import tmx in new version. 

This would create example_tm.TMD(data file) and example.TMI(index file) in MEM folder
If there are "data" provided, no "sourceLang" required and vice versa - base64 data should be base64 encoded .tm file(which is just archive that contains .tmd and .tmi files 
If there are no "data" - new tm would be created, "sourceLang" should be provided and should be match with lang in languages.xml

Code Block
languagejs
titleResponse
collapsetrue
Code Block
languagejs
titleResponse
collapsetrue
Request example:{ "name": "mem_internal_format", "sourceLang": "bg-BG", "data":"UEsDBBQACAgIAPmrhVQAAAAAAAAAAAAAAAAWAAQAT1RNXy1JRDE3NS0wXzJfNV9iLk1FTQEAAADtzqEKgDAQgOFTEHwNWZ5swrAO0SBys6wfWxFBDILv6uOI2WZQw33lr38GbvRIsm91baSiigzFEjuEb6XHEK\/myX0PXtXsyxS2OazwhLDWeVTaWgEFMMYYY\/9wAlBLBwhEWTaSXAAAAAAAAAAACAAAAAAAAFBLAwQUAAgICAD5q4VUAAAAAAAAAAAAAAAAFgAEAE9UTV8tSUQxNzUtMF8yXzVfYi5UTUQBAAAA7d3Pa5JxHMDxz+Ns09phDAYdPfaDyQqWRcYjS9nGpoYZhBeZMCISW2v2g5o6VkqQONk\/0KVzh4IoKAovnboUo1PHbuuwU8dSn8c9Pk2yTbc53y+R5\/P9fL7P1wf5Ps9zep5vIOy3iMiSiPLn0yPrQ7In+rStTQARi\/bV9chEyHcxGPIKAGDnPonl21SsHNmUYNgfHZ70nnKNDo9ET0dHozFn2L+Ll9uxZPzazPz1mYQAAAAAAAAAAAAAAAAAAAAAAAAAANDtBkXRoj5Zk7OqSFZ9q35Vn6khNa6W2wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdBKbKHK4Em1omT5DxV6J7FrmkKFypBKt9FczvYaKtr+2DLpiqPTWVayGiq2uYjFUpC7VI6aElN8F8JPn\/QEAAAAAAAAAAAAAAAAAAAAAAAAAAAAA2ANW7U0Ag9Iv60MnT4j8uLBZ\/X5+7dxn1ztX6Uy5AgAAAAAAAAAAAAAAAAAAgA6nL1qFjmc1rAO2IwNN9bL9u4ulVUeEfcQqQAfxSNtltshZaytB7jalZZ2a5KhFGT3Qr\/ztv1pkzAnP1v06+F7UxL22tRzSNf6aFq08MdoiY078\/znmkTZo5Qm2YdoOSLSyDdbaVUop\/Cj3cDm14I6\/uqf++nDUN1u4lS+k9MbKXL4QK72+775U+phOpp8sucdK728X5nK5hVT+weJqbTiHjMiNzWG1yNxWvI8rvxZ9cTfycj71NH1nsZgbf54uJlKryWy6GFlueBT6xHrzJRupDqkPXc9eyyduJmbLkf6\/mlYRDgQDPtO++3\/uYvsazANfYHx68vLEsSvOKedxqa\/hAGowD4Jh\/1X\/dH1X5sEBZpoH6E6\/AVBLBwj3gRyzjAIAAAAAAAAAAAEAAAAAAFBLAwQUAAgICAD5q4VUAAAAAAAAAAAAAAAAFgAEAE9UTV8tSUQxNzUtMF8yXzVfYi5UTUkBAAAA7d3PS9NhHMDxz\/Y1nbp0zfw2Vw6CEjooJkkFPs9DZZaFCiIRHRxKoJUIFXk06iB0kS5Fvw6dhDp28FDgOSqiIKQ\/ICQMhIIuYVnJt2f7eK2M2Ps1xp49b8Y+fP6ArXegJy4iV0RiPx6BNAXyT6ysrKhXlLZ49PwlkKP9hw\/19XcKAOD3PZX42+PDP0+JWN9AT765u3P33vbm1nxbvj0\/3DLQ0y3r5uClsZGhC2eGxgUAAAAAAAAAAAAAAAAAAAAAAAAAgFKXllh0ahQbLHeInDb3Xc6NWrF77Jibcr22zC2YY6bVLNoX5qp97Pa5SbPc8ci8sqHpd1k7a2+ZN+6eFQAAAAAAAAAAAAAAAAAAAAAAAAAAAAD4YxISk8bVUyq6eVa905dtqtxO3fBlqyqnkrW+ZFVZCGp8aVDl9ZeELxlVjhRNsEWVa+UffAlVuf78rC\/1eoK20JfNqnzt3OhLnSp1DZW+bFJl\/467vqRUuVxV5UutKts\/JX2pUWUyXvie9OopE5U7QWEHSfWZXdmPvlSr8i75xJcqVT7fPOdLpSqj5+t9Sahy8UBhOxWqLEph6nJVHhZNvUFPXbS3MlXyYWFvgSon3xf2FldlpGiCmCoPiiYQVbLR3or\/ZT0tS04AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMC6K4t+ZSAtOWkKQpOSeTfnZty0m3CDrsu1uNB9swv2pZ21IlN23J6w1uZsuV0y82bOzJhpM2EGTZdpMaERAAAAAAAAAAAAAAAAAAAAAAAAAAAAAPjrUmteK0RypXifid5n1tyX6j7+9\/vvUEsHCGo104BhAgAAAAAAAAAAAQAAAAAAUEsBAgAAFAAICAgA912FVERZNpJcAAAAAAgAABYABAAAAAAAAAAAALSBAAAAAE9UTV8tSUQxNzUtMF8yXzVfYi5NRU0BAAAAUEsBAgAAFAAICAgA\/F2FVPeBHLOMAgAAAAABABYABAAAAAAAAAAAALSBrAAAAE9UTV8tSUQxNzUtMF8yXzVfYi5UTUQBAAAAUEsBAgAAFAAICAgA\/F2FVGo104BhAgAAAAABABYABAAAAAAAAAAAALSBiAMAAE9UTV8tSUQxNzUtMF8yXzVfYi5UTUkBAAAAUEsGBiwAAAAAAAAAHgMtAAAAAAAAAAAAAwAAAAAAAAADAAAAAAAAANgAAAAAAAAAOQYAAAAAAABQSwYHAAAAABEHAAAAAAAAAQAAAFBLBQYAAAAAAwADANgAAAA5BgAAAAA=" }
Response example:{
"name": "examle_tm"
}

TM already exists:
{
  "ReturnValue": 65535,
  "ErrorMsg": ""
}


...

Clone TM localy

PurposeCreates TM with the provided name
RequestPost /%service%/%tm_name%/clone
Params

Required: name, sourceLang

Endpoint is sync(blocking)

Code Block
languagejs
titleResponse
collapsetrue
Request example 
{    "newName": "examle_tm"
}

Response example:
Success: 
{
    "msg": "newBtree3_cloned2 was cloned successfully",
    "time": "5 ms"
}

 Failure: 
{
    "ReturnValue": -1,
    "ErrorMsg": "'dstTmdPath' = /home/or/.t5memory/MEM/newBtree3_cloned.TMD already exists; for request for mem newBtree3; with body = {\n    \"newName\": \"newBtree3_cloned\"\n}"
}


...

Delete TM

PurposeDeletes .TMD, .TMI, .MEM files 
RequestDelete /%service%/%tm_name%/
Params

-


Code Block
languagejs
titleResponse
collapsetrue
Response example:
success:
{
    "newBtree3_cloned2": "deleted"
},


Code Block
languagejs
titleResponse
collapsetrue
Response example:
failed:
{
    "newBtree3_cloned2": "not found"
}


...

Import provided base64 encoded TMX file into TM

PurposeImport provided base64 encoded TMX file into TM. Starts another thead for import. For checking import status use status call
RequestPOST /%service%/%tm_name%/import
Params

{"tmxData": "base64EncodedTmxFile" }

TM must exist
It's async, so check status using status endpoint, like with reorganize in 0.5.x and up

Code Block
languagejs
titleResponse
collapsetrue
Request example:{
  "tmxData":   "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0idXRmLTgiPz4KPHRteCB2ZXJzaW9uPSIxLjQiPgogIDxoZWFkZXIgY3JlYXRpb250b29sPSJTREwgTGFuZ3VhZ2UgUGxhdGZvcm0iIGNyZWF0aW9udG9vbHZlcnNpb249IjguMCIgby10bWY9IlNETCBUTTggRm9ybWF0IiBkYXRhdHlwZT0ieG1sIiBzZWd0eXBlPSJzZW50ZW5jZSIgYWRtaW5sYW5nPSJlbi1HQiIgc3JjbGFuZz0iYmctQkciIGNyZWF0aW9uZGF0ZT0iMjAxNTA4MjFUMDkyNjE0WiIgY3JlYXRpb25pZD0idGVzdCIvPgogIDxib2R5PgoJPHR1IGNyZWF0aW9uZGF0ZT0iMjAxODAyMTZUMTU1MTA1WiIgY3JlYXRpb25pZD0iREVTS1RPUC1SNTlCT0tCXFBDMiIgY2hhbmdlZGF0ZT0iMjAxODAyMTZUMTU1MTA4WiIgY2hhbmdlaWQ9IkRFU0tUT1AtUjU5Qk9LQlxQQzIiIGxhc3R1c2FnZWRhdGU9IjIwMTgwMjE2VDE2MTMwNVoiIHVzYWdlY291bnQ9IjEiPgogICAgICA8dHV2IHhtbDpsYW5nPSJiZy1CRyI+CiAgICAgICAgPHNlZz5UaGUgPHBoIC8+IGVuZDwvc2VnPgogICAgICA8L3R1dj4KICAgICAgPHR1diB4bWw6bGFuZz0iZW4tR0IiPgogICAgICAgIDxzZWc+RXRoIDxwaCAvPiBkbmU8L3NlZz4KICAgICAgPC90dXY+CiAgICA8L3R1PgogIDwvYm9keT4KPC90bXg+Cg=="
}Response example:Error in case of errorFrom v0_2_15
{ "%tm_name%":""} in case of errorFrom v0_2_15
{ "%tm_name%":"deleted"} in case of success
Check status of import using status call of success
Check status of import using status call




Reorganize TM

PurposeReorganizes tm and fixing issues.
RequestGET /%service%/%tm_name%/reorganize
Headers

Accept - applicaton/xml

up to v0.4.x reorganize is sync, so t5memory 
starting from 0.5.x is async, so you can check status of reorganize similar to how you can check status for importTMX

Under the hood it creates new tm with $Org- prefix, then reimport all segments one-by-one, and then deletes original TM and rename reorganized TM to replace original. 
This request should flush tm(from RAM to the disk) before reorganizing


reorganize would check this condition

if (fValidXmlInSrc && fValidXmlInTrg && (pProposal->getSourceLen() != 0) && (pProposal->getTargetLen() != 0) &&
(szTargetLanguage[0] != EOS) && (szTagTable[0] != EOS) )

, and in case if this condition is true and then it passes segment to putProposal function, which is also used by UpdateRequest and ImportTmx request, so other 
issues could be connected to updating new tm.


In 0.4.48 reorganize responce would look like this

{
"too_long_reorg_0_4":"reorganized",
"time":"37 sec",
"reorganizedSegmentCount":"9424",
"invalidSegmentCount":"0"
}
so if there is invalid segments, inspect t5memory log





Reorganize TM

PurposeReorganizes tm and fixing issues.RequestGET /%service%/%tm_name%/reorganizeHeaders

Accept - applicaton/xml

Export TMX from TM

PurposeCreates TMX from tm.
RequestGET /%service%/%tm_name%/
Headers

Accept - applicaton/xml


This endpoint should flush tm before execution

Code Block
languagejs
titleResponse
collapsetrue
Response example:<?xml version="1.0" encoding="UTF-8" ?>
<tmx version="1.4">
<header creationtoolversion="0.2.14" gitCommit="60784cf * refactoring and cleanup" segtype="sentence" adminlang="en-us" srclang="en-GB" o-tmf="t5memory" creationtool="t5memory" datatype="xml" />
<body>
  <tu tuid="1" datatype="xml" creationdate="20190401T084052Z">
     <prop type="tmgr:segNum">10906825</prop>
     <prop type="tmgr:markup">OTMXML</prop>
     <prop type="tmgr:docname">none</prop>
     <tuv xml:lang="en-GB">
          <prop type="tmgr:language">English(U.K.)</prop>
          <seg>For > 100 setups.</seg>
     </tuv>
     <tuv xml:lang="de-DE">
          <prop type="tmgr:language">GERMAN(REFORM)</prop>
     <seg>Für > 100 Aufstellungen.</seg>
     </tuv>
     </tu>
   </body>
</tmx>


...

Export in internal format

PurposeCreates and exports archive with .TMD, .TMI, .MEM files of TM
RequestGET /%service%/%tm_name%/
Headers

application/zip

returns archive(.tm file) consists with .tmd and .tmi files
This should flush tm before execution

Code Block
languagejs
titleResponse
collapsetrue
Response example:%binary_data%


...

Get the status of TM

RequestGET /%service%/%tm_name%/status
Params

-

Would return status of TM. It could be 'not found', 'available' if it's on the disk but not loaded into the RAM yet, and 'open' with additional info. In case if there was at least one try to import tmx or reorganize tm since it was loaded into the RAM, additional fields would appear and stay in the statistics till memory would be unloaded. 

Code Block
languagejs
titleResponse
collapsetrue
Response example:
{//just opened tm, without import\reorganize called
    "status": "open",
    "lastAccessTime": "",
    "creationTime": "20230703T122212Z",
    "tmCreatedInT5M_version": "0:5:1"
}

{// after reorgainize was called called 
    "status": "open",
    "reorganizeStatus": "available",
    "reorganizeTime": 100,
    "reorganizeTime": "Overall reorganize time is      : 0:00:02\n",
    "statussegmentsReorganized": "open"1112,
    "reorganizeStatusinvalidSegments": "available"10,
    "reorganizeProgressinvalidSegmentsRCs": "5005:10; 100",
    "reorganizeTimefirstInvalidSegments": "Overall123; reorganize432; time554; is623; 659; 675; 741; 742; 753; : 0:00:02\n755; ",
    "segmentsReorganizedinvalidSymbolErrors": 1739,
    "invalidSegments": 2-1,
    "reorganizeErrorMsg": "",
    "lastAccessTime": "",
    "creationTime": "20230703T122212Z20230810T095233Z", 
    "tmCreatedInT5M_version": "0:5:110"
}


   {//not opened but available on the disk 
	"status": "available"
}
{//not found tm {
    "status": "not found",
    "res": 48 // 48- both tmi and tmd files are no found, 16- only TMD file not found, 32 - only TMI file not found
}
  The tmxImportStatus could be "available", "import" or "failed" if the import had errors. If there were at least one import to that tm, new fields would appear
{//tm in process of import
    "status": "open",
    "tmxImportStatus": "import",    
	"importProgress" : 56,    
	"importTime": "00:00:13",    
	"segmentsImported": 1356,    
	"invalidSegments": 23,    
	"invalidSymbolErrors": 2,    
	"importErrorMsg": "", 
    "lastAccessTime":  "%lastAccessTime",
    "creationTime": "20230703T122212Z",
    "tmCreatedInT5M_version": "0:5:1" 
}


...

Update entry

PurposeUpdates entry\translation 
RequestPOST /%service%/%tm_name%/entry
Params

Only sourceLang, targetLang, source and target are required


This request would made changes only in the filebuffer(so files on disk would not be changed)
To write it to the disk just call request which would flush tm to the disk as part of execution(exportTMX, exportTM, cloneTM) or using SaveAllTms request 

Code Block
languagejs
titleResponse
collapsetrue
Request example:
{
    "source": "The end",
    "target": "The target",
    "sourceLang": "en",
    "targetLang": "de", 
    ["documentName": "Translate5 Demo Text-en-de.xlf"],
    ["segmentNumber": 8,]
    ["author": "Thomas Lauria"],
    ["timeStamp": "20210621T071042Z"],
    ["context": "2_2"],
    ["addInfo": "2_2"], 
    ["type": "Manual"],
    ["markupTable": "OTMXUXLF"],
	["loggingThreshold": 0] 
}
Response example:


...

Delete entry

PurposeDeletes entry\translation 
RequestPOST /%service%/%tm_name%/entrydelete
Params

Only sourceLang, targetLang, source, and target are required

Deleting based on strict match(including tags and whitespaces) of target and source

This request would made changes only in the filebuffer(so files on disk would not be changed)
To write it to the disk just call request which would flush tm to the disk as part of execution(exportTMX, exportTM, cloneTM)  or using SaveAllTms request 

Code Block
languagejs
titleResponse
collapsetrue

Request example:
{
  "sourceLang": "bg",
  "targetLang": "en",
  "source": "The end",
  "target": "Eth dne"
  ["documentName": "my file.sdlxliff",]
  ["segmentNumber": 1,]
  ["markupTable": "translate5",]
  ["author": "Thomas Lauria",]
  ["type": "",]
  ["timeStamp": ""],
  ["context": "",]
   ["addInfo": ""] ,  ["loggingThreshold": 0] 
}


...

Working directory

PathDescription
~/.t5memoryThe main directory of service. Should always be under the home directory. Consists of nested folders and t5memory.conf file(see Config file). All directories\files below are nested
LOG

lIncludes log files. It should be cleanup manualy. One session(launch of service) creates two files Log_Thu May 12 10:15:48 2022 .log and Log_Thu May 12 10:15:48 2022 .log_IMPORTANT
Last have logs reduced to level Warning and higher. 

MEMMain data directory.  All tm files is stored here. One TM should include .TMD(data file), .TMI(index file), .MEM(properties file) with the same name as TM name
TABLEServices reserved readonly folder with tagtables, languages etc.
TEMPFor temporary files that were created for mainly import\export. On low debug leved(DEVELOP, DEBUG) should be cleaned manualy
t5memory.confMain config file(see config file)


Config directory should be located in a specific place




Config file - obsolete - use commandline flags instead

fielddefaultDescription
namet5memoryname of service that we use under %service% in address
port8080

service port

timeout3600service timeout
threads1
logLevel2logLevel - > see logging
AllowedRAM_MB1500

Ram limit to operate openning\closing TM(see Openning and closing TM)

Doesn't include services RAM
in Megabytes

TriplesThreshold33

Level of pre-fuzzy search filtering based on combinations  of triples of tokens(excluding tags). Could impact fuzzy search perfomance. For higher values service is faster, but could skip some segments in result. Not always corelated with resulted fuzzyRate 

Config file should be located under ~/.t5memory/t5memory.conf

Anyway, all field has default values so the service could start without the conf file

Reading\applying configs happen only once at service start

Once service started you should be able to see setup values in logs. 
Config file example:

Code Block
languagejs
titleResponse
collapsetrue
name=t5memory
port=4040
timeout=3600
threads=1
logLevel=0
AllowedRAM_MB=200
TriplesThreshold=5                 


...

Openning and closing TM

In first concept it was planned to implement routines to open and close a TM. While concepting we found some problemes with this approach:

  • First one is the realization: opening and closing a TM by REST would mean to update the TM Resource and set a state to open or close. This is very awkward.
  • Since in translate5 multiple tasks can be used to the same time, multiple tasks try to access one TM. Closing TMs is getting complicated to prevent race conditions in TM usage.
  • Since OpenTM2 loads the whole TM in memory, OpenTM2 must control itself which TMs are loaded or not.

This leads to the following conclusion in implementation of opening and closing of TMs:

OpenTM2 has to automatically load the requested TMs if requested. Also OpenTM2 has to close the TMs after a TM was not used for some time. That means that OpenTM2 has to track the timestamps when a TM was last requested.


Concept endpoints, not implemented

http://opentm2/translationmemory/[TM_Name]/openHandle

GET – Opens a memory for queries by OpenTM2

Note: This method is not required as memories are automatically opened when they are accessed for the first time.

http://opentm2/translationmemory/[TM_Name]/openHandle

DELETE – Closes a memory for queries by OpenTM2

Note: This method is not required as memories are automatically opened when they are accessed for the first time.


For now we open  TM in case of call to work with it. TM stays opened till the shutdown we wouldn't try to open more TM's, exceeding the RAM limit setupped in config file. 
In that case we would close TM in order of longest not used, till we would fit in limit including TM that we try to open.
 TM size is calcucated basicaly as sum .TMD and .TMI files
Ram limit doesn't include service RAM and temporary files




TM files structure and other related info

Info below is actual for version 0_5_x

Starting from version 0_5_0 .mem file is excluded from TM files - tm now consists only with .tmd and .tmi files. That files have 2kb headers which have some useful information, like creation date and version in which that file was created. In general, changing mid_version number means binary incompatible files. During reorganize there would be created new empty tm and then segments would be reimported from previous, and then old files would be deleted and new ones would be renamed to replace old files. That means that reorganize would also update creation t5memory version of files to the newest.


TM file is just archive with tmi and tmd files. 

tmd and tmi files should be flushed in a safe way - saved on disk with temporary filename and then replacing old files.(Should be implemented)

There is tmmanager(as singletone) which have list of tm, and one tm instance have two binary trees(for both (tmd)data and (tmi)index files), with each have own filebuffer instance(before there used to be a pool of filebuffers and it's files operation functions, like write, read, close and open was handling requests). 

Request handler - it's an instance of class in request handler hierarhy classes. For each type of requests there is class to handle it. In general it have private functions "parseJSON"(would parse json if provided and would return error if json is invalid), "checkData"(whould check if all required fields was provided), "requestTM"(would request readOnly, write or service tm handlers. It would load tm if it is not loaded in RAM yet) and "execute" - original requests code. And also it has public function "run" which is stategy template to operate listed private function. 

The TMs is saved in TMManager using smart pointers(it's pointer which track references to itself and call destructor automaticaly). That means that on request it's possible to clear list from some TM, while it would still be active in other thread(like in fuzzy search). Then ram would be freed at the end of last request handling that TM.
In case if in the middle of some request(like fuzzy search) there was a call to delete tm, first we clear TMlist(but we keep smart pointer in fuzzy requests thread, so this is not calling destructor yet, but would after fuzzy request would be done).  Destructor would try to flush filebuffer into filesystem but because there is no files in the disk, filebuffers would not create them again and it would just clean the RAM(in that case log would be writen about filebuffer flush not founding file in the folder).  

From TMManager, request could ask for one of 3 types of tm handers - readonly, write or service. ReadOnly\write requests here have it's name from inside-tm perspective(so operations with tm files in filesystem is service requests).
ReadOnly(concordance search, fuzzy search, exportTmx) would be provided if there is no write handlers, for write handlers(deleteEntry, updateEntry, importTmx) there should be no other write handlers and no readOnly handlers. Service handlers could mean different for different requests. For example status request should be able to access something like readonly handler, but it shouldn't be blocked if there is any write requests, since it's used for checking import\reorganize status and progress. For some filesystem requests(deleteTM, createTM, cloneTM, importTM, exportTM(internal format)) there should be other blocking mechanism, since most of them even doesn't require to load tm into the ram. 

 In case if tm is not in RAM, requesting handler from TMManager would try to load TM into the RAM, considering RAM limit explained in this document. 


TAG REPLACEMENT


Tag replacement

...

Previous documentation:

Code Block
languagejs
titleResponse
collapsetrue

http://opentm2/translationmemory/
POST - creating a new or importing an existing filebased binary OpenTM2 TM

The Parameter „name“ contains the TM Name as a string. The string has a maxlength of 256 chars. It can contain any characters except the characters backslash (\), slash(/), colon (:), question mark (?), asterisk (*), vertical line (|), less than sign (<), and greater than sign (>).

Uploading a file is optional, omitting a file means creating a empty TM only.

If an empty TM is created, the POST request contains only the JSON structure with the TM Name.

If an existing binary OpenTM2 file should be additionally imported to the new TM, the POST must be encoded as multipart/form-data.

The JSON structure with the meta data will then be in the first chunk of the multiparted request, the chunk must be named “meta”.

The second chunk contains the plain binary file content and must be named “data”. This binary data contains the TM content

The resulting body contains the name of the TM, as given in the POST request.




To OpenTM2 – without data / creating an empty TM:

{
sourceLang: “en”, // the source language is required for a new TM
name: „TM Name“,
[loggingThreshold:"2"]
}



Raw POST to OpenTM2 – with provided import file:

POST http://opentm2/translationmemory HTTP/1.1
Content-Type: multipart/form-data; boundary="autogenerated"

-- autogenerated
Content-Type: application/json; charset=utf-8
Content-Disposition: form-data; name=meta

{"name":"TM Name", sourceLang:"en"}

--autogenerated
Content-Type: image/jpeg
Content-Disposition: form-data; name=data; filename=Original Filename.jpg
...TM content ...
--autogenerated--





In both cases from OpenTM2 - HTTP 200 OK:

{

name: „TM Name“

}

Errors:
400 Bad Request – if parameters are missing or are not well formed.
409 Conflict – if a memory with the given name already exists.
500 Server Error – for other technical problems.
http://opentm2/translationmemory/[TM_Name]/import
POST import a TMX file into an existing OpenTM2 TM

To OpenTM2:

multipart/form-data like on POST above, expect that no separate JSON section is needed here.

Call answers directly after the upload is done, but before the import starts with HTTP 201 – this means: Import is created and will be started now.

From OpenTM2 - HTTP 201 OK:

{ // empty JSON object, since no data expected as result here!

}

Errors:
400 Bad Request – if parameters are missing or are not well formed.
404 Not Found – if the memory of the given name does not exist
500 Server Error – for other technical problems.
http://opentm2/translationmemory/[TM_Name]/status
GET status of a TM

To OpenTM2:

multipart/form-data like on POST above, expect that no separate JSON section is needed here.

From OpenTM2 - HTTP 200 OK:

{

‘status’:’import’ //allowed status values: import, available, error

}

Errors:
400 Bad Request – if parameters are missing or are not well formed.
404 Not Found – if the memory of the given name does not exist
500 Server Error – for other technical problems.
http://opentm2/translationmemory/
GET – retrieving a list of available TM Files

To OpenTM2: -

From OpenTM2 - HTTP 200 OK:

[{

name: 'my nice TM'

}]

Errors:
500 Server Error – for other technical problems.
http://opentm2/translationmemory/[TM_Name]/
TM_Name is URL-encoded
GET – retrieving a single TM File

To OpenTM2: -

From OpenTM2 - HTTP 200 OK:

Same as POST from OpenTM2 result.

Errors:
404 Not Found – if TM file to given [TMID] in URL was not found
500 Server Error – for other technical problems.
DELETE – deletes an existing TM File

Adressed by the given URL, no body needed.

Errors:
404 Not Found – if TM file to given [TMID] in URL was not found
500 Server Error – for other technical problems.
PUT – updating an existing TM File in one request

Currently not needed, would be only to change the TM name

GET – list of all segments from TM

Currently not needed.




http://opentm2/translationmemory/[TM_Name]/entry/
POST – creates a new entry or updates target entry if match pair already exists

This method updates an existing proposal when a proposal with the same key information (source text, language, segment number, and document name) exists.

Parameters sourceLang and targetLang are containing the languages as RFC5646.

Parameters source and target are containing the entry contents to be stored. Format? plain string?

Attribute Parameters:

documentName: contains the filename where the segment resides in Translate5.
context: evaluates to Translate5 segment mid.
markupTable: OpenTM2 gets a new markup table named „translate5“, so this is the value which is delivered by Translate5.
timestamp: this parameter is not set by translate5, but calculated automatically and delivered from OpenTM2 to translate5.
author: contains the named user which provides the update / new entry
In addition there are the following OpenTM2 Attributes currently not used by translate5:
segmentNumber
additional info
type

To OpenTM2:

{

sourceLang: 'de',

targetLang: 'en',

source: „Das ist das Haus des Nikolaus“,

target: „This is the house of St. Nicholas“,

documentName: 'my file.sdlxliff',

segmentNumber: ,

markupTable: 'translate5',

author: „Thomas Lauria“,

type: '',

timeStamp: '',

context: '123',

addInfo: '',

[loggingThreshold:"2"]

}

The result from the server contains the same data as posted to the server. No additonal ID is added, since the entries are identified by the whole source string instead by an ID, only the timestamp is added.

From OpenTM2 – HTTP 200 OK:

{

sourceLang: 'de',

targetLang: 'en',

source: „Das ist das Haus des Nikolaus“,

target: „This is the house of St. Nicholas“,

documentName: 'my file.sdlxliff',

segmentNumber: 123,

markupTable: 'translate5',

timestamp: '2015-05-12 13:46:12',

author: „Thomas Lauria“

}

Errors:
404 Not Found – if TM file to given [TM_Name] in URL was not found
500 Server Error – for other technical problems.
400 Bad Request – if JSON parameters are missing or are not well formed.





http://opentm2/translationmemory/[TM_Name]/fuzzysearch/
POST– Serves a memory lookup based on the provided search criteria

To OpenTM2:

{

sourceLang: 'de',

targetLang: 'en-US',

source: „Das ist das Haus des Nikolaus“,

documentName: 'my file.sdlxliff', // can be empty

segmentNumber: 123, // can be empty

markupTable: 'translate5', // can be empty

context: „xyz“, // can be empty

[loggingThreshold:"2"]

}

From OpenTM2 HTTP 200 OK:

{

'NumOfFoundProposals': 2,

'results':

[{

source: „Das ist das Haus des Nikolaus“,

target: „This is the house of St. Nicholas“,

sourceLang: 'de', ← rfc5646

targetLang: 'en', ← rfc5646

matchRate: '100',

documentName: 'my file.sdlxliff',

DocumentShortName: 'shortnam.txt',

id: 'identifier',

type: 'Manual',

matchType: 'Exact',

segmentNumber: 123,

markupTable: 'XYZ',

timestamp: '2015-05-12 13:46:12',

author: „Thomas Lauria“.

context: '',

addInfo: ''

},{

source: „Das ist das Haus des Nikolaus“,

target: „This is the house of St. Nicholas“,

sourceLang: 'de', ← rfc5646

targetLang: 'en', ← rfc5646

matchRate: '100',

documentName: 'my file.sdlxliff',

DocumentShortName: 'shortnam.txt',

id: 'identifier',

type: 'Manual',

matchType: 'Exact',

segmentNumber: 123,

markupTable: 'XYZ',

timestamp: '2015-05-12 13:46:12',

author: „Thomas Lauria“.

context: '',

addInfo: ''

}]}

Errors:
400 Bad Request – if search, query or language parameters are missing or are not well formed.
404 Not Found – if TM file to given [TM_Name] in URL was not found
500 Server Error – for other technical problems.





http://opentm2/translationmemory/[TM_Name]/concordancesearch /?
POST – Performs a context search of the given search string in the proposals contained in a memory. Returns one proposal per request.









To OpenTM2:

{

searchString: 'Haus des Nikolaus',

searchType: 'source', // values can be source or target

searchPosition: 123// can be empty; Position where a search should start in the memory, see below


numResults: 1,

msSearchAfterNumResults: 100 //number of milliseconds the search will continue, after the first result is found. All additional results that are found in this additional time will also be returned until numResults is reached. If numResults is reached before msSearchAfterNumResults is reached, the search will abort. If msSearchAfterNumResults is reached before numResults is reached, search is also aborted. All found results are delivered in both cases.

[loggingThreshold:"2"]

}

From OpenTM2 HTTP 200 OK:

{

NewSearchPosition: '123:54', /returns NULL, if end of TM is reached, see below


results:[{

source: „Das ist das Haus des Nikolaus“,

target: „This is the house of St. Nicholas“,

sourceLang: 'de', ← rfc5646

targetLang: 'en', ← rfc5646

matchRate: '100',

documentName: 'my file.sdlxliff',

DocumentShortName: 'shortnam.txt',

id: 'identifier',

type: 'Manual',

matchType: 'Exact',

segmentNumber: 123,

markupTable: 'XYZ',

timestamp: '2015-05-12 13:46:12',

author: „Thomas Lauria“.

context: '',

addInfo: ''

},{

source: „Das ist das Haus des Nikolaus“,

target: „This is the house of St. Nicholas“,

sourceLang: 'de', ← rfc5646

targetLang: 'en', ← rfc5646

matchRate: '100',

documentName: 'my file.sdlxliff',

DocumentShortName: 'shortnam.txt',

id: 'identifier',

type: 'Manual',

matchType: 'Exact',

segmentNumber: 123,

markupTable: 'XYZ',

timestamp: '2015-05-12 13:46:12',

author: „Thomas Lauria“.

context: '',

addInfo: ''

}]}

Errors:
400 Bad Request – if search, query or language parameters are missing or are not well formed.
404 Not Found – if TM file to given [TM_Name] in URL was not found
500 Server Error – for other technical problems.                  


...