Developer

Page tree

Versions Compared

Old Version 58

changes.mady.by.user Orest Dragush

Saved on

compared with

New Version 59

changes.mady.by.user Orest Dragush

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Added info about multithreading, mutexes, timeouts etc.

...

Openning and closing TM

In first concept it was planned to implement routines to open and close a TM. While concepting we found some problemes with this approach:

  • First one is the realization: opening and closing a TM by REST would mean to update the TM Resource and set a state to open or close. This is very awkward.
  • Since in translate5 multiple tasks can be used to the same time, multiple tasks try to access one TM. Closing TMs is getting complicated to prevent race conditions in TM usage.
  • Since OpenTM2 loads the whole TM in memory, OpenTM2 must control itself which TMs are loaded or not.

This leads to the following conclusion in implementation of opening and closing of TMs:

OpenTM2 has to automatically load the requested TMs if requested. Also OpenTM2 has to close the TMs after a TM was not used for some time. That means that OpenTM2 has to track the timestamps when a TM was last requested.


Concept endpoints, not implemented

http://opentm2/translationmemory/[TM_Name]/openHandle

GET – Opens a memory for queries by OpenTM2

Note: This method is not required as memories are automatically opened when they are accessed for the first time.

http://opentm2/translationmemory/[TM_Name]/openHandle

DELETE – Closes a memory for queries by OpenTM2

Note: This method is not required as memories are automatically opened when they are accessed for the first time.


For now we open  TM in case of call to work with it. TM stays opened till the shutdown we wouldn't try to open more TM's, exceeding the RAM limit setupped in config file. 
In that case we would close TM in order of longest not used, till we would fit in limit including TM that we try to open.
 TM size is calcucated basicaly as sum .TMD and .TMI files
Ram limit doesn't include service RAM and temporary files



Multithreading

In 0.6.44 multithreading are implemented this way

  1. you can set number of sevice threads with --servicethreads 8 commandline argument. This would be a number of threads which would handle requests, but also there would be 2-3 proxygen service threads running, and also for every import and reorganize - they would be new threads created. 
  2. there are mutexes for shared resources, like filesystem and some shared files in the ram, which are not configurable
  3. there are 3 configurable recursive timed mutexes, which can be also be used as non-timed mutexes(that means that it would not have timeout, but would wait till mutex would become free). To use them that way, they need to be set to 0

    This mutexes are -
  4. requestTMMutex - mutex for whole requestTM functions, which can just find tm in tm list, or reserve a place for tm in that list(tm would be opened later). Probably could be disabled to optimize code, was implemented as first high level mutex
  5. tmListMutex - every operation with tm list, like search, adding or deleting elements is managed with that mutex. 
  6. tmMutex - most requests which have tm_name in url, except status request, would be blocking - they would occupy tm for whole execution time(after request data would be parsed and checked). The reason for that is opentm2 code, which still have too many low level chunks, which makes multithreading impossible. 
  7. by default in import or reorganize threads(non request handlers - that would have regular mutexes, but threads, which are created by that handlers, which would run after you would receive response from your reorganize or import tmx requests) would be used non-timed mutexes. So this threads would wait till tm would be free.  You can change that with commandline argument 

    UseTimedMutexesForReorganizeAndImport 1

  8. You can set default values for tmRequestLockDefaultTimeout, tmLockDefaultTimeout, tmRequestLockDefaultTimeout using commandline arguments with this names. Value would be set in ms, default value is 0, which means that timeouts would be disabled. That change would apply for all requests without body and for requests with body if other value is not provided. For import and reorganize threads by default would be used non-timed mutexes, but if it's changed with commandline argument, would be used value from corresponding request(if provided, or default if not). 
  9. saveAllTms request could be used in 2 ways - with non timed mutexes for tms, or with timed mutexes, and in case of timeout, tm would not be saved and request would skip to the next tm. In response you would see message about saved and timeouted tms. 
  10. shutdown request would internally use saveAllTms with hardcoded non-timed mutexes. But it could fail on tmListMutexTimeout when checking how many tm are in import and reorganize status
  11.  resources request would use tmListMutex and timeout, when listing tms. It case of timeout, instead of list of tms, Failed to lock tm list: (timeout_error_msg) would be returned. But that wouldn't be threated as request error.   
  12. in case of timeout fail, you would get back (and into the logs) errorcode 506 and one of the next messages(where (msg) is generated service message about location of failed lock and backtrace):

    (msg); Failed to lock tm list:(msg) /(msg); Failed to lock requestTm:(msg) / (msg); Failed to lock tm:(msg)

    like this : {
    "ReturnValue":506,
    "ErrorMsg":"Failed to acquire the lock(tmMutex:18) within the timeout(1ms)(location: requestTM:339); Failed to lock tm:(location: requestTM:342); "
    }
  13. you can see default timeout values in initMsg. every timeout value is ms value.
  14. for requests with body, you can provide value for each type of mutexes as integer with this names:

   {

       "tmMutexTimeout": 5000, 

       "tmListMutexTimeout": 4000,

       "requestTMMutexTimeout": 15000,

...

}







TM files structure and other related info

Info below is actual for version 0_5_x

Starting from version 0_5_0 .mem file is excluded from TM files - tm now consists only with .tmd and .tmi files. That files have 2kb headers which have some useful information, like creation date and version in which that file was created. In general, changing mid_version number means binary incompatible files. During reorganize there would be created new empty tm and then segments would be reimported from previous, and then old files would be deleted and new ones would be renamed to replace old files. That means that reorganize would also update creation t5memory version of files to the newest.


TM file is just archive with tmi and tmd files. 

tmd and tmi files should be flushed in a safe way - saved on disk with temporary filename and then replacing old files.(Should be implemented)

There is tmmanager(as singletone) which have list of tm, and one tm instance have two binary trees(for both (tmd)data and (tmi)index files), with each have own filebuffer instance(before there used to be a pool of filebuffers and it's files operation functions, like write, read, close and open was handling requests). 

Request handler - it's an instance of class in request handler hierarhy classes. For each type of requests there is class to handle it. In general it have private functions "parseJSON"(would parse json if provided and would return error if json is invalid), "checkData"(whould check if all required fields was provided), "requestTM"(would request readOnly, write or service tm handlers. It would load tm if it is not loaded in RAM yet) and "execute" - original requests code. And also it has public function "run" which is stategy template to operate listed private function. 

The TMs is saved in TMManager using smart pointers(it's pointer which track references to itself and call destructor automaticaly). That means that on request it's possible to clear list from some TM, while it would still be active in other thread(like in fuzzy search). Then ram would be freed at the end of last request handling that TM.
In case if in the middle of some request(like fuzzy search) there was a call to delete tm, first we clear TMlist(but we keep smart pointer in fuzzy requests thread, so this is not calling destructor yet, but would after fuzzy request would be done).  Destructor would try to flush filebuffer into filesystem but because there is no files in the disk, filebuffers would not create them again and it would just clean the RAM(in that case log would be writen about filebuffer flush not founding file in the folder).  

From TMManager, request could ask for one of 3 types of tm handers - readonly, write or service. ReadOnly\write requests here have it's name from inside-tm perspective(so operations with tm files in filesystem is service requests).
ReadOnly(concordance search, fuzzy search, exportTmx) would be provided if there is no write handlers, for write handlers(deleteEntry, updateEntry, importTmx) there should be no other write handlers and no readOnly handlers. Service handlers could mean different for different requests. For example status request should be able to access something like readonly handler, but it shouldn't be blocked if there is any write requests, since it's used for checking import\reorganize status and progress. For some filesystem requests(deleteTM, createTM, cloneTM, importTM, exportTM(internal format)) there should be other blocking mechanism, since most of them even doesn't require to load tm into the ram. 

 In case if tm is not in RAM, requesting handler from TMManager would try to load TM into the RAM, considering RAM limit explained in this document. 

...