tables are provided in format - name(internal key): "position) idInTable:value;"
When adding new entry, for new Id t5memory uses basicaly size of the table before adding+1, so Id should start from id=1. 
When doing fuzzy search, some field could also be saved to the table, to generate then id(or find matching) to compare with saved data.
There are 2 types of entries- older, fixed size, and long filename. For older, every entry has the same size, 39 bytes+endOfLine. You can have up to 389 entries saved in that way in each of the tables. Longname table manage size dynamically, so it's not clear what's the maximum size, but for filenames there are duplicates in old style docname table and in longname(which are used only for this type of data). For filename search, first used longname table, then generated on tm loading caseignore longname table, then short name table. Usually entries in tables are caseignore. 

In the record t5memory saves only id to the table. Maybe it make sense to implement endpoint to edit that table, if something is missing or corrupted. Exception for saving id are context and additional info, some data, that could be saved as int(segId, time), and source and target.
To clear or fix tag tables, you need to do reorganize, but if some entry is missing from the table, you would not restore it on export or segment exctraction via id. So tables could be used to check health of the tm. And for some ways for concordance search or some other checks. 

Response example:
{
    "tableName(additional info)(INTERNAL_KEY_OF_THE_TABLE)": "position)idInTable: value;",
    "tagTablesTable(4)": "0)0: \"\"; ",
    "langTable(1)": "0)2: \"DE-DE\"; 1)1: \"FR-FR\"; 2)0: \"\"; ",
    "langGroupTable(virtual)(8)":"0)1: \"F\"; 1)2: \"G\", "authors(3)": "0)36: \"ALINA GEDGAUDIENE\"; 1)7: \"ANAIS KLUCZKA\"; 2)4: \"ANNA MAYER\"; 3)17: \"ARTHUR LE PIVERT\"; 4)35: \"CHARLOTTE FOGLIA\"; 5)10: \"CLAUDIA RICHLING\"; 6)18: \"DANIEL THURSFIELD\"; 7)39: \"DANIELA BARSCH\"; 8)8: \"DANIELA NOEBEL\"; 9)15: \"DUYGU ZORBA\"; 10)48: \"EDITA KAZLAUSKAITE\"; 11)40: \"FRANK ERVEN\"; 12)1: \"FRéDéRIC MOULIN\"; 13)14: \"GIL DéNIEL (DE6U3268)\"; 14)31: \"HEIDRUN LUDOLF\"; 15)55: \"HEIKE REINER\"; 16)30: \"INDRE BARTKIENE\"; 17)23: \"ISABELLE OUELLET\"; 18)19: \"JP COSTEROUSSE (DE6U3269)\"; 19)41: \"JüRGEN KUKLINSKI\"; 20)46: \"LINA PAULAUSKIENE\"; 21)53: \"LINDA PORRECA HEIMROTH\"; 22)58: \"LUCIE BOYN\"; 23)26: \"MARIUS EITMINAVICIUS\"; 24)50: \"MARLIES VAN DEN HURK\"; 25)43: \"MARTIN HOLTMANN\"; 26)54: \"NADINE SCHAELE\"; 27)22: \"NELSON RUSK\"; 28)49: \"RAIMONDA JAZUKEVICIENE\"; 29)25: \"RITA DANZè\"; 30)6: \"UNBEKANNT\"; 31)13: \"UNBEKANNT 1\"; 32)33: \"UNKNOWN 1\"; 33)38: \"VILMA JURGELIONYTE\"; 34)32: \"VILMA JURGELIONYTE2\"; 35)12: \"VINCENT HENRY\"; 36)47: \"VOLKER ZEYER\"; 37)29: \"VéLINA GABROVSKA\"; 38)28: \"W W\"; 39)11: \"[AMPLEXOR]\"; 40)42: \"[BUEHLER, IHRE ABTEILUNG (AUF VS-BUEHLE\"; 41)57: \"[BüHLER AG, TECHNISCHE DOKUMENTATION (\"; 42)24: \"[CLS]\"; 43)56: \"[ERVEN.BIZ - TRAINING, CONSULTING & SUP\"; 44)44: \"[ERVEN]\"; 45)20: \"[EUROSCRIPT DEUTSCHLAND GMBH, IHRE ABTE\"; 46)2: \"[ITL]\"; 47)51: \"[KAVO DENTAL GMBH, IT (AUF DENBIBSVAC00\"; 48)45: \"[LIONBRIDGE]\"; 49)9: \"[OLD - EUROSCRIPT]\"; 50)16: \"[OLD_AMPLEXOR]\"; 51)21: \"[RAPTRAD IMAGINE]\"; 52)37: \"[STO SE & CO.KGAA, (ON DEWEAT06)]\"; 53)52: \"[STOQUART SA, (ON THERIVER)]\"; 54)3: \"[TRANSLINE EUROPE, (ON TEU-SERVER)]\"; 55)5: \"[TRANSLINE]\"; 56)27: \"[WEINOR, MARKETING (AUF NTS-ACROSS)]\"; 57)34: \"[WIENERS]\"; 58)0: \"\"; ",
"filenames(long)(5)": "0)1: \"none\"; 1)0: \"\"; ",
"filenames(long, caseign, virtual)(5)": "0)1: \"NONE\"; 1)0: \"\"; ",
"filenames(short)(2)": "0)1: \"none\"; 1)0: \"\"; "}

Would return status of TM. It could be 'not found', 'available' if it's on the disk but not loaded into the RAM yet, and 'open' with additional info. In case if there was at least one try to import tmx or reorganize tm since it was loaded into the RAM, additional fields would appear and stay in the statistics till memory would be unloaded. 

"en-GB",    // langs would be checked with languages.xml 
  "targetLang":"de",   
  "source":"For > 100 setups.", 

 // optional fields  
  ["documentName":"OBJ_DCL-0000000845-004_pt-br.xml"],   
  ["segmentNumber":15],   
  ["markupTable":"OTMXUXLF"],    //if there is no markup, default OTMXUXLF would be used. 
								 //Markup tables should be located inside ~/.t5memory/TABLE/%markup$.TBL 
  ["context":"395_408"],  
  ["numOfProposals":20],   // num of expected segments in output. By default it's 5
  ["loggingThreshold": 0]
}





Response example:

Success:
{
    "ReturnValue": 0,
    "ErrorMsg": "",
    "NumOfFoundProposals": 1,
    "results": [
        {
            "source": "The end",
            "target": "The target",
            "segmentNumber": 0,
            "id": "",
            "documentName": "Te2.xlf",
            "sourceLang": "de-DE",
            "targetLang": "EN-GB",
            "type": "Manual",
            "author": "THOMAS LAURIA",
            "timestamp": "20231228T171821Z",
            "markupTable": "OTMXUXLF",
            "context": "2_3",
            "additionalInfo": "",
            "internalKey": "7:1",
            "matchType": "Fuzzy",
            "matchRate": 50,
            "fuzzyWords": 0,
            "fuzzyDiffs": 0
        }
    ]
} 

example 2
{
  "ReturnValue": 0,
  "ErrorMsg": "",
  "NumOfFoundProposals": 1, 
  "results": [
  {
     "source": "For > 100 setups.",
     "target": "Für > 100 Aufstellungen.",
     "segmentNumber": 10906825,
     "id": "",
     "documentName": "none",
    "documentShortName": "NONE",
    "sourceLang": "en-GB",
    "targetLang": "de-DE",
    "type": "Manual",
    "matchType": "Exact", // could be exact or fuzzy
    "author": "",
    "timestamp": "20190401T084052Z",
    "matchRate": 100,
     "fuzzyWords": -1, // for exact match it would be -1 here and in diffs
     "fuzzyDiffs": -1, // otherwise here would be amount of parsed words and diffs that was 
					   // used in fuzzy matchrate calculation    
	 "markupTable": "OTMXML",
     "context": "",
     "additionalInfo": ""
   }
 ]
}

Not found:

{
"ReturnValue": 133,
"ErrorMsg": "OtmMemoryServiceWorker::concordanceSearch::"
}
For exact match used function that's comparing strings ignoring whitespaces. First normalized strings(without tags).
If it's the same string, then t5memory is checking string with tags and could return 100 or 97 match rate depending on result.

Then it's checking context match rate and if document name is the same(non case sensitive)

Then it's checking and modifying exactMatchRate according to code in code block below.
After that it would store exact matches only with usMatchLevel>=100. If there would be no exact matches, fuzzy match calculations would begin.
In case if there is at least one exact match, any fuzzy matches would be skipped.
In case if we have only one exact exact match, it's rate would be set to 102


For equal matches with 100% word matches but different whitespaces/newlines, each whitespace/newline diffs would be count as -1%. For punctuation, at least for 0.4.50, each punctuation would count as word token. This would be changed in future to count punctuation as whitespaces. 

For fuzzy calculation tags would be removed from text, except t5:np tags, which would be replaced with their "r" attribute to be counted as 1 word per tag. 
 

For fuzzy rate calculation we count words and then diffs in normalized string(without tags), using this formula: 
  if (usDiff < usWords )
  {
    *pusFuzzy = (usWords != 0) ? ((usWords - usDiff)*100 / usWords) : 100;
  }
  else
  {
    *pusFuzzy = 0;
  } /* endif */ Regarging Number Protection feature, tags from number protection would be replaced with their regexHashes from their attributes, so they would be count as 1 word each. NP with the same regex would be counted as equal
 To count diffs, t5memory go throuht both segments to find matching tokens, to find something called snake- line of matching tokens. 
 Then It marks unmatched as INSERTED or DELETED tokens, and based on that it calculates diffs.

if it's 100% rate, we add tags and compare it again
if then it's not equal, here is how match rate would be changed - probably this would never happens, because we have exact match test before fuzzy, 
and we do exact test even if triplesHashes is different(which is pre-fuzzy calculation and if it's equal, it could be flag that trigger exact test)

  if ( !fStringEqual )
  {
    if ( usFuzzy > 3 )
    {
      usFuzzy -= 3;
    }
    else
    {
       usFuzzy = 0;
    } /* endif */
    usFuzzy = std::min( (USHORT)99, usFuzzy );
  } /* endif */

then depending on type of translation it could tweak rate
if ( (usModifiedTranslationFlag == TRANSLFLAG_MACHINE) && (usFuzzy < 100) )
{
  // ignore machine fuzzy matches
}
else if ( usFuzzy > TM_FUZZINESS_THRESHOLD )
{
  /********************************************************/
  /* give MT flag a little less fuzziness */
  /********************************************************/
  if ( usModifiedTranslationFlag == TRANSLFLAG_MACHINE )
  {
    if ( usFuzzy > 1 )
    {
      usFuzzy -= 1;
    }
    else
    {
      usFuzzy = 0;
    } /* endif */
  } /* endif */
  if (usFuzzy == 100 && (pGetIn->ulParm & GET_RESPECTCRLF) && !fRespectCRLFStringEqual )
  { // P018279!
    usFuzzy -= 1;
  }
   add to resulting set
} /* endif */
} /* endif */


At the end fuzzy request replaces tags in proposal from TM with tags from request, and if matchRate >= 100, it calculates whitespace diffs and apply matchRate-= wsDiffs

ExactMatchRate calculation:so, before usExact is equal to 97 or 100, depending if strings with tags are equal ignoring whitespaces  and then code do some tweaks.
 pClb is struct that have proposals from TM, pGetIn is fuzzy requests data

 // loop over CLBs and look for best matching entry
{
  LONG lLeftClbLen; // left CLB entries in CLB list
  PTMX_TARGET_CLB pClb; // pointer for CLB list processing
  #define SEG_DOC_AND_CONTEXT_MATCH 8
  #define DOC_AND_CONTEXT_MATCH 7
  #define CONTEXT_MATCH 6
  #define SAME_SEG_AND_DOC_MATCH 5
  #define SAME_DOC_MATCH 4
  #define MULT_DOC_MATCH 3
  #define NORMAL_MATCH 2
  #define IGNORE_MATCH 1
  SHORT sCurMatch = 0;

  // loop over all target CLBs
  pClb = pTMXTargetClb;
  lLeftClbLen = RECLEN(pTMXTargetRecord) -
  pTMXTargetRecord->usClb;
  while ( ( lLeftClbLen > 0 ) && (sCurMatch < SAME_SEG_AND_DOC_MATCH) )
  {
    USHORT usTranslationFlag = pClb->bTranslationFlag;
    USHORT usCurContextRanking = 0; // context ranking of this match
    BOOL fIgnoreProposal = FALSE;
    // apply global memory option file on global memory proposals
   if ( pClb->bTranslationFlag == TRANSLFLAG_GLOBMEM ) // pClb it's segment in TM
   {
       if ( (pGetIn->pvGMOptList != NULL) && pClb->usAddDataLen ) // pGetIn it's fuzzy requests segment
       {

           USHORT usAddDataLen = NtmGetAddData( pClb, ADDDATA_ADDINFO_ID, pContextBuffer, MAX_SEGMENT_SIZE );
           if ( usAddDataLen )
           {
               GMMEMOPT GobMemOpt = GlobMemGetFlagForProposal( pGetIn->pvGMOptList, pContextBuffer );
               switch ( GobMemOpt )
               {
                  case GM_SUBSTITUTE_OPT: usTranslationFlag = TRANSLFLAG_NORMAL; break;
                  case GM_HFLAG_OPT : usTranslationFlag = TRANSLFLAG_GLOBMEM; break;
                  case GM_HFLAGSTAR_OPT : usTranslationFlag = TRANSLFLAG_GLOBMEMSTAR; break;
                  case GM_EXCLUDE_OPT : fIgnoreProposal = TRUE; break;
               } /* endswitch */
          } /* endif */
     } /* endif */ 

     if ( pClb == pTMXTargetClb )
    {
       usTargetTranslationFlag = usTranslationFlag;
    } /* endif *
  } /* endif */ 


  // check context strings (if any)
  if ((!fIgnoreProposal)
       && pGetIn->szContext[0]
       && pClb->usAddDataLen )
   {
       USHORT usContextLen = NtmGetAddData( pClb, ADDDATA_CONTEXT_ID, pContextBuffer, MAX_SEGMENT_SIZE );
       if ( usContextLen != 0 )
       {
            usCurContextRanking = NTMCompareContext( pTmClb, pGetIn->szTagTable, pGetIn->szContext, pContextBuffer );
       } /* endif */
    } /* endif */


  // check for matching document names
  if ( pGetIn->ulParm & GET_IGNORE_PATH )
  {
     // we have to compare the real document names rather than comparing the document name IDs
     PSZ pszCLBDocName = NTMFindNameForID( pTmClb, &(pClb->usFileId), (USHORT)FILE_KEY );
     if ( pszCLBDocName != NULL )
     {
        PSZ pszName = UtlGetFnameFromPath( pszCLBDocName );
        if ( pszName == NULL )
        {
           pszName = pszCLBDocName;
         } /* endif */
      fMatchingDocName = stricmp( pszName, pszDocName ) == 0;
    }
    else
    {
       // could not access the document name, we have to compare the document name IDs
      fMatchingDocName = ((pClb->usFileId == usGetFile) || (pClb->usFileId == usAlternateGetFile));
    } /* endif */
  }
  else
  {
     // we can compare the document name IDs
     fMatchingDocName = ((pClb->usFileId == usGetFile) || (pClb->usFileId == usAlternateGetFile));
  } /* endif */


  if ( fIgnoreProposal )
  {
    if ( sCurMatch == 0 )
    {
      sCurMatch = IGNORE_MATCH;
    } /* endif */
  }
  else if ( usCurContextRanking == 100 )
  {
    if ( fMatchingDocName && (pClb->ulSegmId >= (pGetIn->ulSegmentId - 1)) && (pClb->ulSegmId <= (pGetIn->ulSegmentId + 1)) )
    {
      if ( sCurMatch < SEG_DOC_AND_CONTEXT_MATCH )
      {
         sCurMatch = SEG_DOC_AND_CONTEXT_MATCH;
        pTMXTargetClb = pClb; // use this target CLB for match
        usTargetTranslationFlag = usTranslationFlag;
        usContextRanking = usCurContextRanking;
      }
    }
    else if ( fMatchingDocName )
    {
    if ( sCurMatch < DOC_AND_CONTEXT_MATCH )
    {
      sCurMatch = DOC_AND_CONTEXT_MATCH;
      pTMXTargetClb = pClb; // use this target CLB for match
      usTargetTranslationFlag = usTranslationFlag;
      usContextRanking = usCurContextRanking;
     }
     else if ( sCurMatch == DOC_AND_CONTEXT_MATCH )
     {
       // we have already a match of this type so check if context ranking
       if ( usCurContextRanking > usContextRanking )
       {
          pTMXTargetClb = pClb; // use newer target CLB for match
          usTargetTranslationFlag = usTranslationFlag;
          usContextRanking = usCurContextRanking;
       }
       // use time info to ensure that latest match is used
       else if ( usCurContextRanking == usContextRanking )
       {
         // GQ 2015-04-10 New approach: If we have an exact-exact match use this one, otherwise use timestamp for the comparism
         BOOL fExactExactNewCLB = fMatchingDocName && (pClb->ulSegmId >= (pGetIn->ulSegmentId - 1)) && (pClb->ulSegmId <= (pGetIn->ulSegmentId + 1));
         BOOL fExactExactExistingCLB = ((pTMXTargetClb->usFileId == usGetFile) || (pTMXTargetClb->usFileId == usAlternateGetFile)) &&
         (pTMXTargetClb->ulSegmId >= (pGetIn->ulSegmentId - 1)) && (pTMXTargetClb->ulSegmId <= (pGetIn->ulSegmentId + 1));
         if ( fExactExactNewCLB && !fExactExactExistingCLB )
         {
           // use exact-exact CLB for match
           pTMXTargetClb = pClb;
           usTargetTranslationFlag = usTranslationFlag;
           usContextRanking = usCurContextRanking;
         }
         else if ( (fExactExactNewCLB == fExactExactExistingCLB) && (pClb->lTime > pTMXTargetClb->lTime) )
         {
           // use newer target CLB for match
           pTMXTargetClb = pClb;
           usTargetTranslationFlag = usTranslationFlag;
           usContextRanking = usCurContextRanking;
         }
       } /* endif */
     } /* endif */
   }
   else
   {
     if ( sCurMatch < CONTEXT_MATCH )
     {
     sCurMatch = CONTEXT_MATCH;
     pTMXTargetClb = pClb; // use this target CLB for match
     usTargetTranslationFlag = usTranslationFlag;
     usContextRanking = usCurContextRanking;
     }
     else if ( sCurMatch == CONTEXT_MATCH )
     {
       // we have already a match of this type so check if context ranking
      if ( usCurContextRanking > usContextRanking )
      {
        pTMXTargetClb = pClb; // use newer target CLB for match
        usTargetTranslationFlag = usTranslationFlag;
        usContextRanking = usCurContextRanking;
      }
      // use time info to ensure that latest match is used
     else if ( (usCurContextRanking == usContextRanking) && (pClb->lTime > pTMXTargetClb->lTime) )
     {
       pTMXTargetClb = pClb; // use newer target CLB for match
       usTargetTranslationFlag = usTranslationFlag;
       usContextRanking = usCurContextRanking;
      } /* endif */
    } /* endif */
  } /* endif */
 }
 else if ( fMatchingDocName && (pClb->ulSegmId >= (pGetIn->ulSegmentId - 1)) && (pClb->ulSegmId <= (pGetIn->ulSegmentId + 1)) )
 {
   // same segment from same document available
   sCurMatch = SAME_SEG_AND_DOC_MATCH;
   pTMXTargetClb = pClb; // use this target CLB for match
   usContextRanking = usCurContextRanking;
   usTargetTranslationFlag = usTranslationFlag;
 }
 else if ( fMatchingDocName )
 {
    // segment from same document available
    if ( sCurMatch < SAME_DOC_MATCH )
    {
       sCurMatch = SAME_DOC_MATCH;
       pTMXTargetClb = pClb; // use this target CLB for match
       usTargetTranslationFlag = usTranslationFlag;
       usContextRanking = usCurContextRanking;
     }
     else if ( sCurMatch == SAME_DOC_MATCH )
     {
       // we have already a match of this type so
       // use time info to ensure that latest match is used
       if ( pClb->lTime > pTMXTargetClb->lTime )
       {
         pTMXTargetClb = pClb; // use newer target CLB for match
         usTargetTranslationFlag = usTranslationFlag;
         usContextRanking = usCurContextRanking;
       } /* endif */
     } /* endif */
   }
    else if ( pClb->bMultiple )
    {
       // multiple target segment available
       if ( sCurMatch < MULT_DOC_MATCH )
       {
         // no better match yet
         sCurMatch = MULT_DOC_MATCH;
         pTMXTargetClb = pClb; // use this target CLB for match
         usTargetTranslationFlag = usTranslationFlag;
         usContextRanking = usCurContextRanking;
       } /* endif */
     }
     else if ( usTranslationFlag == TRANSLFLAG_NORMAL )
     {
        // a 'normal' memory match is available
        if ( sCurMatch < NORMAL_MATCH )
        {
           // no better match yet
           sCurMatch = NORMAL_MATCH;
           pTMXTargetClb = pClb; // use this target CLB for match
           usTargetTranslationFlag = usTranslationFlag;
           usContextRanking = usCurContextRanking;
         } /* endif */
     } /* endif */

    // continue with next target CLB
    if ( sCurMatch < SAME_SEG_AND_DOC_MATCH )
    {
      lLeftClbLen -= TARGETCLBLEN(pClb);
      if (lLeftClbLen > 0)
      {
        usTgtNum++;
        pClb = NEXTTARGETCLB(pClb);
      }
    } /* endif */
} /* endwhile */


{
  BOOL fNormalMatch = (usTargetTranslationFlag == TRANSLFLAG_NORMAL) ||
  (usTargetTranslationFlag == TRANSLFLAG_GLOBMEM) ||
  (usTargetTranslationFlag == TRANSLFLAG_GLOBMEMSTAR);
  switch ( sCurMatch )
  {
    case IGNORE_MATCH :
      usMatchLevel = 0;
       break;
  case SAME_SEG_AND_DOC_MATCH :
      usMatchLevel = fNormalMatch ? usEqual+2 : usEqual-1;
      break;
   case SEG_DOC_AND_CONTEXT_MATCH :
       usMatchLevel = fNormalMatch ? usEqual+2 : usEqual-1; // exact-exact match with matching context
       break;
    case DOC_AND_CONTEXT_MATCH :
       if ( usContextRanking == 100 )
       {
         // GQ 2015/05/09: treat 100% context matches as normal exact matches
         // usMatchLevel = fNormalMatch ? usEqual+2 : usEqual-1;
         usMatchLevel = fNormalMatch ? usEqual+1 : usEqual-1;
        }
        else
        {
          usMatchLevel = fNormalMatch ? usEqual+1 : usEqual-1;
        } /* endif */
       break;
  case CONTEXT_MATCH :
    if ( usContextRanking == 100 )
    {
      // GQ 2015/05/09: treat 100% context matches as normal exact context matches
      // usMatchLevel = fNormalMatch ? usEqual+2 : usEqual-1;
      // GQ 2016/10/24: treat 100% context matches as normal exact matches
      usMatchLevel = fNormalMatch ? usEqual : usEqual-1;
    }
    else
    {
      usMatchLevel = fNormalMatch ? usEqual : usEqual-1;
     } /* endif */
     break;
  case SAME_DOC_MATCH :
    usMatchLevel = fNormalMatch ? usEqual+1 : usEqual-1;
    break;
  case MULT_DOC_MATCH :
     usMatchLevel = fNormalMatch ? usEqual+1 : usEqual-1;
     break;
  default :
     usMatchLevel = fNormalMatch ? usEqual : usEqual-1;
     break;
  } /* endswitch */
}
}

Here is structure of the segment from responses

{
"source":"in Verbindung 2 fds fdsa amit Befestigungswinkel fdsaf MS-...-WPE-B zur Wandmontage eines sfg Einzelgeräts", // source that was saved
"sourceNPRepl":"in Verbindung 2 fds fdsa amit Befestigungswinkel fdsaf MS-...-WPE-B zur Wandmontage eines sfg Einzelgeräts",// np replaced source - used for fuzzy and triples thresholds - here there are no NP tags, but their hashes
"sourceNorm":"in Verbindung 2 fds fdsa amit Befestigungswinkel fdsaf MS-...-WPE-B zur Wandmontage eines sfg Einzelgeräts",// normalized source - used for fuzzy calclulation - there are no tags at all
"target":"In combinahgfd tion with mounting bracket MS-...-WPE-B for wall mounting an individual component ",// saved target
"segmentNumber":1, // previously segmentNumber  - internal Id generated in tm, or provided with update call. Can be used together with internalKey as a primary number in tm 
"id":"", // dummy field, 
"documentName":"Audioscript_Hybrides_Arbeiten.xlsx.sdlxliff",
"sourceLang":"DE-DE",// that langs in requests would be searched in languages.xml, and would be used best match(or preffered) 
"targetLang":"EN-GB",
"type":"Manual",
"author":"PROJECT MANAGER",
"timestamp":"",// if empty, current time would be used. 
"markupTable":"OTMXUXLF", // the same all the time, in the future could be refactored and deleted
"context":"390",// context and addinfo would be saved as additional field in cbl(internal data struct, which saves variants of other variables for the same translation, which have position pointed below - "11:1"
"additionalInfo":"",
"internalKey":"11:1"// internal position of the segment inside tmd file. could be shifted with deleting some other segments. both numbers should not be zero
}

Search is made segment-by segment, and it's checking segment if it fits selected filters. You can search for EXACT or CONCORDANCE matches in this fields:
source, target, document, author, addInfo, context
To set filter, use it's SearchMode field, otherwise filter would be disabled. So you have sourceSearchMode, targetSearchMode, documentSearchMode,
authorSearchMode, addInfoSearchMode, contextSearchMode 

Search mode should be set explicitly to CONTAINS/CONCORDANCE or EXACT, otherwise filter would be ignored. But also each searchMode could have additional search parameters "CONTAINS, caseinsensetive, WHITESPACETOLERANT, INVERTED", all that values is not important, as well as delimiter. By default search is case sensetive. If you add Inverted option, check for that filter would be reverted. 
To check how filters would be parsed, check json in responce.  Field with that info could look like this:

"Filters":"
Search filter, field: SOURCE FilterType::CONTAINS SearchStr: 'THE'; Options: SEARCH_FILTERS_NOT|SEARCH_CASEINSENSITIVE_OPT|SEARCH_WHITESPACETOLERANT_OPT|;\n
Search filter, field: TARGET FilterType::EXACT SearchStr: ''; Options: SEARCH_CASEINSENSITIVE_OPT|;\n
Search filter, field: ADDINFO FilterType::CONTAINS SearchStr: 'some add info'; Options: SEARCH_WHITESPACETOLERANT_OPT|;\n
Search filter, field: CONTEXT FilterType::EXACT SearchStr: 'context context'; Options: ;\nSearch filter, field: AUTHOR FilterType::CONTAINS SearchStr: ''; Options: ;\n
Search filter, field: DOCUMENT FilterType::CONTAINS SearchStr: 'evo3_p1137_reports_translation_properties_de_fr_20220720_094902'; Options: SEARCH_FILTERS_NOT|;\n
Search filter, field: TIMESTAMP FilterType::RANGE Range: 20000121T115234Z - 20240121T115234Z Options: ;\n",

It's possible to apply filter just with SearchMode, like if you would type "authorSearchMode": "exact",but there would be no "author" field, it would look for segments, where author field is empty.

Also there are  timespan parameter, to set it, use this fields and format:

"timestampSpanStart":"20000121T115234Z",
"timestampSpanEnd":"20240121T115234Z",

You should set both parameters to apply filter, otherwise you would get error as return.  Check output to see how it was parsed and applied.
By default all mentioned filters is applied in logical and combination, but you can change that globaly with adding 

"logicalOr": 1, 
Then all mentioned filters would be applied in logical or combination(please, use 1 to set this to true, boolean type is not supported by json parser in t5memory). Supported since 0.6.5

"onlyCountSegments":1

Instead of returning segments, just count them and return counter in 

"NumOfFoundSegments":22741

Also there are lang filters, they would always be applied to selection of segments that passed previous filters, so value of  "logicalOr": 1,  wouldn't be applied to that.
To set language filters, use this fields:

"sourceLang":"en-GB",
"targetLang":"de",

Lang filters could be applied with major lang feature, so source lang in this case would be applied as exact filter for source lang, but target lang would check if langs is in the same lang group. That check is done in languages.xml file with isPreferred flag.
Lang filters and if filters is combined in logical or or logical and you can check in  GlobalSearchOptions  field of responce. It could look like this: 

"GlobalSearchOptions":"SEARCH_FILTERS_LOGICAL_OR|SEARCH_EXACT_MATCH_OF_SRC_LANG_OPT, lang = en-GB|SEARCH_GROUP_MATCH_OF_TRG_LANG_OPT, lang = de",

Other that you can send is:

"searchPosition":"8:1",
"numResults":2,
"msSearchAfterNumResults":250
"loggingThreshold": 4 - check other requests,

So search position is position where to start search internaly in btree. This search is limited by num of found segment(set by numResults) or timeout(set by msSearchAfterNumResults), but timeout would be ignored in case if there are no segments in the tm to fit params.  Max  numResults is 200.

You can send empty json and search would work fine, but it would just return first 5 segments in tm
You can go through all segment  with using this 2 fields
"searchPosition":"8:1",
"numResults":200 

{     
    "logicalOr": 1,
     "source":"the",
    "sourceSearchMode":"CONTAINS, CASEINSENSETIVE, WHITESPACETOLERANT, INVERTED",
    
    "target":"",
    "targetSearchMode":"EXACT, CASEINSENSETIVE",
    
    "document":"evo3_p1137_reports_translation_properties_de_fr_20220720_094902",
    "documentSearchMode":"CONTAINS, INVERTED",
    
     "author":"some author",
     "authorSearchMode":"CONTAINS",

    "timestampSpanStart": "20000121T115234Z",
    "timestampSpanEnd": "20240121T115234Z",

    "addInfo":"some add info",
    "addInfoSearchMode":"CONCORDANCE, WHITESPACETOLERANT",


    "context":"context context",
    "contextSearchMode":"EXACT",
    
    "sourceLang":"en-GB", 
    "targetLang":"SV",  
    "searchPosition": "8:1",
    "numResults": 2,
    "msSearchAfterNumResults": 25,
    "loggingThreshold": 3
}



So here search would be done in logical or way, so if any of source, target, document, context, author, timestamp filters returns true, result would be added to set, which then would be filtered out by sourceLang on exact match check and targetLang on groupLang check.
Search would start from position "8:1"(tm data start at "7:1" but if you wan't to start from the beggining, just avoid that param. 
numResuts:2 - so if there would be 2 segments found, search would end
"msSearchAfterNumResults": 25 - 25ms after first found segment, search would end, even if more segments was found, responce would contain "NewSearchPosition": "10:1", which can be used in searchPosition to continue search

Response example:Success:
example{
"Filters": "Search filter, field: SOURCE FilterType::CONTAINS SearchStr: 'THE'; Options: SEARCH_FILTERS_NOT|SEARCH_CASEINSENSITIVE_OPT|SEARCH_WHITESPACETOLERANT_OPT|;\n
Search filter, field: TARGET FilterType::EXACT SearchStr: ''; Options: SEARCH_CASEINSENSITIVE_OPT|;\n
Search filter, field: ADDINFO FilterType::CONTAINS SearchStr: 'some add info'; Options: SEARCH_WHITESPACETOLERANT_OPT|;\n
Search filter, field: CONTEXT FilterType::EXACT SearchStr: 'context context'; Options: ;\n
Search filter, field: AUTHOR FilterType::CONTAINS SearchStr: ''; Options: ;\n
Search filter, field: DOCUMENT FilterType::CONTAINS SearchStr: 'evo3_p1137_reports_translation_properties_de_fr_20220720_094902'; Options: SEARCH_FILTERS_NOT|;\n
Search filter, field: TIMESTAMP FilterType::RANGE Range: 20000121T115234Z - 20240121T115234Z Options: ;\n",
"GlobalSearchOptions": "SEARCH_FILTERS_LOGICAL_OR|SEARCH_EXACT_MATCH_OF_SRC_LANG_OPT, lang = en-GB|SEARCH_GROUP_MATCH_OF_TRG_LANG_OPT, lang = sv",
"ReturnValue": 0,
"ReturnMessage": "FOUND",
"NewSearchPosition": "10:1",
"results": [
{
"source": "Congratulations on the purchase of a <ph x=\"101\"/> machine control system.",
"target": "Gratulerar till köpet av maskinstyrningsystemet <ph x=\"101\"/>.",
"segmentNumber": 5740419,
"id": "",
"documentName": "none",
"sourceLang": "en-GB",
"targetLang": "SV-SE",
"type": "Manual",
"author": "",
"timestamp": "20170327T091814Z",
"markupTable": "OTMXUXLF",
"context": "",
"additionalInfo": "",
"internalKey": "8:1"
},
{
"source": "The <ph x=\"101\"/> System is an ideal tool for increasing productivity in all aspects of the construction earthmoving industry.",
"target": "Systemet <ph x=\"101\"/> är ett verktyg som lämpar sig perfekt för att öka produktiviteten inom alla delar av bygg- och anläggningsområdet.",
"segmentNumber": 5740420,
"id": "",
"documentName": "none",
"sourceLang": "en-GB",
"targetLang": "SV-SE",
"type": "Manual",
"author": "",
"timestamp": "20170327T091814Z",
"markupTable": "OTMXUXLF",
"context": "",
"additionalInfo": "",
"internalKey": "9:1"
}
]
}
SearchPosition / NewSearchPositionFormat: "7:1"
First is segment\record number, second is target number
The NextSearchposition is an internal key of the memory for the next position on sequential access. Since it is an internal key, maintained and understood by the underlying memory plug-in (for EqfMemoryPlugin is it the record number and the position in one record),
no assumptions should be made regarding the content. It is just a string that, should be sent back to OpenTM2 on the next request, so that the search starts from there.
So is the implementation in Translate5: The first request to OpenTM2 contains SearchPosition with an empty string, OpenTM2 returns than a string in NewSearchPosition, which is just resent to OpenTM2 in the next request.

Not found:{
"ReturnValue": 0,
"NewSearchPosition": null,
"ErrorMsg": ""
}TM not found:{
"ReturnValue": 133,
"ErrorMsg": "OtmMemoryServiceWorker::concordanceSearch::"
}

This request would made changes only in the filebuffer(so files on disk would not be changed)
To write it to the disk just call request which would flush tm to the disk as part of execution(exportTMX, exportTM, cloneTM) or using SaveAllTms request 

Request example:
{
    "source": "The end",
    "target": "The target",
    "sourceLang": "en", // langs would be checked with languages.xml
    "targetLang": "de", 
//additional field
    ["documentName": "Translate5 Demo Text-en-de.xlf"],
    ["segmentNumber": 8,]
    ["author": "Thomas Lauria"],
    ["timeStamp": "20210621T071042Z"], // if there is no timestamp, current time would be used
    ["context": "2_2"], // context and addInfo would be saved in TM in the same field
    ["addInfo": "2_2"], 
    ["type": "Manual"], // could be GlobalMemory, GlobalMemoryStar, MachineTranslation, Manual, by default Undefined         
    ["markupTable": "OTMXUXLF"], //if there is no markup, default OTMXUXLF would be used. 
								 //Markup tables should be located inside ~/.t5memory/TABLE/%markup$.TBL
    ["loggingThreshold": 0],
(in 0.7)  ["newEntryIfOnlyContextAndTimestampDifferent"]: 0,
(in 0.7)["saveDifferentTargetsForSameSource"]: 0, 
	["save2disk": 0]   // flag if we need to flush tm to disk after update. by default is true
}


both newEntryIfOnlyContextAndTimestampDifferent and saveDifferentTargetsForSameSource impact how similar translation would be saved. In case if source and other field(doc, author, lang) are the same, but target different, and newEntryIfOnlyContextAndTimestampDifferent is set to true, you would have 2 translation, otherwise you would have 1 translation saved - with higher updateTime. but if you have different attributes, like document, you would have 2 translation anyway. 
newEntryIfOnlyContextAndTimestampDifferent would also increase number of saved segments. If values was not provided, defaule values from flags would be used.
here are data struct used for search, so you can see max numbers of symbols
typedef struct _LOOKUPINMEMORYDATA
{
  char szMemory[260];
  wchar_t szSource[2050];
  wchar_t szTarget[2050];
  char szIsoSourceLang[40];
  char szIsoTargetLang[40];
  int lSegmentNum;
  char szDocName[260];
  char szMarkup[128];
  wchar_t szContext[2050];
  wchar_t szAddInfo[2050];
  wchar_t szError[512];
  char szType[256];
  char szAuthor[80];
  char szDateTime[40];
  char szSearchMode[40]; // only for concordance search
  char szSearchPos[80]; // only for concordance search
  int iNumOfProposals;
  int iSearchTime;
  wchar_t szSearchString[2050];
} LOOKUPINMEMORYDATA, *PLOOKUPINMEMORYDATA;

Response example:success:
example_new{
  "source": "The end",
  "sourceNPRepl": "The end",
  "sourceNorm": "The end",
  "target": "The target",
  "segmentNumber": 0,
  "id": "",
  "documentName": "Te2.xlf",
  "sourceLang": "DE-DE",
  "targetLang": "EN-GB",
  "type": "Manual",
  "author": "THOMAS LAURIA",
  "timestamp": "",
  "markupTable": "OTMXUXLF",
  "context": "2_3",
  "additionalInfo": "addInfo2",
  "internalKey": "8:1"
}

example_old
{
"sourceLang": "de-DE",
"targetLang": "en-GB",
"source": "The end",
"target": "The target",
"documentName": "Translate5 Demo Text-en-de.xlf",
"segmentNumber": 222,
"markupTable": "OTMXUXLF",
"timeStamp": "20210621T071042Z",
"author": "Thomas Lauria"
}

in case if similar record exists, t5memory comparing source text, 
if it's the same, t5memory would compare docName, 
if it's the same,t5memory would compare timestamps and would leave only newer one

in case if TM is alreade reached it's limit, you would get 
{
"ReturnValue": 5034,
"ErrorMsg": ""
}or{
"ReturnValue": 5035,
"ErrorMsg": ""
}

 --v is glogs flag, for t5memory it make sense to set it to 0(by default) for production or to 2 for debuging.glog have it's own log levels and flags, but we are not touching them, default is okay, but it have just INFO, WARNING, ERROR. t5memory have it's own system, that was implemeted before proxygen, so it have 6 log levels (0=develop, 1=debug, 2=info, 3=warning, 4=error, 5=fatal, 6=transaction), which would be streamed to glog streams this way:
1. develop, debug, info, transaction would be streamed to glogs INFO stream
2. warning to WARNING
3. error and fatal error to ERROR stream, but also when first error log would happen, cached info about tm name and body of request that caused error would be flushed once per request. with next errors in the same request, you should not see "...with body... etc" in the log.--v and t5log level are just two separate filters for logs. when you set --v=0, glog allow only ERROR stream to work, so if you set t5loglevel to [0,1,2,3,4] wouldn't matter. but you can set to 5 to skip regular errors and have only fatal errors. in that mode also transaction log level is downgraded to be just info log levelwhen you set --v=2 you disable glogs filter, so you would have a lot of logs and now you can set logging intensivity with --t5loglevel. also t5transaction would be the highest log level now, you you can skip all info logs(with --t5loglevel=4) and still have transaction logs(which are not warnings or errors, usually something about that request handling begin or end, if you enable that). 
Shortly: for production just leave defaults(–v=0, --t5loglevel=2(info)), for debugging you need to set --v=2 and --t5loglevel to 0,1,2. Sometimes it make sense to use 1 or 2, because for 0 t5loglevel would print a ton of logs and t5memory would be slow.

Logging could impact application speed very much, especially during import or export. In t5memory there are 2 systems of logs - one from glog library and could be set in launch as commandline parameter and one is internal to filter out logs based on their level, can be set with every request that have json body with additional ["loggingThreshold": 0]  parameter or at startup with flag. 
[loggingThreshold:"2"]
Like here 

POST http://localhost:4040/t5memory/example_tm/

{
sourceLang: “en”, // the source language is required for a new TM
name: „TM Name“,
loggingThreshold:"2"
}
This would set the logging level to INFO just before the main work of creating mem endpoint starts. DEVELOP could be used in really low level debugging, but most of the time DEBUG log is more useful, since DEVELOP would log a lot of logs.  Transaction logs have the highest level of severity but it's severity is also changes with -v parameter, so with --v=2 it would be the highest log level(this log is not used often, it's only to track something like end or start of request) but with default --v=0 it's severity is belowe WARNING

gLog part - it have it's own configuration with command line flags. you can see all possible flags for t5memory with ./t5memory --help command.
main parameter here is --v and you can set it to 2 or 0(default). 
By default it set to 0,  in that case all not-errors would be avoided in logs, except startup. 
idea of --v=1 was to have logBuffer to keep log in some stream and in case of error show previous logs for that request, but it seems not so usefull, so it was not fixed and it's not working properly
--v=2 is basicaly disables that buffering, so 
In case of error or fatalError, log would be written with info about what request caused that log to happen(but that info would be truncated to 3000 symbols, this is important for importTMX), but if there are second error with the same request, new logs would not have that requests info

Some parameters combinations:
Default - --t5loglevel=2(T5INFO), --v=0,  in this case you could see only init messages and errors only, with info about requests that caused error to happen
Change only --v=2 - t5loglevel would be set by default to 2(T5INFO), so you could see T5INFO, T5WARNING, T5ERROR, T5FATAL, T5TRANSACTION messages
Debug production --t5loglevel=1(T5DEBUG), --v=2 - should be enough to have some info about issues, a lot of logs, but not as much as with Develop
Develop --t5loglevel=0(T5DEVELOP), --v=2 - all possible logs, includes entering to some functions, some step-by-step mechanisms logs(like how t5memory is parsing and hashing strings) etc. Useful only when you can reproduce issue so you don't get lost in logs from just normal behaviour or when it's crashing etc.

It's possible to change t5loglevel with some requests, so for example for some specific update request, you can set it to some lower log level and then set it back. It would affect other threads, but since in logs you have info about thread, it could be useful tool. 

Seems like --v parameter it's not quite useful, maybe should be refactored, since with --v=0 you wouldn't get any messages with severity lower than T5ERROR, except init process. 
But gLog library could be connected to some other libs in proxygen package

Here are all glog flags:
Flags from src/logging.cc:
    -alsologtoemail (log messages go to these email addresses in addition to
      logfiles) type: string default: ""
    -alsologtostderr (log messages go to stderr in addition to logfiles)
      type: bool default: false 
    -colorlogtostderr (color messages logged to stderr (if supported by
      terminal)) type: bool default: false
    -drop_log_memory (Drop in-memory buffers of log contents. Logs can grow
      very quickly and they are rarely read before they need to be evicted from
      memory. Instead, drop them from memory as soon as they are flushed to
      disk.) type: bool default: true
    -log_backtrace_at (Emit a backtrace when logging at file:linenum.)
      type: string default: ""
    -log_dir (If specified, logfiles are written into this directory instead of
      the default logging directory.) type: string default: ""
      currently: "/root/.t5memory/LOG/"
    -log_link (Put additional links to the log files in this directory)
      type: string default: ""
    -log_prefix (Prepend the log prefix to the start of each log line)
      type: bool default: true
    -logbuflevel (Buffer log messages logged at this level or lower (-1 means
      don't buffer; 0 means buffer INFO only; ...)) type: int32 default: 0
    -logbufsecs (Buffer log messages for at most this many seconds) type: int32
      default: 30
    -logemaillevel (Email log messages logged at this level or higher (0 means
      email all; 3 means email FATAL only; ...)) type: int32 default: 999
    -logfile_mode (Log file mode/permissions.) type: int32 default: 436
    -logmailer (Mailer used to send logging email) type: string
      default: "/bin/mail"
    -logtostderr (log messages go to stderr instead of logfiles) type: bool
      default: false
    -max_log_size (approx. maximum log file size (in MB). A value of 0 will be
      silently overridden to 1.) type: int32 default: 1800
    -minloglevel (Messages logged at a lower level than this don't actually get
      logged anywhere) type: int32 default: 0
    -stderrthreshold (log messages at or above this level are copied to stderr
      in addition to logfiles.  This flag obsoletes --alsologtostderr.)
      type: int32 default: 2
    -stop_logging_if_full_disk (Stop attempting to log to disk if the disk is
      full.) type: bool default: false

 

Config directory should be located in a specific place

Config file should be located under ~/.t5memory/t5memory.conf

In first concept it was planned to implement routines to open and close a TM. While concepting we found some problemes with this approach:

• First one is the realization: opening and closing a TM by REST would mean to update the TM Resource and set a state to open or close. This is very awkward.
• Since in translate5 multiple tasks can be used to the same time, multiple tasks try to access one TM. Closing TMs is getting complicated to prevent race conditions in TM usage.
• Since OpenTM2 loads the whole TM in memory, OpenTM2 must control itself which TMs are loaded or not.

This leads to the following conclusion in implementation of opening and closing of TMs:

OpenTM2 has to automatically load the requested TMs if requested. Also OpenTM2 has to close the TMs after a TM was not used for some time. That means that OpenTM2 has to track the timestamps when a TM was last requested.

Concept endpoints, not implemented

http://opentm2/translationmemory/[TM_Name]/openHandle

GET – Opens a memory for queries by OpenTM2

Note: This method is not required as memories are automatically opened when they are accessed for the first time.

http://opentm2/translationmemory/[TM_Name]/openHandle

DELETE – Closes a memory for queries by OpenTM2

Note: This method is not required as memories are automatically opened when they are accessed for the first time.

For now we open  TM in case of call to work with it. TM stays opened till the shutdown we wouldn't try to open more TM's, exceeding the RAM limit setupped in config file. 
In that case we would close TM in order of longest not used, till we would fit in limit including TM that we try to open.
 TM size is calcucated basicaly as sum .TMD and .TMI files
Ram limit doesn't include service RAM and temporary files

In 0.6.44 multithreading are implemented this way

1. you can set number of sevice threads with --servicethreads 8 commandline argument. This would be a number of threads which would handle requests, but also there would be 2-3 proxygen service threads running, and also for every import and reorganize - they would be new threads created. 
2. there are mutexes for shared resources, like filesystem and some shared files in the ram, which are not configurable
3. there are 3 configurable recursive timed mutexes, which can be also be used as non-timed mutexes(that means that it would not have timeout, but would wait till mutex would become free). To use them that way, they need to be set to 0
   
   This mutexes are -
4. requestTMMutex - mutex for whole requestTM functions, which can just find tm in tm list, or reserve a place for tm in that list(tm would be opened later). Probably could be disabled to optimize code, was implemented as first high level mutex
5. tmListMutex - every operation with tm list, like search, adding or deleting elements is managed with that mutex. 
6. tmMutex - most requests which have tm_name in url, except status request, would be blocking - they would occupy tm for whole execution time(after request data would be parsed and checked). The reason for that is opentm2 code, which still have too many low level chunks, which makes multithreading impossible. 
7. by default in import or reorganize threads(non request handlers - that would have regular mutexes, but threads, which are created by that handlers, which would run after you would receive response from your reorganize or import tmx requests) would be used non-timed mutexes. So this threads would wait till tm would be free.  You can change that with commandline argument 

   UseTimedMutexesForReorganizeAndImport 1

8. You can set default values for tmRequestLockDefaultTimeout, tmLockDefaultTimeout, tmRequestLockDefaultTimeout using commandline arguments with this names. Value would be set in ms, default value is 0, which means that timeouts would be disabled. That change would apply for all requests without body and for requests with body if other value is not provided. For import and reorganize threads by default would be used non-timed mutexes, but if it's changed with commandline argument, would be used value from corresponding request(if provided, or default if not). 
9. saveAllTms request could be used in 2 ways - with non timed mutexes for tms, or with timed mutexes, and in case of timeout, tm would not be saved and request would skip to the next tm. In response you would see message about saved and timeouted tms. 
10. shutdown request would internally use saveAllTms with hardcoded non-timed mutexes. But it could fail on tmListMutexTimeout when checking how many tm are in import and reorganize status
11.  resources request would use tmListMutex and timeout, when listing tms. It case of timeout, instead of list of tms, Failed to lock tm list: (timeout_error_msg) would be returned. But that wouldn't be threated as request error.   
12. in case of timeout fail, you would get back (and into the logs) errorcode 506 and one of the next messages(where (msg) is generated service message about location of failed lock and backtrace):

    (msg); Failed to lock tm list:(msg) /(msg); Failed to lock requestTm:(msg) / (msg); Failed to lock tm:(msg)

    like this : {
    "ReturnValue":506,
    "ErrorMsg":"Failed to acquire the lock(tmMutex:18) within the timeout(1ms)(location: requestTM:339); Failed to lock tm:(location: requestTM:342); "
    }
13. you can see default timeout values in initMsg. every timeout value is ms value.
14. for requests with body, you can provide value for each type of mutexes as integer with this names:

   {

       "tmMutexTimeout": 5000, 

       "tmListMutexTimeout": 4000,

       "requestTMMutexTimeout": 15000,

...

}

Mutexes and request handling details:

Can you explain more, why the tm list mutex is needed?
Should it not be enough to block a new request from accessing a TM that is already in use?

it is mutex to block access to the tm list.
It's not quite a list, internally it's a map, hash table, or dictionary- so it has a key, which is tm name, and value, which is tm data, and its auto sorting
also, that means, as with any non-fixed size array, that I can't be sure about its memory location, because during, for example, search in one thread, it could be completely reallocated to another size in the memory, if some thread would try to add some new tm object to that list. so even read operations should be blocking
so yes, I would explain about simplier mutexes first
In 0.5 I made every request handler as a class, which have the same abstract ancestor, which implements the same strategy pattern to execute any request, but each request implements it's own methods as it need it to be implemented
here is a code to run every request, but each request type implements it's own parseJSON, checkData and execute methods.
TimedMutexGuard  is a class that I implemented to make timed recursive mutexes and be flexible and that could be used as RAII, and  tmLockTimeout is a timeout class, which I implemented to have timeout fields, flag that some timeout is out of time, so in case of that, execution should be rolled back with an error code. Also, in case of an error, that class writes down a string with all functions names and a line numbers, that was calling for that mutex even if it's nested, to trace location of failed mutex.  if timeout would be set to 0, it would be used as non-timed mutex
//pseudo-code but it's close to real 

int RequestData::run(){ 
  if(!res) res = parseJSON();
  if(!res) res = checkData();
  if(!res) res = requestTM();
  if(!res)
  {
    if(isLockingRequest())
    {
        TimedMutexGuard l(mem->tmMutex, tmLockTimeout, "tmMutex") ; // mutexes used as RAII, to open the lock automatically 
        if(tmLockTimeout.failed()){
          return buildErrorReturn(506, tmLockTimeout.getErrMsg(), _rc_);  // would return "Failed to lock tm:...   

        }      
        res = execute();
      }// here mutex would be destroyed 

      else

      { // request that doesn't require lock, so it's in general without tmName in URL + status request

          res = execute();    

      }

  }  
  buildRet(res);
  //reset pointer
  if(mem != nullptr)  mem.reset(); 

  return res;
} 
so here you can see how tmLock is used
it's blocking selected tm for execution time only

then there are requestTM function, which would, depending on request type, request writePointer, readPointer or service pointer (some requests, like list of tm's , that don't have tm in url, or status request, as exception)
requesting writePointer or readPointer would check tm list and return tm if tm is found
else, it would init tm with found tm name and add that to tmList(it used to open tm also, but now it's separated)
service pointer wouldn't add nor open tm if it's not in list, but, can also return tm, if it's in list, without blocking it, for status request)
and that requestTM function have it's own mutex. it was implemented first, so only one tm could be requested at a time. Maybe with that tmList mutexes that I would explain later here, we don't need that requestTM function, but anyway... requestTM could also try to free some space, closing tms in the list, so at least that probably should be managed and synced.
so we would see if it's needed

and TmList mutex
so, every access to tm list block that mutex
few versions ago I had simple mutexes, but then it became recursive mutexes to simplify code and make it safer.
Then I had 2 versions of functions, safe, with mutex, and unsafe, that was used in functions that had that mutex locked on higher level
for example, I had this most primitive function

bool TMManager::IsMemoryInList(strMemName, tmListTimeout)

{  

  TimedMutexGuard l {mutex_access_tms, tmListTimeout, "tmListMutex"};// lock tms list
  if(tmListTimeout.failed()){     

    returnError(".Failed to lock tm list:");    

    return false;  

} // if TM mutex is failed in some nested function, tmListTimeout would be marked as spoiled so every other mutex that would use that timeout would be failed after first fail. in execution boolean function would return false, but also check if mutex was spoiled is needed to find out if function returned false because it didn't find (in this case) tm in list, or because its timeout is spoiled. But all checks is placed in code for now. 

  return TMManager::find(strMemName); // if lock was successful - try to find mem
}

 so it's boolean, but making it return some custom data type would make it harder to use
it's used everywhere, for example, to check if memory is loaded, but without having pointer, we have this function

bool TMManager::IsMemoryFailedToLoad(strMemName, tmListTimeout){  
 TimedMutexGuard l{mutex_access_tms, tmListTimeout, "tmListMutex"};
    bool res = false;
    if(tmListTimeout.failed())

   {      

      tmListTimeout.addToErrMsg(".Failed to lock tm list:");    

      return false;  

   }

   if(IsMemoryInList(strMemName, tmListTimeout)  
    && tms[strMemName]->isFailedToLoad())

        {    

                 res = true;    

        }

  if(tmListTimeout.failed())

  {     // if timeout was spoiled,  errMessage would be extended with new, so you would have backtrace with functions and lines in the file in the outputting message  

     tmListTimeout.addToErrMsg(".Failed to lock tm list:");

     return false;  

}

  return res;
}
 which also have blocking IsMemoryInList  but also blocks the same mutex because it's also working with tm list directly, and theoretically some other thread could change tm list between these lines

IsMemoryInList(strMemName, tmListTimeout)  // memory was checked to be present in the list
    && tms[strMemName]→isFailedToLoad()) // slow change that list was changed from the line above in another thread

so that type of boolean functions in case of timeout would return false, and then places where they were called should check if they returned false and timeout is not exiped. In case if it expired once, it would fail every next mutex lock, so even if that check is missing, that should now be handled in a right way. every addToErrMsg  sets that tmMutex is expired and adds some comment, function name and line number, so they could be tracked.

and tm list could be used not only when just requesting tm, but, for example, for resource request, or to free some space, or flush tm during shutdown.
so access to tm list should be managed and synced
that 2 classes TimedMutexGuard and MutexTimeout(tmListTimeout- is an object of that class) makes it more time consuming to implement that mutexes, because of that requiremnt
They should provide RAII possibility, be recursive and timed, but also supports non-timed mode, collect info in case if timeout failed, and also secure that in case if one timeout would fail, next mutexes would fail automatically, collect, provide and log data, where it failed

Regarding
and that requestTM function have it's own mutex. it was implemented first, so only one tm could be requested at a time
so I understand it right, as long as one TM is requested for opening (not for reading) another one can not be opened at the same time? That does not make sense to me. Why that? That means, a request for a small TM has to wait, until a large TM has been loaded into the RAM.

Load call is outside of mutex_requestTM mutex, so it wouldn't be blocked in current version.  Openning(loading) of the tm files is happening outside of the
mutex_requestTM  area.

So an active mutex to the tm list would still block every request then, also to other TMs, rigth? How long will it be blocked for example, if I import a new TMX into one TM? The whole time the TMX is imported? And same question for update?
And what about read requests? I understood, they will also block the TM list, right? But why I do not understand.

no, it wouldn't be blocked whole time for another tm's
it would be blocked for the time to check if tm is in list, and then if not, it would add it to the list. TMListLock is used only to prevent rearranging list when accessing its data. 
during that it would calculate how much space is needed and if we have enough space free
in not enough, it would delete tms from the list starting from the one that was not in use for the longest time
but if that tm is blocked, that wouldn't prevent for deletion from the list, because there are smart pointers used there, so tm would be closed when last pointer would be freed.
but it could take longer if there are some processes which uses tm list

blocking tm is necessary because of big chunk of low-level code which exists in opentm2 which operated with pointers to the memory (RAM) for a long time
I implemented read pointer and write pointer to be able in future have multiple read pointers and only single write pointer at the time, but with old code of opentm2 we need to treat every request as write request, because it can lead to memory reallocations that leads to crashes
For example if you have 2 projects translating with the same tm assigned, they would send both requests to the same tm, and that leads to crashes. that was actually one of the things that I fixed in recent versions
so tm should be blocked at least till some legacy code would be removed. like lookup tables.

most up-to-date version for this ticket is 0.6.75, where there are new flags and functionality to manipulate tcp stack.
--http_listen_backlog, default was 1024, in 0.6.75 it's 128, suppose to set tcp backlog for proxygen server, but seems like in reality it's just a hint, because requests over that limit is not dropping, except of timeout

--add_premade_socket - this is used to create socket and bind it to proxygen server instead of just providing ip address tot the server to open socket inside, should be set to true to enable, log_tcp_backog_events and socket_backlog flags

--log_tcp_backog_events  if set to true allow to test tcp backog, for that also recomended to set  --v=2 --t5loglevel=4. Require  add_premade_socket  to be set to true. You would see then in logs behaviour of tcp backlog

--socket_backlog is simillar to http_listen_backlog, but for socket.  But this require add_premade_socket to be set to true

--limit_num_of_active_requests, this would limit num of requests that could be handled at the same time in a way, when only n-1 of n created worker threads could be executed at the same time. last one would send 503 error and message that service is busy. I think that it make sense to play with num of worker threads and measure performance, for example try service with 32 threads on 8 cores. in that case service would handle properly 31 thread but 32nd would be responded with error.

--debug_sleep_in_request_run just make sleep  n microseconds(1/1000000 s) in every requests  to artificially slow them down. 

to test behaviour of tcp backlog you can use attached python script via command: 
python(3) sendNrequests4.py -n 40 
this would send 40 request on default local t5memory address
feel free to edit script if needed
flags for

to test tcp backlog you can set --add_premade_socket=1 --t5loglevel=4 --v=2 --debug_sleep_in_request_run=10000000 --log_tcp_backog_events=true --log_every_request_end=1 --log_every_request_start=1 --http_listen_backlog=4 --socket_backlog=2

and other flags as you wish

This would make every request at least 10 sec longer, every tcp backlog action would be logged, and also start and end of request handler execution, proxygens http tcp backog would be set to 4(or set it to some other value), and sockets backlog to 2
add_premade_socket is required to set sockets backlog and also tcp backlogs event logs.

other approach is to set docker containers environment, but seems like it's also just a hint and could be ignored by os
in docker-compose.yaml:
  myt5m:
    image: translate5/t5memory:0.6.75
    sysctls:
      net.core.somaxconn: 1
      net.ipv4.tcp_max_syn_backlog: 1
      net.ipv4.tcp_abort_on_overflow: 1
    ports:
      - '4086:4086'

import asyncio
import aiohttp
import argparse
import time
import traceback

async def fetch(session, url, request_id):
    try:
        async with session.get(url, timeout=60) as response:
            text = await response.text()
            if response.status != 200:
                print(f"Request {request_id}: Error with status {response.status}. Response:")
                print(text)
            else:
                print(f"Request {request_id}: Success with status {response.status}")
            return response.status, text
    except Exception as e:
        print(f"Request {request_id}: Exception occurred: {e}")
        traceback.print_exc()  # Print the full traceback for the exception
        return e  # Return the exception for further handling

async def main(num_requests, url, delay):
    async with aiohttp.ClientSession() as session:
        tasks = []
        for i in range(num_requests):
            tasks.append(asyncio.create_task(fetch(session, url, i)))
            if delay > 0:
                await asyncio.sleep(delay)
        results = await asyncio.gather(*tasks, return_exceptions=True)

    success_count = 0
    failure_count = 0
    for idx, result in enumerate(results):
        if isinstance(result, Exception):
            failure_count += 1
            print(f"Request {idx} raised an exception: {result}")
        else:
            status, text = result
            if status is None or status != 200:
                failure_count += 1
                print(f"Request {idx}: Failed. Status: {status}. Response: {text}")
            else:
                success_count += 1

    print(f"\nTotal successes: {success_count}")
    print(f"Total failures: {failure_count}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        description="Send multiple HTTP GET requests concurrently with an optional delay between requests"
    )
    parser.add_argument("-n", "--num_requests", type=int, default=200,
                        help="Number of parallel requests to send (default: 200)")
    parser.add_argument("-u", "--url", type=str, default="http://127.0.0.1:4080/t5memory",
                        help="URL to send requests to (default: http://127.0.0.1:4080/t5memory)")
    parser.add_argument("-d", "--delay", type=float, default=0.1,
                        help="Delay in seconds between starting each request (default: 0.1)")
    args = parser.parse_args()
    
    asyncio.run(main(args.num_requests, args.url, args.delay))