Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Added info about new concordance search

...

nSearchnSearchnSearchnSearchnSearchnSearch

New Concordance search

PurposeReturns entries\translations that fits selected filters. 
RequestPOST /%service%/%tm_name%/search
Params

Required: NONE

iNumOfProposal -  limit of found proposals - max is 20200, if 0 → use default value '5' 

Search is made segment-by segment, and it's checking segment if it fits selected filters. You can search for EXACT or CONCORDANCE matches in this fields:
source, target, document, author, addInfo, context
To set filter, use it's SearchMode field, otherwise filter would be disabled. So you have sourceSearchMode, targetSearchMode, documentSearchMode,
authorSearchMode, addInfoSearchMode, contextSearchMode 

Search mode should be set explicitly to CONTAINS/CONCORDANCE or EXACT, otherwise filter would be ignored. But also each searchMode could have additional search parameters "CONTAINS, caseinsensetive, WHITESPACETOLERANT, INVERTED", all that values is not important, as well as delimiter. By default search is case sensetive. If you add Inverted option, check for that filter would be reverted. 
To check how filters would be parsed, check json in responce.  Field with that info could look like this:

"Filters":"
Search filter, field: SOURCE FilterType::CONTAINS SearchStr: 'THE'; Options: SEARCH_FILTERS_NOT|SEARCH_CASEINSENSITIVE_OPT|SEARCH_WHITESPACETOLERANT_OPT|;\n
Search filter, field: TARGET FilterType::EXACT SearchStr: ''; Options: SEARCH_CASEINSENSITIVE_OPT|;\n
Search filter, field: ADDINFO FilterType::CONTAINS SearchStr: 'some add info'; Options: SEARCH_WHITESPACETOLERANT_OPT|;\n
Search filter, field: CONTEXT FilterType::EXACT SearchStr: 'context context'; Options: ;\nSearch filter, field: AUTHOR FilterType::CONTAINS SearchStr: ''; Options: ;\n
Search filter, field: DOCUMENT FilterType::CONTAINS SearchStr: 'evo3_p1137_reports_translation_properties_de_fr_20220720_094902'; Options: SEARCH_FILTERS_NOT|;\n
Search filter, field: TIMESTAMP FilterType::RANGE Range: 20000121T115234Z - 20240121T115234Z Options: ;\n"
,

It's possible to apply filter just with SearchMode, like if you would type "authorSearchMode": "exact",but there would be no "author" field, it would look for segments, where author field is empty.

Also there are  timespan parameter, to set it, use this fields and format:

"timestampSpanStart":"20000121T115234Z",
"timestampSpanEnd":"20240121T115234Z",

You should set both parameters to apply filter, otherwise you would get error as return.  Check output to see how it was parsed and applied.
By default all mentioned filters is applied in logical and combination, but you can change that globaly with adding 

"filtersCombinedAs":"or",
Then all mentioned filters would be applied in logical or combination.

Also there are lang filters, they would always be applied to selection of segments that passed previous filters, so value of  filtersCombinedAs wouldn't be applied to that.
To set language filters, use this fields:

"sourceLang":"en-GB",
"targetLang":"de",

Lang filters could be applied with major lang feature, so source lang in this case would be applied as exact filter for source lang, but target lang would check if langs is in the same lang group. That check is done in languages.xml file with isPreferred flag.
Lang filters and if filters is combined in logical or or logical and you can check in  GlobalSearchOptions  field of responce. It could look like this: 

"GlobalSearchOptions":"SEARCH_FILTERS_LOGICAL_OR|SEARCH_EXACT_MATCH_OF_SRC_LANG_OPT, lang = en-GB|SEARCH_GROUP_MATCH_OF_TRG_LANG_OPT, lang = de",

Other that you can send is:

"searchPosition":"8:1",
"numResults":2,
"msSearchAfterNumResults":250
"loggingThreshold": 4 - check other requests,

So search position is position where to start search internaly in btree. This search is limited by num of found segment(set by numResults) or timeout(set by msSearchAfterNumResults), but timeout would be ignored in case if there are no segments in the tm to fit params.  Max  numResults is 200.

You can send empty json and search would work fine, but it would just return first 5 segments in tm
You can go through all segment  with using this 2 fields
"searchPosition":"8:1",
"numResults":200 

and just updating  searchPosition with  NewSearchPosition
from responce.


Code Block
languagejs
titleResponse
collapsetrue
{
    "filtersCombinedAs": "or",
    "source":"the",
    "sourceSearchMode":"CONTAINS, CASEINSENSETIVE, WHITESPACETOLERANT, INVERTED",
    
    "target":"",
    "targetSearchMode":"EXACT, CASEINSENSETIVE",
    
    "document":"evo3_p1137_reports_translation_properties_de_fr_20220720_094902",
    "documentSearchMode":"CONTAINS, INVERTED",
    
    "authorSearchMode":"CONTAINS",

    "timestampSpanStart": "20000121T115234Z",
    "timestampSpanEnd": "20240121T115234Z",

    "addInfo":"some add info",
    "addInfoSearchMode":"CONCORDANCE, WHITESPACETOLERANT",


    "context":"context context",
    "contextSearchMode":"EXACT",
    
    "sourceLang":"en-GB", 
    "targetLang":"SV",  
    "searchPosition": "8:1",
    "numResults": 2,
    "msSearchAfterNumResults": 25
}

So here search would be done in logical or way, so if any of source, target, document, context, author, timestamp filters returns true, result would be added to set, which then would be filtered out by sourceLang on exact match check and targetLang on groupLang check.
Search would start from position "8:1"(tm data start at "7:1" but if you wan't to start from the beggining, just avoid that param. 
numResuts:2 - so if there would be 2 segments found, search would end
"msSearchAfterNumResults": 25 - 25ms after first found segment, search would end, even if more segments was found, responce would contain "NewSearchPosition": "10:1", which can be used in searchPosition to continue search

Response example:Success:
example{
"Filters": "Search filter, field: SOURCE FilterType::CONTAINS SearchStr: 'THE'; Options: SEARCH_FILTERS_NOT|SEARCH_CASEINSENSITIVE_OPT|SEARCH_WHITESPACETOLERANT_OPT|;\
n
Search filter, field: TARGET FilterType::EXACT SearchStr: ''; Options: SEARCH_CASEINSENSITIVE_OPT|;\
n
Search filter, field: ADDINFO FilterType::CONTAINS SearchStr: 'some add info'; Options: SEARCH_WHITESPACETOLERANT_OPT|;\
n
Search filter, field: CONTEXT FilterType::EXACT SearchStr: 'context context'; Options: ;\
n
Search filter, field: AUTHOR FilterType::CONTAINS SearchStr: ''; Options: ;\
n
Search filter, field: DOCUMENT FilterType::CONTAINS SearchStr: 'evo3_p1137_reports_translation_properties_de_fr_20220720_094902'; Options: SEARCH_FILTERS_NOT|;\
n
Search filter, field: TIMESTAMP FilterType::RANGE Range: 20000121T115234Z - 20240121T115234Z Options: ;\n",
"GlobalSearchOptions": "SEARCH_FILTERS_LOGICAL_OR|SEARCH_EXACT_MATCH_OF_SRC_LANG_OPT, lang = en-GB|SEARCH_GROUP_MATCH_OF_TRG_LANG_OPT, lang = sv",
"ReturnValue": 0,
"ReturnMessage": "FOUND",
"NewSearchPosition": "10:1",
"results": [
{
"source": "Congratulations on the purchase of a <ph x=\"101\"/> machine control system.",
"target": "Gratulerar till köpet av maskinstyrningsystemet <ph x=\"101\"/>.",
"segmentNumber": 5740419,
"id": "",
"documentName": "none",
"sourceLang": "en-GB",
"targetLang": "SV-SE",
"type": "Manual",
"author": "",
"timestamp": "20170327T091814Z",
"markupTable": "OTMXUXLF",
"context": "",
"additionalInfo": "",
"internalKey": "8:1"
},
{
"source": "The <ph x=\"101\"/> System is an ideal tool for increasing productivity in all aspects of the construction earthmoving industry.",
"target": "Systemet <ph x=\"101\"/> är ett verktyg som lämpar sig perfekt för att öka produktiviteten inom alla delar av bygg- och anläggningsområdet.",
"segmentNumber": 5740420,
"id": "",
"documentName": "none",
"sourceLang": "en-GB",
"targetLang": "SV-SE",
"type": "Manual",
"author": "",
"timestamp": "20170327T091814Z",
"markupTable": "OTMXUXLF",
"context": "",
"additionalInfo": "",
"internalKey": "9:1"
}
]
}
SearchPosition / NewSearchPositionFormat: "7:1"
First is segment\record number, second is target number
The NextSearchposition is an internal key of the memory for the next position on sequential access. Since it is an internal key, maintained and understood by the underlying memory plug-in (for EqfMemoryPlugin is it the record number and the position in one record),
no assumptions should be made regarding the content. It is just a string that, should be sent back to OpenTM2 on the next request, so that the search starts from there.
So is the implementation in Translate5: The first request to OpenTM2 contains SearchPosition with an empty string, OpenTM2 returns than a string in NewSearchPosition, which is just resent to OpenTM2 in the next request.

Not found:{
"ReturnValue": 0,
"NewSearchPosition": null,
"ErrorMsg": ""
}TM not found:{
"ReturnValue": 133,
"ErrorMsg": "OtmMemoryServiceWorker::concordanceSearch::"
}







Concordance search

PurposeReturns entries\translations that contain requested segment
RequestPOST /%service%/%tm_name%/concordancesearch
Params

Required: searchString - what we are looking for , searchType ["Source"|"Target"|"SourceAndTarget"] - where to look

iNumOfProposal -  limit of found proposals - max is 20, if 0 → use default value '5' 


Code Block
languagejs
titleResponse
collapsetrue
Request example:
{
    "searchString": "The",
    "searchType": "source", // could be Source, Target, SourceAndTarget - says where to do search
    ["searchPosition": "",] 
    ["numResults": 20,]
    ["msSearchAfterNumResults": 250,]
	["loggingThreshold": 0]
}
Response example:Success:
example_new{
  "ReturnValue": "ENDREACHED_RC",
  "NewSearchPosition": null,
  "results": [
    {
      "source": "The end",
      "target": "The target",
      "segmentNumber": 0,
      "id": "",
      "documentName": "Te2.xlf",
      "sourceLang": "de-DE",
      "targetLang": "EN-GB",
     "type": "Manual",
     "author": "THOMAS LAURIA",
     "timestamp": "20231228T171821Z",
      "markupTable": "OTMXUXLF",
      "context": "2_3",
      "additionalInfo": "",
      "internalKey": "7:1"
    }
  ]
}

example_old
{
  "ReturnValue": 0,
  "NewSearchPosition": null,
  "results": [
  {
     "source": "For > 100 setups.",
     "target": "Für > 100 Aufstellungen.",
     "segmentNumber": 10906825,
     "id": "",
     "documentName": "none",
     "documentShortName": "NONE",
     "sourceLang": "en-GB",← rfc5646     
     "targetLang": "de-DE",← rfc5646
     "type": "Manual",
     "matchType": "undefined",
     "author": "",
     "timestamp": "20190401T084052Z",
     "matchRate": 0,
     "markupTable": "OTMXML",
     "context": "",
     "additionalInfo": ""
   }
  ],
 "ErrorMsg": ""
}

Success, but with NewSearchPosition - not all TM was checked, use this position to repeat search:
{
  "ReturnValue": 0,
  "NewSearchPosition": "8:1",
  "results": [
  {
     "source": "For > 100 setups.",
     "target": "Für > 100 Aufstellungen.",
     "segmentNumber": 10906825,
     "id": "",
    "documentName": "none",
    "documentShortName": "NONE",
    "sourceLang": "en-GB",
    "targetLang": "de-DE",
    "type": "Manual",
     "matchType": "undefined",
     "author": "",
     "timestamp": "20190401T084052Z",
     "matchRate": 0,
     "markupTable": "OTMXML",
     "context": "",
     "additionalInfo": ""
   }
  ],
 "ErrorMsg": ""
}
SearchPosition / NewSearchPositionFormat: "7:1"
First is segmeng\record number, second is target number
The NextSearchposition is an internal key of the memory for the next position on sequential access. Since it is an internal key, maintained and understood by the underlying memory plug-in (for EqfMemoryPlugin is it the record number and the position in one record),
no assumptions should be made regarding the content. It is just a string that, should be sent back to OpenTM2 on the next request, so that the search starts from there.
So is the implementation in Translate5: The first request to OpenTM2 contains SearchPosition with an empty string, OpenTM2 returns than a string in NewSearchPosition, which is just resent to OpenTM2 in the next request.

Not found:{
"ReturnValue": 0,
"NewSearchPosition": null,
"ErrorMsg": ""
}TM not found:{
"ReturnValue": 133,
"ErrorMsg": "OtmMemoryServiceWorker::concordanceSearch::"
}


...