Using matcher:find-document-matches-by-options-name to check whether a document has any matches, reports in the section of the return matching documents along with their score and matches () feedback. However, there is not match entry for match based on dbl-metaphone algorithm. Is it natural?

As a USER
I want to TITLE
SO that over time, as I have multiple merges that have occurred on a document, I can still specify how merging should happen for each property instead of rely only on top-level document metadata.

For example, if I have a mastered doc with information derived from three sources, the lastUpdated value of each of those properties may be different, while the mastered document will retain the lastUpdated value for any change to the entire document.

Currently, if I have a fourth source that I want to use to merge into that mastered record and I want to keep the most recent properties, Smart Mastering only knows the lastUpdated information for the entire merged doc, not the individual properties. This will lead to unexpected behavior based on my use case, as the lastUpdated info for the doc can be very different than the lastUpdated info for a given property.

I am not able to add merge configuration to minimal-project using sm-merge-options Rest api, Although I was successful in adding match configuration.

Steps followed to setup.

1. Cloned the Smart mastering core repository and ran gradle.bat publishToMavenLocal command
2. Smart mastering added to local maven repository at location .m2\repository\com\marklogic\community\smart-mastering-core\1.0.0-beta.1.
3. Traversed to minimal-project and ran gradle mlDeploy
4. Verified the resources by hitting the end point
   http://localhost:8800/v1/config/resources
5. Access the below endpoint to put/post match & merge options
   http://localhost:8800/v1/resources/sm-match-options?rs:name=mlw-match
   http://localhost:8800/v1/resources/sm-merge-options?rs:name=mlw-merge
   Smart_Mastering_Resources.txt
   mdm-merge-options.txt

Let me know if I missed any step in setting the project.

In order to conduct fielded searches, I would like to be able to POST a pseudo-document to sm-match and get matches as if I had started with a document in the database.

For example, I would like to be able to POST the following to /LATEST/resources/sm-match?rs:options=mlw-match

{document: {PersonGivenName: "maurise"}}

When I try this now, I get the following error:

2018-07-10 16:52:52.762 Notice: XDMP-AS: (err:XPTY0004) $uri as xs:string -- Invalid coercion: () as xs:string
2018-07-10 16:52:52.762 Notice:+in /com.marklogic.smart-mastering/matcher-impl/matcher-impl.xqy, at 59:21,
2018-07-10 16:52:52.762 Notice:+in match-impl:find-document-matches-by-options(object-node{"PersonGivenName":text{"maurise"}}, fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options, xs:int("1"), xs:int("200"), xs:double("1"), fn:false(), fn:false()) [1.0-ml]
2018-07-10 16:52:52.762 Notice:+  $document = object-node{"PersonGivenName":text{"maurise"}}
2018-07-10 16:52:52.762 Notice:+  $options = fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options
2018-07-10 16:52:52.762 Notice:+  $start = xs:int("1")
2018-07-10 16:52:52.762 Notice:+  $page-length = xs:int("200")
2018-07-10 16:52:52.762 Notice:+  $minimum-threshold = xs:double("1")
2018-07-10 16:52:52.762 Notice:+  $lock-on-search = fn:false()
2018-07-10 16:52:52.762 Notice:+  $include-matches = fn:false()
2018-07-10 16:52:52.762 Notice:+  $options = fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options
2018-07-10 16:52:52.762 Notice:+  $tuning = fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options/matcher:tuning
2018-07-10 16:52:52.762 Notice:+  $property-defs = fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options/matcher:property-defs
2018-07-10 16:52:52.762 Notice:+  $thresholds = fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options/matcher:thresholds
2018-07-10 16:52:52.762 Notice:+  $scoring = fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options/matcher:scoring
2018-07-10 16:52:52.762 Notice:+  $algorithms = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>...function...)
2018-07-10 16:52:52.762 Notice:+  $query = cts:or-query((cts:element-value-query(fn:QName("","PersonGivenName"), "maurise", ("case-insensitive","lang=en"), 6), cts:element-value-query(fn:QName("","PersonGivenName"), ("Maurie", "MURIAL", "KRIS"), ("case-insensitive","lang=en"), 6)), ())
2018-07-10 16:52:52.762 Notice:+  $serialized-query = <boost-query><cts:or-query xmlns:cts="http://marklogic.com/cts"><cts:element-value-query .../>...</cts:or-query></boost-query>
2018-07-10 16:52:52.762 Notice:+  $minimum-threshold-combinations = (cts:element-value-query(fn:QName("","PersonGivenName"), "maurise", ("case-insensitive","lang=en"), 0), cts:element-value-query(fn:QName("","PersonGivenName"), ("Maurie", "MURIAL", "KRIS"), ("case-insensitive","lang=en"), 0))
2018-07-10 16:52:52.762 Notice:+in /com.marklogic.smart-mastering/matcher.xqy, at 102:2,
2018-07-10 16:52:52.762 Notice:+in matcher:find-document-matches-by-options(object-node{"PersonGivenName":text{"maurise"}}, fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options, 1, 200, fn:false()) [1.0-ml]
2018-07-10 16:52:52.762 Notice:+  $document = object-node{"PersonGivenName":text{"maurise"}}
2018-07-10 16:52:52.762 Notice:+  $options = fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options
2018-07-10 16:52:52.762 Notice:+  $start = xs:int("1")
2018-07-10 16:52:52.762 Notice:+  $page-length = xs:int("200")
2018-07-10 16:52:52.762 Notice:+  $include-matches = fn:false()
2018-07-10 16:52:52.762 Notice:+in /marklogic.rest.resource/sm-match/assets/resource.xqy, at 60:4,
2018-07-10 16:52:52.762 Notice:+in xdmp:function(fn:QName("http://marklogic.com/rest-api/resource/sm-match","post"), "/marklogic.rest.resource/sm-match/assets/resource.xqy")(map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}) [1.0-ml]
2018-07-10 16:52:52.762 Notice:+  $context = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $input = document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}
2018-07-10 16:52:52.762 Notice:+  $uri = "/none"
2018-07-10 16:52:52.762 Notice:+  $input-root = object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}
2018-07-10 16:52:52.762 Notice:+  $document = object-node{"PersonGivenName":text{"maurise"}}
2018-07-10 16:52:52.762 Notice:+  $options = fn:doc("/com.marklogic.smart-mastering/options/algorithms/mlw-match.xml")/matcher:options
2018-07-10 16:52:52.762 Notice:+  $start = 1
2018-07-10 16:52:52.762 Notice:+  $page-length = 200
2018-07-10 16:52:52.762 Notice:+  $include-matches = fn:false()
2018-07-10 16:52:52.762 Notice:+in /MarkLogic/rest-api/lib/extensions-util.xqy, at 945:44,
2018-07-10 16:52:52.762 Notice:+in extut:call-service("sm-match", "POST", xdmp:function(fn:QName("http://marklogic.com/rest-api/resource/sm-match","post"), "/marklogic.rest.resource/sm-match/assets/resource.xqy"), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}) [1.0-ml]
2018-07-10 16:52:52.762 Notice:+  $extension-name = "sm-match"
2018-07-10 16:52:52.762 Notice:+  $method = "POST"
2018-07-10 16:52:52.762 Notice:+  $service = xdmp:function(fn:QName("http://marklogic.com/rest-api/resource/sm-match","post"), "/marklogic.rest.resource/sm-match/assets/resource.xqy")
2018-07-10 16:52:52.762 Notice:+  $context = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $service-params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $input = document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}
2018-07-10 16:52:52.762 Notice:+in /MarkLogic/rest-api/lib/extensions-util.xqy, at 898:20,
2018-07-10 16:52:52.762 Notice:+in function() as item()*() [1.0-ml]
2018-07-10 16:52:52.762 Notice:+in /MarkLogic/rest-api/lib/extensions-util.xqy,
2018-07-10 16:52:52.762 Notice:+in xdmp:invoke(function() as item()*, <options xmlns="xdmp:eval"><isolation>same-statement</isolation><ignore-amps>...</ignore-amps></options>) [1.0-ml]
2018-07-10 16:52:52.762 Notice:+in /MarkLogic/rest-api/lib/extensions-util.xqy, at 896:12,
2018-07-10 16:52:52.762 Notice:+in extut:invoke-service("sm-match", "POST", "query", xdmp:function(fn:QName("http://marklogic.com/rest-api/resource/sm-match","post"), "/marklogic.rest.resource/sm-match/assets/resource.xqy"), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}, fn:false()) [1.0-ml]
2018-07-10 16:52:52.762 Notice:+  $extension-name = "sm-match"
2018-07-10 16:52:52.762 Notice:+  $method = "POST"
2018-07-10 16:52:52.762 Notice:+  $default-txn-mode = "query"
2018-07-10 16:52:52.762 Notice:+  $service = xdmp:function(fn:QName("http://marklogic.com/rest-api/resource/sm-match","post"), "/marklogic.rest.resource/sm-match/assets/resource.xqy")
2018-07-10 16:52:52.762 Notice:+  $context = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $service-params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $input = document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}
2018-07-10 16:52:52.762 Notice:+  $in-txn = fn:false()
2018-07-10 16:52:52.762 Notice:+  $txn-curr = "query"
2018-07-10 16:52:52.762 Notice:+  $txn-mode = "query"
2018-07-10 16:52:52.762 Notice:+in /MarkLogic/rest-api/models/resource-model-query.xqy, at 269:20,
2018-07-10 16:52:52.762 Notice:+in rsrcmodqry:resource-post("sm-match", map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}, fn:false(), local:rsrcmod-callback#6) [1.0-ml]
2018-07-10 16:52:52.762 Notice:+  $resource-name = "sm-match"
2018-07-10 16:52:52.762 Notice:+  $context = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $resource-params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $input = document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}
2018-07-10 16:52:52.762 Notice:+  $in-txn = fn:false()
2018-07-10 16:52:52.762 Notice:+  $responder = local:rsrcmod-callback#6
2018-07-10 16:52:52.762 Notice:+  $service = xdmp:function(fn:QName("http://marklogic.com/rest-api/resource/sm-match","post"), "/marklogic.rest.resource/sm-match/assets/resource.xqy")
2018-07-10 16:52:52.762 Notice:+in /MarkLogic/rest-api/models/resource-model-query.xqy, at 236:4,
2018-07-10 16:52:52.762 Notice:+in rsrcmodqry:exec-post(map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}, local:rsrcmod-callback#6) [1.0-ml]
2018-07-10 16:52:52.762 Notice:+  $headers = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $endpoint-params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $input = document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}
2018-07-10 16:52:52.762 Notice:+  $responder = local:rsrcmod-callback#6
2018-07-10 16:52:52.762 Notice:+in /MarkLogic/rest-api/endpoints/resource-service-query.xqy, at 78:8 [1.0-ml]
2018-07-10 16:52:52.762 Notice:+  $headers = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $method = "POST"
2018-07-10 16:52:52.762 Notice:+  $body = document{object-node{"document":object-node{"PersonGivenName":text{"maurise"}}}}
2018-07-10 16:52:52.762 Notice:+  $params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-07-10 16:52:52.762 Notice:+  $extra-names = ()
2018-07-10 16:52:53.127 Info: Status 500: XDMP-AS: (err:XPTY0004) $uri as xs:string -- Invalid coercion: () as xs:string

The issue may be the way I am calling the endpoint. This is how I called the old am-match endpoint, but I may need a different incantation now.

For completeness, the contents of mlw-match are:

<?xml  version="1.0" encoding="UTF-8"?>
<options xmlns="http://marklogic.com/smart-mastering/matcher">
<property-defs>
<property namespace="" localname="IdentificationID" name="ssn">
</property>
<property namespace="" localname="PersonGivenName" name="first-name">
</property>
<property namespace="" localname="PersonSurName" name="last-name">
</property>
<property namespace="" localname="AddressPrivateMailboxText" name="addr1">
</property>
<property namespace="" localname="LocationCity" name="city">
</property>
<property namespace="" localname="LocationState" name="state">
</property>
<property namespace="" localname="LocationPostalCode" name="zip">
</property>
<property namespace="" localname="PersonBirthDate" name="dob">
</property>
</property-defs>
<algorithms>
<algorithm name="std-reduce" function="standard-reduction">
</algorithm>
<algorithm name="std-reduce-query" function="standard-reduction-query">
</algorithm>
<algorithm name="dbl-metaphone" function="double-metaphone">
</algorithm>
<algorithm name="thesaurus" function="thesaurus">
</algorithm>
</algorithms>
<scoring>
<add property-name="ssn" weight="50">
</add>
<add property-name="last-name" weight="8">
</add>
<!--

      We want 12 points for the first name. Six here, plus six from a thesaurus match, which includes the original
      target name.
    
-->
<add property-name="first-name" weight="6">
</add>
<!--
     <add property-name="addr1" weight="5"/> 
-->
<!--
     <add property-name="city" weight="3"/>
    <add property-name="state" weight="1"/>
    <add property-name="zip" weight="3"/> 
-->
<!--

    <expand property-name="first-name" algorithm-ref="dbl-metaphone" weight="6">
      <dictionary>first-name-dictionary.xml</dictionary>
      <distance-threshold>50</distance-threshold>
    </expand>
    
-->
<expand property-name="first-name" algorithm-ref="thesaurus" weight="6">
<thesaurus>/mdm/config/thesauri/first-name-synonyms.xml</thesaurus>
<distance-threshold>50</distance-threshold>
</expand>
<expand property-name="first-name" algorithm-ref="dbl-metaphone" weight="6">
<dictionary>first-name-dictionary.xml</dictionary>
</expand>
<expand property-name="last-name" algorithm-ref="dbl-metaphone" weight="8">
<dictionary>last-name-dictionary.xml</dictionary>
<!--
defaults to 100 distance 
-->
</expand>
<reduce algorithm-ref="std-reduce" weight="4">
<all-match>
<property>last-name</property>
<property>addr1</property>
</all-match>
</reduce>
</scoring>
<thresholds>
<threshold above="1" label="Possible Match">
</threshold>
<threshold above="15" label="Likely Match" action="merge">
</threshold>
<threshold above="50" label="Definitive Match" action="merge">
</threshold>
<!--
 below 30 will be NOT-A-MATCH or no category 
-->
</thresholds>
<tuning>
<max-scan>200</max-scan>
<!--
 never look at more than 200 
-->
<initial-scan>20</initial-scan>
</tuning>
</options>

A user may want to experiment with a new set of match options. Given a set of options and a query that identifies a set of URIs to work with (could be a cts:document-query with a bunch of URIs), run the matcher on all the documents and produce a report of what the results would be and what actions would be taken.

Currently the collection names for content, archiving, and other purposes are hard-coded using constants.xqy. Revise this to provide configurable control.

The standard-reduction.xqy module has two functions: standard-reduction and standard-reduction-query. It doesn’t look like standard-reduction-query is actually used anywhere, although there are orphaned references to it in several of our example match options. The standard-reduction-query likely can't work, since it relies on combined queries, but a cts:and-query doesn't take a weight.

Actions:

• remove implementation of standard-reduction-query
• document that standard-reduction requires matches
• consider whether there's a way to implement this that doesn't require matches

The sm-merge extension takes parameters primary-uri and secondary-uri, but the underlying library accepts an arbitrary number of URIs. Modify sm-merge to take multiple URIs.

The zip.xqy algorithm is easy to understand because it includes a config example in the module comments. I think each OOTB algorithm would benefit from this - unless there's a separate plan for documenting them e.g. on the github Wiki or in another place. I like the example in the module though because it's next to the code then and easier to understand.

If a user creates a merge config to use a custom timestamp, the user must also include source information in the expected place in all documents, or else the merging by timestamp or recency silently fails.

As a USER
I want to understand exactly what algorithms are available for matching and for merging, what they do, and how to declare each one with examples in both XML and JSON.

Currently, a merged document inherits the collections from all source documents being merged, with the "mdm-merged" collection being added.

To support business workflows, smart mastering should allow users to select how collections are handled during a merge, including what collections should be added, and what collections should be removed from a merged document. Without having control of the collections, batch processing of documents requires writing more complex queries to isolate documents for processing. And being able to put merged documents into custom collections allows users to master multiple types of entities in the same database, instead of having multiple entities all being in the "mdm-merged" collection.

Ideally, this type of workflow should be supported:
(1) User puts all documents to be mastered in a "toBeMastered" collection
(2) A batch process runs by selected documents in the "toBeMastered" collection
(3) New documents are put into a user-specified collection, including the option to bring forward collections from source documents or not, like "masterPerson".
(4) Users can remove collections, like "toBeMastered" from merged and original documents, so the next time the batch process runs to select documents in the "toBeMastered" collection, it won't rerun against the same documents.

Add a documentation page for troubleshooting. Include:

• I expected these documents to match, but they aren't. How do I figure out why they didn't?
• How do find out what matches happened?
• How do I find the scores from matching?
• Outcome not as expected (eg no matches/merges) - how do I diagnose whether the error is in my code calling smart mastering? did Smart Mastering run?

As a USER
I want to TITLE
so if I have a URI scheme I want to deploy, it can be honored while using smart mastering. In addition, as information is updated, the URI for the same resource won't need to continually change with each merge that may be providing only very little information.

The current smart-mastering DHF example uses DHF 3. It would be nice to have not just an updated example project but also to provide more detailed instructions on how users can integrate it with a DHF project (basically fleshing out https://marklogic-community.github.io/smart-mastering-core/how-to-use/).

When saving match options using matcher.saveOptions(), options are saved with "mdm-match-options" collection.

However saving merge options using merging.saveOptions(), options are save with "mdm-merge" collection. It would be nice that when saving merge options, collection would be "mdm-merge-options".

As a USER
I want to title
so I can fix and resubmit my request, without having to consult documentation unless absolutely necessary

E.g, a call to sm-match without an rs:options or an options property defined in the body will result in a blank 200 response. Should send a 400 instead with details of what's missing.

14 Endpoints, may support a couple verbs each

Make sure each endpoint requesting required parameters

As a USER
I need to understand the data model requirements in order to use smart mastering (/envelope/instance/{entityName}/{entityProperties} along with any required namespace if XML)
and would like to receive a meaningful error message if I try to use SM features with documents that don't meet the data model requirements.

By default, the MarkLogic cts: matching query is always returned in the response. Would be nice to have an option to not return this.

As a USER
I was to TITLE
So I don't have to manually declare each property's merge options if I want all of my properties to be merged the same way, including the merge algorithm and max-values.

The default options should be overridden by any more specific merge option defined within the merging property. So if I have documents with 100 properties, I can set the default merge option once (e.g., max-value=1 and longest value), and override it for a specific property that I want handled in a different way (max-value=2 and source).

When processMatchandMerge is run with a uri, if there is no match, there should be an option that treats the document as a mastered document, and received the proper collections (see related issue #190).

As it stands now, documents come into an "mdm-content" collection, which will contain duplicates until the mastering process is run. By adopting the approach in issue 190, users can import data into a temp collection in the FINAL db, run processMatchAndMerge against them, and any merged documents will be put into the "mastered" collection, but any documents that don't have a match may still be considered an authoritative record for the entity, and also be put into the mastered collection. In that case, the un-merged document can be updated in place with the appropriate collections to avoid making another copy.

Given the document below (redacted: ask me for the whole thing) at /com.marklogic.smart-mastering/merged/ffcc223a-7479-4af0-9dfc-61797d8ebb84.xml, when I call:

/LATEST/resources/sm-history-document?rs:uri=/com.marklogic.smart-mastering/merged/ffcc223a-7479-4af0-9dfc-61797d8ebb84.xml

I only get:

{activities: []}

The document:

{
"envelope": {
"headers": {
"sources": [
{
"name": "MMIS", 
"user": "admin", 
"dateTime": "2018-07-09T13:23:34.647238Z", 
"action": "Harmonization", 
"rawDoc": "1191448670-4"
}, 
{
"name": "MEDS", 
"user": "admin", 
"dateTime": "2018-07-09T20:22:13.638788Z", 
"action": "Harmonization", 
"rawDoc": "/raw/meds/medstestdata_21.xml"
}, 
{
"name": "MMIS", 
"user": "admin", 
"dateTime": "2018-07-09T13:23:34.647238Z", 
"action": "Harmonization", 
"rawDoc": "1191448670-1"
}, 
{
"name": "MMIS", 
"user": "admin", 
"dateTime": "2018-07-09T13:23:34.647238Z", 
"action": "Harmonization", 
"rawDoc": "1191448670-2"
}, 
{
"name": "CURAM", 
"user": "admin", 
"dateTime": "2018-07-09T20:26:52.945146Z", 
"action": "Harmonization", 
"rawDoc": "/raw/curam/curamtestdata_21.xml"
}, 
{
"name": "MMIS", 
"user": "admin", 
"dateTime": "2018-07-09T13:23:34.647238Z", 
"action": "Harmonization", 
"rawDoc": "1191448670-3"
}
], 
"merges": [
{
"document-uri": "/person//raw/meds/medstestdata_21.xml.json"
}, 
{
"document-uri": "/person/1191448670-4.json"
}, 
{
"document-uri": "/person/1191448670-1.json"
}, 
{
"document-uri": "/person/1191448670-2.json"
}, 
{
"document-uri": "/person//raw/curam/curamtestdata_21.xml.json"
}, 
{
"document-uri": "/person/1191448670-3.json"
}
], 
"id": "ffcc223a-7479-4af0-9dfc-61797d8ebb84"
}, 
"triples": [], 
"instance": {
"MDM": {
"Person": {
"Person": {
"EligibilityHistory": {
"Eligibility": {
"DateEffectiveEligibility": "2014-12-31", 
"DateIneligible": "2015-12-31"
}
}
}
}
}
}
}

We've retired smart-mastering-demo, but there's still a link to it in the docs.

https://marklogic-community.github.io/smart-mastering-core/how-to-use/

has this link:

See the smart-mastering-demo for an example.

As a developer it would be helpful to call process:process-match-and-merge with a $commit option to specify whether the merged result and collection changes should be committed or not. This would allow more rapid testing of changes to match and merge options as you wouldn't have to reset the collections on your data before each test.

I will work on building this and try to submit a PR for it soon.

Configuration will control the permissions that get applied to documents at various times. Configuration will be part of the merge options.

  <algorithms xmlns="http://marklogic.com/smart-mastering/merging">
    <permissions >
      <on-merge function="union" at="/some/dir/code.xqy" ns="some-namespace"/>
      <on-archive function="no-change" at="/some/dir/code.xqy" ns="some-namespace"/>
      <on-no-match function="no-change" at="/some/dir/code.xqy" ns="some-namespace"/>
      <on-notification function="union" at="/some/dir/code.xqy" ns="some-namespace"/>
    </permissions >
  </algorithms>

The on-merge strategy will determine what permissions are applied to newly created merged documents. Default strategy: union of all permissions on source documents, plus $const:CONTENT-COLL. Comment if there's interest in having an intersection plus $const:CONTENT-COLL strategy available out of the box.

The on-archive strategy will determine what permissions are applied to documents that get archived (merged into other documents). Default strategy: no change to permissions.

The on-no-match strategy will determine what permissions are applied to documents passed to process:process-match-and-merge but do find any matches. Default strategy: no change to the document's permissions.

The on-notification strategy will determine what permissions are applied to newly created notification documents. Default strategy: notification documents will get the union of the source document permissions.

For each type of permission strategy, we'll define an API that can be used to make custom strategies.

See attachments for input record and match config.

A request to the REST match endpoint returned 0 results and this query:

  cts:and-query((
    cts:true-query(), 
    cts:boost-query(
      cts:and-query((
        cts:collection-query("mdm-content"), 
        cts:not-query(cts:document-query("/mmisDoc/CRM/Person/6986792174.xml-0-1"), 1), 
        cts:or-query(
          cts:element-value-query(fn:QName("","B834-SOCIAL-SECURITY-NUMBER"), "500000001", ("case-insensitive","lang=en"), 0), ())), ()),
        cts:or-query(cts:element-value-query(fn:QName("http://marklogic.com/data-hub/envelope","B834-SOCIAL-SECURITY-NUMBER"), "500000001", ("case-insensitive","lang=en"), 20), ())
      )
      ), 
    ()) 

The query was actually taken from logging $query at line 300 of match-impl.xqy (I don't have the JSON query representation from the REST response).

Note that the namespace is missing from the first element-value-query on B834-SOCIAL-SECURITY-NUMBER. Adding the proper namespace fixes the query, and we are pretty sure the missing namespace causes the failure to match in the REST call.

Team,

I am facing below mentioned issue while running "gradle prepDemo" from \examples\smart-mastering.

Task :deployMatchOptions FAILED

FAILURE: Build failed with an exception.

• What went wrong:
  Execution failed for task ':deployMatchOptions'.

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?

Is there any configuration changes needs to be done for this ?

Is there collateral/documentation that lays out what roles and methods need to be in place to implement the weighting values to give to potential customers so they know the resources necessary to implement?

The double-metaphone algorithm uses an element range index to populate the dictionary and constructs an element value query. This allows for pulling in values from sources other than the intended target, if other elements share the same QName. Using path range indexes would be more precise.

As a USER
I need to ensure the options files I have created are syntactically valid before I enter them
to help decrease errors and make troubleshooting bad match/merge options easier

This would ideally be a separate function that can be called by a user, but that would also be automatically invoked before saving any new match or merge options to prevent bad options from being persisted.

Merge option files shown in the documentation include a source-weights / sourceWeights property, however (unless missed) this property is not explained and what the name property refers too.

As a USER
I want the ability to troubleshoot why matching and merging isn't happening as I expect
and to do that, enabling a tracing mode that shows the values of all the inputs to matching and or/merging would be helpful (for example, everything that goes into the match-impl:find-document-matches-by-options() function for matching).

Two documents get merged, but only one document has a value for a property. In some cases, the document without a value for the property will still have the XML to hold the property. When happens, the auditing code sees that both have the property, and so it attributes the one value to both.

Solution: when considering attribution, check whether there is actually any content in the property XML.

I get the following error when passing a set of JSON options to the sm-match endpoint regardless whether I pass an rs:uri param or a document property in the body to match against. Works as expected when using the rs:options param.

Error:

{
    "errorResponse": {
        "statusCode": 500,
        "status": "Internal Server Error",
        "messageCode": "INTERNAL ERROR",
        "message": "XDMP-AS: (err:XPTY0004) $options as element(matcher:options) -- Invalid coercion: object-node{\"options\":object-node{\"propertyDefs\":object-node{\"property\":array-node{...}}, ...}} as element(matcher:options) . See the MarkLogic server error log for further detail."
    }
}

Postman request:

POST /v1/resources/sm-match?rs:uri=/CSV_FILE/Person/199.xml HTTP/1.1
Host: localhost:8011
Content-Type: application/json
Authorization: Digest username="admin", realm="public", nonce="", uri="/v1/resources/sm-match?rs:uri=/CSV_FILE/Person/199.xml", algorithm="MD5", response="70623c0eefe70ddb44108f31eeb0e458"
Cache-Control: no-cache
Postman-Token: c380c9d8-850a-4b3c-afb4-1d70eeb39221

{
  "options": {
    "propertyDefs": {
      "property": [
        { "namespace": "", "localname": "IdentificationID", "name": "ssn" },
        { "namespace": "", "localname": "PersonGivenName", "name": "first-name" },
        { "namespace": "", "localname": "PersonSurName", "name": "last-name" },
        { "namespace": "", "localname": "AddressPrivateMailboxText", "name": "addr1" },
        { "namespace": "", "localname": "LocationCity", "name": "city" },
        { "namespace": "", "localname": "LocationState", "name": "state" },
        { "namespace": "", "localname": "LocationPostalCode", "name": "zip" }
      ]
    },
    "algorithms": {
      "algorithm": [
        { "name": "std-reduce", "function": "standard-reduction" },
        { "name": "dbl-metaphone", "function": "double-metaphone" },
        { "name": "thesaurus", "function": "thesaurus" }
      ]
    },
    "scoring": {
      "add": [
        { "propertyName": "ssn", "weight": "50" },
        { "propertyName": "last-name", "weight": "8" },
        { "propertyName": "first-name", "weight": "6" },
        { "propertyName": "addr1", "weight": "5" },
        { "propertyName": "city", "weight": "3" },
        { "propertyName": "state", "weight": "1" },
        { "propertyName": "zip", "weight": "3" }
      ],
      "expand": [
        {
          "propertyName": "first-name",
          "algorithmRef": "thesaurus",
          "weight": "6",
          "thesaurus": "/mdm/config/thesauri/first-name-synonyms.xml",
          "distanceThreshold": "50"
        },
        {
          "propertyName": "last-name",
          "algorithmRef": "dbl-metaphone",
          "weight": "8",
          "dictionary": "name-dictionary.xml"
        }
      ],
      "reduce": [
        {
          "algorithmRef": "std-reduce",
          "weight": "4",
          "allMatch": { "property": ["last-name", "addr1"] }
        }
      ]
    },
    "actions": {
      "action": {
        "name": "my-custom-action",
        "function": "custom-action",
        "namespace": "http://marklogic.com/smart-mastering/action",
        "at": "/custom-action.xqy"
      }
    },
    "thresholds": {
      "threshold": [
        { "above": "30", "label": "Possible Match" },
        { "above": "50", "label": "Likely Match", "action": "notify" },
        { "above": "68", "label": "Definitive Match", "action": "merge" },
        { "above": "75", "label": "Custom Match", "action": "my-custom-action" }
      ]
    },
    "tuning": { "maxScan": "200" }
  }
}

When running match functions like findDocumentMatchesByOptionsName, the results indicate what documents matched, what threshold those matches hit, and what action should be taken with them. It does not run those actions, but provides enough information that the caller can run them. When calling processMatchAndMerge, the actions are run automatically.
Clarify the documentation to make it clear that the match functions are a read-only activity.

On the "Smart Mastering with Libraries" page of the documentation available here: https://marklogic-community.github.io/smart-mastering-core/docs/libraries/

If on the "Matching" (first) paragraph, you click on see "Match Results" link, you get the following error:

404
Page not found :(
The requested page could not be found.

URI of a merged json document shows .xml extension in mdm-merged collection.

When using the rs:includeMatches param on the sm-match endpoint, I get the following error:

{
"errorResponse": {
"statusCode": 500,
"status": "Internal Server Error",
"messageCode": "INTERNAL ERROR",
"message": "XDMP-AS: (err:XPTY0004) $include-matches as xs:boolean -- Invalid coercion: "true" as xs:boolean . See the MarkLogic server error log for further detail."
}
}

The timestamp information in the merge-impl:get-sources function isn't returned correctly for JSON documents, as the ns:map is always populated, and (I'm guessing) the following xdmp:unpath call tries to grab a path with a namespace that doesn't exist on the JSON doc.

One option I tried was checking for the existence of any attributes on line 1066 (if fn:exists($ns-path/@*), which seemed to work. If so, the documentation should be updated to inform users not to define any namespace attributes on the $ns-path element when using JSON.

The MDMImport input flow output collections are capitalized, like MDM-CONTENT, which makes the Harmonize flow fail. I think the collection name should be in lowercase like they are in the /com.marklogic.smart-mastering/constants.xqy

In matcher.xqy, we have

• matcher:get-option-names-as-xml()
• matcher:get-option-names-as-json()
• matcher:get-options-as-xml($options-name as xs:string)
• matcher:get-options-as-json($options-name as xs:string)

In merging.xqy, we have

• merging:get-option-names($format as xs:string)
• merging:get-options($options-name, $format as xs:string)

Change the matcher functions to use the same convention as merging. Mark the existing functions as deprecated for a couple releases, then remove.

In order to make filter queries easier to use, allow them to be added to the match options. There would be a new filter-query element:

<filter-query xmlns="http://marklogic.com/smart-mastering/matcher">
  <collection>my-collection</collection>
  <directory>my-directory</directory>
</filter-query>

The collection and directory elements are short-cuts for defining cts:collection-query and cts:directory-query. Whatever children are added to filter-query will be combined using AND semantics (in other words, added to a cts:and-query).

A directory element may take a depth attribute, with values passed to the $depth parameter of cts:directory-query.

For more complex queries, filter-query may have a serialized cts query as a child element.

If a filter-query is provided in the match options and the $filter-query parameter is used when calling a match or match-and-merge function, the function parameter will override the configured query.

Matching options (https://marklogic-community.github.io/smart-mastering-core/docs/matching-options/) list the ability to use a thesaurus algorithm but there is not detail documentation regarding thesaurus document "actions" : should it be referenced using thsr:load, should it be just ingested in a specific collection, etc.

See: https://marklogic-community.github.io/smart-mastering-core/docs/merging-options/

The namespace on the merge options sample is incorrect. It should instead be: "http://marklogic.com/smart-mastering/merging"

Should be able to follow the same approach as ml-unit-test, where a client project can use it like this:

mlRestApi "com.marklogic:smart-mastering-code:(version)"

And not have to do anything else.

Also, as part of publishing the project, jcenter requires there to be a "sources" jar, which is basically the same as the regular jar/zip - that's the approach I'm using for ml-unit-test.

I'll get a PR together for this soon.

Running the following code (see below) one would expect to all saved merge options but it looks like it is always returning one (first) entry even if multiple merge options have been saved.

let constSmartMastering = require('/com.marklogic.smart-mastering/constants.xqy');
let mergeSmartMastering = require('/com.marklogic.smart-mastering/merging.xqy');

mergeSmartMastering.getOptionNames(constSmartMastering.FORMATJSON)

It seems that the /sm-match endpoint only returns one result, instead of all the results

Note that the following response says there is a total of 8, but only shows a single result. Also, the am-match endpoint used to include matches inside each result. I would have expected to see that here on PersonGivenName, but no matches are included.

{
  "results": [
    {
      "boost-query": {
        "or-query": {
          "queries": [
            {
              "element-value-query": {
                "element": [
                  "PersonGivenName"
                ],
                "option": [
                  "case-insensitive"
                ],
                "text": [
                  {
                    "_value": "maurise",
                    "lang": "en"
                  }
                ],
                "weight": 6
              }
            },
            {
              "element-value-query": {
                "element": [
                  "PersonGivenName"
                ],
                "option": [
                  "case-insensitive"
                ],
                "text": [
                  {
                    "_value": "Maurie",
                    "lang": "en"
                  },
                  {
                    "_value": "MURIAL",
                    "lang": "en"
                  },
                  {
                    "_value": "KRIS",
                    "lang": "en"
                  }
                ],
                "weight": 6
              }
            }
          ]
        }
      },
      "match-query": {
        "and-query": {
          "queries": [
            {
              "collection-query": {
                "uri": "mdm-content"
              }
            },
            {
              "or-query": {
                "queries": [
                  {
                    "element-value-query": {
                      "element": [
                        "PersonGivenName"
                      ],
                      "option": [
                        "case-insensitive"
                      ],
                      "text": [
                        {
                          "_value": "maurise",
                          "lang": "en"
                        }
                      ],
                      "weight": 0
                    }
                  },
                  {
                    "element-value-query": {
                      "element": [
                        "PersonGivenName"
                      ],
                      "option": [
                        "case-insensitive"
                      ],
                      "text": [
                        {
                          "_value": "Maurie",
                          "lang": "en"
                        },
                        {
                          "_value": "MURIAL",
                          "lang": "en"
                        },
                        {
                          "_value": "KRIS",
                          "lang": "en"
                        }
                      ],
                      "weight": 0
                    }
                  }
                ]
              }
            }
          ]
        }
      },
      "page-length": "200",
      "result": {
        "action": "",
        "index": "8",
        "score": "6",
        "threshold": "Possible Match",
        "uri": "/person/1234068650-2.json"
      },
      "start": "1",
      "total": "8"
    }
  ]
}

I am about to file a PR for this.

We discovered that the sm-history-properties was returning empty-string keys for certain fields, when the merged field was were the result of simple JSON fields.

For example, it would return:

{
  ...,
  "PersonGender": {"": { ... } }
  ...
}

if the field in the document was simple:

{
  envelope: {
    headers: { ... },
    instance: {
      ...,
      "PersonGender": "F"
    }
  }
}

The desired result would have been:

{
  ...,
  "PersonGender": {"F": { ... } }
  ...
}

It did work for JSON fields where the value was an object.

Although smart-mastering handles XML and JSON content, most examples use XML & XQuery implementations. It would be nice to have examples also with JSON content and SJS implementation.

these two lines should coerce to node() instead of object-node() because they generate the following error:

matcher:results-to-json($results) -- Invalid coercion: document{object-node{"results":array-node{object-node{"total":text{"8"}, "page-length":text{"200"}, ...}}}} as object-node()
https://github.com/marklogic-community/smart-mastering-core/blob/develop/src/main/ml-modules/root/com.marklogic.smart-mastering/matcher-impl/matcher-impl.xqy#L415

and

https://github.com/marklogic-community/smart-mastering-core/blob/develop/src/main/ml-modules/root/com.marklogic.smart-mastering/matcher.xqy#L208