handling greedy full text indexes about network-of-terms HOT 7 OPEN

Note that the GTAA service layer also provides score information, so that results can be relevance ranked.

Thanks, @wmelder, that’s the hint that solved our issue: the GeoNames query wasn’t sorting results by relevance score yet. Fixed in #1066.

from network-of-terms.

it should expand the search query with an AND clause when multiple terms are used.

We already do so for ?virtuosoQuery; you can try using that in your SPARQL query from the NoT, although the backend seems to be Fuseki rather than Virtuoso.

We could also add a third query variable besides ?query and ?virtuosoQuery. Something like ?intersectionQuery, which joins all query words with AND but does not quoting like ?virtuosoQuery does.

from network-of-terms.

I think it would indeed be very useful to add to add another variable like that. Treating the phrase as "OR"ed words seems to be default behaviour of the Jena Full Text Search. We have a similar problem with the current Geonames implementation. Searching for "Sas van Gent" returns 195 results, all most all of them are irrelevant. Rewriting the query to "(sas AND gent AND van)" returns the two relevant ones, see https://api.triplydb.com/s/8DBt_82Qy.

Note: The GTAA also uses the Jena Full Text Search but does not have this problem, searching for Wim de Bie returns the correct results. Maybe we should approach @wmelder to asks for details on his config settings?

from network-of-terms.

Thanks for signalling this @EnnoMeijers. I agree, using an AND clause for multiple terms should preferrably be standard.

@ddeboer what would be the optimal solution in your opinion? Would adding ?virtuosoQuery to the queries work for all datasets?

from network-of-terms.

Sounds familiar, the problem of too many hits with full text search. The GTAA settings for the Lucene index here.
The service points to the text datastore instead of the triple datastore. The text datastore points to an index and the triple datastore.

@prefix fuseki:  <http://jena.apache.org/fuseki#> .
@prefix rdf:     <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs:    <http://www.w3.org/2000/01/rdf-schema#> .
@prefix tdb1:    <http://jena.hpl.hp.com/2008/tdb#> .
@prefix tdb2:    <http://jena.apache.org/2016/tdb#> .
@prefix text:    <http://jena.apache.org/text#> .
@prefix ja:      <http://jena.hpl.hp.com/2005/11/Assembler#> .
@prefix :        <#> .

[] rdf:type fuseki:Server .

<#service_public> rdf:type fuseki:Service ;
    fuseki:name "public" ;
    fuseki:label "Service Layer Triple Store (for public access)" ;
    fuseki:serviceQuery "query" , "sparql" ;
    fuseki:serviceReadGraphStore "get" ;
    fuseki:dataset <#text_public> ;
    fuseki:serviceUpdate "update" ;
    fuseki:serviceUpload "upload" ;
    fuseki:serviceReadWriteGraphStore "data" ;
.


<#text_public> rdf:type     text:TextDataset ;
    text:dataset   <#tdb_public> ;
    text:index     <#index_public> ;
.

<#index_public> rdf:type text:TextIndexLucene ;
    text:directory "/opt/fuseki/databases/public_text" ;
    text:entityMap <#entity_map_public> ;
    text:analyzer [
	rdf:type        text:ConfigurableAnalyzer ;
	text:tokenizer  text:StandardTokenizer ;
	text:filters    (text:ASCIIFoldingFilter text:LowerCaseFilter)
    ] ;
.

<#entity_map_public> rdf:type text:EntityMap ;
    text:defaultField     "prefLabel" ;
    text:entityField      "uri" ;
    text:uidField         "uid" ;
        text:map (
            [
	        text:field "prefLabel" ;
		text:predicate <http://www.w3.org/2004/02/skos/core#prefLabel>
	    ]
	    [
		text:field "altLabel" ;
		text:predicate <http://www.w3.org/2004/02/skos/core#altLabel>
	    ]
	    [
		text:field "hiddenLabel" ;
		text:predicate <http://www.w3.org/2004/02/skos/core#hiddenLabel>
	    ]
        ) ;
.

<#tdb_public> rdf:type tdb1:DatasetTDB ;
    tdb1:location "/opt/fuseki/databases/public" ;
    ja:context [ ja:cxtName "arq:queryTimeout" ;  ja:cxtValue "120000" ] ;
.

Note that the GTAA service layer also provides score information, so that results can be relevance ranked.

Please let me know if you need more information.

from network-of-terms.

Ah, thanks for sharing @wmelder! It looks quite similar to the Geonames configuration that we currently use. I can't explain the differences in behaviour for searching between these two, can you @ddeboer?

from network-of-terms.

Using the ?virtuosoQuery var instead of the regular ?query var in the sparql query seems to fix the problems with the jena full text search. Updated the geonames.rq accordingly, see #1065

from network-of-terms.