DESCRIBE using Shapes about sparql-dev HOT 29 OPEN

The interoperability problems could be lessened by adding an argument to DESCRIBE that requests a particular triple boundary for the results:

DESCRIBE ?person AS ex:SCBD WHERE { ... }

This asks for the description of persons, as a symmetric concise bounded description (SCBD). The specific IRIs that are available, and their meaning, would not be defined as part of SPARQL, but be left to each service implementation. This would make the existing extension point—describe handlers—more interoperable by allowing clients to request a particular describe handler.

One would hope that the community agrees on IRIs for the most common options, like CBD and SCBD. If the IRI is a SHACL shape, then services could treat that as a request for triples covered by the shape. Services could also offer things like ex:tiny, ex:short, ex:full.

from sparql-dev.

At TopQuadrant we are increasingly using SHACL for such use cases. A typical SHACL shape would define sh:property shapes that declare which predicates to walk (from the start resource) and use sh:node to link to other shapes that shall be walked recursively to collect all relevant triples. It's a straight-forward algorithm, although the handling of complex sh:path (SPARQL 1.1) expressions adds some complexity, if that is needed. At each reached focus node, it could check the constraints defined at the current shape to exclude certain triples. I can elaborate if needed. A specific example would help to frame this discussion.

Related to this, we are also using SHACL to describe links to external resources. This article describes our new support to link to Wikidata resources (or other graphs with a SPARQL endpoint), including the ability to "infer" triples from wikidata to augment local resources.

https://www.topquadrant.com/technology/shacl/wikidata/

Finally, we are using SHACL to automatically define GraphQL schemas which can then be queried by client applications. The schema serves as a bounded description (also using sh:node and sh:class to walk into adjacent shapes). In contrast to DESCRIBE-like scenarios, GraphQL allows the client to choose which fields to retrieve. More on our GraphQL technology is at

https://www.topquadrant.com/technology/graphql/

from sparql-dev.

@namedgraph - I don't think we are anywhere near specs.

A general comment - the way forward for many of the themes in the issues list is to try for real, understand the value. We have a technology ecosystem; nothing is stopping people innovate with that. When starting, specs may need to lead or be near to the leading edge, but their role moved to being capturing experience.

from sparql-dev.

I think such a DESCRIBE ... AS ... feature could be relevant in connection with the "content negociation by profile" draft : https://www.w3.org/TR/2018/WD-dx-prof-conneg-20181218/

from sparql-dev.

Feel free to generate DESCRIBE queries from SHACL, just don't create a dependency on SHACL in SPARQL.
Specifications are supposed to be orthogonal. This is an implementation question really.

from sparql-dev.

Here are a couple flavors of @VladimirAlexiev's proposal:

1. Return all arcs in and out of the nodes reachable from a starting node/shape association. This is like a more constrained Concise Bounded Description.
2. Return only the triples described by the shapes involved in 1.

In an online example of 1, you can click [controls V] button, next to load data click ☐ slurp, and then [validate]. The data box will be replaced with the slurped triples (and comments describing their fetch time).

That gives you the entire neighborhood for every visited node but does not limit to the triples directly involved in validation. For that, you can walk through the TestedTriples in the validation proof at the bottom as does this code.

This approach has been used successfully in a clinical decision support system, as well as for downloading coherent wikidata items.

from sparql-dev.

from sparql-dev.

Unless you extend ShEx and SHACL with specific code for (real) CBDs, I think you have a choice between shapes and CBDs. IMO, that's a pretty reasonable choice to make. A shape would say "I want to crawl out from the query node as specified by this shape. A CBD says "I want you to crawl all arcs out (and maybe in), recursing until you can uniquely discriminate this node from any other node currently in the store."

The two seem like they have kinda different use cases.:

1. Shapes are useful when the data is well enough understood to be documented in a schema, This schema is likely to include unique identifiers that CBDs would have to discover by exhaustive search.
2. CBDs are more useful if you are doing faceted browsing and don't know much about the nodes coming back.

So if one just wants a cheap way to uniquely identify some node, e.g. for archiving or relaying to a user, I'd prefer shapes, and fall back to CBDs if no shape is known to identify the node. Of course, how the user finds such a shape is a question of metadata.

from sparql-dev.

One strategy that @tpluscode is currently working on is SHACL -> CONSTRUCT QUERY generation over in https://github.com/hypermedia-app/shape-to-query. The main limiting factor there is handling recursive paths such as that described here.

IMO its worth considering allowing CONSTRUCT WHERE queries with complex paths such as

CONSTRUCT WHERE {
   ?s ( foaf:knows | ex:aquantance )+ / foaf:name ?o .
}

where any triple used to form the path is included in the result-set of the construct query.

It keeps SHACL and SPARQL orthogonal, but also allows much more easily turn arbitrary SHACL constraints into construct queries.

from sparql-dev.

from sparql-dev.

supports sh:node at any level

We support that too. This is what I meant by walks through all the possible "paths" in the shapes : traversing all sh:node to reach all possible nodes. The difference I think is that we use VALUES with the property URIs because we need to fetch sometimes quite a lot of properties, while you have one UNION per property on each node.

from sparql-dev.

Found 3 new works:

• Shape Fragments
• Shacl Engine by @bergos
• Extracting CBD using Shape Templates by @pietercolpaert

Thanks for referencing my work! The last commits of today make the codebase actually functional. If a shape is defined, it’s going to create and process something I call a Shape Template, which looks like this:

class Shape {
    requiredPaths: Path[]; // Can trigger an HTTP request if not set
    optionalPaths: Path[]; // Also include the deeper down paths, if they are set
    nodelinks: NodeLink[]; // If this path is set, we need to re-do the algorithm on that named node with the shape linked in the nodelink.
    atLeastOneLists: [ Shape[] ] ; // Each atLeastOneList is an array of at least one shape with one or more required paths that must be set, or it will trigger an HTTP request.
}
class NodeLink {
    shape: Shape;
    path: Path;
}

It is thus going to ignore more specific cardinalities or specific literal contraints, it’s going to simplify all conditionals towards atLeastOneLists, etc. This is a design decision: we’re not fully validating the shape while extracting it - instead we are doing a best effort to describe the entity by taking hints from the shape.

What it does more than what is needed here: It also defined the triggers on which an HTTP request should be done to the current focus node. This was needed for the TREE hypermedia specification, where for some tree:Collections, you may decide to describe your entities out of band. Through the shape defined on the collection, the member extraction algorithm can this way provide the triggers for HTTP requests.

Motivated by use cases encountered in LDES -- where we may have multiple representation versions of the same entity in the same collection -- it also extracts all triples in the named graph of the to be described entity.

from sparql-dev.

That makes sense to me. I know that it's a useful feature, but I don't know that it's a useful SPARQL feature. I think the real use case for integration would be if there were some analog of the SERVICE query which allowed you to compose your subquery as shapes. (While we're at it, I also want a kayak integrated into my bicycle for when there are no convenient bridges.) Barring that, you could use a ShEx API URL in a SPARQL FROM and get the same effect, just without the readability.

from sparql-dev.

One other perspective to keep in mind: part of the challenge in using RDF is that the ecosystem is fragmented: we have to deal with multiple different languages and tools to get the job done. It would be really nice if we could (eventually) figure out a higher-level language that would enable the whole job to be expressed. I don't know what that should look like, but I encourage everyone to think and share ideas about how that might be achieved.

(I also half expect the JavaScript folks to yell "JavaScript" at this point, ;) and maybe they're right. But I'd be interested to hear other ideas as well. )

from sparql-dev.

from sparql-dev.

Removing "SPARQL: " on transferred issue.

from sparql-dev.

Interesting idea, indeed! I remember way back, when we tried to fully exploit the fact that we were using an open standard, we found the lack of DESCRIBE interoperability the largest obstacle to jump from one SPARQL engine to another, it was a pain to the extent that I proposed standardizing a default.

I'm not sure this use case is that prominent anymore, but it seems like a good solution, though I can see the point of not having a normative reference to shapes too.

from sparql-dev.

A great suggestion!

@cygri @ericprud @HolgerKnublauch can you write SHACL/ShEx shapes to formally capture CBD and SCBD?

from sparql-dev.

https://twitter.com/jindrichmynarz/status/1470109531040624645

from sparql-dev.

One strategy that @tpluscode is currently working on is SHACL -> CONSTRUCT QUERY generation over in https://github.com/hypermedia-app/shape-to-query.

We did implement such a query generation algorithm in SHACL Play : https://shacl-play.sparna.fr/play/sparql#documentation-1 - this takes into account a limited subset of SPARQL constraints; it walks through all the possible "paths" in the shapes and generates one UNION clause per path to fetch the metadata on one possible kind of node that can be reached from a starting point. The output queries look like

CONSTRUCT 
  { 
    ?s ?p ?o .
  }
WHERE
  { ?s  ?p  ?o
    { SELECT  ?this
      WHERE
        { ?this  ex:Country  <http://data.sparna.fr/country/FRANCE> }
    }
      { BIND(?this AS ?s)
        VALUES ?p { ex:streetAddress ex:firstName ex:language dcterms:creator ex:country ex:locality ex:lastName ex:postalCode }
      }
    UNION
      { ?this  ex:country  ?s . 
        VALUES ?p { ex:languageCountry ex:ContryName ex:CodeCountry }
      }
    UNION
      { ?this  dcterms:creator  ?s . 
        ?s  rdf:type  foaf:Person
        VALUES ?p { rdf:type rdfs:label }
      }
    UNION
      { ?this  dcterms:creator  ?s . 
        ?s  rdf:type  org:Organization
        VALUES ?p { rdf:type skos:prefLabel }
      }
  }

Query generation code is at https://github.com/sparna-git/shacl-play/tree/master/shacl-sparql-query-gen/src/main/java/fr/sparna/rdf/shacl/sparqlgen

Our use case was : given a large knowledge graph, how can we automate the generation of a dataset that is a subset of this large KG, based on a SHACL description of the content of this dataset ?

from sparql-dev.

@tfrancart the UNION looks similar but there are quite some differences:

• the CONSTRUCT patterns are not s/p/o but precisely reflect the input shape
• supports sh:node at any level

I do not yet support the constraints or targets but soon. And the end game is SHACL Advanced Features (most importantly sh:values with node expressions)

Here's an example of the latest version (WIP at https://visitor--shape-to-query.netlify.app/)

PREFIX hydra: <http://www.w3.org/ns/hydra/core#>
PREFIX schema: <http://schema.org/>
PREFIX sh: <http://www.w3.org/ns/shacl#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>

[
  a sh:NodeShape ;
  sh:property
    [
      sh:path
        [
          sh:alternativePath
            (
              hydra:title
              rdfs:label
            )
        ] ;
    ],
    [
      sh:path hydra:description ;
    ],
    [
      sh:path schema:mainEntity ;
      sh:node
        [
          sh:property
            [
              sh:path hydra:member ;
              sh:node
                [
                  sh:property
                    [
                      sh:path
                        [
                          sh:zeroOrMorePath
                            schema:parent
                        ] ;
                    ] ;
                ] ;
            ] ;
        ] ;
    ] ;
] .

PREFIX hydra: <http://www.w3.org/ns/hydra/core#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
PREFIX schema: <http://schema.org/>

CONSTRUCT {
  <http://example.com> hydra:title ?resource2 .
  <http://example.com> rdfs:label ?resource3 .
  <http://example.com> hydra:description ?resource4 .
  <http://example.com> schema:mainEntity ?resource5 .
  ?resource5 hydra:member ?resource6 .
  ?resource8 schema:parent ?resource7 .
}
WHERE {
  {
    {
      <http://example.com> hydra:title ?resource2 .
      BIND(?resource2 as ?resource1)
    } UNION {
      <http://example.com> rdfs:label ?resource3 .
      BIND(?resource3 as ?resource1)
    }
  } UNION {
    <http://example.com> hydra:description ?resource4 .
  } UNION {
    <http://example.com> schema:mainEntity ?resource5 .
  } UNION {
    <http://example.com> schema:mainEntity ?resource5 .
    ?resource5 hydra:member ?resource6 .
  } UNION {
    <http://example.com> schema:mainEntity ?resource5 .
    ?resource5 hydra:member ?resource6 .
    {
      BIND (?resource6 as ?resource7)
    } UNION {
      ?resource6 schema:parent* ?resource8 .
      ?resource8 schema:parent ?resource7 .
    }
  }
}

from sparql-dev.

Thank you for the explanation. I now understand better what is happening in your query.

UNION per property on each node.

And I'm a little anxious if that will not blow up, when there really is a ton of UNIONed graph patterns...

from sparql-dev.

And I'm a little anxious if that will not blow up, when there really is a ton of UNIONed graph patterns...

Yes, we have an option to generate multiple SPARQL queries instead of a single big one, just in case :-)
But the complexity of more UNION clauses should be linear.

from sparql-dev.

@tpluscode and @tfrancart Thanks a lot for your contribution!
The biggest sort-of "DESCRIBE" query I've written in my life is this:
https://vocab.getty.edu/doc/queries/#All_Data_For_Subject

It collects all triples from 13 nodes, using 13 wildcard patterns:

I wonder:

1. Does the shape of these UNIONs match what your tools are generating?
2. Whether this "business object" can be captured in SHACL and the above query generated from SHACL?

from sparql-dev.

1. Roughly yes, it matches. The difference is that in our approach we always bind the ?s variable instead of using a different subject variable in each union. And in the construct there is only CONSTRUCT { ?s ?p ?o } instead of different variables.
2. I think yes, maybe except for the ?p1 != gvp:narrowerExtended && ?p1 != skos:narrowerTransitive clause

from sparql-dev.

1. My approach is slightly different because I do not generate an ?s ?p ?o for any given focus node. The current implementation UNIONs only properties explicitly defined in the shapes at all nesting levels.
2. I agree with @tfrancart. There isn't anything which should prevent the capturing your data using SHACL although you may need to make concessions. That said, many complex requirements can be expressed using SHACL Advanced Features. I'm linking to the draft, which includes many additional features, some of which I have already been implementing

from sparql-dev.

My approach is slightly different because I do not generate an ?s ?p ?o for any given focus node. The current implementation UNIONs only properties explicitly defined in the shapes at all nesting levels.

This is what we do too : listing only properties explicitely defined in the shapes. The point is that the variables are always named ?s ?p ?o in every UNION clauses (combined with a VALUES ?p {...} to fetch only properties explicitely defined), so that the CONSTRUCT clause is simply CONSTRUCT { ?s ?p ?o }

from sparql-dev.

Found 3 new works:

• https://github.com/Shape-Fragments by @thomas-delva. Includes
  • spec at https://shape-fragments.github.io/shape-fragments-spec/ and https://github.com/Shape-Fragments/shape-fragments-spec
  • python implementation https://github.com/Shape-Fragments/pySHACL-fragments
  • https://github.com/Shape-Fragments/SHACL2SPARQL translator
  • a couple of papers: https://github.com/Shape-Fragments/full-provenanceshacl-paper, https://github.com/Shape-Fragments/old-shapefragments-paper
• https://www.bergnet.org/2023/03/2023/shacl-engine/ by @bergos
• https://github.com/pietercolpaert/extract-cbd-shape by @pietercolpaert

from sparql-dev.

About SHACL2SPARQL:
For example, the WATDIV test query F1 https://dsg.uwaterloo.ca/watdiv/basic-testing.shtml

SELECT ?v0 ?v2 ?v3 ?v4 ?v5 WHERE {
	?v0	og:tag	%v1% .
	?v0	rdf:type	?v2 .
	?v3	sorg:trailer	?v4 .
	?v3	sorg:keywords	?v5 .
	?v3	wsdbm:hasGenre	?v0 .
	?v3	rdf:type	wsdbm:ProductCategory2 .
}

is translated to https://github.com/Shape-Fragments/SHACL2SPARQL/blob/main/real_shacl_testfiles/watdiv/F1.ttl:

:F1shape a sh:NodeShape ;
    sh:targetSubjectsOf sorg:trailer ; #target v3 and get v4
    sh:property [
        sh:path sorg:keywords ;
        sh:minCount 1 ;
        ] ;
    sh:property [
        sh:path wsdbm:hasGenre ;
        sh:qualifiedMinCount 1 ;
        sh:qualifiedValueShape [
            sh:and (
                [
                    sh:path ogp:tag ;
                    sh:hasValue wsdbm:Topic241 ;
                    ]
                [
                    sh:path rdf:type ;
                    sh:minCount 1 ;
                    ]
                ) ;
            ] ;
        ] ;
    sh:property [
        sh:path rdf:type ;
        sh:hasValue wsdbm:ProductCategory2 ;
        ] .

from sparql-dev.