opengeometadata / geocombine Goto Github PK
View Code? Open in Web Editor NEWA Ruby toolkit for managing geospatial metadata
Home Page: https://github.com/OpenGeoMetadata/GeoCombine
License: Other
A Ruby toolkit for managing geospatial metadata
Home Page: https://github.com/OpenGeoMetadata/GeoCombine
License: Other
Some OpenGeoMetadata repositories contain out-of-date records and have no current contact person, so they will be archived on GitHub to indicate they're no longer in use.
We want GeoCombine to avoid harvesting these repositories entirely, since the records should no longer be used.
Maybe this is by design... but I'm not seeing several OGM repos when I clone all repositories:
ewlarson@beanburrito .internal_test_app % bundle exec rake geocombine:clone --trace
** Invoke geocombine:clone (first_time)
** Execute geocombine:clone
Cloned https://github.com/OpenGeoMetadata/shared-repository.git
Cloned https://github.com/OpenGeoMetadata/edu.stanford.purl.git
Cloned https://github.com/OpenGeoMetadata/edu.princeton.arks.git
Cloned https://github.com/OpenGeoMetadata/edu.virginia.git
Cloned https://github.com/OpenGeoMetadata/edu.nyu.git
Cloned https://github.com/OpenGeoMetadata/edu.harvard.git
Cloned https://github.com/OpenGeoMetadata/edu.umn.git
Cloned https://github.com/OpenGeoMetadata/edu.tufts.git
Cloned https://github.com/OpenGeoMetadata/edu.columbia.git
Cloned https://github.com/OpenGeoMetadata/edu.lclark.git
Cloned https://github.com/OpenGeoMetadata/gov.data.git
Cloned https://github.com/OpenGeoMetadata/geobtaa.git
Cloned https://github.com/OpenGeoMetadata/edu.uarizona.git
Cloned https://github.com/OpenGeoMetadata/edu.berkeley.git
Cloned https://github.com/OpenGeoMetadata/edu.cornell.git
Cloned https://github.com/OpenGeoMetadata/edu.vt.git
Cloned https://github.com/OpenGeoMetadata/edu.upenn.git
Cloned https://github.com/OpenGeoMetadata/edu.mit.git
Cloned https://github.com/OpenGeoMetadata/ca.frdr.geodisy.git
Cloned https://github.com/OpenGeoMetadata/edu.wisc.git
Cloned 20 repositories
Perhaps all of the missing repositories are Big Ten affiliates?
@karenmajewicz @thatbudakguy -- Does the main geobtaa repo supplant these? or is the API call that lists all the repos missing these repos for whatever reason:
https://github.com/OpenGeoMetadata/GeoCombine/blob/main/lib/geo_combine/harvester.rb#L97-L103
We have OGP to GeoBlacklight schema code here https://github.com/geoblacklight/geoblacklight-schema/blob/master/tools/ogp/2_transform.rb that could be migrated piecemeal, and the OGP schema has the full FGDC metadata as a field, so we have a couple options for supporting the OGP schema for import into GeoCombine.
This would be a good thing to have as part of full aardvark support. See related OpenGeoMetadata/metadata-issues#3.
If we do this by updating the existing iso2gbl.xsl
, it might be a good chance to resolve some longstanding issues with it:
We should be using persistent http/s connections in the geocombine:index
rake task
This would be preferred for resolving geoblacklight/geoblacklight#722 and avoiding an approach like geoblacklight/geoblacklight@master...jrgriffiniii:issues-722-jrgriffiniii-geocombine-xslt please
Currently the ENVELOPE
generated for the solr_geom
field is invalid and throws an exception in Solr.
https://github.com/OpenGeoMetadata/GeoCombine/blob/master/lib/xslt/iso2geoBL.xsl#L465-L487
This should be in the format
ENVELOPE(minX, maxX, maxY, minY)
We should probably move away from just puts
ing everything so that output can be controlled in a more granular way.
Hello GeoCombine Dev
Sitting at the Geo4LibCamp 2017 working through a tutorial on getting bulk data from geocombine, I ran into the issue above.
gem geo_combine
to my Gemfilebundle install
bundle exec rake geocombine:index
and hit the Don't know how to build task 'geocombine:index'
error.be rake -T| grep geo_combine
be rake -T| grep geocombine
OctoKit offers a nice ruby wrapper around the GitHub API, which would let us handle things like pagination, filtered queries for repos, etc. with a little more flexibility. And ideally if the API changed, all we would need to do is update the gem.
Running the iso2geoBL.xsl
transform on metadata from GeoNetwork produces a Solr doc that is rejected with the following response:
RSolr::Error::Http - 400 Bad Request
Error: {'responseHeader'=>{'status'=>400,'QTime'=>1},'error'=>{'msg'=>'Invalid Date String:\'2010-11-08\'','code'=>400}}
This appears to be in the layer_modified_dt
field. I believe this requires a full timestamp.
https://github.com/geoblacklight/geoblacklight-schema/blob/master/conf/schema.xml#L72
https://cwiki.apache.org/confluence/display/solr/Working+with+Dates
May need to update https://github.com/OpenGeoMetadata/GeoCombine/blob/master/lib/xslt/iso2geoBL.xsl#L218-L229
So the fallback can generate a fake timestamp (maybe with xs:time
?).
Rename the 'master' branch to 'main'.
When a record fails to index, the cause could be one of a number of things:
In all of these cases, it'd be better to log the failure and continue on โ currently indexing just stops. In some of them, it might also be useful to retry the request (possibly after a delay) using faraday-retry
or similar.
Some GeoBlacklight instances expect harvester requests to include a format=json
parameter. This is added as a default param to the first request:
However, subsequent requests are taken from the next
link of the previous document and the parameter has been removed:
https://earthworks.stanford.edu/?f%5Baccess%5D%5B%5D=public&f%5Bdct_provenance_s%5D%5B%5D=MassGIS&format=json&per_page=100&q=%2A&page=1
This breaks harvesting functionality on some sites.
Ensure that this parameter is appended to all requests.
related to #105.
for folks not using aardvark, we want to support the possibility of harvesting scoped to earlier schema versions. the "simple" way to do this is just e.g. looking for /aardvark/
in the path and asking folks to put their records in an "aardvark" folder, but ideally we would do it the "smart" way and actually look at the schema version in the record at the point we see the record. if this isn't possible for performance reasons, we can fall back to the "simple" strategy.
Currently, OGM allows for two styles of naming. One is to have all of your metadata in folders (usually some sort of pair tree), and the file itself is just named geoblacklight.json (see Stanford). However, a fair amount of people don't use this naming convention and instead just have a few directories of files with .json (see Harvard).
This involves allowing any '*.json', but excluding explicitly layers.json (which is a standard file under the Stanford model)
Move everything to GitHub Actions. Here's an example setup from GeoBlacklight:
https://github.com/geoblacklight/geoblacklight/blob/db24a6e19472190c093721d5aa2762b61fee55c3/.github/workflows/ruby.yml
We probably only need one ruby version, though.
GeoCombine::GeoBlacklightIndexer
should read GeoCombine::GeoBlacklightHarvester
task :index, [:site] => [:environment] do |_t, args|
Should be implemented as a GeoBlacklight md enhancement
Pseudo code:
if !'layer_geom_type_s'
search dc_subject for terms containing "point", "polygon" etc
end
So I found this XSL which actually seems to do a decent semantic job of going to HTML for both FGDC and HTML.
See two sample conversion here:
https://gist.github.com/mejackreed/2f9d78df58c61bf5c392
For ISO md I would like some better dt's but its not that terrible
Thoughts @gravesm , @drh-stanford , @eliotjordan , @kimdurante ??
When converting from ISO/FGDC into GeoBlacklight, all of the solr_
Solr schema fields should be auto-generated from their authorative elements.
See the "Implementation" section here for the list of Solr fields:
http://journal.code4lib.org/articles/9710
In the latest version of GeoBlacklight, we've cut the fields down to solr_geom
, solr_year_i
, and solr_issued_dt
I've been playing around with GeoCombine for Aardvark metadata harvesting.
Harvesting individual institutions has been working well:
bundle exec rake geocombine:clone\[edu.umn\]
=> 5480 docs
But cloning all the repos, fails:
ewlarson@beanburrito GeoDiscovery % bundle exec rake geocombine:clone
rake aborted!
SystemStackError: stack level too deep
/Users/ewlarson/.rbenv/versions/3.2.1/bin/bundle:25:in `load'
/Users/ewlarson/.rbenv/versions/3.2.1/bin/bundle:25:in `<main>'
Tasks: TOP => geocombine:clone
(See full trace by running task with --trace)
ewlarson@beanburrito GeoDiscovery % cd tmp/opengeometadata
ewlarson@beanburrito opengeometadata % ls -la
total 0
drwxr-xr-x 10 ewlarson staff 320 Mar 9 08:19 .
drwxr-xr-x 15 ewlarson staff 480 Mar 9 08:18 ..
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:19 edu.harvard
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:19 edu.nyu
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:19 edu.princeton.arks
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:18 edu.stanford.purl
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:19 edu.tufts
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:19 edu.umn
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:19 edu.virginia
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:18 shared-repository
ewlarson@beanburrito GeoDiscovery % bundle exec rake geocombine:clone
rake aborted!
SystemStackError: stack level too deep
/Users/ewlarson/.rbenv/versions/3.2.1/bin/bundle:25:in `load'
/Users/ewlarson/.rbenv/versions/3.2.1/bin/bundle:25:in `<main>'
Tasks: TOP => geocombine:clone
(See full trace by running task with --trace)
ewlarson@beanburrito GeoDiscovery % cd tmp/opengeometadata
ewlarson@beanburrito opengeometadata % ls -la
total 0
drwxr-xr-x 10 ewlarson staff 320 Mar 9 08:43 .
drwxr-xr-x 15 ewlarson staff 480 Mar 9 08:41 ..
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:43 edu.harvard
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:43 edu.nyu
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:42 edu.princeton.arks
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:41 edu.stanford.purl
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:43 edu.tufts
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:43 edu.umn
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:43 edu.virginia
drwxr-xr-x 3 ewlarson staff 96 Mar 9 08:41 shared-repository
Can anyone else confirm? Seems to likely StackError the same place each clone run...
denylist is more inclusive phrasing
GeoCombine/lib/geo_combine/ogp.rb
Line 210 in b0053ae
Trying to account for all the possibilities of the FGDC is a losing battle. At least, I've made my retreat, and we've been using a simplified XSL on our site.
Rather than relying on custom code for each element of the FGDC metadata structure, our stylesheet preserves the nesting structure of the XML elements, labeling each with the human-readable label found in a separate "fgdc_labels.xml" file. The result is a much-simplified XSL.
If you want to take a look, I cleaned out our institution-specific stuff, and put a copy in this gist:
https://gist.github.com/kgjenkins/fa8b46d619f1dd1d90befaadd1e793c3
What would you think about using an approach like that?
Ingest ISO19139 or FGDC and validate them.
Enable to_geoblacklight
and validate
for ISO19139 and FGDC
# Input an ISO 19139 record
iso_record = GeoCombine::Record.new(iso19139_metadata)
# Validate the record
iso_record.validate # returns true
# Convert the record
iso_record.to_geoblacklight # Converts a record to GeoBlacklight-Schema
# Input a FGDC record
fgdc_record = GeoCombine::Record.new(fgdc_metadata)
#Validate the record
fgdc_record.validate # returns true
# Convert the record
fgdc_record.to_geoblacklight # Returns an object with formatted fields, could do to_json or to_xml
I've installed the "full" stack using the docker images provided here running in a single Amazon EC2 instance. I've found the address of the container running solr (listed in the geoblacklight container's hosts file) but don't know how to tell geocombine that I'm not running solr locally at 127.0.0.1:8983, and provide a different location instead. I can see that "geo_combine.rake" specifies 127.0.0.1, but I don't have the file locally after following the install and can't change the IP.
Apologies if the question is overly simplistic; I have little experience with ruby or docker & am just trying to get a feel for how the pieces fit together. Any guidance would be appreciated.
currently, the CLI is only calling out to rake tasks without much logic. can we just use the rake tasks directly? then we won't have to maintain the CLI code, or do we want to migrate the rake logic into the CLI class?
Currently, the logic for indexing when no specific repo is specified is to pull all repositories with edu., uk., or org.* and anything in the allowlist (currently only big10). If orgs have made their repositories and didn't follow this, they are being excluded. The proposal is to either
Since GeoBlacklight v2.0.0, and the introduction of our bbox overlap ratio feature, Solr will automatically generate fields and values for these four coordinates:
These fields are returned during the GeoBlacklightHarvester's harvesting. But, you cannot pass these fields/values back into Solr during the index, without seeing Solr errors like:
RSolr::Error::Http (RSolr::Error::Http - 400 Bad Request)
Error: {
"responseHeader":{
"status":400,
"QTime":869},
"error":{
"metadata":[
"error-class","org.apache.solr.common.SolrException",
"root-error-class","java.lang.IllegalArgumentException"],
"msg":"Exception writing document id stanford-cg357zz0321 to the index; possible analysis error: DocValuesField \"solr_bboxtype__minX\" appears more than once in this document (only one value is allowed per field)",
"code":400}}
The fix is simple. We just need to delete these fields via the document_transformer.
def document_transformer
@document_transformer || ->(document) do
document.delete('_version_')
document.delete('score')
document.delete('timestamp')
document.delete('solr_bboxtype__minX')
document.delete('solr_bboxtype__minY')
document.delete('solr_bboxtype__maxX')
document.delete('solr_bboxtype__maxY')
document
end
end
SCHEMA_FIELD_MAP
in the migrator class)Identify Aardvark implications for GeoCombine.
geocombine:index
rake task has a hard-coded layer_id_s
field statement. That field is changed to gbl_wxsIdentifier_s
in Aardvark.) (addressed by f05d344)layer_slug_s
and dc_source_s
is present in spec/lib/geo_blacklight_harvester_spec.rb
)As the iso2geoBL.xsl
file stands, it generates the Solr doc content, but the default update handler (at least in the version of Solr) expects the same add
and doc
as the XML.
{
"add": {
"doc": {
"uuid": ""
...
}
}
}
The current behavior is to download all OGM repositories that aren't on the configured denylist:
GeoCombine/lib/geo_combine/harvester.rb
Lines 12 to 22 in 7a42aaa
Not sure if other folks do it the same way, but at Stanford we have code to instead only harvest repositories on an allowlist. This avoids accidentally harvesting our own metadata from OGM (and possibly duplicating it) and also ensures that adding new institutional metadata is an intentional process via pull request.
It would be nice to have GeoCombine support this behavior without any additional logic.
FGDC XML to Aardvark schema json transformation to be used with GeoCombine
Draft version 1 is available on GeoCombine branch:
https://github.com/OpenGeoMetadata/GeoCombine/blob/draft_fgdc2Aardvark/lib/xslt/fgdc2Aardvark_draft_v1.xsl
Notes on further enhancement needs/decision points embedded in comments in xsl.
The iso2geoBL.xsl transforms the publisher field to the multi-valued element dc_publisher_sm.
However, the official GBL Metadata Schema indicates the singular dc_publisher_s.
Since the Big Ten project uses both the XSLT with GIS records, and NYU's plugin for Omeka with maps, we discovered that we had a mix of dc_publisher_s and dc_publisher_sm in our Solr. We would prefer to use dc_publisher_sm. Any reason why we wouldn't want to enable having multiple publishers?
Add rubocop as a development dependency
This speeds up the indexing process considerably, and is not super difficult using faraday's net_http_persistent
adapter.
JSON output generated by iso2geoBL.xsl
is invalid as there is mixed XML content.
Right now in the geocombine:clone
task, git clone
works using git://
protocol URLs. It should support https://
for secure connections and for firewalls that block git protocol traffic.
Most people are using GeoCombine in the context of GeoBlacklight; if it happens to be running in an environment where Blacklight.default_index
is defined, we can just use that connection instead of asking for SOLR_URL
or similar.
allowlist is more inclusive phrasing.
GeoCombine/lib/tasks/geo_combine.rake
Line 13 in b0053ae
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.