| 
A Ruby toolkit for managing geospatial metadata, including:
- tasks for cloning, updating, and indexing OpenGeoMetadata metadata
- library for converting metadata between standards
Add this line to your application's Gemfile:
gem 'geo_combine'And then execute:
$ bundle installOr install it yourself as:
$ gem install geo_combine# Create a new ISO19139 object
> iso_metadata = GeoCombine::Iso19139.new('./tmp/opengeometadata/edu.stanford.purl/bb/338/jh/0716/iso19139.xml')
# Convert ISO to GeoBlacklight
> iso_metadata.to_geoblacklight
# Convert that to JSON
> iso_metadata.to_geoblacklight.to_json
# Convert ISO (or FGDC) to HTML
> iso_metadata.to_htmlYou can use the GeoCombine::Migrators to migrate metadata from one schema to another.
Currently, the only migrator is GeoCombine::Migrators::V1AardvarkMigrator which migrates from the GeoBlacklight v1 schema to the Aardvark schema
# Load a record in geoblacklight v1 schema
record = JSON.parse(File.read('.spec/fixtures/docs/full_geoblacklight.json'))
# Migrate it to Aardvark schema
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record).runSome fields cannot be migrated automatically. To handle the migration of collection names to IDs when migrating from v1 to Aardvark, you can provide a mapping of collection names to IDs to the migrator:
# You can store this mapping as a JSON or CSV file and load it into a hash
id_map = {
'My Collection 1' => 'institution:my-collection-1',
'My Collection 2' => 'institution:my-collection-2'
}
GeoCombine::Migrators::V1AardvarkMigrator.new(v1_hash: record, collection_id_map: id_map).runSome of the tools and scripts in this gem use Ruby's Logger class to print information to $stderr. By default, the log level is set to Logger::INFO. For more verbose information, you can set the LOG_LEVEL environment variable to DEBUG:
$ LOG_LEVEL=DEBUG bundle exec rake geocombine:clone$ bundle exec rake geocombine:cloneWill clone all edu.*, org.*, and uk.* OpenGeoMetadata repositories into ./tmp/opengeometadata. Location of the OpenGeoMetadata repositories can be configured using the OGM_PATH environment variable.
$ OGM_PATH='my/custom/location' bundle exec rake geocombine:cloneYou can also specify a single repository:
$ bundle exec rake geocombine:clone[edu.stanford.purl]Note: If you are using zsh, you will need to use escape characters in front of the brackets:
$ bundle exec rake geocombine:clone\[edu.stanford.purl\]$ bundle exec rake geocombine:pullRuns git pull origin master on all cloned repositories in ./tmp/opengeometadata (or custom path with configured environment variable OGM_PATH).
You can also specify a single repository:
$ bundle exec rake geocombine:pull[edu.stanford.purl]Note: If you are using zsh, you will need to use escape characters in front of the brackets:
$ bundle exec rake geocombine:pull\[edu.stanford.purl\]To index into Solr, GeoCombine requires a Solr instance that is running the GeoBlacklight schema:
$ bundle exec rake geocombine:indexIf Blacklight is installed in the ruby environment and a solr index is configured, the rake task will use the solr index configured in the Blacklight application (this is the case when invoking GeoCombine from your GeoBlacklight installation). If Blacklight is unavailable, the rake task will try to find a Solr instance running at http://localhost:8983/solr/blacklight-core.
You can also set a the Solr instance URL using SOLR_URL:
$ SOLR_URL=http://www.example.com:1234/solr/collection bundle exec rake geocombine:indexGeoCombine provides a Harvester class and rake task to harvest and index content from GeoBlacklight sites (or any site that follows the Blacklight API format). Given that the configurations can change from consumer to consumer and site to site, the class provides a relatively simple configuration API. This can be configured in an initializer, a wrapping rake task, or any other ruby context where the rake task our class would be invoked.
bundle exec rake geocombine:geoblacklight_harvester:index[YOUR_CONFIGURED_SITE_KEY]Only the sites themselves are required to be configured but there are various configuration options that can (optionally) be supplied to modify the harvester's behavior.
GeoCombine::GeoBlacklightHarvester.configure do
{
commit_within: '10000',
crawl_delay: 1, # All sites
debug: true,
SITE1: {
crawl_delay: 2, # SITE1 only
host: 'https://geoblacklight.example.edu',
params: {
f: {
dct_provenance_s: ['Institution']
}
}
},
SITE2: {
host: 'https://geoportal.example.edu',
params: {
q: '*'
}
}
}
endCrawl delays can be configured (in seconds) either globally for all sites or on a per-site basis. This will cause a delay for that number of seconds between each search results page (note that Blacklight 7 necessitates a lot of requests per results page and this only causes the delay per page of results)
Solr's commitWithin option can be configured (in milliseconds) by passing a value under the commit_within key.
You may need to transform documents that are harvested for various purposes (removing fields, adding fields, omitting a document all together, etc). You can configure some ruby code (a proc) that will take the document in, transform it, and return the transformed document. By default the indexer will remove the score, timestamp, and _version_ fields from the documents harvested. If you provide your own transformer, you'll likely want to remove these fields in addition to the other transformations you provide.
GeoCombine::GeoBlacklightIndexer.document_transformer = -> (document) do
# Removes "bogus_field" from the content we're harvesting
# in addition to some other solr fields we don't want
%w[_version_ score timestamp bogus_field].each do |field|
document.delete(field)
end
document
endTo run the tests, use:
$ bundle exec rake spec- Fork it ( https://github.com/[my-github-username]/GeoCombine/fork )
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create a new Pull Request
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent