When doing:

$ catmandu convert Solr --url http://localhost:8984/solr --id_field myfield

I get records where the content of myfield is put into the _id field of the resulting records:

---
_id: archive.ugent.be:00003832-ADE7-11E0-92CA-80A237D8FA8C
accessRights:
- ugent
date: 2016-08-26T08:55:51.45Z

When doing:

$ catmandu export Solr --url http://localhost:8984/solr --id_field myfield

the _id has an undefined content:

---
_id: ~
accessRights:
- ugent
date: 2016-08-26T08:55:51.45Z

Fix for old versions of Solr (without cursorMark):

https://lucidworks.com/blog/2013/12/12/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/

Using the Perl API of Catmandu one can access data in a Solr installation that doesn't have the notion of 'bags':

  my $store = Catmandu->store('Solr');
  $store->bag->searcher(query => 'any solr query')->each(sub {
  });

This is probably a bug that the bag doesn't have any effect in this code. But it is also a very nice feature to be able to support "bagless" Solr installations.

However, with the command line tools the "bag" is enforced in the Solr query on line 33: https://github.com/LibreCat/Catmandu-Solr/blob/master/lib/Catmandu/Store/Solr/Bag.pm#L33

This gives no results in "bagless" installations

  $ catmandu export Solr to YAML

Can searcher keep the support for bagless queries. Can this support also be added to the generator?

E.g. it is an undocumented feature that extra parameters in the searcher are passed verbatim via the searcher to the backend Solr endpoint.

The current importer doesn't seem to support the bulk DataImportHandler method to add data to a Solr index in bulk. Pushing data record by record is a slow, error-prone process since it seems to re-trigger the indexing process each time a new record is pushed and committed to the index. The DataImportHandler method circumvents this.

We've implemented this method of indexing in the Datahub::Factory application (which is heavily based on the Catmandu architecture)

Would it be viable to reuse this code in this module as a separate importer?

See: https://github.com/thedatahub/Datahub-Factory/blob/master/lib/Datahub/Factory/Indexer/Solr.pm

The above module expects two inputs:

• The local location of the JSON file which contains data to be uploaded.
• The URL defined by the DataImportHandler in the Solr configuration.

Implementation looks like this:

my filename = "/tm/bulk.json"
my $requestHandler = "http://localhost:8983/solr/blacklight-core/update/json"
my $indexer = Datahub::Factory->indexer('Solr')->new(
    'file_name' = $filename,
    'request_handler' => $requestHandler
);
$indexer->import();
$indexer->commit();

Both methods will return the response of the handler API as a perl hash.
Both methods throw a Catmandu::HTTP:Error at the moment if something goes wrong.

This query throws an error

$ catmandu export Solr --id "archive.ugent.be:6B4577BE-ADCD-11E2-9873-C6B0A558A163" 
Can't use string ("archive.ugent.be:6B4577BE-ADCD-1"...) as a HASH ref while "strict refs" in use at /home/librecat/.plenv/versions/5.20.2/lib/perl5/site_perl/5.20.2/Catmandu/Store/Solr/Bag.pm line 254.

Shouldn't the $_ on line 254 be an $item?

https://github.com/LibreCat/Catmandu-Solr/blob/master/lib/Catmandu/Store/Solr/Bag.pm#L254

It seems that _build_id_key was incorrectly named in 2dc89b8 and should've been _build_bag_key.