Solr allows atomic indexing, which means only changed fields are updated.

This can add a lot of efficiency to the Solr Update that hooks in to DataObject save operations.

Apache Solr includes the ability to set up a cluster of Solr servers.
https://lucene.apache.org/solr/guide/6_6/solrcloud.html

It includes a handy Collections API as well which is nice to have for future enhancements.
https://lucene.apache.org/solr/guide/6_6/collections-api.html

Command: bin/solr -e cloud

https://github.com/minimalcode-org/search

More easily add criteria to a base query (boosting etc.)

Currently it's Debug::message() mostly. This needs to switch to the logger.

Adding geospatial awareness to the search would be great. Solr can do this easily and thereby reducing the need for PHP overhead

For some reason, messages on my test environment are all marked to be in #general, and not in the actual channel they were posted in

A queued job to run at specific intervals to index a specific group instead of just running a CLI command.

Benefits:
No user interaction needed to index everything
Solr and PHP are less likely to run out of memory
More control from the CMS

Indexing can error out due to Java IO or PHP memory errors. This should be caught by the system and create a new queuedjob, depending #20 so the group that failed would be attempted to be re-added/updated soon.

Managed schema is great, but it's not updating from schema.xml. Until managed schema is fully implemented, it's better to use the classic schema

Currently, there is only one copyfield definition. Multiple copyfields are requested

Currently, we're returning an ArrayData set, which is nice, but it doesn't contain the actual objects and therefore possibly missing the Link() etc.

Solution:
Use both. The Data set is small enough and highlighting etc. can be done differently

Schema.xml is ignored after Solr switched it to the managed schema.

This should be covered in the code changes as well. Right now, any changes to Schema.xml are ignored because Solr isn't aware of them, once a managed schema is created

Results are currently not controlled by the CMS user

Problem:

• Elevating results to the top, disregarding the score, is not possible

Solution:

• Add Elevation and upload the elevation to Solr automatically after saving
• Reload the elevation settings after every change

A lot of the features are "duplicating" from the old Fulltext Search module. But under a new name.

Some stubs need to be added to support backward compatibility, making it easier to migrate. Possibly as a "Solr-FTS-Compat" module?

Custom copyFields should be populated in types.ss where necessary

If Solr responds with an error, it is good to check if the instance is actually running and possibly report back
https://solarium.readthedocs.io/en/stable/queries/ping-query/

Items that need to be managable:

• Elevation
• Boosting
• Fields to be indexed
• Facets

Add search history to the BaseQuery.

Search History gives engineers the option to show a user's most recent searches. An often required feature.

The value is available but not yet used as protected $history on the BaseQuery

SubSites should do this themselves obviously, but adding support for the SubsiteID is not a bad idea.

There is no impact on the index, but useable.

When objects are changed or deleted, the Solr instance should be updated too

There's a lack of documentation around methods at the moment, this should be fixed before a release.

Please keep updating docblocks while working on this

I miss the cow

Currently it's just the Highlight component from Solarium

Post store is the primary way to create cores where possible. The File Storage is not ideal

AC's:

• Use a post
• Enable uploading custom files through post
• Let Solr store the files in it's own data folder
• Skip .solr folder creation in project

Option to review searches and query count, timings and efficiency of boosting/elevation

Indexing should simply add all documents. The FTS implementation of %% id % count is not reliable and might skip items that are added after start. A cleaner solution would be to simply build batches of 1000 documents, starting from 0, limit to 1000 and start from there.

This would prevent overhead of calculation as well.

Possibly use a queued job to update in the background, which would be a lot better!

Preferably avoided, but if needed it can be added.

To prevent PHP errors, the index task should be optional to run in the background, as FTS originally did.

FTS has a "quick start" script (albeit broken).

It's worth looking in to setting this up on this repo too.

Please check the following:
https://scrutinizer-ci.com/g/Firesphere/silverstripe-solr-search/
https://codecov.io/gh/Firesphere/silverstripe-solr-search

It should tell you where to start cleaning up code if possible.

A name for the yml allows overriding

There are no tests yet, as I'm more discovering how Solarium works etc. than actually testing things.

Unit tests to be done. I'd like some input from Guy for this, as he seems to know a thing or two about testing the old FTS module

There's currently no process in place to manage the schema, even though it's a wished feature

Fix the API to be consistent

Problem:

• Fields does not do what is expected

Solution:

• Update the field queries etc. to not do something different from what Solarium provides.

Add spellchecking from Solr if requested

XML Generation is cluttering up the index, using an abstraction would clear things up massively

Boosting implementation currently adds boosting at index time. Adding at query time boosting would be very valuable.
Resolved in #c613ac5e87348eda7eccfa6f89f562b56060472f

Using YML is faster than running an init.

This is resolved in #c613ac5e87348eda7eccfa6f89f562b56060472f (Which has a wrong commit message, apologies)

Yeah.... ehm, it doesn't do that properly yet

Figure out how to do an exclude filter, as Solarium does not have this by default

types.ss is hardcoded in the base XML. This needs to be split back in to an editable/configurable file.

No documents are indexed yet
Requirements:

• Get fields to index from Solr
• Determine classes to index
• Get lookup chain
• Generate doc
• Update docs in batches of {Solr limit} (~1000)

Standard get/set/add methods that do nothing but update the protected variable do not need actual testing. A simple testing stub that does them all would be enough.

e.g.

class SetGetAddTester extends SapphireTest
{
    testAll()
    {
         foreach($class as $testClass) { foreach ($variable as $testVar) { $this->assertIsset()); }}}

^^ Above code is a stub, not actually functional :)

No boosting is currently in place.

Solarium does support boosting, it's just not yet implemented

Problem:

• Boosting is not working
• Clients/users would like to have control over what is and isn't boosted

Solution:

• Boosting has a sane default (e.g. Title)
• Boosting is CMS controllable on a high level
• Boosting applies the proper setting to filterQueries and Terms

No tests are written yet, nor is there any integration with a CI.

Problem:

• Lack of knowledge on what's going on actually

Solution:

• Add CircleCI

Caveats:

• CircleCI is not free for closed source projects it seems

Current indexing and results do not take the canView in to account.

This can be mitigated by adding the canView as a field in Solr to filter on.

Requirements:

• canView is added as a field
• canView contains a list of groups that are allowed to view
• canView returns a group-level or logged-in level value

Outcome:
Solr can filter on a multi-valued field. E.g. if the user is administrator, the filterQuery should say group:administrator
This should always return true from the method, and be in the index.
But, Group:content-authors, might not always be in the list of allowed viewers, therefore, Solr would automatically not select these items through a filterQuery.

Solarium has the support to do the above

The query is passed on to the index which doesn't do a check for empty values at the moment. This should be fixed