Deleting a document should also delete the file as long as it isn't associated with any other documents too (check for this). This behavior may need to be changed depending on how distributed storage works.

Currently there are duplicate copies of dataspecs to match what the publication step generates. Deduplicate these.

Track documents that have been deleted (maybe by hash?) so that they can also be removed on public instance.

Remove the instance-specific dataspecs from the DM repository and move them to another repository with instructions on how to use them. These will probably still be used from time to time, but we should move towards users creating custom dataspecs via the UI on the hosted system rather than having to work with the raw JSON files.

Add the default values for fields that should be searched for each document type by default to the dataspec.

Add the query needed for this issue: TransparencyToolkit/ArchiveAdministrator#1

Deleted documents (maybe since last batch) tracked with #10 should be removed from public instance when republish is run. Since they are deleted, it is difficult to know if they match the conditions for publication. So perhaps all deleted should be removed each time publish is run.

This is a major upgrade because document types were removed (and these are used heavily within TT). Most likely the document types should be separate indexes or in the same index but managed as part of the document ID.

Hi, after running the rake db:create db:migrate, I get the following error:

rake aborted!
NoMethodError: undefined method client= for Elasticsearch::Persistence:Module
/opt/DocManager/config/initializers/elasticsearch.rb:2:in <main>

Just wondering if this is an issue due to version or am I missing anything in previous steps. Thanks for all your help.

Currently the publish function only sends documents updated since it was last run. But someone could choose a different set of documents or fields to publish. So, instead, the following should be checked-

1. Publication date
2. Document published
3. Fields document published with

The publication date should be used if the document was already published WITH the same fields. If it hasn't been updated and is same field combo, document should not be published.

If it doesn't match and the document matches publication query, it should be resent. If the document wasn't published, it should be republished.

Currently, changing the title of a document or other field used in the ID on pre-publication archive may result in creation of a second document on the public instance. The document ID should either be passed to the public instance (and constant) or a based on a non-user-changeable (but unique) field.

Ensure that-

1. The language for the index is saved with the project spec
2. This language is actually set as the index language

Probably there also needs to be a function to change the index language when this updates, but this may be rare. More likely is someone will want a second index for another language.

Properly update the modified field when documents updated after being added.