Look into this. Would likely need to-

• Take union of dataspecs
• Show joint field facets at top
• Loop through dataspecs and find non-joint facet fields
• Print data source name
• Print non-joint facets

What should we put in the top upper left for the image for SII?

We often manually process part of a dataset, but also want to OCR the documents and run extraction tools on it. To do this within the normal TT pipeline, we need to add something to Harvester that matches up this csv file with the docs and OCRs them. First, we should make a gem that does this with the following inputs-

• csv or json file (convert to json if csv)
• field used for file path
• optional prefix for file path

This should then loop through the items in the JSON and run parsefile on each corresponding file. Then it should save these with the text.

Add the transfers database, for which I need to-

• Get it into a JSON format
• Extract out the links into their own field (split source field)
• Generate title
• Write dataspec
• Associate with companies based on ID (and add to that dataspec)

Current test version is down. Update it and send

Make repository with processing scripts and data

• Remove annote
• Remove year collected from raw data
• Remove year collected from dataspec

Create an additional page with info on other resources (bugged planet, spy files, blue cabinet). Write draft, add to dataspec, add to routes, and share for review.

If name is present, use that
If not, if Document-Title is present, use that (or maybe reverse)
If neither is present, use Type-Of-Media

Probably best to put this after current processing

Add associations between LookingGlass models

In the sii theme for LookingGlass, the drop down in the upper left allowing people to choose which fields to search seems to be unreadable because the text is too light.

The icons in the SII instance of LookingGlass seem to be rotated, so each field is associated with the wrong icon

Company reports don't have proper names. In initialization scripts, generate these with company_name + Report.

For each company, go through list of documents (may need to change gen order) and generate a list of links to associated documents. May need to add configurable prefix to dataspec/instancespec to make this work. Will need to add field to dataspec to show this.

Figure out how to make this link display work with dataset associations

Update it to work with the new data file

I'm not sure this is the right repo for this, but it would be useful to have UUIDs so that they can be linked to Wikidata entries, for instance.

Generate company descriptions in the data parsing scripts with location, what they work on, etc

Just to be clearer

Make sure this actually shows up

Add configuration for tab order to LG and instance spec config. Make document tab first on PI theme

Add the Document Cloud PDF viewer (https://github.com/documentcloud/document-viewer) to LG. Then add a function that lets users choose in the config which viewer to use. Make sure the SII docs work on it.

Add csv matching/OCR to CrawlerManager and test with Harvester.

Consider adding blocks of code to this to app to handle extraction

Figure out ABBYY lock error