wipacrepo / file-catalog-indexer Goto Github PK

REST requesting is (A LOT) faster than check-summing a file. So, first, check if the file already exists in the FC before collecting its metadata.

Only proceed with collecting metadata if --patch is included. (depends on #23)

Replace --no-patch with --patch. Since not patching is the most common usage, it shouldn't require a command-line option.

We can now use:

• wipac-file-catalog (pending WIPACrepo/file_catalog#127)
• iceprod (pending WIPACrepo/iceprod#307)
• wipac-rest-tools[telemetry]

We will only need the flake8 and mypy jobs in wipac-cicd.yml, at a minimum. Packaging the repo could have benefits (which means publishing to PyPI, etc), but this is not necessary, though tempting so let's do it! (required for @blinkdog's new disk pipeline).

See WIPACrepo/wipac-dev-tools#20

Edit: Indexer will now be published as a package (see #43)

Currently, "content_status" is solely based on whether the .i3 file can be read.

There are also the "good runs" list files. Do we want to consider these? This could be a new field in the FC record.

Optionally, we wait until we have an event-based store since this matches the "good run" granularity.

Currently, the top year is 2020 (IC86-10). We will need to increase this list, and potentially expand outward. Unrecognized years/seasons will cause this indexer to fail fast.

There's a potential race condition when indexing L2 files, if the client script is using index_file() directly and sharing a single MetadataManager instance between threads. This isn't an issue for using index().

Solutions include

• add a threading.Lock() context manager around the above code
• creating an instance-attribute dict (self.L2_dir_data) keyed on dir_path (instead of a single self.dir_path & self.real_l2_dir_metadata)

Find metadata fields for files in /data/sim and index into File Catalog.

Check these directories:

• resources/path_collector/
• resources/filename_patterns/