Figure out a good way to root out duplicate records about warn-transformer HOT 9 CLOSED

The pandas stuff is just to explore the data and document what I find. I think our ultimate solution will be something else, and will likely also involve some upstream patches to our scrapers.

from warn-transformer.

I've identified some here. It looks like our hashing method does seem to condense the detection

https://github.com/biglocalnews/warn-transformer/blob/main/_notebooks/duplicates.ipynb

from warn-transformer.

Is notebook/pandas approach for experimentation purposes? Wondering if we could just use sets + tuples to perform a basic dedupe in the production pipeline code to keep things light-weight...

from warn-transformer.

Alabama is clearly caused by the source

from warn-transformer.

California dupes also appear in the raw source materials. Not sure why, but we're not introducing it.

from warn-transformer.

Same goes for Texas

from warn-transformer.

many of these files have numerous dupes. Could be not complete dupes, with some date differences, and slight changes in dates or number of employees. From what I understand, if there is any change, the company submits a new WARN notice to make sure they comply and it gets added. Additionally, if there are multiple locations, the data may be reflecting multiple layoffs from the same company though, so something to watch for.

from warn-transformer.

Gotcha. Now that I've verified these "exact dupes" come from the source, and aren't introduced as bugs in our code, I think I'm going to take the step of eliminating them from our dataset at this point.

from warn-transformer.

makes sense

from warn-transformer.