Multiple revisions of same layoff in WI data about warn-transformer HOT 16 CLOSED

@zstumgoren, yes, the Wisconsin system is based on three factors: the revised flag, the sort order and the company name. It passes my eyeball test.

I'm considering having it throw a hard error rather than a log when no ancestor filing is found. That might prevent some new data entry practice by the state from introducing silent bugs.

The policy is limited to WI thus far. The code in the current PR wouldn't affect anywhere else.

from warn-transformer.

If the state's raw data has an amendment column, could we use that to strike certain records?

from warn-transformer.

Another thought, would a true/false boolean called something like is_amendment be a reasonable way to start flagging these?

from warn-transformer.

Take a look at #68, @chriszs

from warn-transformer.

Yeah, good start! I think what we really want to know is if something was amended, but obviously harder to flag.

from warn-transformer.

In Wisconsin's case, revisions seem to always appear below the original, could be used to exclude records superseded by amendments.

In this case, we want to keep revision 2, discard revision 1 and the original.

from warn-transformer.

Hmm. There must be some way to suss this out.

from warn-transformer.

I think it's going to be a state-by-state fight, because the date and amount can change and I suspect in rare cases the location can change, too. It's pretty clear to me what do in WI, but other states may not be so straightforward.

from warn-transformer.

Gotcha. I think a first step is to go state by state and try to mark amendments. I'm going to try that first, and then we can figure out how to handle them next.

from warn-transformer.

I went through all of the CSV files. I only found one other state — Iowa — that had a clear and obvious indicator of an amendment. That was added here #69

Have you see other states where you know there to be amendments, @chriszs?

I wonder if @zstumgoren has some wisdom for us here, as well.

from warn-transformer.

I've drafted an addition to our transformer system for excluding the ancestors of amendments. You can find that here. It encodes what are my latest opinions on WI and IA. For WI, it strikes the previous record in the sheet if the name is the same. For IA, it does nothing, because I can't spot any duplication introduced by the amendments.

from warn-transformer.

Yes, I think. IL comes to mind because I did that recently. It's at the bottom of the spreadsheet there. I'd have to go back and look at other states to see how they handle. Some just add like "UPDATE" to it and then update the record, so that may not require special handling.

from warn-transformer.

Hey @palewire I hadn't noticed amendments in other states, but I'd defer to you and @chriszs at this point on where else this may apply. I do like the strategy of giving precedence to the most recently filed amendment, which feels very much in line with how FEC filings work. Alas, sounds like WARN-land we need to devise a heuristic for identifying related records in an "amendment chain". @palewire Am I reading your PR correctly that the v1 strategy is based primarily on identification of a repeated company name?

from warn-transformer.

@chriszs, when you have a minute can you point out to me what you see in the IL data?

In the meantime, I am going to merge this PR.

from warn-transformer.

So this may be moot if the state's live data portal provides a better alternative to the archive page we're using (I've put this in a ticket), but archive has these supplemental notices at bottom of sheets:

Employee number is additional, so may not inflate that, but unclear whether handling this well in every case.

from warn-transformer.

I think we got this handled. Correct me if we dont.

from warn-transformer.