KeyError: 'gilded' about redarc HOT 12 CLOSED

Pushshift data isn't normalized and there are inconsistencies sometimes. Working on a fix

from redarc.

5da45f2

This commit should fix it. Let me know if that works for you

from redarc.

@ohhdemgirls

from redarc.

Prefect, thanks. Seems to be working, going to take awhile to get everything imported mind you 🤣

from redarc.

Spoke too soon, same file failed differently toward the end...

Identifier inserted succesfully x9zhzi
====================
ERROR: A string literal cannot contain NUL (0x00) characters.

(also spelling typo in your code 😝 )

from redarc.

Working on a fix. I'm also considering removing the SQL rollback; it's not ideal to have to reinsert every submission/comment after an error especially for very large dump files

from redarc.

it's not ideal to have to reinsert every submission/comment after an error especially for very large dump files

True, once I'm able to get everything into the DB it's going to take some tuning for it to be lean but I've already provisioned all nvme storage and a terabyte of ram for it so we should be able to host a static historical version of what PS had available on it's API before people started wanting things removed and reddit nuked it!

There's going to be more edge cases in this data I'm sure so I'm very glad you're able to put in some time wrangling it, get on discord if you want to knock heads on this. Realizing who you are I've been a fan of your previous works 👌

from redarc.

I completely revamped the scripts. It now tries to insert everything first and then logs the problematic rows.

from redarc.

I also added .replace("\u0000", "") to handle null characters in user submitted fields like selftext and title

from redarc.

@ohhdemgirls

from redarc.

real 383m8.352s

Worked that time, but now lets work on speeding up ingest if possible 😋

from redarc.

It's the print statements that's making it slow. Btw, did you check the log file for any warnings/errors

from redarc.