r888888888 / archives Goto Github PK

When no arguments are specified, Process.daemon redirects all output to /dev/null; therefore, if the service is run directly via bundle exec ruby services/sqs_processor.rb --logfile=my.log --pidfile=my.pid, no output is logged. After changing the line to Process.daemon(true, true), it executes fine and logs output as expected.

Disclaimer: I have no idea what sets the RUN environment variable, because googling that turned out quite difficult.

Sorry to bother you with newb stuff.

Is there any way I could get instructions for deploying this? iqdb/iqdbs and archives seem less straight-forward than danbooru is.

I vaguely remember @r888888888 mentioning there's no guarantee that messages will get delivered/processed in order they were pushed. Due to the nature of versioning service this spells disaster, because if two versions for the same post get delivered out of order, the diffs will get calculated and stored incorrectly.

Some thoughts:

1. AWS supports FIFO mode as far as I can tell, they advertise it on their front page; it should be enabled if it isn't already
2. Version ID can be generated from a sequence on a main danbooru site and pushed to the queue as well. That way, if versions get delivered out of order, we at least have a chance of reordering everything properly: calculate diff to the nearest prior version and recalculate all diffs for later versions. Version merging would interfere, though, so it's best to make sure that queue is processed in order.

When I create a post or a pool, the operation is successful but Archives' database is not updated due to the "booru_id" column being null.
I should probably specify that I'm using Archives without AWS, after I substituted the cloud service with a local service reading from a UNIX socket (it's just a ruby server with two threads writing to and reading from a shared Queue object.)
Despite that radical, the data is sent correctly to Archives from the main Danbooru and in fact inspecting it I could see that all the expected fields are there with a value, except "booru_id" which is missing from the JSON and thus null.
I solved this by creating a new migration dropping the "booru_id" column, but I don't know if this is the correct solution.

I don't know if this is a one-off error or something else, but I noticed the following when browsing through the pool versions.

http://danbooru.donmai.us/pool_versions?search%5Bpool_id%5D=12054

It basically created 5 versions for 5 different pool adds done at pool creation.