omarroth / archive Goto Github PK

if i restart it continues with the last few missing ones. if it still cant download all of them i have to repeat again until it got all. then it uploads and starts a new batch.

niemand@akaralan:~/archive/docker$ sudo docker build -t archive .
Sending build context to Docker daemon 2.048kB
Step 1/6 : FROM node:10
10: Pulling from library/node
no matching manifest for unknown in the manifest list entries

what package do i have to install that includes this command? i already installed gcrystal.

i don't know if this much attention is needed or wanted but jan 15th seems very close to me. is there any way to see the progress?

no idea why but some items in the beginning of the batch failed

(node:10616) UnhandledPromiseRejectionWarning: RequestError: Error: socket hang up at new RequestError (/home/niemand/archive/node/node_modules/request-promise-core/lib/errors.js:14:15) at Request.plumbing.callback (/home/niemand/archive/node/node_modules/request-promise-core/lib/plumbing.js:87:29) at Request.RP$callback [as _callback] (/home/niemand/archive/node/node_modules/request-promise-core/lib/plumbing.js:46:31) at self.callback (/home/niemand/archive/node/node_modules/request/request.js:185:22) at emitOne (events.js:116:13) at Request.emit (events.js:211:7) at Request.onRequestError (/home/niemand/archive/node/node_modules/request/request.js:881:8) at emitOne (events.js:116:13) at ClientRequest.emit (events.js:211:7) at TLSSocket.socketErrorListener (_http_client.js:387:9) (node:10616) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 170)
i hope this doesnt lead to incomplete data

i know that some instructions seem trivial but i think there are many possible users for who completely copy pastable instructions mean the difference between getting it working or not.

It appears that everything has been backed up now. CONGRATULATIONS! the script also doesnt upload anything anymore. wouldnt it be beneficial to still recieve uploads for possible error corrections or are there possibly other sources for video ids? especially since there are now so many active workers.

uaing the crystal version i got this error:

GC Warning: Repeated allocation of very large block (appr. size 33558528):
May lead to memory leak and poor performance
All annotations collected (28.1 MiB)
Compressing...
Committing...
All annotations compressed (2.6 MiB)
Uploading to S3...
Unhandled exception: Status 204 should not have a body (ArgumentError)
from /usr/share/crystal/src/http/client/response.cr:0:11 in 'exec_internal_single'
from /usr/share/crystal/src/http/client.cr:499:5 in 'exec'
from /usr/share/crystal/src/http/client.cr:342:3 in 'post'
from src/worker.cr:192:5 in '__crystal_main'
from /usr/share/crystal/src/kernel.cr:453:5 in 'main'
from /usr/share/crystal/src/string.cr:4202:5 in '__libc_start_main'
from ???
niemand@akaralan:~/archive$

Download Docker from https://download.docker.com/win/stable/29211/Docker%20for%20Windows%20Installer.exe

download archive from https://github.com/omarroth/archive/archive/master.zip

extract and open powershell in docker subdirectory

then procede like on linux

recently i am always getting 10.3-10.6 mb uncompressed and 1.9mb compressed. i hope the uploaded data is still valid.

you should add:
sudo npm install request-promise-native
sudo npm install sqlite

to the node.js part of the readme because those were not present when i installed npm on linux mint. the sqlite install failed so personally i unfortunately cannot use it anyway. so there could be even more dependencies missing.

i am getting invalid size errors all the time now

Continuing 57f8b4c2-16c9-4ef1-899f-f4063cf75054...
GC Warning: Repeated allocation of very large block (appr. size 33558528):
May lead to memory leak and poor performance
All annotations collected (39.5 MiB)
Compressing...
Committing...
Invalid size for 57f8b4c2-16c9-4ef1-899f-f4063cf75054
Continuing 57f8b4c2-16c9-4ef1-899f-f4063cf75054...
GC Warning: Repeated allocation of very large block (appr. size 67112960):
May lead to memory leak and poor performance
All annotations collected (39.5 MiB)
Compressing...
Committing...
Invalid size for 57f8b4c2-16c9-4ef1-899f-f4063cf75054
Continuing 57f8b4c2-16c9-4ef1-899f-f4063cf75054...
GC Warning: Repeated allocation of very large block (appr. size 67112960):
May lead to memory leak and poor performance
All annotations collected (39.5 MiB)
Compressing...
Committing...
Invalid size for 57f8b4c2-16c9-4ef1-899f-f4063cf75054