uchidalab / book-dataset Goto Github PK

Hey i need updated datasets every day i want to run a cron that would help me to get all the data of books on end of the day.
I see you have data so there must be a way to do that.Can you tell me how we can do that.

Label names are almost sorted.
But Christian Books & Bibles is not located correct place.
It must be placed between Children's Books and Comics & Graphic Novels.

i am trying to run this dataset but it is quite time consuming and heave.is there form like ciphar10 binary or pickled dataset of above mention datasets

When I try to run the download images.py I keep getting an attribute error. I'm new to the Argparse library and haven't been able to figure out how to fix this

[Download images into "Desktop/book_coverart"]
0%| | 8/207573 [00:03<11:06:21, 5.19it/s]joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/joblib/externals/loky/process_executor.py", line 418, in _process_worker
r = call_item()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/joblib/externals/loky/process_executor.py", line 272, in call
return self.fn(*self.args, **self.kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 567, in call
return self.func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/joblib/parallel.py", line 225, in call
for func, args, kwargs in self.items]
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/joblib/parallel.py", line 225, in
for func, args, kwargs in self.items]
File "download_images.py", line 39, in download_image
inner_output_dirpath = os.path.join(args.output_dirpath, category)
AttributeError: 'Namespace' object has no attribute 'output_dirpath'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "download_images.py", line 52, in
Parallel(n_jobs=-1)(delayed(download_image)(i) for i in trange(len(csv)))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/joblib/parallel.py", line 934, in call
self.retrieve()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/joblib/parallel.py", line 833, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
return future.result(timeout=timeout)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
AttributeError: 'Namespace' object has no attribute 'output_dirpath

If a ConnectionResetError exception is thrown while an image is being download, the script will crash. There should be some sort of auto-recovery mechanism that attempts to recover the script instead of the existing logic which results in lost progress (since the script will have to be re-run again if an crash occurs).