Giter Club home page Giter Club logo

thingiversecrawler's Introduction

Thingiverse Crawler

About:

Thingiverse Crawler is a simple script to batch download things from Thingiverse. It does not use the thingiverse API because non-web application flow is not yet supported.

The script dynamically update wait time to prevent it being blocked by Thingiverse website.

This script has been modified to accomplish the following:

  • Use thingiverse's built in search functionality to grab models matching a user defined query
  • Remove required input of number of models when using the above mentioned search
  • Remove repeated calls to the same page. Original script would loop until it hit the user defined number even if there were no more pages

Example:

Retrieve information of the newest featured 1000 things:

$ ./thingiverse_crawler.py -n 1000

Retrieve information of things matching a search query

$ ./thingiverse_crawler.py -s "kobayashi cube"

Retrieve limited number of things matching a search query

$ ./thingiverse_crawler.py -s "kobayashi cube" -n 1000

A summary.csv will be created. It contians the following information:

  • thing_id
  • file_id
  • file: where to store the output file.
  • license: info about the license it is published in.
  • link: direct link for download.

To download all 1000 files:

$ ./download_model.py summary.csv

To grab context of each thing (i.e. tags, dates, categories):

$ ./grab_context.py summary.csv

To get a single file using its thing_id

$ ./get_file.py thing_id

Modifed By :

Mike Gleason

Web Developer

Original Author:

Qingnan Zhou

New York University

https://www.cs.nyu.edu/~qnzhou/

thingiversecrawler's People

Contributors

lejafar avatar qnzhou avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

thingiversecrawler's Issues

No longer downloading models

Thingiverse has been updating their site heavily lately, which may be the reason why, but I can't get the download_model.py script to actually download anything. I have tried Python 2 & 3, and I managed to get to the following error using Python 3.

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "multiprocessing\pool.py", line 119, in worker
File "multiprocessing\pool.py", line 44, in mapstar
File "C:\Users\Bradley\Documents\Extracts\python-3.5.1-embed-win32\ThingiverseCrawler\download_model.py", line 18, in download_single_file
r = call(command.split());
File "subprocess.py", line 560, in call
File "subprocess.py", line 950, in init
File "subprocess.py", line 1220, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "./ThingiverseCrawler/download_model.py", line 64, in
main();
File "./ThingiverseCrawler/download_model.py", line 60, in main
pool.map(download_single_file, entries[start:end]);
File "multiprocessing\pool.py", line 260, in map
File "multiprocessing\pool.py", line 608, in get
FileNotFoundError: [WinError 2] The system cannot find the file specified

The thingiverse_crawler.py is still working, and the grab_context.py seemed to work on about half of the files in summary.csv. I'm very new to python and am just learning to use it for webscrapping. I'm just not sure how to get past this error and start downloading some .stl files.

Specify category

Dear,

Absolutely love the ThingiverseCrawler you made! But I was wondering if there is a way to automatically download models that are tagged as a specific object or category. For instances automatically downloading all models from cats or chairs.

Sincerely,
Judith

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.