Giter Club home page Giter Club logo

scrapix-cli's Introduction

Scrapix

A TypeScript-based CLI tool for scraping and downloading Google images ๐Ÿš€.

experimental

Description

Scrapix is an interactive CLI for scraping and downloading Google images easily. The core of this CLI is the images-scraper package, which uses a puppeteer-based headless browser to automatically search and return the URLs of the images. This scrapix CLI can help developers and non-developers to build image datasets for computer vision tasks i.e. image classification, object detection, face recognition.

Installation

$ npm install -g scrapix-cli

To resolve puppeteer installation issues on Linux based OS: -

$ sudo npm install -g scrapix-cli --unsafe-perm=true

Usage

To start the CLI, Type the scrapix command in your CMD or terminal: -

$ scrapix

Modes

default The search keywords are defined in the terminal directly
file The search keywords are defined in a .json file.
Default mode: usage

Required Parameters: -

  • keywords: - The names of images you want to download seperated by commas (i.e. roses,daisy,hibiscus)
  • number: - The number of images you want to download
File mode: usage

Required parameters: -

  • Name of .json file in the base directory to load the keyword and number of images to be scraped and downloaded for each keyword.
Structure of the .json file content
{
    "images": [
        {"keyword": "roses", "number":20},
        {"keyword":"daisy", "number":25},
        {"keyword":"hibiscus", "number":30},
    ]
}

View Video Demonstration

TODOS

  • Image validation
  • Support for custom image processing actions i.e. resizing, compression
  • Improve error handling
  • Provide support for other search engines i.e. Bing, Wikipedia
  • Improve general performance
  • Clean code and develop a better documentation

Contributing Guide

Coming soon

scrapix-cli's People

Contributors

rexsimiloluwah avatar

Watchers

 avatar

scrapix-cli's Issues

Optional maximum concurrency parameter

I think it would be a good idea to allow users to define some sort of maximum concurrent browser instance limit so even if they try to pull images for a large number of queries it won't just tank their computer

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.