Light

guidj / manga-mirror Goto Github PK

View Code? Open in Web Editor NEW

1.0 2.0 0.0 30 KB

Crawl manga from mangareader.net

License: Apache License 2.0

Go 98.89% Makefile 1.11%

manga-mirror's Introduction

Manga-Mirror

mangareader.net crawler.

Build

$ make

It builds an executable under $GOPATH/bin/ named manga-mirror.

Usage

Parameters:

  -db [string]
	   Path for crawler sync index (default "_mrdb")
  -dir [string]
    	Path to store downlaoded media (default "_media")
  -url [string]
    	Web URL to crawl for media
  -filter [string]
    	Regex pattern to filter URIs, e.g. 'mangareader.net|naruto'
  -log [string]
    	log file name, for logging (default STDOUT)
  -user-agent string
    	UserAgent for HTTP HEADER (default "mng-rdr")

$ manga-mirror -url [some-manga-url] -filter [some-regex-pattern]

Misc

This crawler can be used to crawl other sites, and download images. Though it is ever hardly used or tested at that task. Note that this crawler respects robots.txt configurations on which paths can be crawled, but not on the crawling interval.

manga-mirror's People

Contributors

Stargazers

Watchers

manga-mirror's Issues

Accept a regex pattern to filter URIs

Accept a user provided regex pattern to filter URIs to crawl

Respect robots.txt

When parsing read and respect robots.txt file from sites

Parametize runtime

Use User set params to run (e.g. output dir, db path)

Image done queue is not processed

The image done queue is not handled by the OperateNotifier function in main. After a while, the buffer fills in the channel, and the crawler stops downloading images

Track crawling activity

Keep track of which pages have been crawled, and which files have been downloaded

Avoid repeated crawling

Some pages are crawled multiple times, because they're added to the queue channel everything they're encountered by the crawler and haven't yet been processed.

To avoid this, once an URI is added to the waiting queue, it should be registered to the DB with INQ state.
This way we avoid duplicate entries to the waiting queue
Once processed, we can mark the URI as DONE

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.