Comments (8)
That would be great! โจ
For the cli module I'd recommend using tilda
.
Also, I would say to create a different project for the cli tool (e.g. scrape-it-cli
).
Let me know if you have any questions! ๐
from scrape-it.
I thought about docopt...
it's based on this original project
The idea is that it takes usage string as DSL of your CLI. So all you have to write is this:
docopt(`
Usage:
scrape-it <url> [--some-option]
scrape-it -
scrape-it -h | --help | --version
Options:
--config=FILE Config file [default: ./scrape-it.conf.js]
`)
And all of sudden you get something like this:
{
url: 'http://...',
config: './cfg.js',
some-option: true
}
If you're okay with that I'd be happy to do it (and yeah it can be in its own repo)
from scrape-it.
@cztomsik Sure, feel free to use whatever you're comfortable with and share a link once it's on GitHub. ๐
I'll be happy to add a link to it. ๐
from scrape-it.
ok :)
from scrape-it.
Here's the very first version, it's not even published to npm yet, what do you think? Any ideas?
I was thinking about some default config or maybe an init
command so people could start quickly?
Something like
npm i -g scrape-it-cli
scrape-it init
scrape-it http://github.com
and instantly get:
{ title: 'How people build software ยท GitHub' }
from scrape-it.
@cztomsik Cool! ๐
from scrape-it.
@cztomsik Hey! Did you do any progress on this? Maybe we can work together for a stable release of scrape-it-cli
.
from scrape-it.
Hey, not exactly, basic version worked fine but I've actually found myself in need of something much more powerful (dozens of concurrent crawls backed in one big database) which I am working on and hopefully will open-source soon - I think you might like it and OFC we could join forces then :)
Anyway if you have ideas about scrape-it-cli I can certainly help you a little (or brainstorm) or even transfer the repo to you
from scrape-it.
Related Issues (20)
- What is the reason for this jump in popularity? HOT 1
- Encoding issue with Spanish accents HOT 3
- Cannot find name Cheerio / CheerioSelector / CheerioStatic HOT 1
- Add support for conditional selector HOT 1
- Meta Data HOT 1
- no space between tag HOT 1
- Move scrapeHTML into its own package HOT 1
- Headers of type HTTP/2 HOT 1
- scrapeHTML is not defined HOT 1
- Alternative selectors for one element HOT 1
- Scrape a table with image links. HOT 1
- Parsing tables and xpath
- Can't select nth element HOT 1
- Scrape Behind Login HOT 1
- commas on next line in readme HOT 1
- $ is not a function in a Remix App HOT 1
- Doesn't return tag script attribute value HOT 1
- Doesn't work with :nth-child selector HOT 1
- [Feature request] "OR" query
- Can't use :not(:fist-child) nor eq in listItem HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrape-it.