Gain Improvements - Ludaro about gain HOT 5 OPEN

Accept pr.

from gain.

@gaojiuli,I see we cannot install this project from pypi because that version is flawed. Can I expect this project to accept PR's or should I continue with my own fork?

from gain.

For people reading this message, here is my fork of the project

Changes

• Bug fixes
• Improved default url extraction
• Proxy can now be a generator so every fetch can now yield a proxy
• Implemented aiocache, with Redis as default (but you can choose) so you can cache every single requests and define which urls shouldn't be cached with cache_disabled_urls in case you are daily scraping a feed or blog and you are only interested in the new urls. Just set cache_enabled=True and run the docker instance that I have mentioned in the README.md file.
• There is now a spider test flag so you can test a single parsed page to improve your css or figure out issues with your code. Together with the cache feature this makes a very quick and reliable trial and error cycle. Increase the amount of tests to be run with the max_requests
• With the limit_requests flag you can adjust the max_requests value to actually limit the maximum amount of external requests.
• Replaced Pyquery code with the lxml library and made it really easy to write extract data with css code. It's written with backwards compatibility in mind but if you use jquery you can still use the pyquery code by changing your Css class in Item to the Pyq class.

Extraction

With the new Css class the following can be done:

• Get the dict of an element
• Extract all elements into a list
• HTML tables to dict
• Control the index
• Iterate throught texts
• Request text and text content

In case you need to cleanup data or extract stuff like a phone number or email address, the following manipulate options are available:

• clean_string; Will become clean in the future, so it cleans list, dicts and strings to give much more control
• extract_email
• extract_phone
• extract_website
• to_date_iso

The order in which you supply manipulate options, is the order of execution so you can actually combine these manipulations.

And many more will be added in the future. I have written tests for all features so take a look at this file if you are interested.

With the current version on my dev branch you can:

1. First focus on parsing pages with the right xpath or regex (go for this one, since it's very reliable in my version)
2. Set cache_enabled to true and go through all pages to cache them
3. Use the test feature to start writing your extraction code
4. Once done, re-run your code on the Redis cache and push it to any datasource like csv, postgres, etc

I hope @gaojiuli , is interested in the way I moved forward with this project so we can merge our code once I am satisfied with a production version. I kept the philosophy of creating a scraper for everyone and with that in mind I changed the way we extract data.

from gain.

@gaojiuli, great news, happy that I can share my code.

Before the PR, the following I have to do:

1. Update the clean code
2. Ensure users can get back there url
3. Go through all the code once more to make sure we can publish to Pypi

For the item.py I have a question regarding this code:

        if hasattr(self, 'save_url'):
            self.url = getattr(self, 'save_url')
        else:
            self.url = "file:///tmp.data"

Are you using this code or this junk code that can be removed?

from gain.

1. Just pull request.
2. I will review your code.
3. I will realize a new version after I merge your pull request.

(Welcome any kind of optimization)

from gain.