Giter Club home page Giter Club logo

free-proxy's Introduction

Free-proxy

Version 1.1.1

The FreeProxy class scrapes proxies from https://www.sslproxies.org/, https://www.us-proxy.org/, https://free-proxy-list.net/uk-proxy.html, and https://free-proxy-list.net and checks to make sure that it works. You can filter proxies by country, and specify an acceptable timeout. You can also randomize the list of proxies, rather than going in the order that they are scraped in.

You can use this to send requests through a custom proxy, with Selenium, or with anything else.

Returns proxy as string:

'http://113.160.218.14:8888'

Requirements

Installation

pip install free-proxy

asciicast

Usage examples

First import Free Proxy that way:

from fp.fp import FreeProxy

Options

Parameter Type Example Default value
country_id list ['US', 'BR'] None
timeout float > 0 0.1 0.5
rand bool True False
anonym bool True False
elite bool True False
google bool,None False None
https bool True False
proxy = FreeProxy().get()
proxy = FreeProxy(country_id=['US', 'BR']).get()
proxy = FreeProxy(country_id=['US']).get()
proxy = FreeProxy(country_id=['GB']).get()
  • timeout parameter Timeout is the parameter for checking if a proxy is valid. If the server does not respond in specified time, the script will mark the proxy as invalid. Default timeout=0.5. You can change it by specifying a timeout eg. timeout=1.
proxy = FreeProxy(timeout=1).get()
  • rand parameter Shuffles the order of the proxy list from https://www.sslproxies.org/ instead of going from newest to oldest (as listed on the website). Defaults to rand=False
proxy = FreeProxy(rand=True).get()
  • anonym parameter Return only proxies marked as anonymous. Defaults to anonym=False
proxy = FreeProxy(anonym=True).get()
  • elite parameter Return only proxies marked as 'elite proxy'. Defaults to elite=False.
proxy = FreeProxy(elite=True).get()

Please note: elite proxies are always anonymous. If you set elite=True, you will also be eliminating any non-anonymous proxies.

  • google parameter If True it will only return proxies marked as "google". If False, it will not return proxies marked as "google". Defaults to google=None, which returns all proxies.
proxy = FreeProxy(google=True).get()
  • https parameter If True it will only return proxies marked as HTTPS. Defaults to https=False - i.e. HTTP proxy (for HTTP websites).

    Please note: HTTPS proxies work for both HTTP and HTTPS websites.

proxy = FreeProxy(https=True).get()

You can combine parameters:

proxy = FreeProxy(country_id=['US', 'BR'], timeout=0.3, rand=True).get()

If there are no working proxies with the provided parameters, the script will raise FreeProxyException with the message There are no working proxies at this time..

CHANGELOG


[1.1.1] - 2023-02-18

  • Fixed https parameter error
  • Fixed additional loop issue when no proxy was found

[1.1.0] - 2022-11-12

[1.0.6] - 2022-01-23

  • Added google parameter
  • Added https parameter

[1.0.5] - 2022-01-07

  • Added elite parameter
  • Add exception class and raise exception instead of system exit
  • Change lxml version to 4.6.5

[1.0.4] - 2021-11-13

  • Fix proxy list default length

[1.0.3] - 2021-08-18

  • Change XPatch due to SSL proxies page update
  • Change lxml version

[1.0.2] - 2020-09-03

  • Added anonym parameter

[1.0.1] - 2020-03-19

  • Fix typos in readme
  • Fix urrlib3 exception urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None,
  • Fix imports

[1.0.0] - 2019-02-04

  • Initial release

License


MIT

Free Software!

free-proxy's People

Contributors

dependabot[bot] avatar diggidydev avatar dimitryzub avatar edgarrmed avatar jundymek avatar la55u avatar lukeprofits avatar rafahuelin avatar vkraut avatar vol1ura avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

free-proxy's Issues

failed to get proxy

I haven't received a proxy yet, is it possible that it no longer works?
I will install with pip
Here is my code

    def proxy():
        i = 1
        while True:
            try:
                proxy = FreeProxy(google=True, https=True).get()
            except:
                status_fail = Fore.RED + 'FAIL'
                print(f'failed to get proxy {status_fail}')
                time.sleep(2)
                if i == 50:
                    status_fail = Fore.RED + 'proxy not found'
                    sys.exit(f'{status_fail}')
                i += 1
            else:
                status_ok = Fore.GREEN + 'OK'
                print(f'proxy {proxy} status {status_ok}')
                break
        return proxy

    proxy = proxy()

50 lines always "failed to get proxy FAIL"
Any ideas?
Thank you

For some reason it does not seem use the proxy to make the request, even after declaring the param correctly.

As said above. I have tried printing/logging the proxy it gets and it gets printed. It just doesn't get used while making the request. What could the reason be?

As you can see below, the proxy being used is the localIP while the proxy being pulled by the library is also being printed successfully.

https://i.imgur.com/aoqpvLf.png

Snippet:

proxyObject = FreeProxy(country_id=['US', 'IN'], rand=True)
proxy = {"http": proxyObject.get()}
print(proxy)
url = 'https://superkicks-in.translate.goog/wp-json/wc/store/products?stock_status=instock&_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=en-GB&_x_tr_pto=nui'
s = requests.Session()
response = s.get(url=url, headers=headers, proxies=proxy, timeout=15)
print(response)
print(response.text)

`IndexError: list index out of range` when FreeProxy().get()

I get the an IndexError when after using the .get() method. Below is an example:

>> from fp.fp import FreeProxy
>> obj = FreeProxy(country_id=['US'])
>> obj.get()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/fp/fp.py", line 38, in get
    proxy_list = self.get_proxy_list()
  File "/usr/local/lib/python3.8/dist-packages/fp/fp.py", line 28, in get_proxy_list
    proxies = [f'{tr_elements[i][0].text_content()}:{tr_elements[i][1].text_content()}' for i in
  File "/usr/local/lib/python3.8/dist-packages/fp/fp.py", line 30, in <listcomp>
    if tr_elements[i][2].text_content() in self.country_id
IndexError: list index out of range

Caching the list of servers

The get() method is convenient to retrieve individual proxies but since it retrieves the whole list on every invocation it is rather slow and puts a high strain on the sslproxies site unnecessarily.

A better way would be to store the returned list within the class instance and remove individual proxies from the list till it is empty, and only then fetch further proxies (or maybe even have a setting to reuse the old list and rotate through them that way).

What do you think?

ImportError: No module named 'fp'

This happens on WinXP 32bit, Python 3.4.
Apparently the installation was successful:

C:\Dropbox\python>c:\python34\scripts\pip install free-proxy
Collecting free-proxy
  Downloading https://files.pythonhosted.org/packages/2e/fa/d01b586a6cca1ae2c75657dac7a2e3b43767182c4d1a2bbcbe7f641acd8a/free_proxy-1.0.0-py3-none-any.whl
Requirement already satisfied: lxml>=4.3.0 in c:\python34\lib\site-packages (from free-proxy) (4.3.1)
Requirement already satisfied: requests>=2.21.0 in c:\python34\lib\site-packages (from free-proxy) (2.21.0)
Requirement already satisfied: certifi>=2017.4.17 in c:\python34\lib\site-packages (from requests>=2.21.0->free-proxy) (2021.5.30)
Requirement already satisfied: idna<2.9,>=2.5 in c:\python34\lib\site-packages (from requests>=2.21.0->free-proxy) (2.7)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in c:\python34\lib\site-packages (from requests>=2.21.0->free-proxy) (1.23)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\python34\lib\site-packages (from requests>=2.21.0->free-proxy) (3.0.4)
Installing collected packages: free-proxy
Successfully installed free-proxy-1.0.0

But then I can't load the module:

from fp.fp import FreeProxy
ImportError: No module named 'fp'

The same script worked in a different machine (Win7 64bit, Python 3.7)
But as I recall free-proxy version is the same in both.

Proxy country not corrected

Looks like it doesnt filter correctly by nation anymore. Before this worked:

FreeProxy(country_id=['US'], https=True, rand=True).get()

Now the resulting proxy is not anymore from US but from other countries. In this moment I verified that other proxies from US are effectively working but they are not picked by Freeproxy.
Thx for the attention

Edit: weid, after some hours it acts correctly, writing "There are no working proxies at this time."

Can't use the imported package

Hello,

I successfully installed the package using 'pip install free-proxy', but fp.fp is nowhere to be found when I am using the line you provided for importing the package into my code. Any idea why ?

Thanks :)

Setting https=True gives error

When I try to get https proxies using

from fp.fp import FreeProxy
proxy = FreeProxy(https=True).get()

I get the following error
Exception has occurred: ValueError check_hostname requires server_hostname

Setting https to False or completely omitting the parameter works fine. Other parameters also work as intended but I just can't get the https parameter to work.
Am I missing something?

Library not working. Giving IndexError: list index out of range

File "c:/Users/tanuja jain/OneDrive/Desktop/mblazr/edgarapp/static/routine/test.py", line 2, in
proxy = FreeProxy(rand=True).get()
File "c:\Users\tanuja jain\OneDrive\Desktop\mblazr\env\lib\site-packages\fp\fp.py", line 38, in get
proxy_list = self.get_proxy_list()
File "c:\Users\tanuja jain\OneDrive\Desktop\mblazr\env\lib\site-packages\fp\fp.py", line 24, in get_proxy_list
proxies = [f'{tr_elements[i][0].text_content()}:{tr_elements[i][1].text_content()}' for i in
File "c:\Users\tanuja jain\OneDrive\Desktop\mblazr\env\lib\site-packages\fp\fp.py", line 24, in
proxies = [f'{tr_elements[i][0].text_content()}:{tr_elements[i][1].text_content()}' for i in
IndexError: list index out of range

TypeError : 'NoneType' object is not callable

Are the deps alright in this library? When I try to import:
from fp.fp import FreeProxy

An error is thrown:
PythonException: TypeError : 'NoneType' object is not callable File "Assets/StreamingAssets/python-3.7.9/lib/site-packages/lxml/html/__init__.py", line 88, in <module> namespaces={'x':XHTML_NAMESPACE}) File "Assets/StreamingAssets/python-3.7.9/lib/site-packages/fp/fp.py", line 5, in <module> import lxml.html as lh File "search.py", line 2, in <module> from fp.fp import FreeProxy

Installed via PyPi. Any ideas what's going on?

Suggestion: make test site customable?

Sometimes we may want to use other website than google.com for testing. How about keep google.com as default, and add a option for users to customize it?

I think the code is wrong

Traceback (most recent call last):
File "test.py", line 1, in
from free_proxy import FreeProxy
ModuleNotFoundError: No module named 'free_proxy'

AttributeError: __enter__

When trying to run this on python 3.7, I get the following error

self = <fp.fp.FreeProxy object at 0x7f11da8c2490>
proxies = {'http': 'http://104.41.54.53:3128'}

    def check_if_proxy_is_working(self, proxies):
>       with requests.get('http://www.google.com', proxies=proxies, timeout=self.timeout, stream=True) as r:
E       AttributeError: __enter__

Add package to conda-forge?

We use fp as a dependency in scholarly and we are considering making scholarly available in conda-forge. To do that, we need to get all our dependencies also in conda-forge and fp is the only one that is not in. As a part of this effort, could I raise a PR to get fp into conda-forge as well?

lxml etree issue in AWS Lambda

Hello all,
I'm getting this error message due to the library lxml and the modulo etree in AWS Lambda. I'm using python 3.9.

[ERROR] Runtime.ImportModuleError: Unable to import module 'src/lambda_function': cannot import name 'etree' from 'lxml' (/var/task/lib/lxml/init.py)
Traceback (most recent call last):

Do you have any help for me please :'c

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.