nbulaj / proxy_fetcher Goto Github PK

The documentation states that the .url method of Proxy "returns a formatted URL like http://IP:PORT" but it does not seem to actually return the protocol. Looks like it's just "#{addr}:#{port}" .

Is it possible to make HTTP request with chosen proxy?

This gem looks great but in attempting to use it I can't get ProxyFetcher::Client.get( 'https://example.com/resource') to return anything. In my rails console it just hangs. I'm able to manually create a manager and fetch a list of proxies fine but the Client.get just never returns any results. Is there something I'm missing?

I've tried both the master version and release 0.6.3

Is it possible to check multiple providers e.g.

    ProxyFetcher.configure do |config|
      config.provider = [:free_proxy_list, :free_proxy_list_ssl, ...]
    end

Thanks

In the clean up process response time for each proxy could be calculated.

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler found conflicting requirements for the Ruby version:
  In Gemfile:
    Ruby (~> 2.3.8.0)

    nokogiri (~> 1.8) was resolved to 1.11.5, which depends on
      Ruby (>= 2.5.0)

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler found conflicting requirements for the Ruby version:
  In Gemfile:
    Ruby (~> 2.3.8.0)

    nokogiri (~> 1.8) was resolved to 1.11.3, which depends on
      Ruby (>= 2.5.0)

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

This gem assumes returned proxy list is not rotated (i.e. if proxy if found non-connecting it is permanently removed from proxy list instead being re-added in rotation period). For rotating proxies invalid proxy should be considered invalid only for rotation period (e.g. 10 minutes, configurable).

The documentation states you can filter by country but this does not seem to work (version 0.6.3). See output:

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler found conflicting requirements for the Ruby version:
  In Gemfile:
    Ruby (~> 2.3.8.0)

    nokogiri (~> 1.8) was resolved to 1.11.4, which depends on
      Ruby (>= 2.5.0)

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

i'm just trying out your plugin... and this is what i get:

/usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/free_proxy_list.rb:28:in Integer': invalid value for Integer(): "08080" (ArgumentError) from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/free_proxy_list.rb:28:in block in to_proxy'
from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/free_proxy_list.rb:26:in tap' from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/free_proxy_list.rb:26:in to_proxy'
from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/base.rb:10:in block in fetch_proxies!' from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/base.rb:10:in map'
from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/base.rb:10:in fetch_proxies!' from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/base.rb:15:in fetch_proxies!'
from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/manager.rb:38:in block in refresh_list!' from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/manager.rb:34:in each'
from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/manager.rb:34:in refresh_list!' from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/lib/proxy_fetcher/manager.rb:19:in initialize'
from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/bin/proxy_fetcher:60:in new' from /usr/local/var/rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/proxy_fetcher-0.6.3/bin/proxy_fetcher:60:in <top (required)>'
from /usr/local/var/rbenv/versions/2.3.3/bin/proxy_fetcher:22:in load' from /usr/local/var/rbenv/versions/2.3.3/bin/proxy_fetcher:22:in

From what I see this gem supports only proxies that don't require Authorization header to be set:

    def initialize(proxy_addr, proxy_port)
      timeout = ProxyFetcher.config.proxy_validation_timeout

      @http = HTTP.follow.via(proxy_addr, proxy_port.to_i).timeout(connect: timeout, read: timeout)
    end

it would be nice if there was such possibility

ruby runner.rb                    
/Users/username/.rvm/rubies/ruby-2.4.0/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- proxy-fetcher (LoadError)
	from /Users/username/.rvm/rubies/ruby-2.4.0/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:55:in `require'
	from runner.rb:6:in `<main>'
[~/Desktop]  $ 

the code im trying:

require 'proxy-fetcher'
manager = ProxyFetcher::Manager.new
manager.raw_proxies

Hi -

While experimenting with the CLI interface, I found this bug:

ubuntu@a13bff005883:/opt/app$ proxy_fetcher --timeout 1
/usr/local/lib/ruby/gems/2.4.0/gems/proxy_fetcher-0.5.0/bin/proxy_fetcher:44:in `<top (required)>': undefined method `connection_timeout=' for #<ProxyFetcher::Configuration:0x0055ba1a9c99b8> (NoMethodError)
	from /usr/local/bin/proxy_fetcher:22:in `load'
	from /usr/local/bin/proxy_fetcher:22:in `<main>'

Indeed, there is no connection_timeout= method in lib/proxy_fetcher/configuration.rb

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler found conflicting requirements for the Ruby version:
  In Gemfile:
    Ruby (~> 2.3.8.0)

    nokogiri (~> 1.8) was resolved to 1.11.6, which depends on
      Ruby (>= 2.5.0)

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

Ruby version: 2.5.0
Gem version: 0.6.3

Code

manager = ProxyFetcher::Manager.new(refresh: false)
manager.cleanup!
manager.refresh_list!(maxtime: '4000')

manager.proxies.each do |proxy|
...
end

Error

ArgumentError:invalid value for Integer(): "08080"
.../proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/free_proxy_list.rb:28:in `Integer'
.../proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/free_proxy_list.rb:28:in `block in to_proxy'
.../proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/free_proxy_list.rb:26:in `tap'
.../proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/free_proxy_list.rb:26:in `to_proxy'
.../proxy_fetcher-0.6.3/lib/proxy_fetcher/providers/base.rb:10:in `block in fetch_proxies!'

Dependabot can't resolve your Ruby dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Bundler::VersionConflict with message: Bundler found conflicting requirements for the Ruby version:
  In Gemfile:
    Ruby (~> 2.3.8.0)

    nokogiri (~> 1.8) was resolved to 1.11.7, which depends on
      Ruby (>= 2.5.0)

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

manager = ProxyFetcher::Manager.new

Net::ReadTimeout: Net::ReadTimeout
from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/protocol.rb:158:in rescue in rbuf_fill' from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/protocol.rb:152:in rbuf_fill'
from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/protocol.rb:134:in readuntil' from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/protocol.rb:144:in readline'
from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http/response.rb:39:in read_status_line' from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http/response.rb:28:in read_new'
from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:1413:in block in transport_request' from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:1410:in catch'
from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:1410:in transport_request' from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:1383:in request'
from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:1376:in block in request' from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:853:in start'
from /Users/leal/.rvm/rubies/ruby-2.2.0/lib/ruby/2.2.0/net/http.rb:1374:in request' from /Users/leal/.rvm/gems/ruby-2.2.0/gems/proxy_fetcher-0.6.4/lib/proxy_fetcher/utils/http_client.rb:38:in fetch'
from /Users/leal/.rvm/gems/ruby-2.2.0/gems/proxy_fetcher-0.6.4/lib/proxy_fetcher/utils/http_client.rb:51:in fetch' from /Users/leal/.rvm/gems/ruby-2.2.0/gems/proxy_fetcher-0.6.4/lib/proxy_fetcher/providers/base.rb:37:in load_document'
... 10 levels...

I was working fine several days ago but today it's raising a exception.