Proxy setup. about crawly HOT 6 CLOSED

You will need to add in request options for HTTPoison through the RequestOptions request middleware . The format needs to match the option format in the HTTPoison docs

https://hexdocs.pm/crawly/Crawly.Middlewares.RequestOptions.html

https://hexdocs.pm/httpoison/HTTPoison.Request.html#content

from crawly.

@Unumus did you search through the docs before opening this issue? If you did but didnt' find the RequestOptions middleware, we might need to update the docs for it

from crawly.

@Ziinc I was also thinking about making an explicit example somewhere in docs...

from crawly.

@Ziinc, I searched. But not found. I think it will be helpful to mention here: https://hexdocs.pm/crawly/configuration.html#proxy-binary the middleware.requests to complete proxy configuration.

from crawly.

@Unumus ah I see, the docs got out of sync, the proxy setting on :config is from an older version that is a breaking change. We'll need to update that.

@oltarasenko I think perhaps a few common use case examples can be shown in the middleware module doc? Besides proxy, what others would be good?

• remove proxy setting from config.

from crawly.

@oltarasenko You might need to do a patch release to update hexdocs

from crawly.