INFO: How to scrape a API using POST method calls about crawly HOT 6 CLOSED

Fetcher docs: https://hexdocs.pm/crawly/Crawly.Fetchers.Fetcher.html#content

Example fetcher: https://github.com/oltarasenko/crawly/blob/master/lib/crawly/fetchers/httpoison_fetcher.ex

override_settings callback: https://hexdocs.pm/crawly/Crawly.Spider.html#c:override_settings/0

override_settings example: https://github.com/oltarasenko/crawly/blob/7bf04b0f23e02c9bc7c5b12dc962efb25ca88371/test/settings_test.exs#L47

from crawly.

You can create a custom fetcher that uses the Poison.post function and then override settings in the spider.

from crawly.

Thanks for the response as I am new to Elixir and never used crawly would you happen to have a sample. I can there are plenty of samples around scrapping website but none covering this use case.

from crawly.

@Ziinc thanks for helping!

@ahsandar But why do you need to use POST requests? Otherwise, if it's about fetching ajax parts of your page, you could try to use the browser rendering approach!

from crawly.

@oltarasenko i am looking for fetching data from APIs not in websites which could be using POST. As I want to use same tool to consistently use to collect data from websites and APIs and Crawly looks to be a great tool to achieve this

from crawly.

I would assume this is resolved now. Please re-open if it's not the case!

from crawly.