Stop Iteration Error while using run_in_reactor about crochet HOT 6 CLOSED

Can you please show the code for def _crawler_Stop() ? And also my imagePipeline and deltafetch middleware are also not working??

from crochet.

Solved The Issue Thanks i was passing Crawlrunner object as Parameter, as the function stop. The Object gets destroyed. This leads to Stop Iteration Error I guess

from crochet.

Hi, i have the same error occuring when i run a spider within a flask app but i don't get a response from the spider
i am using flask, crochet(1.12.0), scrapyd, twisted(19.10.0)

from crochet.

Solved The Issue Thanks i was passing Crawlrunner object as Parameter, as the function stop. The Object gets destroyed. This leads to Stop Iteration Error I guess

I am using the same method and facing same issue.
how did you resolve it. any help would be great

from crochet.

Hello my brother I have the same issue.

from crochet.

The Reactor Code look like this, which call upon GeneralSpider which fetches all the necessary data based on the pk as primary key.

runner = CrawlerRunner()
output = []
@run_in_reactor
def scraped_by_id(pk):
    global runner
    dispatcher.connect(_crawler_result, signal=signals.item_scraped)
    eventual = runner.crawl(GeneralSpider, id=pk)
    dispatcher.connect(_crawler_Stop, signals.engine_stopped)
    return eventual


def _crawler_result(item, response, spider):
    global output_data
    output_data.append(dict(item))


def _crawler_Stop():
    # Perform any operation which is relevant to your application, like notify user
    pass

And my GeneralSpider class look like this. Where Job contains info like url to scrape, fields to scrape, etc.

class GeneralSpider(scrapy.Spider):
    def __init__(self, id, **kwargs):
        self.id = id
        self.custom_settings = {"LOG_LEVEL": "DEBUG"}

        job = Job.objects.get(pk=id)

Hope this helps @MarwaKH29 @HimavarshaVS @dzhen19

from crochet.