Field coverage always 100% about spidermon HOT 5 CLOSED

Hello, spidermon calculates coverage based on the fields existing on the item. If you are using a item class that always defines all their fields with some default value like None, "". Alternatively, items like dict and scrapy.Items should work fine.

from spidermon.

regardless of whether the item field is empty or not.

Yes, there is a difference between non-empty and non-existent, and coverage is calculated based on the latter.

from spidermon.

Thank you for the clarification. If I do not load the Item with the variable at all I still get no spidermon_item_scraped_count for the variable so I am a little unsure what is "triggering" these counts (since neither non-existent nor empty values seem to be counted). Could I bother you with briefly explaining what would be defined as 'non-existent' in this case? It would be very useful to have item coverage working...
Thanks!

from spidermon.

If I do not load the Item with the variable at all I still get no spidermon_item_scraped_count for the variable

Which is good?

I am a little unsure what is "triggering" these counts (since neither non-existent nor empty values seem to be counted).

Per your original report empty values were counted while non-existent ones were not, which is the expected behavior.

Could I bother you with briefly explaining what would be defined as 'non-existent' in this case?

Non-existent means the item doesn't have the field at all. Note that setting it to None is the same as not setting it at all, as None values are ignored by ItemLoader, not sure if it's related to your questions though.

from spidermon.

Thank you for the clarification, I got it working!

The thing was that my pipeline errored out with a key error if I did not specify a value. For this reason I was passing None values where the value were not found. Once I understood that instead of doing (in the pipeline)

thing.variable = item['variable']

I could do
thing.variable = item.get('variable')

The pipeline allowed non-existent values and the item coverage works as you describe.

from spidermon.