Support restricting the coverage monitor fields about spidermon HOT 4 CLOSED

The alternative of SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS looks promising in my point of view. We can set the "depth" where we want to check the coverage, so with an item like:

{
    "field_1": "some_value",
    "field_2": "some_value",
    "field_3": {
        "field_3_1": "some_value",
        "field_3_2": "some_value",
    }
}

If SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS = 1, so the stats will be like:

{
    "spidermon_field_coverage/dict/field1/": "some_value",
    "spidermon_field_coverage/dict/field2/": "other_value",
    "spidermon_field_coverage/dict/field3/": "other_value",
}

If SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS = 2, so the stats will be like:

{
    "spidermon_field_coverage/dict/field1/": "some_value",
    "spidermon_field_coverage/dict/field2/": "other_value",
    "spidermon_field_coverage/dict/field3/": "other_value",
   "spidermon_field_coverage/dict/field3/field_3_1": "other_value",
   "spidermon_field_coverage/dict/field3/field_3_2": "other_value",
}

What do you think???

from spidermon.

@curita

Skipping the coverage of certain patterns may not solve the problem, considering that for data not well structured, we may get keys unexpected from our targets. Regarding the stats limit in Scrapy Cloud, maybe changing what HubStorageStatsCollector uploads as stats is a more reliable alternative and is specific to the platform, so we don't introduce new Zyte specific code to Spidermon.

from spidermon.

I like the depth approach the most 👍 Mauricio implemented SPIDERMON_DICT_FIELDS_COVERAGE_LEVELS as a separate setting in his PR, though both SPIDERMON_LIST_FIELDS_COVERAGE_LEVELS and SPIDERMON_DICT_FIELDS_COVERAGE_LEVELS could be unified into a single setting if we consider so.

from spidermon.

Fixed by #433

from spidermon.