Filter data based on year about comcrawl HOT 3 CLOSED

Hi Michael, I had been looking for historical data for chocolates from websites like bigbasket and Amazon. However, when I select data for 2019-2020, I get very few records for say 'cadbury' brand. (as few as 25 records for the entire brand not even product wise). Are all the historical records not maintained well in commoncrawl? I tried similar for Amazon for 'Hershey' brand, still no luck Below is my attempt in getting the data. Can you please help if possible if anything is getting wrong in my understanding? from comcrawl import IndexClient import pandas as pd from bs4 import BeautifulSoup import re from comcrawl.utils import fetch_available_indexes indexes = [index for index in fetch_available_indexes() if re.match("2019|2020", index)] client = IndexClient(indexes) client.search('https://www.bigbasket.com/pd/*', threads=2) client.results client.results = [res for res in client.results if res['status'] == '200'] client.results ecom_df = pd.DataFrame(client.results) sorted_results_df = ecom_df.sort_values(by="timestamp") filtered_results_df = (sorted_results_df.drop_duplicates("urlkey", keep="last")) chocolate_df = filtered_results_df[filtered_results_df['url'].str.contains("cadbury")] client.results = chocolate_df.to_dict("records") client.download() c_df = pd.DataFrame(client.results) c_df.to_csv('chocolate_df.csv', index=False) Regards, Prachi