Removed

The search results in Pubmed are output successfully, but the search results in Google scholar are not output.

I was searching for the human gene symbol "USPL1" in Pubmed and Google scholar, and I got the following error in Google scholar.

% python3 scrappaper.py

Initiating... please wait.

Please paste search URL and press Enter:https://scholar.google.com/scholar?as_ylo=2023&q=%22USPL1%22&hl=en&as_sdt=0,5
Input is from: Google Scholar.

Waiting for a few secs...
Waiting done. Continuing...

Traceback (most recent call last):
  File "/Users/file/to/path/scrappaper.py", line 167, in <module>
    search_results = soup.find_all("div", class_="gs_ab_mdw")[1].text
IndexError: list index out of range

Following the description in the article, I searched as follows;

I ran the git clone repeatedly several times and it succeeded only once, but after that I get the above error.

How should I respond ?

Thank you！

Thank you for developing such a great python script. This truly stands out over the other 10 I tried - finally a script that actually searches for direct keywords within the search results.

There are a couple small things I ran into, that if modified would add monumental additional power and abilities to the script. Currently, it's nearly impossible to complete running the script when its parsing any searches that have more than 49 pages results, due to Captcha. Or if I realize result pages start to have irrelevant studies and I abort, it breaks the csv output process.

1. A simple solution: A way to use a simple parameter input that pre-defines how many search pages to parse.
   Example to parse only the first 25 search pages: (-s = search pages) python scrappaper.py -s 25

2. Allow the script to receive a new URL that starts on for example the 50th results page. So i can switch VPN IP and continue from the point where it stopped due to Capcha. Currently it still restarts from the 1st results page even if the URL points to the 50th results page.

3. Have the script automatically name the output csv file with the keywords used in the google scholar/pubmed search with a date and time stamp. That way its clear which CSV file has which information and it doesn't keep overwriting the same output file.

One other suggestion, but less important:

4. To prevent losing the csv output if a user aborts the process. Create another parameter input (-w = write _to_file) that has the app write_to_file and append the csv file after each search page is completed. This will allow the user to abort the process if current pages have irrelevant data, without breaking the CSV output for the articles already parsed.

Thanks in advance, I'm sure many users would benefit greatly from these small updates.