lorae / roundup Goto Github PK

appears to now have issues collecting author names from website

Script appears to not be collecting data. Issue may relate to Chromium driver.

The following text appeared incorrectly on the StreamLit app:

"Two in five Americans have medical debt, nearly half of whom owe at least $2,500. Concerned by this burden, governments and private donors have undertaken large, high-profile efforts to relieve medical debt. We partnered with RIP Medical Debt to conduct two randomized experiments that relieved medical debt with a face value of $169 million for 83,401 people between 2018 and 2020. We track outcomes using credit reports, collections account data, and a multimodal survey. There are three sets of results. First, we find no impact of debt relief on credit access, utilization, and financial distress on average. Second, we estimate that debt relief causes a moderate but statistically significant reduction in payment of existing medical bills. Third, we find no effect of medical debt relief on mental health on average, with detrimental effects for some groups in pre-registered heterogeneity analysis."

Something is happening to make it italicized unintentionally.

does not have any 2024 entries yet

Hasn't found a new working paper since Sept 2023

Currently, there are three lists of scraper IDs, each contained in compare.py, streamlit_app.py, or runall.py, which look like the following:

source_order = ['NBER', 'FED-BOARD', 'FED-BOARD-NOTES', 'FED-ATLANTA', 'FED-BOSTON', 'FED-CHICAGO', 'FED-CLEVELAND', 'FED-DALLAS', 'FED-KANSASCITY', 'FED-NEWYORK', 'FED-PHILADELPHIA', 'FED-RICHMOND', 'FED-SANFRANCISCO', 'FED-STLOUIS', 'BEA', 'BFI', 'BIS', 'BOE', 'ECB', 'IMF']

Integrating a new scraper module in the project involves updating all three lists, which is unintuitive and prone to producing bugs.

Resolving this issue would involve creating a global order of sources that can be called by all 3 scripts.

I suspect it has to do with Selenium.

It appears to be cutting date string off - e.g. "Fri, 19 Jan 2" rather than "Fri, 19 Jan 2024"

Change the way entries display so that hyperlinked title is above a posted pubdate and estimate pubdate, authors, and abstract.

Fed_Boston.py is not able to navigate to 2024 publications. Adjust the web scraper accordingly. Explore API options.

Abstract data Xpath index is not working. Revisit this script and potentially use an API instead

Should be based on whether the current month is January.

Current scrapers have a mix of logic, but have been migrating toward always scraping the current and former year's entries. This may mean overuse of computing resources.