mr-devs / py_misinfo_exposure Goto Github PK

The block of code in get_misinfo_exposure_score starting at line 172 can be removed by building the defaultdict(list) directly within the _get_users_data function itself.

Instead of building results as a list of tuples, we can just build the defaultdict(list) as we get data from Twitter.

Currently, the package works by processing all users provided and - once it is done - converts everything to a dataframe that is then saved all at once.

It would be good if the function could smoothly handle keyboard interruptions without losing the data it has already gathered. This should be simple by using a try/except KeyboardInterrupt block.

Include more details in the rate-limits and more control section of the README about save_friends_to_disk and output_dir, etc.

Should we run get_users_misinfo_exposure_scores.py for a very large set of users - it will crash at some point because it requires that all of the gathered data is held in memory at once.

A solution to this problem would be to redesign things so that they save individual files for each queried user that contain all of their friends and their user IDs. That way we would never have to hold more than one users total number of friends at once and we could (after all friends have been gathered) iterate through the list of files, load them individually, calculate their score, store it, rinse, and repeat.

Create a command-line script that that takes in a file of user IDs and outputs a file with the results.

Potential arguments:

• -u/--user-file : the input file of user IDs
• -v/--verbose : boolean to indicate whether or not to print updates
  • --updates : only works with -v and indicates when to provide updates
• -o/--output-file: name of the output file