I am adding all of us but likely I will handle most of this myself.

ToDo List

The below is for when the repo is ready and all data has been updated.

• Make sure that the README docs are correct in all directories
• Update the web-based documentation (see this) which is quite outdated at this point.
  • Nice to have: Can we create a diagram of the infrastructure and the data processing pipeline? (use https://app.diagrams.net/)

• data model
• FIB-index function
• etc..

The logic from the two scripts listed below do not work.

• calc_crowdtangle_fib_all_months.sh
• calc_twitter_fib_all_months.sh

They are not crucial to the monthly logic so they can either:

1. Be removed
2. Be updated to loop through the directories that we have symlinks for
3. Be fixed some other way

It would be nice to have the convenience of these scripts in case something happens.

Currently, there are two scripts that are used to calculate the fib information.

They are:

• scripts/calc_crowdtangle_fib_indices.py
• scripts/calc_twitter_fib_indices.py

They can be merged into one file if we do the following:

• Update the argparse inputs to include:
  • Number of spreaders (currently hardcoded)
  • Type of spreader (currently hardcoded)
  • Platform
• Move both data extraction functions to the package/top_fibers_pkg/fib_helpers.py
  • Another option would be to combine them into one function that does different things depending on a platform flag (which could be taken from above) — this would create a very large function though...
• Other small things like: importing both data models, setting output files based on flags, etc.

Right now, the code works by ingesting all tweet objects from base tweets, retweets, and quote tweets. This means that, if something is retweeted from a very old post, it could have had a year or more to gain retweets. This is not necessarily a fair comparison so we may want to filter out tweets that are not originated during the time frame observed.

We want to be able to calculate the FIB indices for all months in 2021.

In order to do this, we will need to have one file for each month, going back three months from Jan '21. Thus, the script needs to be called 15 different times for each month of data.

• December (pull in January; offset -1)
• November (offset -1)
• October (offset -2)
• September (offset -3)
• August (offset -4)
• July (offset -5)
• June (offset -6)
• May (offset -7)
• April (offset -8)
• March (offset -9)
• February (offset -10)
• January (offset -11)
• December (offset -12)
• November (offset -13)
• October (offset -14)

I merged #35 despite requesting some small cosmetic changes. I did this to get the pipeline ready for the cronjob that is going to occur over the weekend.

When you have a chance, please create a new PR to address what I left in the comments of that PR.

Thanks!

As you can see in the most updated rendering of this page here — the date in the referenced line above is outdated.

Having the last updated date in two places is likely to lead to this issue. Use only the Last modified note from the YAML.

You can find the list of domains to use in data/iffy_files. Will probably be easier to do this after you merge #36 first (as the file is on that branch).

The data should be repulled for all months that we had previously. That is all of last year, up to the month prior to whatever is the previous month. As today is 4/28, that would be up to and including 2023-03. However, in two days, we are going to need the April data as well.

Please let me know if anything isn't clear. Thanks!

API requests page: https://www.facebook.com/help/contact/908993259530156

Once the project is finalized, do you think we need to add others to this variable? I know that many of the other projects send emails to the larger group email list and Fil gets updates.

I will let you decide how you'd like to manage this as it will ultimately become your project to manage after I am gone. That said, I would recommend adding at least one other person's email from the developers' team.

Also, I will add myself to keep an eye on this while I am still around.

• Create a clean version of the iffy list with https and wildcards removed
• Update the code so it loads the clean version and doesn't need to clean the domains on the fly

Start by pulling only Facebook data.

See #9 for a bit more details on incorporating Instagram.

This script will need to be updated to log its progress and any errors.

For the FB download, this could be built into the script.

For the Twitter pipeline, this will need to be thought through a bit..

Create a logging directory that has different subdirectories for each type of script running.

• data collection
• analysis
• etc

On the "Accounts" page, it looks like we can no longer click the user icon to visit an account's twitter/facebook page. I think that Fil had asked you to make the text no longer look like a hyperlink when we added the "Unfollow" button but I would still like to be able to click the icon to visit. So we would be able to visit both ways, via the unfollow button but also the icon. Can you please add that functionality back in when you get a chance?

Thanks!

Figure out how to automatically upload top FIBer files to zenodo.

Hi Pasan, can you please update the FAQ sections outlined below with the following text? These have changed since we've changed the list of domains slightly.

Thanks!

How do you define misinformation?

We adopt a common definition of misinformation utilized in academic research, which focuses on a source of information “that mimics news media content in form but not in organizational process or intent” (Lazer et al, Science, 2018). With this definition, we search for posts that contain at least one link to sources within a list that is curated by an independent third party, Iffy.news. Specifically, we include sources that have been marked by Media Bias Fact Check (MBFC) as having a "low" or "very-low" "MBFC Factual" score.

According to MBFC methodology, a source in these categories "rarely uses credible sources and is not trustworthy for reliable information" and "need[s] to be fact-checked for intentional fake news, conspiracy, and propaganda." Since a source's MBFC Factual score can change, we update our list of sources each month prior to releasing a new Top FIBers monthly report. These updates do not affect prior reports.

How do you collect your data?

Facebook

Facebook data are gathered using the CrowdTangle API . Specifically, we utilize the /posts/search/ endpoint. As a result of utilizing CrowdTangle, we are limited to collecting public posts (see the CrowdTangle documentation for more details).

Data for all months in 2022 as well as January--March of 2023 were gathered during the week of April 24, 2023. After that point, Facebook data are collected within the first week of the following month (depending on how long it takes for all data to download). For example, April 2023 data were collected during the first week of March 2023. We collect public posts linking to at least one of the low-credibility sources (see How do you define misinformation? for more details). please link the bold portion to the section above.

Twitter

Twitter data are collected with the enterprise-level Decahose endpoint. The Decahose delivers a 10% random sample of all tweets in real time. From this, we then collect all tweets that link to at least one of the credibility sources.

As Twitter's recent API changes have made continuing this data collection virtually impossible, we will no longer be able to continue analyzing Twitter's biggest superspreaders of misinformation.

Some ideas for account visualization:

• Top ten domains shared
• Top hashtags utilized
• Misinfo links shared per day
• Mean low-credibility tweets per day/week/etc

need to figure out a way to check for task success accurately.

Just noticed that the Contents section of this README does not include a line item for the profile_links.py file. Can you please add one? Thanks!

See: https://github.com/mr-devs/top-fibers/blob/2e74079ab1581037087f3d4b40b5851375bd368e/docs/documentation.md

Towards the end there is a typo of the word "repository" that needs to be fixed.

We need to update the master bash script so that it sends an email to Nick when something fails.

We have already added Nick's email to the crontab file using the MAILTO variable but need to make all of the exit lines return 1.

I will take care of this.

Technically, we will have the ability to pull Instagram data as well.

Decide if we want to do this. If we do, I think it would require rewriting a lot of the code...

It looks like there is some documentation for the package here but it can not be reached on the page as nothing points to it.