A curated list of awesome academic research, books, code of ethics, newsletters, principles, podcast, reports, tools, regulations and standards related to Responsible AI and Human-Centered AI.

• Academic Research
• Books
• Code of Ethics
• Data Sets
• Institutes
• Newsletters
• Principles
• Podcasts
• Reports
• Tools
• Regulations
• Standards

• Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (Schwartz, Reva et al., 2022) NIST

• Underspecification presents challenges for credibility in modern machine learning. (D'AMOUR, Alexander, et al., 2020) Google

• FreaAI: Automated extraction of data slices to test machine learning models (Ackerman, S. et al. 2021) IBM
• Machine Learning Model Drift Detection Via Weak Data Slices (Ackerman, S. et al. 2021) IBM

• Efficient Data Representation by Selecting Prototypes with Importance Weights (Gurumoorthy et al., 2019) Amazon Development Center IBM Research
• Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives (Dhurandhar et al., 2018) University of Michigan IBM Research
• Contrastive Explanations Method with Monotonic Attribute Functions (Luss et al., 2019)
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier (LIME) (Ribeiro et al. 2016, Github) University of Washington
• A Unified Approach to Interpreting Model Predictions (SHAP) (Lundberg, et al. 2017, Github) University of Washington
• Teaching AI to Explain its Decisions (Hind et al., 2019) IBM Research
• Boolean Decision Rules via Column Generation (Light Edition) (Dash et al., 2018) IBM Research
• Generalized Linear Rule Models (Wei et al., 2019) IBM Research
• Improving Simple Models with Confidence Profiles (Dhurandhar et al., 2018) IBM Research
• Towards Robust Interpretability with Self-Explaining Neural Networks (Alvarez-Melis et al., 2018) MIT
• Leveraging Latent Features for Local Explanations (Luss et al., 2019) IBM Research University of Michigan

• Caton, S., & Haas, C. (2020). Fairness in machine learning: A survey. ACM Computing Surveys
• LiFT: A Scalable Framework for Measuring Fairness in ML Applications (Vasuvedan et al., 2020) LinkedIn

• Building Inclusive Products Through A/B Testing (Saint-Jacques et al, 2020) LinkedIn

• Energy and policy considerations for deep learning in NLP (Strubell, E. et al. 2019)
• Quantifying the carbon emissions of machine learning. (Lacoste, A. et al. 2019)
• Carbon emissions and large neural network training. (Patterson, D. et al. 2021)
• The Energy and Carbon Footprint of Training End-to-End Speech Recognizers. (Parcollet, T., & Ravanelli, M. 2021)
• Sustainable AI: AI for sustainability and the sustainability of AI (van Wynsberghe, A. 2021). AI and Ethics, 1-6
• Green Algorithms: Quantifying the carbon emissions of computation (Lannelongue, L. et al. 2020)
• Machine Learning: The High Interest Credit Card of Technical Debt (Sculley, D. et al. 2014) Google

• Google Research on Responsible AI: https://research.google/pubs/?collection=responsible-ai Google
• Pipeline-Aware Fairness: http://fairpipe.dssg.io

• Papers with Code
• Papers without Code

• Interpretable Machine Learning (Molnar, C., 2021) Explainability Interpretability Transparency R
• Explanatory Model Analysis (Biecek et al., 2020) Explainability Interpretability Transparency R

• Trust in Machine Learning (Varshney, K., 2022) Safety Privacy Drift Fairness Interpretability Explainability
• Interpretable AI (Thampi, A., 2022) Explainability Fairness Interpretability
• AI Fairness (Mahoney, T., Varshney, K.R., Hind, M., 2020 Report Fairness
• Practical Fairness (Nielsen, A., 2021) Fairness
• Hands-On Explainable AI (XAI) with Python (Rothman, D., 2020) Explainability
• AI and the Law (Kilroy, K., 2021) Report Trust Law
• Responsible Machine Learning (Hall, P., Gill, N., Cox, B., 2020) Report Law Compliance Safety Privacy
• Privacy-Preserving Machine Learning
• Human-In-The-Loop Machine Learning: Active Learning and Annotation for Human-Centered AI
• Interpretable Machine Learning With Python: Learn to Build Interpretable High-Performance Models With Hands-On Real-World Examples
• Responsible AI (Hall, P., Chowdhury, R., 2023) Governance Safety Drift

• ACS Code of Professional Conduct by Australian ICT (Information and Communication Technology)
• AI Standards Hub
• Association for Computer Machinery's Code of Ethics and Professional Conduct
• IEEE Global Initiative for Ethical Considerations in Artificial Intelligence (AI) and Autonomous Systems (AS)
• ISO/IEC's Standards for Artificial Intelligence

• An ImageNet replacement for self-supervised pretraining without humans
• Huggingface Data Sets

• Ada Lovelace Institute United Kingdom
• European Centre for Algorithmic Transparency
• Center for Responsible AI
• Montreal AI Ethics Institute
• Munich Center for Technology in Society (IEAI) Germany
• National AI Centre's Responsible AI Network Australia
• Open Data Institute United Kingdom
• Stanford University Human-Centered Artificial Intelligence (HAI) United States of America
• The Institute for Ethical AI & Machine Learning
• University of Oxford Institute for Ethics in AI United Kingdom

• Import AI
• The AI Ethics Brief
• The Machine Learning Engineer

• Allianz's Principles for a responsible usage of AI Allianz
• Asilomar AI principles
• European Commission's Guidelines for Trustworthy AI
• Google's AI Principles Google
• IEEE's Ethically Aligned Design IEEE
• Microsoft's AI principles Microsoft
• OECD's AI principles OECD
• Telefonica's AI principles Telefonica
• The Institute for Ethical AI & Machine Learning: The Responsible Machine Learning Principles

Additional:

• FAIR Principles Findability Accessibility Interoperability Reuse

• The Human-Centered AI Podcast
• Responsible AI Podcast
• Trustworthy AI

• AI Incident Database
• Inferring Concept Drift Without Labeled Data, 2021 Drift
• Interpretability, Fast Forward Labs, 2020 Interpretability
• ML Commons Safety Benchmark for general purpose AI chat model
• State of AI - from 2018 up to now -

• balance Python Facebook

• CausalAI Python Salesforce
• CausalNex Python
• CausalImpact R
• Causalinference Python
• CIMTx: Causal Inference for Multiple Treatments with a Binary Outcome R
• dagitty R
• DoWhy Python Microsoft
• mediation: Causal Mediation Analysis R
• MRPC R

• BackPACK Python
• DataSynthesizer: Privacy-Preserving Synthetic Datasets Python Drexel University University of Washington
• diffpriv R
• Diffprivlib Python IBM
• Discrete Gaussian for Differential Privacy Python IBM
• Opacus Python Facebook
• PyVacy: Privacy Algorithms for PyTorch Python
• SEAL Python Microsoft
• SmartNoise Python OpenDP
• Tensorflow Privacy Python Google

• Alibi Detect Python
• Deepchecks Python
• drifter R
• Evidently Python
• nannyML Python
• phoenix Python

• Aequitas' Bias & Fairness Audit Toolkit Python
• AI360 Toolkit Python R IBM
• EDFfair: Explicitly Deweighted Features R
• Fairlearn Python Microsoft
• Fairmodels R University of California
• fairness R
• FairRankTune Python
• FairPAN - Fair Predictive Adversarial Network R
• Themis ML Python
• What-If Tool Python Google

• Mercury Python BBVA
• Deepchecks Python

• AI360 Toolkit Python R IBM
• aorsf: Accelerated Oblique Random Survival Forests R
• breakDown: Model Agnostic Explainers for Individual Predictions R
• captum Python PyTorch
• ceterisParibus: Ceteris Paribus Profiles R
• DALEX: moDel Agnostic Language for Exploration and eXplanation Python R
• DALEXtra: extension for DALEX Python R
• ecco article Python
• eli5 Python
• eXplainability Toolbox Python
• ExplainerHub in github Python
• fasttreeshap Python LinkedIn
• FAT Forensics Python
• flashlight R
• Human Learn Python
• hstats R
• innvestigate Python Neural Networks
• intepretML Python
• interactions: Comprehensive, User-Friendly Toolkit for Probing Interactions R
• kernelshap: Kernel SHAP R
• lime: Local Interpretable Model-Agnostic Explanations R
• Network Dissection Python Neural Networks MIT
• Shap Python
• Shapash Python
• shapviz R
• Skater Python Oracle
• survex R
• TCAV (Testing with Concept Activation Vectors) Python
• truelens Python Truera
• truelens-eval Python Truera
• pre: Prediction Rule Ensembles R
• Vetiver R Python Posit
• vivid R
• XAI - An eXplainability toolbox for machine learning Python The Institute for Ethical Machine Learning
• Zennit Python

• auditor R
• automl: Deep Learning with Metaheuristic R
• AutoKeras Python
• Auto-Sklearn Python
• DataPerf Python Google
• deepchecks Python
• EloML R
• Featuretools Python
• LOFO Importance Python
• forester R
• metrica: Prediction performance metrics R
• NNI: Neural Network Intelligence Python Microsoft
• performance R
• TensorFlow Model Remediation Python Google
• TPOT Python
• Unleash Python
• Yellowbrick Python
• WeightWatcher Python

• Nightshade University of Chicago
• Glaze University of Chicago

• Dr. Why R Warsaw University of Technology
• Responsible AI Widgets R Microsoft
• The Data Cards Playbook Python Google

• Modelscan Python
• NB Defense Python
• Rebuff Playground Python

• Code Carbon Python
• Azure Sustainability Calculator Microsoft
• Computer Progress

• Data Protection Laws of the Word
• SCL Artificial Intelligence Contractual Clauses

• General Data Protection Regulation GDPR - Legal text for the EU GDPR regulation 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC
• GDPR.EU Guide - A project co-funded by the Horizon 2020 Framework programme of the EU which provides a resource for organisations and individuals researching GDPR, including a library of straightforward and up-to-date information to help organisations achieve GDPR compliance (Legal Text).
• AI ACT

• State consumer privacy laws: California (CCPA and its amendment, CPRA), Virginia (VCDPA), and Colorado (ColoPA).
• Specific and limited privacy data laws: HIPAA, FCRA, FERPA, GLBA, ECPA, COPPA, VPPA and FTC.
• EU-U.S. and Swiss-U.S. Privacy Shield Frameworks - The EU-U.S. and Swiss-U.S. Privacy Shield Frameworks were designed by the U.S. Department of Commerce and the European Commission and Swiss Administration to provide companies on both sides of the Atlantic with a mechanism to comply with data protection requirements when transferring personal data from the European Union and Switzerland to the United States in support of transatlantic commerce.
• Executive Order on Maintaining American Leadership in AI - Official mandate by the President of the US to Privacy Act of 1974 - The privacy act of 1974 which establishes a code of fair information practices that governs the collection, maintenance, use and dissemination of information about individuals that is maintained in systems of records by federal agencies.
• Privacy Protection Act of 1980 - The Privacy Protection Act of 1980 protects journalists from being required to turn over to law enforcement any work product and documentary materials, including sources, before it is disseminated to the public.
• AI Bill of Rights - The Blueprint for an AI Bill of Rights is a guide for a society that protects all people from IA threats based on five principles: Safe and Effective Systems, Algorithmic Discrimination Protections, Data Privacy, Notice and Explanation, and Human Alternatives, Consideration, and Fallback.

What are standards?

Standards are voluntary, consensus soluctions. They document an agreement on how a material, product, process, or serice should be specified, performed or delivered. They keep people safe and ensure things work. They create confidence and provide security for investment.

• NIST AI Risk Management Framework

Additional standards can be found using the Standards Database.