Thomas Roca, Phd, Researcher and Data Officer @Agence Française de Développement

Stay in touch via Twitter & Github

Whether they are massive - big data - or more traditional (census, household surveys, administrative data, etc.), we are the witness of the explosion of the use of data for decision and policy making; first within the private sector then in the administration and lately at the crossroads between sectors and public affairs (e.g. Cambridge analytics, project google election, Facebook monitoring fake news etc.)

Algorithms now "make decisions" based on real time data. But these algorithms are mostly black boxes, parsing and computing data that are not open data (which providers have never been so centralized - GAFA, etc.). This raises societal and democratic challenges.

Are we entereing a Post-StatistiK world? What seems obvisous today is that data and statistics production are no longer a State monoply. What are the promises and challenges of the data revolution ?

Seminar organization: 1h30: 3x 20 mins presentations + 10 mins QA
Slide available in this folder: Big_Data_Public_Policy_columbia_univ_SIPA.ipynb, + link to nbviewer

Slide available in this folder: Dataviz_workshop_Columbia_Science_PO_SIPA.ipynb; + link to nbviewer

• The open movement
• Towards more accountable States
• More efficient administration ?
• Towards new data partenerships with in the private sector?
• What about Development Assistance ?

Conclusion: data is about people - A new data ecostystem - A new power distribution - A new ethic

• "Traditional" sources of data:
  • Census and Survey data
  • Administrative data
  • Africa's Statisticall Tragedy
• Big Data: when it's raining information:
  • Sensor data (Sat., IoT, Cell-phone)
  • Social network data
  • Use case:
    • vulnerability to flood using statellite imagery (use case: Cloud to Street /AFD study)
• Big Data: big difficulty to get access to it
  • Privacy, security, business
  • About Data ethics

• The history of the digital humanitarian movement
  • Haïti 2010
  • Crisis mapping (Humanitarian OpenStreet Map)
• Official statistics & Big Data in emergency context
  • When saving time saves life, real time data matters
    • Early assessment use case: - Using cell phone to monitor displacement (use case: Flowminder 2015 EarthQuake in Nepal)
    • Coordination of the emergency response (UN-OCHA HDX)

Conclusion: new tools, new skills and analytics strategy: AI everywhere ?

• Data Glossary - Data-Pop Alliance
• Mapping Data Sources for development
• Data-Pop Alliance, 2015, “Beyond Data Literacy: Reinventing Community Engagement and Empowerment in the Age of Data.” Data-Pop Alliance White Paper Series. Bhargava, R & al.
• Data-Pop Alliance, 2015, "Leveraging Algorithms for Positive Disruption: On data, democracy, society and statistics", Data-Pop Alliance White Paper Series, Letouzé, E., Sangokoya, D.
• Flowminder, 2016, Rapid and Near Real-Time Assessments of Population Displacement Using Mobile Phone Data Following Disasters: The 2015 Nepal Earthquake, Plos current disasters - Link
• Lanier, J. 2013, Who owns the future ?
• Meyer, P. 2015, How Big Data is Changing the Face of Humanitarian Response
• O'Neil, C. 2014, On Being a Data Skeptic, ebook-pdf
• O'Neil, C. 2016, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
• Roca, T. 2015, Toward data-driven development: Big improvement or big buzz?
• Roca, T. 2016, Open algorithms: A new paradigm for using private data for social good
• Schartum, D.W., 2016, Law and algorithms in the public domain, Nordic Journal of Applied Ethics n°1 2016
• Schwarz, B. & al. 2017, Socio-Physical Vulnerability To Flooding In Senegal, AFD, Cloud to Street, Link
• Schwarz, B., Roca, T. 2016, Data-driven preparedness for disaster, devex - link
• Slatalla, M., Quittner, J. 1995, Master of Decepetion, the Gang that Ruled the Cyberspace, HarperPerennial
• UN 2015, Data Revolution Report, A WORLD THAT COUNTS
• UN-OCHA, 2012, humanitarianism in the network age, Link
• UN-OCHA, 2014, Humanitarian Innovation:The State of the Art link
• World Bank, 2016 World Development Report: Digital Dividend

Part II. June the 2nd

This folder contains information and script for the dataviz workshop

For this workshop, we are going to use Python 3.4.3 and JavaScript. I recommend installing Python and a code editor (e.g. Bracket or Notepad++) we are also going to use https://jsfiddle.net/

Here the thematic this workshop will cover:

1. Basic instroduction: (1h30)

• using packages with R and Python (15 min)
• SIG using Carto and leaflet
• Basic introduction to HTML, CSS
• Introduction to SVG
• JavaScript & dataviz
  • quick intro to JavaScript
  • dataviz the example of highchart and d3.js

2. Getting real time data: the API revolution: (1h)

• Introduction: what is an API ?
  • Intro to JSON
• Example: Using twitter API in python
• Use cases:
  • An application leveraging World Bank API
  • Buidling an API based Velib' application
  • Scrapping the web with YQL(and twitter again)
  • example: wacky twitter race !

• Lupi, H., Posavec, S.,2016, "Dear Data", http://www.dear-data.com/
• Roca, T. 2014, "Web programming and dataviz with Stata" - Link
• Yau, N. 2011, visualize this: the flowingdata guide to design, visualization, and statistics
• https://www.w3schools.com
• https://www.highcharts.com/blog/