The UNICEF AI4D Relative Wealth Project aims to develop open datasets and machine learning (ML) models for poverty mapping estimation across nine countries in Southeast Asia (SEA).
We also aim to open source all the scripts, experiments and other artifacts used for developing these datasets and models in order to allow others to replicate our work as well as to collaborate and extend our work for their own use cases.
This project is part of Thinking Machines's overall push for open science through the AI4D (AI for Development) Research Bank which aims to accelerate the development and adoption of effective machine learning (ML) models for development across Southeast Asia.
Documentation geared towards our methodology and experiments can be found here.
Our final trained models and their use to produce nationwide estimates can replicated through our notebooks, assuming you've followed the Data and Local Development setup below.
-
For countries with available DHS training data (Cambodia, Myanmar, Philippines, and Timor-Leste), please refer to the notebooks here: https://github.com/thinkingmachines/unicef-ai4d-poverty-mapping/tree/main/notebooks/2023-02-21-single-country-rollouts
-
For the other countries without DHS training data (Indonesia, Laos, Malaysia, Thailand, and Vietnam), please refer to the notebooks here: https://github.com/thinkingmachines/unicef-ai4d-poverty-mapping/tree/main/notebooks/2023-02-21-cross-country-rollouts
All the output files (models, datasets, intermediate files) can all be downloaded from here.
Due to the sensitive nature of the data and the DHS program terms of use, we cannot provide the raw DHS data used in our experiments.
You will have to request for access to raw data yourself on the DHS website.
Generally, for all the experiment notebooks in this repo, they assume that the DHS Stata and Shape zip files contents are unzipped to its own folder under data/dhs/<iso-country-code>/ where the <iso-country-code> is the two-letter ISO country code.
For example, from the data for the Philippines will have this directory structure:
data/
dhs/
ph/
PHGE71FL/
DHS_README.txt
GPS_Displacement_README.txt
PHGE71FL.cpg
PHGE71FL.dbf
PHGE71FL.prj
PHGE71FL.sbn
PHGE71FL.sbx
PHGE71FL.shp
PHGE71FL.shp.xml
PHGE71FL.shx
PHHR71DT/
PHHR71FL.DCT
PHHR71FL.DO
PHHR71FL.DTA
PHHR71FL.FRQ
PHHR71FL.FRW
PHHR71FL.MAP
If you create your own notebook, of course you are free to modify these conventions for filepaths yourself. But out-of-the-box, this is what our notebooks assume.
The only other data access requirement is for the EOG Nightlights Data which requires registering for an account. The notebooks require the use of these credentials (user name and password) to download the nightlights data automatically.
All the other datasets used in this project are publically available and the notebooks provide the code necessary to automatically download and cache the data.
Due to the size of the datasets, please make sure you have enough disk space (minimum 40GB-50GB) to accommodate all the data used in building the models.
This repo assumes the use of miniconda for simplicity in installing GDAL.
- Python 3.9
- make
- miniconda
Run this the very first time you are setting-up the project on a machine to set-up a local Python environment for this project.
- Install miniconda for your environment if you don't have it yet.
wget "https://repo.anaconda.com/miniconda/Miniconda3-latest-$(uname)-$(uname -m).sh"
bash Miniconda3-latest-$(uname)-$(uname -m).sh- Create a local conda env and activate it. This will create a conda env folder in your project directory.
make conda-env
conda activate ./env
- Run the one-time set-up make command.
make setup
- To test if the setup was successful, run the tests. You should get a message that all the tests passed.
make test
At this point, you should be ready to run all the existing notebooks on your local.
Over the course of development, you will likely introduce new library dependencies. This repo uses pip-tools to manage the python dependencies.
There are two main files involved:
requirements.in- contains high level requirements; this is what we should edit when adding/removing librariesrequirements.txt- contains exact list of python libraries (including depdenencies of the main libraries) your environment needs to follow to run the repo code; compiled fromrequirements.in
When you add new python libs, please do the ff:
-
Add the library to the
requirements.infile. You may optionally pin the version if you need a particular version of the library. -
Run
make requirementsto compile a new version of therequirements.txtfile and update your python env. -
Commit both the
requirements.inandrequirements.txtfiles so other devs can get the updated list of project requirements.
Note: When you are the one updating your python env to follow library changes from other devs (reflected through an updated
requirements.txtfile), simply runpip-sync requirements.txt
We are using Quarto to maintain the Unicef AI4D Relative Wealth documentation site.
Here are some quick tips to running quarto/updating the doc site, assuming you're on Linux.
For other platforms, please refer to Quarto's website.
- Download:
wget https://github.com/quarto-dev/quarto-cli/releases/download/v1.2.247/quarto-1.2.247-linux-amd64.deb
- Install:
sudo dpkg -i quarto-1.2.247-linux-amd64.deb
- Preview the site locally (view in http://localhost:4444) :
quarto preview --port 4444 --no-browser
- Update the site (must have maintainer role):
quarto publish gh-pages --no-browser
- Pro-tip : If you are using VS Code as your code editor, install the Quarto extension to make editing/previewing the doc site a lot smoother.
We have created a docker image (ghcr.io/butchtm/povmap-jupyter) of the poverty mapping repo for those who want to view the notebooks or rollout the models for new countries and new data (e.g. new nightlights and ookla years)
To run these docker images please copy and paste the following scripts to run on your linux, mac or windows (wsl) terminals:
- View Jupyter notebooks (Read-only) This will run a jupyter notebook environment containing the poverty mapping notebooks at http://localhost:8888/lab/tree/notebooks
curl -s https://raw.githubusercontent.com/thinkingmachines/unicef-ai4d-poverty-mapping/main/localscripts/run-povmap-jupyter-notebook.sh > run-povmap-jupyter-notebook.sh && \
chmod +x run-povmap-jupyter-notebook.sh && \
./run-povmap-jupyter-notebook.sh- Country-wide rollout This will run an interactive dialog that will rollout the poverty mapping models for different countries and different time periods
curl -s https://raw.githubusercontent.com/thinkingmachines/unicef-ai4d-poverty-mapping/main/localscripts/run-povmap-rollout.sh > run-povmap-rollout.sh && \
chmod +x run-povmap-rollout.sh && \
./run-povmap-rollout.sh- Copy rollout to local directory This will copy the contents of the rollout notebooks and rollout data into your current directory (after running a new rollout) to
rollout-dataandrollout-output-notebooks
curl -s https://raw.githubusercontent.com/thinkingmachines/unicef-ai4d-poverty-mapping/main/localscripts/copy-rollout-to-local.sh > copy-rollout-to-local.sh && \
chmod +x copy-rollout-to-local.sh && \
./copy-rollout-to-local.shNote: These commands assume that
curlis installed and will download the scripts, change their permissions to executable as well as run them. After the initial download, you can just rerun the scripts which would would have been downloaded to your current directory.
Note: The scripts create and use a docker volume named
povmap-datawhich contains the outputs as well as caches the data used for generating the features from public datasets
Note: Rolling out the notebooks requires downloading EOG nightlights data so a user id and password are required as detailed in the previous section above.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent