Author: Amy Wooding
Welcome to the easydata-tutorial repo! If you're attending the Pydata Global Tutorial, "Love your (data science) Neighbour: Reproducible Data Science the Easydata Way", you're in the right place!
You're about to embark on the Easydata Quest for Reproducibility. In preparation, you'll need to get your tools ready. In particular, you will need to have the following basic requirements installed on your machine:
- conda >= 4.8 (via Anaconda or Miniconda)
In addition to conda, you will also need
- GNU make
- git
These can be installed using conda if they are not already on your system; e.g. via
conda create -n easydata python=3 make git
conda activate easydata
Windows installations can be especially gnarly. In that case, we have Windows specific setup instructions.
We are going to use git to clone this repo:
git clone https://github.com/acwooding/easydata-tutorial
cd easydata-tutorial
To test if you're ready, make sure you have the dependencies installed (e.g. activate the easydata
environment you created above) and run:
python quest/am_i_ready.py
(If you get a SyntaxError
, make sure you are using at least python 3.6!)
This git repository is build from the Easydata framework, which aims to make your data science workflow reproducible. The Easydata framework includes:
- tools for managing conda environments in a consistent and reproducible way,
- built-in dataset management (including tracking of metadata such as LICENSES and READMEs),
- a prescribed project directory structure,
- workflows and conventions for contributing notebooks and other code.
For more on Easydata see Getting Started.
LICENSE
Makefile
- Top-level makefile. Type
make
for a list of valid commands.
- Top-level makefile. Type
Makefile.include
- Global includes for makefile routines. Included by
Makefile
.
- Global includes for makefile routines. Included by
Makefile.env
- Command for maintaining reproducible conda environment. Included by
Makefile
.
- Command for maintaining reproducible conda environment. Included by
README.md
- this file
catalog
- Data catalog. This is where config information such as data sources and data transformations are saved.
catalog/config.ini
- Local Data Store. This configuration file is for local data only, and is never checked into the repo.
data
- Data directory. Often symlinked to a filesystem with lots of space.
data/raw
- Raw (immutable) hash-verified downloads.
data/interim
- Extracted and interim data representations.
data/interim/cache
- Dataset cache
data/processed
- The final, canonical data sets ready for analysis.
docs
- Sphinx-format documentation files for this project.
docs/Makefile
: Makefile for generating HTML/Latex/other formats from Sphinx-format documentation.
notebooks
- Jupyter notebooks. Naming convention is a number (for ordering),
the creator's initials, and a short
-
delimited description, e.g.1.0-jqp-initial-data-exploration
.
- Jupyter notebooks. Naming convention is a number (for ordering),
the creator's initials, and a short
quest
- This is where you'll find materials related to the Easydata Quest for Reproducibility.
quest_codewords.md
: QUEST TASK This is the only file in here you'll need to worry yourself with. In fact, go on, take a look at it. It will help you along on your quest.
reference
- Data dictionaries, documentation, manuals, scripts, papers, or other explanatory materials.
reference/easydata
: Easydata framework and workflow documentation.reference/templates
: Templates and code snippets for Jupyterreference/dataset
: resources related to datasets; e.g. dataset creation notebooks and scripts
reports
- Generated analysis as HTML, PDF, LaTeX, etc.
reports/figures
- Generated graphics and figures to be used in reporting.
environment.yml
- The user-readable YAML file for reproducing the conda/pip environment.
environment.(platform).lock.yml
- resolved versions, result of processing
environment.yml
- resolved versions, result of processing
setup.py
- Turns contents of
src
into a pip-installable python module (pip install -e .
) so it can be imported in python code.
- Turns contents of
src
- Source code for use in this project.
src/__init__.py
- Makes
src
a Python module.
- Makes
src/data
- Scripts to fetch or generate data.
src/analysis
- Scripts to turn datasets into output products.
src/tests
- Testing scripts that run via
make test
.
- Testing scripts that run via
This project was built using Easydata, a python framework aimed at making your data science workflow reproducible.