./build.sh
./count.sh
./clean.sh
- Uses map-reduce to count the number of words in a set of text files
- Uses a single container, which uses an environment variable to run a mapper or a reducer
- Creates 9 mappers, each of which exposes a FastAPI endpoint, which when queried writes out
data/counts/{i}.json
- The reducer queries each of the mappers on the FastAPI endpoints, and creates
data/counts/total_count.json
, which is the sum of alldata/counts/{i}.json