##Ghalib Concordance Project
The original data from this project comes from (our) Professor Emerita Frances Pritchett’s carefully transcribed work: http://www.columbia.edu/itc/mealac/pritchett/00ghalib.
This is a work in progress in very early stages. If you are interested in helping out, please be in touch. There is a substantial amount of proof reading and “data munging” that has to happen, so we need many eyes and ears.
We also want this to be an entirely open digital humanities project, so all data and source code will be available at all stages of production. The code/data is licensed under an MIT license. There has not yet been an official release.
Much like programming scientists, we are both programming humanists and not professional coders, so suggestions for best practices are always welcome (e.g. ‘use collections’).
##Current Status
We are proofreading the lemma forms along with the tokens in transliteration, Urdu, and devanagari forms. During the first round of proofing, we are checking the file hiur-lemmas-by-size-proofme.md
in output and noting any errors. This is the stage 1 part of the process. We produce a web/print version of the concordance that will supply the complete representative verses and also link out to fp7's page.
During the next phase, we will markup the text of the poems with the corresponding lemma form for greater accuracy. We will expand to multi-token lemmas, e.g. compound verbs, and add part-of-speech markings.
##Overview of Code
The main program here is in the iPython Notebook generator.ipynb
. It reads the text from input/verses.csv.
The notebook generate_urdu.csv
creates Urdu-(....-ur.html) and devanagari-script (....-hi.html) statistics file output based on earlier transliteration-only versions in output/statistics
. (Devanagari is still rather messy).
##Description of Files
README.md
: this filebin
: binary files directoryfw xxx
: finds word xxx in textload_notebooks
: loads the ipython notebooks with --script parameter to generate .py filesog #
: opens ghazal # on the web
generate_urdu.ipynb
: creates Urdu- and devanagari-script statistics output based on earlier transliteration-only versions inoutput/statistics
.generator.ipynb
: xgenerator.py
: x/graphparser
: submodule of graphparser/input
: input file directoryREADME.md
: some notes on inputokay.csv
: okay (as x), token, lemmaverses-orig.csv
: original input fileverses.csv
: verses file in use (###.##.#), transliterate verse,meter
license.txt
: license/old_code
: old code not in use now/output
: output directory (all are generated)conc_details.csv
: lemma (transliteration),words from lemma (in transliteration)hiur-lemmas-by-size-proofme.md
: temporary file (copy of below) that we are proofreading nowhiur-lemmas-by-size-proofme.md
: markdown (md) version of belowhiur-lemmas-by-size-ul.html
: HTML of transliteration, urdu, devanagari, hyperlinks (sorted by instances of lemma)hiur-lemmas.html
: HTML of transliteration, urdu, devanagari, hyperlinks (sorted by instances of transliteration)izafats.csv
: izafats,countizafatsastokens.csv
: token (izafats are token), countlemmas-by-size.txt
: lemmas (sorted by instances), tokens (with count)
/statistics
: directory of various statistical detailsizafat-freq-ur.csv
: just izafats with Urduizafat-freq.csv
: izafats in transliterationizafatastoken-freq.csv
: tokens where izafat phrases are considered tokensizafatastokens-freq-hiur.csv
: tokens with Urdu, messy nagari where izafat phrases are considered tokensizafatastoken-freq-ur.csv
: tokens where izafat phrases are considered tokens with Urdulemmas-beta-freq-ur.csv
: just lemmas with Urdu (very beta)lemmas-beta-freq.csv
: just lemmas (very beta)uniquetokens-freq-ur.csv
: unique tokens with urduuniquetokens-freq.csv
: unique tokens in transliteration
tocheck.csv
: transliterations to check
texts
A. Sean Pue (@seanpue)
Taimoor Shahid (@MohJeMa)