cost-eltec / eltec Goto Github PK

It would be good to provide metadata tables with a similar set of fields, and all in TSV, for all collections. The fact that many have it make people expect to see them everywhere (and I can't blame them).

At the moment, we have collections with texts using the latin alphabet and others using cyrillic script, possibly some mixed collections or even texts. This cannot entirely be deduced from the language of the texts, in some cases, but probably has significant impact on analyses using the words as tokens. So it might be worth while noting this somewhere.

TEI recommends using "@xml:lang" to register the language and, optionally, the writing system of a text. For this, it recommends using the IANA.org codes, see here: http://www.iana.org/assignments/language-subtag-registry/language-subtag-registry and, more accessibly, here: https://en.wikipedia.org/wiki/IETF_language_tag#Syntax_of_language_tags

Bottom line: it is probably sufficient to update the language codes used as values in "@xml:lang" to have the added information about the script used, like: "de-Latn" (for German in Latin script) or "bg-Cyrl" for Bulgarian in Cyrillic script.

Note that the Serbian texts already do this, with @xml:lang value of "sr-Cyrl", but it might be good to generalize the practice at least for those languages where variation in scripts used is possible.

I've been trying to create a demo app for the aggregated ELTeC Level1 material and noticed several issues in this process. Most important seems to be language codes. I would strongly suggest using consistent ISO codes. Atm repositories use 3-letter codes but in language/@ident 2-letter codes are often used instead. Some are also erroneous (e.g. uk instead of ua for Ukrainian).

Furthermore using hierarchical language codes, e.g. de-ch for Swiss German would make filtering results much easier in the future, because the hierarchy is already reflected in the code. Otherwise a clear taxonomy of language hierarchies is a must.