chozelinek / europarl Goto Github PK

Produce sentence alignment with HunAlign and InterText.

• It might be necessary to align interventions
• It might be helpful to align paragraphs

• dashes
• quotation marks

Sometimes, the annotation of the source language is inconsistent in the HTML files.

Provided that several languages have been processed:

1. align interventions
2. align paragraphs
3. compare sl values

If versions agree, it is OK, if not:

• majority vote?
• manual revision?

Many foreign expressions (e.g. in other languages or latinates, and sometimes to stress irony, etc.) are annotated with italics in the source HTML files. It might be useful to keep this information through the pipeline.

After using proceedings_xml.py, there are values for intervention's attribute role which are not only the relevant information but punctuation and more. Examples:

<intervention speaker_id="photo_generic" name="Algirdas Šemeta" is_mep="True" mode="spoken" role="Member of the Commission.">
<intervention speaker_id="photo_generic" name="László Kovács" is_mep="True" mode="spoken" role="Member of the Commission. −">
<intervention speaker_id="photo_generic" name="Vladimír Špidla" is_mep="True" mode="spoken" role="Member of the Commission. – (CS)">

Solution?

Modify proceedings_xml.py before line 341 (self.intervention_to_xml(x_section, s_intervention)) to clean s_intervention['role'].

Produce a schema or DTD to validate the XML files and use it as documentation.

Format Python scripts properly and write docstrings.

There are a number of codes (references to documents, mainly) which degrade automatic linguistic preprocessing (e.g. tokenization, lemmatization, PoS, sentence splitting...).

An idea would be to annotate them to handle them properly by the automatic linguistic preprocessing.

Use the README.md for documentation of the scripts, but create a tutorial illustrating the pipeline (a n commented/explained version of the compile.sh script).

Add to <p> an attribute called trans to make explicit if the paragraph is:

• original
• translation
• NA (for unknown cases)

Or

• True (translations)
• False (originals)
• NA unknown