ulf1 / sentence-embedding-evaluation-german Goto Github PK

https://github.com/hdaSprachtechnologie/Sentiment-Analysis

https://github.com/redewiedergabe/corpus

• Paper: https://www.dfki.de/~neumann/publications/new-ps/starsem2014-FINAL.pdf
• Dataset: http://www.dfki.de/~neumann/resources/omq_public_email_data.zip
• Website: https://www.dfki.de/~neumann/resources/omqdata.html

TODO

• Download script
• Postprocessing with bs4
• What is the prediction task?

https://github.com/UCSM-DUE/IWG_hatespeech_public

• Paper: https://aclanthology.org/L16-1178/
• Website: http://romanklinger.de/scare/
• Public download: http://www.romanklinger.de/scare/scare_v1.0.0.zip

TODO

• Download script for http://www.romanklinger.de/scare/scare_v1.0.0.zip
• Download the actual texts/reviews???

Error when running on GPU:
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

• Website: https://www.spur.uzh.ch/en/departments/research/textgroup/ArchiMob.html
• Downlaod: https://drive.switch.ch/index.php/s/DZycFA9DPC8FgD9

Setup:

• A language detection algorithm for lang:de should cover German dialects and regional varieties, e.g. lang:nds (Lower German), lang:gsw (Swiss German), lang:de-AT (Austrian German).
• The corpus contains 100% lang:gsw text examples from German-speaking part of Switzerland. Thus, we add 50% Standard German sentence examples with label=0 to the dataset. The result is a 50/50 balance binary classification dataset.

Class labels

• ZU
• LU
• BS
• BE
• ...

Random examples with extra class labels

• Standard German (de-DE)
• Austrian (de-AT)

https://github.com/uds-lsv/GermEval-2018-Data

current backup https://drive.google.com/drive/folders/1Z-l-1cxXGZZi5CBFTyRJwxuXmeWT_KyG

Info

• Paper: https://aclanthology.org/2020.vardial-1.3
• Daten: https://github.com/Helsinki-NLP/LSDC

Setup:

• A language detection algorithm for lang:de should cover German dialects and regional varieties, e.g. lang:nds (Lower German), lang:gsw (Swiss German), lang:de-AT (Austrian German).
• The corpus contains 100% lang:nds text examples from regions in North-West Germany and North-East Netherlands. Thus, we add 25% Standard German and 25% Standard Dutch sentence examples with label=0 to the dataset. The result is a 50/50 balance binary classification dataset.

Class labels

• Achterhoek (ACH),
• Drenthe (DRE),
• Groningen (GRO) (not included; i guess copyright issues)
• Hamburg (HAM),
• Holstein (HOL),
• Mark Brandenburg (MAR),
• Mecklenburg-Vorpommern (MKB),
• Münsterland (MON),
• Northern Lower Saxony (NNI),
• Lower Prussia (NPR) (dead dialect)
• Eastphalia (OFL),
• East Frisia (OFR),
• Western Overijssel (OVY),
• Eastern Westphalia (OWL),
• Sauerland (SUD)
• Twente (TWE)

Add random examples for two further class labels

• Standard German (DE)
• Dutch (NL)

https://github.com/ZurichNLP/xstance

binary: in favor vs against it

Error:
TypeError: linear(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

Look here

When does it happen

params = {
    'datafolder': './datasets',
    'batch_size': 128, 
    'num_epochs': 100,
    'early_stopping': True,
    'split_ratio': 0.1,  # if early_stopping=True
    'patience': 10,  # if early_stopping=True
}

https://github.com/bricksdont/swiss-dialect-identification

Setup:

• A language detection algorithm for lang:de should cover German dialects and regional varieties, e.g. lang:nds (Lower German), lang:gsw (Swiss German), lang:de-AT (Austrian German).
• The corpus contains 100% lang:gsw text examples from German-speaking part of Switzerland. Thus, we add 50% Standard German sentence examples with label=0 to the dataset. The result is a 50/50 balance binary classification dataset.

Class labels

• ZU
• LU
• BS
• BE
• ...

Random examples with extra class labels

• Standard German (de-DE)
• Austrian (de-AT)

• https://ofai.github.io/million-post-corpus/
• newspaper, de-AT
• 10.1145/3077136.3080711

https://fz.h-da.de/iggsa/data

• Code: https://github.com/axenov/politik-news
• Paper: https://aclanthology.org/2021.woah-1.13.pdf
• class: far-left, center-left, center, center-right, far-right

TODO

• crawl the text data and dump into zenodo
• ...

• all validations sets with sklearn stratified
• scale the remaining training data with np.choice an p=w up
• get_class_weights

https://www.deepset.ai/datasets

• https://github.com/spinningbytes/SB-CH
• newspaper, gsw
• https://aclanthology.org/L18-1372/

https://sites.google.com/site/iggsahome/downloads

• Website: https://www.ims.uni-stuttgart.de/forschung/ressourcen/korpora/deisear/
• Download: https://www.ims.uni-stuttgart.de/documents/ressourcen/korpora/emotions/deISEARenISEAR.zip
• Paper: https://arxiv.org/pdf/1905.13618.pdf

TODO

• Download script
• Process into torch dataset

https://github.com/mawic/german-abusive-language-covid-19

TODO

• crawl tweets and dump them in a zenodo repo
• ...

https://corpora.linguistik.uni-erlangen.de/data/konvens/proceedings/papers/germeval/GermEvalSharedTask2019Iggsa.pdf