kylebgorman / citylex Goto Github PK

Here, I don't think that there's any good reason to use ARPAbet, but it would be good to have both IPA transcriptions and X-SAMPA.

Hi! I recently tried to execute the code citylex --all-free from a Google Colab notebook and I got the following error:

INFO: Requesting URL: http://crr.ugent.be/papers/SUBTLEX-UK.xlsx
Traceback (most recent call last):
  File "/usr/local/bin/citylex", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/citylex.py", line 502, in main
    _subtlex_uk(lexicon)
  File "/usr/local/lib/python3.7/dist-packages/citylex.py", line 248, in _subtlex_uk
    ptr.subtlex_uk_freq = freq
TypeError: '' has type str, but expected one of: int, long

I have a feeling this is because something is corrupted in the frequency column of the SUBTLEX-UK data but I do not really know. Just thought you'd like to know that this error is out there.

ETA: If I run citylex --elp --cmudict everything works just fine and it saves a tsv of all of the stats.

There's no reason why we can't provide UDLexicon's UD features as such, or translate UM features back to UD features, and so on. The more's the merrier.

In addition to raw frequencies we should report van Heuven et al.'s "Zipfs". Ignoring the Laplace smoothing they talk about, the equation seems to be:

import math


def zipfs(count: int, total: int) -> float:
    # Subtraction in log-space is division in real-space.
    scale = math.log10(count) - math.log10(total)
    return scale + 9