baidu-research / globallynormalizedreader Goto Github PK

Hi, when I ran the data/download.sh script, the command
cat augmented_data/augmented_zips.zip.z* > augmented_train.json.zip
raised an error:
cat: augmented_data/augmented_zips.zip.z*: No such file or directory

I then changed augmented_zips.zip.z*
to
cat augmented_data/augmented_zips.z01 augmented_data/augmented_zips.z02 augmented_data/augmented_zips.z03 augmented_data/augmented_zips.z04 augmented_data/augmented_zips.z05 augmented_data/augmented_zips.z06 augmented_data/augmented_zips.z07 augmented_data/augmented_zips.z08 augmented_data/augmented_zips.z09 augmented_data/augmented_zips.z10 augmented_data/augmented_zips.zip > augmented_train.json.zip

But am unable to run featurize.py successfully afterwards, encountering UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 15-16: invalid continuation byte when running line 139 of vocab.py (gensim.models.KeyedVectors.load_word2vec_format(path, binary=True)) during "Building word embedding matrix..."

Is there any advice on what modifications I can make?
Thanks!

https://www.aclweb.org/anthology/D17-1112
this link can not download the GNR paper now.

I had glove.840B.300d.txt unzip from glove.840B.300d.zip, is this right?

python3 featurize.py --datadir data --outdir featurized --glove-path ../glove.840B.300d.txt
Using TensorFlow backend.
Constructing vocabularies...
Finished...
Building word embedding matrix...
Traceback (most recent call last):
File "featurize.py", line 317, in
preprocess()
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "featurize.py", line 273, in preprocess
vocab.construct_embedding_matrix(glove_path)
File "/Users/GloballyNormalizedReader/vocab.py", line 85, in construct_embedding_matrix
missing_word_value=lambda: 0.0)
File "/Users/GloballyNormalizedReader/vocab.py", line 219, in load_word_vectors
word2vec = FastWord2vec.get(path)
File "/Users/GloballyNormalizedReader/vocab.py", line 202, in get
WORD2VEC = FastWord2vec(path)
File "/Users/GloballyNormalizedReader/vocab.py", line 141, in init
binary=True)
File "/usr/local/lib/python3.6/site-packages/gensim/models/keyedvectors.py", line 207, in load_word2vec_format
vocab_size, vector_size = map(int, header.split()) # throws for invalid file format
ValueError: invalid literal for int() with base 10: ','