How are the callbacks (or the get_callbacks function) used? In keras thee train method takes callbacks as an argument but this doesn't seem to be the case here. Is it possible to use custom callback functions originally used with keras here? When extending the callbacks list with my custom callback, this gives an error that it's not iterable. I presume this is because the fit_generator() is used rather than the fit() method?

Hi

Thanks for sharing a great project. I would like to explore this library for my project. I faced an import error which seems to be a version conflict embedded in anago. Could you look into this issue?

Thank you!

Running env:
Ubuntu 16.04
Anaconda 5.0.1 for Python 3.6
Install anaGo with pip install anago

When I run

import anago
from anago.reader import load_data_and_labels

I receive

Using TensorFlow backend.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.multiarray failed to import

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.umath failed to import

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
ImportError: numpy.core.umath failed to import

Hi, @Hironsan , I have successfully trained the model, but when I evaluate the model and test the model, It failed. The error is :

TypeError: Failed to convert object of type <type 'tuple'> to Tensor. Contents: (None, 100). Consider casting elements to a supported type.

Can you help me ? Thank you very much !

hi,
i run the program according to the procedure, train, evaluate, tag...
but the result is not in line with yours

I have a problem with train dataset is small, so i think the number word in train dataset can't cover all word in test dataset, its mean some word in test dataset may be not in set word of train. I don't know anago cover this problem or not. If yes, can you explain for me how you solve this problem.
Thanks you

Its mentioned that its possible to use Glove vectors, can you please provide an example ? thanks

This is a feature request (let me know your thoughts).
Would be helpful have a mechanism that allows retraining the model in the last layer.
There are cases where we just want to augment our dataset instead to retrain everything again.

I try to run the download_model.py and evaluate with a 1-word sentence

Test O

However, the model gave the UnimplementedError (see above for traceback): TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.

Would you help me fixing this error?
Thank you very much

The full trace is below

2018-04-25 14:18:39.235184: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2
2018-04-25 14:18:39.240128: E tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_NO_DEVICE
2018-04-25 14:18:39.240182: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: exp
2018-04-25 14:18:39.240204: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: exp
2018-04-25 14:18:39.240255: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 384.111.0
2018-04-25 14:18:39.240314: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 384.111.0
2018-04-25 14:18:39.240336: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 384.111.0
Traceback (most recent call last):
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
    return fn(*args)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1312, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
    status, run_metadata)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
         [[Node: chain_crf_1/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@chain_crf_1/TensorArray"], dtype=DT_FLOAT, element_shape=[?,10], _device="/job:localhost/replica:0/task:0/device:CPU:0"](chain_crf_1/TensorArray, chain_crf_1/TensorArrayStack/range, chain_crf_1/while/Exit_1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_model.py", line 18, in <module>
    model.eval(x_test, y_test)
  File "/home/phuongpham/NER/anago/anago/wrapper.py", line 55, in eval
    evaluator.eval(x_test, y_test)
  File "/home/phuongpham/NER/anago/anago/evaluator.py", line 27, in eval
    f1score.on_epoch_end(epoch=-1)  # epoch takes any integer.
  File "/home/phuongpham/NER/anago/anago/metrics.py", line 132, in on_epoch_end
    y_pred = self.model.predict_on_batch(data)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/keras/engine/training.py", line 1945, in predict_on_batch
    outputs = self.predict_function(ins)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2478, in __call__
    **self.session_kwargs)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 905, in run
    run_metadata_ptr)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1140, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    run_metadata)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnimplementedError: TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
         [[Node: chain_crf_1/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@chain_crf_1/TensorArray"], dtype=DT_FLOAT, element_shape=[?,10], _device="/job:localhost/replica:0/task:0/device:CPU:0"](chain_crf_1/TensorArray, chain_crf_1/TensorArrayStack/range, chain_crf_1/while/Exit_1)]]

Caused by op 'chain_crf_1/TensorArrayStack/TensorArrayGatherV3', defined at:
  File "download_model.py", line 17, in <module>
    model = Sequence.load(dir_path)
  File "/home/phuongpham/NER/anago/anago/wrapper.py", line 77, in load
    self.model = SeqLabeling(config, dummy_embeddings, ntags=len(self.p.vocab_tag))
  File "/home/phuongpham/NER/anago/anago/models.py", line 82, in __init__
    pred = self.crf(x)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/keras/engine/topology.py", line 619, in __call__
    output = self.call(inputs, **kwargs)
  File "/home/phuongpham/NER/anago/anago/layers.py", line 313, in call
    y_pred = viterbi_decode(x, self.U, self.b_start, self.b_end, mask)
  File "/home/phuongpham/NER/anago/anago/layers.py", line 105, in viterbi_decode
    mask)
  File "/home/phuongpham/NER/anago/anago/layers.py", line 146, in _forward
    last, values, _ = K.rnn(_forward_step, inputs, initial_states)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2772, in rnn
    outputs = output_ta.stack()
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 893, in stack
    return self._implementation.stack(name=name)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 291, in stack
    return self.gather(math_ops.range(0, self.size()), name=name)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 305, in gather
    element_shape=element_shape)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 6011, in tensor_array_gather_v3
    flow_in=flow_in, dtype=dtype, element_shape=element_shape, name=name)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3290, in create_op
    op_def=op_def)
  File "/home/phuongpham/anaconda2/envs/py3/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1654, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

UnimplementedError (see above for traceback): TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
         [[Node: chain_crf_1/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@chain_crf_1/TensorArray"], dtype=DT_FLOAT, element_shape=[?,10], _device="/job:localhost/replica:0/task:0/device:CPU:0"](chain_crf_1/TensorArray, chain_crf_1/TensorArrayStack/range, chain_crf_1/while/Exit_1)]]

Hi,

I am trying to train an NER model for which I want to use the entire dataset in my possession. Right now in the train method it looks like the the x_valid and y_valid arguments are optional. However, if I leave them as None and don't pass them at all, I get the following error during training:

TypeError: object of type 'NoneType' has no len()

Which comes from the batch_iter method:

    113 
    114 def batch_iter(data, labels, batch_size, shuffle=True, preprocessor=None):
--> 115     num_batches_per_epoch = int((len(data) - 1) / batch_size) + 1
    116 
    117     def data_generator():

Using a validation set is useful when tuning the hyperparameters of the model, however once this is done, how can I train the final model on the entire dataset without having to split it?

hi，@Hironsan . I have trained a model named 'model_weight.h5' .
Then loaded model to tag:tagger = anago.Tagger(model_config, weights, save_path=SAVE_ROOT, preprocessor=p)
but when I use same sentence return different result,I am puzzling.
Please giving same suggestion.
input:<class 'str'> President Obama is speaking at the White House.
output:
[('President', 'O'), ('Obama', 'O'), ('is', 'PER'), ('speaking', 'PER'), ('at', 'O'), ('the', 'O'), ('White', 'O'), ('House.', 'LOC')]
[('President', 'O'), ('Obama', 'O'), ('is', 'O'), ('speaking', 'MISC'), ('at', 'MISC'), ('the', 'O'), ('White', 'LOC'), ('House.', 'LOC')]
[('President', 'O'), ('Obama', 'O'), ('is', 'O'), ('speaking', 'O'), ('at', 'O'), ('the', 'O'), ('White', 'O'), ('House.', 'O')]

Is there a way to print the accuracy during training epochs (not when an epoch is finished) ? i tried setting metrics=['sparse_categorical_accuracy'] inside trainer.py. But its giving me this message !

AttributeError: 'ChainCRF' object has no attribute 'accuracy'

While using Conll2003/en/ner data for training.

KeyError: 'I-PRT'

I am running tensorflow version 1.0.1 and keras 2.0.5 on ubuntu. I am trying to run the code. The weights are created and saved, however I cannot evaluate or tag sentences. When I want to evaluate the model, I get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-21-0cc4c2a02f0f> in <module>()
      2 
      3 evaluator = anago.Evaluator(model_config, weights, save_path=SAVE_ROOT, preprocessor=p)
----> 4 evaluator.eval(x_test, y_test)

~/Desktop/anago-master/anago/evaluator.py in eval(self, x_test, y_test)
     23         # Prepare test data(steps, generator)
     24         train_steps, train_batches = batch_iter(
---> 25             list(zip(x_test, y_test)), self.config.batch_size, preprocessor=self.preprocessor)
     26 
     27         # Build the model

TypeError: batch_iter() missing 1 required positional argument: 'batch_size'

and when I want to run the tagger, I get this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-1151c9247c1c> in <module>()
      1 # To tag any text, we can use Tagger. Prepare an instance of Tagger class and give text to tag method:
      2 weights = 'model_weights.h5'
----> 3 tagger = anago.Tagger(model_config, weights, save_path=SAVE_ROOT, preprocessor=p)

~/Desktop/anago-master/anago/tagger.py in __init__(self, config, weights, save_path, preprocessor, tokenizer)
     21 
     22         # Build the model
---> 23         self.model = SeqLabeling(config, ntags=len(self.preprocessor.vocab_tag))
     24         self.model.load(filepath=os.path.join(save_path, weights))
     25 

~/Desktop/anago-master/anago/models.py in __init__(self, config, embeddings, ntags)
     50             word_embeddings = Embedding(input_dim=config.vocab_size,
     51                                         output_dim=config.word_embedding_size,
---> 52                                         mask_zero=True)(word_ids)
     53         else:
     54             word_embeddings = Embedding(input_dim=embeddings.shape[0],

~/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py in __call__(self, inputs, **kwargs)
    567                                          '`layer.build(batch_input_shape)`')
    568                 if len(input_shapes) == 1:
--> 569                     self.build(input_shapes[0])
    570                 else:
    571                     self.build(input_shapes)

~/anaconda3/lib/python3.6/site-packages/keras/layers/embeddings.py in build(self, input_shape)
     99             name='embeddings',
    100             regularizer=self.embeddings_regularizer,
--> 101             constraint=self.embeddings_constraint)
    102         self.built = True
    103 

~/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
     86                 warnings.warn('Update your `' + object_name +
     87                               '` call to the Keras 2 API: ' + signature, stacklevel=2)
---> 88             return func(*args, **kwargs)
     89         wrapper._legacy_support_signature = inspect.getargspec(func)
     90         return wrapper

~/anaconda3/lib/python3.6/site-packages/keras/engine/topology.py in add_weight(self, name, shape, dtype, initializer, regularizer, trainable, constraint)
    389         if dtype is None:
    390             dtype = K.floatx()
--> 391         weight = K.variable(initializer(shape), dtype=dtype, name=name)
    392         if regularizer is not None:
    393             self.add_loss(regularizer(weight))

~/anaconda3/lib/python3.6/site-packages/keras/initializers.py in __call__(self, shape, dtype)
    101     def __call__(self, shape, dtype=None):
    102         return K.random_uniform(shape, self.minval, self.maxval,
--> 103                                 dtype=dtype, seed=self.seed)
    104 
    105     def get_config(self):

~/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py in random_uniform(shape, minval, maxval, dtype, seed)
   3444         seed = np.random.randint(10e6)
   3445     return tf.random_uniform(shape, minval=minval, maxval=maxval,
-> 3446                              dtype=dtype, seed=seed)
   3447 
   3448 

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py in random_uniform(shape, minval, maxval, dtype, seed, name)
    227     maxval = 1
    228   with ops.name_scope(name, "random_uniform", [shape, minval, maxval]) as name:
--> 229     shape = _ShapeTensor(shape)
    230     minval = ops.convert_to_tensor(minval, dtype=dtype, name="min")
    231     maxval = ops.convert_to_tensor(maxval, dtype=dtype, name="max")

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/random_ops.py in _ShapeTensor(shape)
     39   else:
     40     dtype = None
---> 41   return ops.convert_to_tensor(shape, dtype=dtype, name="shape")
     42 
     43 

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, preferred_dtype)
    635       name=name,
    636       preferred_dtype=preferred_dtype,
--> 637       as_ref=False)
    638 
    639 

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype)
    700 
    701         if ret is None:
--> 702           ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
    703 
    704         if ret is NotImplemented:

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in _constant_tensor_conversion_function(v, dtype, name, as_ref)
    108                                          as_ref=False):
    109   _ = as_ref
--> 110   return constant(v, dtype=dtype, name=name)
    111 
    112 

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name, verify_shape)
     97   tensor_value = attr_value_pb2.AttrValue()
     98   tensor_value.tensor.CopyFrom(
---> 99       tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
    100   dtype_value = attr_value_pb2.AttrValue(type=tensor_value.tensor.dtype)
    101   const_tensor = g.create_op(

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py in make_tensor_proto(values, dtype, shape, verify_shape)
    439   if numpy_dtype == dtypes.string and not isinstance(values, np.ndarray):
    440     proto_values = _FlattenToStrings(values)
--> 441     tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
    442     return tensor_proto
    443 

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py in <listcomp>(.0)
    439   if numpy_dtype == dtypes.string and not isinstance(values, np.ndarray):
    440     proto_values = _FlattenToStrings(values)
--> 441     tensor_proto.string_val.extend([compat.as_bytes(x) for x in proto_values])
    442     return tensor_proto
    443 

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/compat.py in as_bytes(bytes_or_text, encoding)
     63   else:
     64     raise TypeError('Expected binary or unicode string, got %r' %
---> 65                     (bytes_or_text,))
     66 
     67 

TypeError: Expected binary or unicode string, got None

I ran a test across a GPU based instance, but i didnt see any speed improvments. Is there something that i would need to modify in order to support GPU ? thanks

Hi @Hironsan,
Thanks for making Anago. I have an error, I would be glad if you look into.
While executing the line
embeddings = load_word_embeddings(p.vocab_word, embedding_path, model_config.word_embedding_size)
I am getting this error in line 83 of reader.py. Surprisingly, even if I put some newline and rearrange the lines, the error still stays in line 83!
I have tried changing the encoding to 'utf-8' or 'Latin-1' with the following change in line 82 in 'reader.py', but nothing worked
with open(glove_filename, encoding='utf-8') as f:

Traceback (most recent call last):
  File "<pyshell#17>", line 1, in <module>
    embeddings = load_word_embeddings(p.vocab_word, embedding_path, model_config.word_embedding_size)
  File "G:/Others/Thesis/anago\anago\data\reader.py", line 83, in load_word_embeddings
    line = line.strip().split(' ')
  File "C:\Users\MasumPC\AppData\Local\Programs\Python\Python35\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2776: character maps to <undefined>

How to predict for a new test data?

Hello, thanks for your efforts in this wonderful project.

I am trying to run the example ner_glove.py with the provided data.
However, I encountered the following error: UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2776: character maps to

The full trace back is

Loading data...
14041 train sequences
3250 valid sequences
Traceback (most recent call last):
File "ner_glove.py", line 18, in
embeddings = load_glove(EMBEDDING_PATH)
File "C:\Anaconda\lib\site-packages\anago-0.0.1-py3.6.egg\anago\reader.py", line 105, in load_glove
File "C:\Anaconda\lib\encodings\cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 2776: character maps to <undefined>

Would you give me instructions to solve this problem?

Thank you very much

Could you explain me in general, how are the directories split inside the project?
Regarding the Entity Extraction, I need the wrapper.py in order to call the Sequence class and thus, train the model with the datasets. There is also a Trainer class with its appropriate train method as well.

How do we know, which method is called for what? There are a lot of sub-directories in there, and no documentation to understand it. Plus, the example datasets, are separated in a different way, either they are used for entity extraction (ner), or other actions, correct?

Thank you in advance,
Vasiliki

When Label present in validation set and not present in training set , the train method throws KeyError

import anago
from anago.reader import load_data_and_labels

data_path = "/tmp/anago/data/conll2003/en/ner"
x_train, y_train = load_data_and_labels(data_path + '/train.txt')
x_valid, y_valid = load_data_and_labels(data_path + '/valid.txt')
model = anago.Sequence()
model.train(x_train, y_train, x_valid, y_valid)

Got error

File "/Users/zzz/test/venv/lib/python3.6/site-packages/anago/wrapper.py", line 50, in train
    trainer.train(x_train, y_train, x_valid, y_valid)
  File "/Users/zzz/test/venv/lib/python3.6/site-packages/anago/trainer.py", line 38, in train
    optimizer=Adam(lr=self.training_config.learning_rate),
  File "/Users/zzz/test/venv/lib/python3.6/site-packages/keras/engine/training.py", line 827, in compile
    sample_weight, mask)
  File "/Users/zzz/test/venv/lib/python3.6/site-packages/keras/engine/training.py", line 426, in weighted
    score_array = fn(y_true, y_pred)
  File "/Users/zzz/test/venv/lib/python3.6/site-packages/anago/layers.py", line 321, in loss
    mask = self._fetch_mask()
  File "/Users/zzz/test/venv/lib/python3.6/site-packages/anago/layers.py", line 276, in _fetch_mask
    if self.inbound_nodes:
AttributeError: 'ChainCRF' object has no attribute 'inbound_nodes'

the parameter train_embeddings is never used https://github.com/Hironsan/anago/blob/master/anago/config.py#L71, so there is no way to not train the embeddings layer of the model.

If you define a custom optimizer in a training_config to use it with the Trainer class, the optimizer is never used since Adam is hardcoded as an optimizer. https://github.com/Hironsan/anago/blob/master/anago/trainer.py#L38

Thank you for sharing analog with us. How do we save a model in anago? Is it similar to Keras?

Is this repo the implementation of Bidirectional LSTM-CRF Models for Sequence Tagging?

Hi @Hironsan

thanks for your effort in building such a fine tool.

I'm using anaGo for my school Project, for the Arabic language, but it seems there is a problem.

and this is the dataset I used :

1. WikiFANE_Gold: Gold standard Wikipedia-based Fine-grained Arabic Named Entity Corpus, ~500K tokens
2. NewsFANE_Gold: Gold standard Newswire-based Fine-grained Arabic Named Entity Corpus, ~170K tokens.

this is the problem :

Epoch 1/15
789/789 [==============================] - 5196s 7s/step - loss: 337.9646
Traceback (most recent call last):
File "C:/Users/Osama Bani Salameh/PycharmProjects/anaGoTest/anaGoTestMainFile.py", line 11, in
model.train(x_train, y_train, x_valid, y_valid)
File "C:\Program Files\Python36\lib\site-packages\anago\wrapper.py", line 50, in train
trainer.train(x_train, y_train, x_valid, y_valid)
File "C:\Program Files\Python36\lib\site-packages\anago\trainer.py", line 51, in train
callbacks=callbacks)
File "C:\Program Files\Python36\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "C:\Program Files\Python36\lib\site-packages\keras\engine\training.py", line 2262, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "C:\Program Files\Python36\lib\site-packages\keras\callbacks.py", line 77, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "C:\Program Files\Python36\lib\site-packages\anago\metrics.py", line 124, in on_epoch_end
for i, (data, label) in enumerate(self.valid_batches):
File "C:\Program Files\Python36\lib\site-packages\anago\reader.py", line 137, in data_generator
yield preprocessor.transform(X, y)
File "C:\Program Files\Python36\lib\site-packages\anago\preprocess.py", line 112, in transform
y = [[self.vocab_tag[t] for t in sent] for sent in y]
File "C:\Program Files\Python36\lib\site-packages\anago\preprocess.py", line 112, in
y = [[self.vocab_tag[t] for t in sent] for sent in y]
File "C:\Program Files\Python36\lib\site-packages\anago\preprocess.py", line 112, in
y = [[self.vocab_tag[t] for t in sent] for sent in y]
KeyError: 'O"'

by the way, the dataset has O label within it, so do you know what's going on?

regards,
Osama Bani Salameh

hi @Hironsan

As Feature Support said:

downloading learned model for many tasks. (e.g. NER, POS Tagging, etc...)

but I can not find how to download learned model

I wan't to apply this method on tweets, having read the paper and I understand what's happening. I was wondering if you implementation could be used for a set of 500 million tweets

Training data looks like:

and training results look like:

Also worth adding that whenever I add a new period at the end of each character sequence, it never begins training and stops at Epoch 1/15 with no progress bar. I'll be happy to share training data with you if that will help reproduce the issue.

Hello, I added my custom datasets in the same format as your examples. (Tab delimited Entity and Label, if exists).
Just before when the first epoch is about to end, the scripts crushes, with an error show below:

Traceback (most recent call last):

File "", line 1, in
model.train(x_train, y_train, x_valid, y_valid)

File "anago/wrapper.py", line 50, in train
trainer.train(x_train, y_train, x_valid, y_valid)

File "anago/trainer.py", line 51, in train
callbacks=callbacks)

File "/opt/anaconda2/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)

File "/opt/anaconda2/lib/python2.7/site-packages/keras/engine/training.py", line 2262, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)

File "/opt/anaconda2/lib/python2.7/site-packages/keras/callbacks.py", line 77, in on_epoch_end
callback.on_epoch_end(epoch, logs)

File "anago/metrics.py", line 124, in on_epoch_end
for i, (data, label) in enumerate(self.valid_batches):

File "anago/reader.py", line 137, in data_generator
yield preprocessor.transform(X, y)

File "anago/preprocess.py", line 117, in transform
y = [[self.vocab_tag[t] for t in sent] for sent in y]

KeyError: 'O'

Tha labels of my entities are like that: "Person", "Location", "Organization". Do I jave to follow the same format as yours?
Are there any hints, on what wrong is happening? Is it because of my dataset or something internal in the system?

Thank you in advance!
Vasiliki

I have observed that each line in the train set can take only the word and it's Named Entity tag. Can we pass the POS tag as well for training to obtain better results? As well as doing some feature engineering adding more columns useful for training.

Hi @Hironsan ,
I just want to clarify, for the performance in the test set, is the F-1 score that you obtained is indeed 90.67? I tried your code but I can only obtain around 86 of F1-score.

Then, I tried to download the pretrained model and then call model.evaluate on the test set, and it yields 98 of F1 score on the test set which I found does not make sense. In Lample's paper, the performance is around 90 of F1. Can you comment on this?

Thanks

Seems that this function is not being used anywhere.

Despite the F1 score, instead create your own function, you could use the scikit-learn: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html

This problem is present in Keras 2.1.3, since self.inbound_nodes was changed for self._inbound_nodes #26

Hi,
Anybody knows how to get reproducible results using this code?
I tried to follow https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development but it seems I'm getting different results every time I run the model.

Not sure why this issue is closed. It is still valid and causing confusion + there are several code instances which need to be changed too.

Try to classify a sentence with a single word, will crash.
Seems to be a bug in CRF layer implementation.

nimplementedError (see above for traceback): TensorArray has size zero, but element shape [?,16] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
         [[Node: chain_crf_1/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@chain_crf_1/TensorArray"], dtype=DT_FLOAT, element_shape=[?,16], _device="/job:localhost/replica:0/task:0/gpu:0"](chain_crf_1/TensorArray, chain_crf_1/TensorArrayStack/range, chain_crf_1/while/Exit_1)]]

Is there any way to convert this model for use with CoreML?

When I run the code like below.
I've got stack at the titled error.
why??

Using TensorFlow backend.
2018-05-22 11:47:25.286883: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA
Epoch 1/15
Traceback (most recent call last):
File "test.py", line 9, in
model.train(x_train, y_train, x_valid, y_valid)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/anago/wrapper.py", line 50, in train
trainer.train(x_train, y_train, x_valid, y_valid)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/anago/trainer.py", line 51, in train
callbacks=callbacks)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/keras/engine/training.py", line 2145, in fit_generator
generator_output = next(output_generator)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/keras/utils/data_utils.py", line 770, in get
six.reraise(value.class, value, value.traceback)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/keras/utils/data_utils.py", line 635, in _data_generator_task
generator_output = next(self._generator)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/anago/reader.py", line 137, in data_generator
yield preprocessor.transform(X, y)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/anago/preprocess.py", line 115, in transform
sents, y = self.pad_sequence(words, chars, y)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/anago/preprocess.py", line 148, in pad_sequence
word_ids, sequence_lengths = pad_sequences(word_ids, 0)
File "/Users/norio.kosaka/anaconda3/envs/py36/lib/python3.6/site-packages/anago/preprocess.py", line 197, in pad_sequences
max_length = len(max(sequences, key=len))
ValueError: max() arg is an empty sequence

import anago
from anago.reader import load_data_and_labels

x_train, y_train = load_data_and_labels('./data/train.txt')
x_valid, y_valid = load_data_and_labels('./data/valid.txt')
x_test, y_test = load_data_and_labels('./data/test.txt')

model = anago.Sequence()
model.train(x_train, y_train, x_valid, y_valid)
model.eval(x_test, y_test)
words = 'President Obama is speaking at the White House.'.split()
model.analyze(words)

Is there a way to scale ? i tried with 12 million sentences, and its giving me that it needs 3000 hours to finish !

hi,

when i run model.train i get the following error...

`---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
in ()
----> 1 model.train(x_train, y_train, x_val, y_val)

~/anaconda3/envs/py3/lib/python3.6/site-packages/anago/wrapper.py in train(self, x_train, y_train, x_valid, y_valid, vocab_init)
48 checkpoint_path=self.log_dir,
49 preprocessor=self.p)
---> 50 trainer.train(x_train, y_train, x_valid, y_valid)
51
52 def eval(self, x_test, y_test):

~/anaconda3/envs/py3/lib/python3.6/site-packages/anago/trainer.py in train(self, x_train, y_train, x_valid, y_valid)
36
37 self.model.compile(loss=self.model.crf.loss,
---> 38 optimizer=Adam(lr=self.training_config.learning_rate),
39 )
40

~/anaconda3/envs/py3/lib/python3.6/site-packages/keras/engine/training.py in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, weighted_metrics, target_tensors, **kwargs)
825 with K.name_scope(self.output_names[i] + '_loss'):
826 output_loss = weighted_loss(y_true, y_pred,
--> 827 sample_weight, mask)
828 if len(self.outputs) > 1:
829 self.metrics_tensors.append(output_loss)

~/anaconda3/envs/py3/lib/python3.6/site-packages/keras/engine/training.py in weighted(y_true, y_pred, weights, mask)
424 """
425 # score_array has ndim >= 2
--> 426 score_array = fn(y_true, y_pred)
427 if mask is not None:
428 # Cast the mask to floatX to avoid float64 upcasting in theano

~/anaconda3/envs/py3/lib/python3.6/site-packages/anago/layers.py in loss(self, y_true, y_pred)
319 """Linear Chain Conditional Random Field loss function.
320 """
--> 321 mask = self._fetch_mask()
322 return chain_crf_loss(y_true, y_pred, self.U, self.b_start, self.b_end, mask)
323

~/anaconda3/envs/py3/lib/python3.6/site-packages/anago/layers.py in _fetch_mask(self)
274 def _fetch_mask(self):
275 mask = None
--> 276 if self.inbound_nodes:
277 mask = self.inbound_nodes[0].input_masks[0]
278 return mask

AttributeError: 'ChainCRF' object has no attribute 'inbound_nodes'`

It would be really great if you could suggest a work around for this...

Thank you..

Hi,
I'm wondering what does the Japanese training data look like.
Are they segmented by word or by character?
also the data for training word2vec,
are they segmented in the same way?

Is there any way I can compute output probabilities or scores for each token-label output?

I am make custom data for my language for ner solution. But when use anago.train with our dataset, its stop at epoch 4-5 and don't show any error for fix?

You're showing an example of Japanese language in Readme.
Are you planning to put the pre-trained model in Japanese language also?

Trying to rebuild the model I noticed that the input 'sequence_lengths' is not used, is there any particular reason?

https://github.com/Hironsan/anago/blob/master/anago/models.py#L85

Why can't you just use different dim for x from # of tags?

Thanks for sharing the code. Error happened when training is as follows,
`Epoch 1/15
702/703 [============================>.] - ETA: 0s - loss: 62.9782 - f1: 84.33
703/703 [==============================] - 78s - loss: 62.8931
Epoch 2/15
702/703 [============================>.] - ETA: 0s - loss: 60.5871 - f1: 87.10
703/703 [==============================] - 80s - loss: 60.5013
Epoch 3/15
702/703 [============================>.] - ETA: 0s - loss: 59.9166 - f1: 89.38
703/703 [==============================] - 78s - loss: 59.8314
Epoch 4/15
702/703 [============================>.] - ETA: 0s - loss: 59.1642 - f1: 89.96
703/703 [==============================] - 77s - loss: 59.0806
Epoch 5/15
702/703 [============================>.] - ETA: 0s - loss: 59.6137 - f1: 90.65
703/703 [==============================] - 78s - loss: 59.5299
Epoch 6/15
702/703 [============================>.] - ETA: 0s - loss: 59.4186 - f1: 90.61
703/703 [==============================] - 78s - loss: 59.3342
Epoch 7/15
702/703 [============================>.] - ETA: 0s - loss: 59.5612 - f1: 91.40
703/703 [==============================] - 78s - loss: 59.4771
Epoch 8/15
702/703 [============================>.] - ETA: 0s - loss: 59.3047 - f1: 91.13
703/703 [==============================] - 78s - loss: 59.2204
Epoch 9/15
702/703 [============================>.] - ETA: 0s - loss: 59.5008

UnimplementedError Traceback (most recent call last)
in ()
1 trainer = anago.Trainer(model_config, training_config, checkpoint_path=LOG_ROOT, save_path=SAVE_ROOT,
2 preprocessor=p, embeddings=embeddings)
----> 3 trainer.train(x_train, y_train, x_valid, y_valid)

/home/hongzhi/wp/anago/anago/trainer.pyc in train(self, x_train, y_train, x_valid, y_valid)
52 steps_per_epoch=train_steps,
53 epochs=self.training_config.max_epoch,
---> 54 callbacks=callbacks)
55
56 # Save the model

/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/legacy/interfaces.pyc in wrapper(*args, **kwargs)
85 warnings.warn('Update your ' + object_name + 86 ' call to the Keras 2 API: ' + signature, stacklevel=2)
---> 87 return func(*args, **kwargs)
88 wrapper._original_function = func
89 return wrapper

/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
2040 outs = self.train_on_batch(x, y,
2041 sample_weight=sample_weight,
-> 2042 class_weight=class_weight)
2043
2044 if not isinstance(outs, list):

/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/engine/training.pyc in train_on_batch(self, x, y, sample_weight, class_weight)
1760 ins = x + y + sample_weights
1761 self._make_train_function()
-> 1762 outputs = self.train_function(ins)
1763 if len(outputs) == 1:
1764 return outputs[0]

/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.pyc in call(self, inputs)
2271 updated = session.run(self.outputs + [self.updates_op],
2272 feed_dict=feed_dict,
-> 2273 **self.session_kwargs)
2274 return updated[:len(self.outputs)]
2275

/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
787 try:
788 result = self._run(None, fetches, feed_dict, options_ptr,
--> 789 run_metadata_ptr)
790 if run_metadata:
791 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
995 if final_fetches or final_targets:
996 results = self._do_run(handle, final_targets, final_fetches,
--> 997 feed_dict_string, options, run_metadata)
998 else:
999 results = []

/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1130 if handle is None:
1131 return self._do_call(_run_fn, self._session, feed_dict, fetch_list,
-> 1132 target_list, options, run_metadata)
1133 else:
1134 return self._do_call(_prun_fn, self._session, handle, feed_dict,

/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _do_call(self, fn, *args)
1150 except KeyError:
1151 pass
-> 1152 raise type(e)(node_def, op, message)
1153
1154 def _extend_graph(self):

UnimplementedError: TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: chain_crf_2/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@chain_crf_2/TensorArray"], dtype=DT_FLOAT, element_shape=[?,10], _device="/job:localhost/replica:0/task:0/gpu:0"](chain_crf_2/TensorArray, chain_crf_2/TensorArrayStack/range, chain_crf_2/while/Exit_1)]]
[[Node: training_1/Adam/gradients/bidirectional_2/while_1/Merge_2_grad/Switch/_887 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_5628_training_1/Adam/gradients/bidirectional_2/while_1/Merge_2_grad/Switch", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

Caused by op u'chain_crf_2/TensorArrayStack/TensorArrayGatherV3', defined at:
File "/home/hongzhi/anaconda2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/home/hongzhi/anaconda2/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/main.py", line 3, in
app.launch_new_instance()
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/traitlets/config/application.py", line 653, in launch_instance
app.start()
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/kernelapp.py", line 474, in start
ioloop.IOLoop.instance().start()
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/zmq/eventloop/ioloop.py", line 162, in start
super(ZMQIOLoop, self).start()
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tornado/ioloop.py", line 887, in start
handler_func(fd_obj, events)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 276, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 228, in dispatch_shell
handler(stream, idents, msg)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/kernelbase.py", line 390, in execute_request
user_expressions, allow_stdin)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/ipykernel/zmqshell.py", line 501, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2717, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2827, in run_ast_nodes
if self.run_code(code, result):
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2881, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 3, in
trainer.train(x_train, y_train, x_valid, y_valid)
File "anago/trainer.py", line 39, in train
model = SeqLabeling(self.model_config, self.embeddings, len(self.preprocessor.vocab_tag))
File "anago/models.py", line 83, in init
pred = self.crf(x)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 602, in call
output = self.call(inputs, **kwargs)
File "anago/layers.py", line 314, in call
y_pred = viterbi_decode(x, self.U, self.b_start, self.b_end, mask)
File "anago/layers.py", line 106, in viterbi_decode
mask)
File "anago/layers.py", line 147, in _forward
last, values, _ = K.rnn(_forward_step, inputs, initial_states)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2551, in rnn
outputs = output_ta.stack()
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 334, in stack
return self.gather(math_ops.range(0, self.size()), name=name)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 360, in gather
element_shape=element_shape)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 1814, in _tensor_array_gather_v3
element_shape=element_shape, name=name)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/home/hongzhi/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in init
self._traceback = _extract_stack()

UnimplementedError (see above for traceback): TensorArray has size zero, but element shape [?,10] is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: chain_crf_2/TensorArrayStack/TensorArrayGatherV3 = TensorArrayGatherV3[_class=["loc:@chain_crf_2/TensorArray"], dtype=DT_FLOAT, element_shape=[?,10], _device="/job:localhost/replica:0/task:0/gpu:0"](chain_crf_2/TensorArray, chain_crf_2/TensorArrayStack/range, chain_crf_2/while/Exit_1)]]
[[Node: training_1/Adam/gradients/bidirectional_2/while_1/Merge_2_grad/Switch/_887 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_5628_training_1/Adam/gradients/bidirectional_2/while_1/Merge_2_grad/Switch", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]
My environmentsanago (0.0.1)
Keras (2.0.8)
lazy-object-proxy (1.2.1)
tensorflow-gpu (1.2.0)
tensorflow-tensorboard (0.1.6)
`
Any idea to fix it is appreciated, thanks.