deeppavlov / ner Goto Github PK
View Code? Open in Web Editor NEWNamed Entity Recognition
License: Apache License 2.0
Named Entity Recognition
License: Apache License 2.0
I have other dataset then CoNLL 2003, where i have my own custom entites. But i am unable to evaluate model as it is unable to find phrases in training set. any solution?
Hi, thanks for the great and a long-awaited tool.
I used to have tensorflow 1.7.0 for CPU installed (initially from provided binaries, then built from source).
In both cases I've bumped into the following issue:
$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ner
Traceback (most recent call last):
File "<stdin>", line 1, in <module>$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
>>> tensorflow.__version__
'1.7.0'
>>> import ner
>>>
File "/usr/local/lib/python3.5/dist-packages/ner/__init__.py", line 1, in <module>
from .extractor import Extractor
File "/usr/local/lib/python3.5/dist-packages/ner/extractor/__init__.py", line 1, in <module>
from .extractor import Extractor
File "/usr/local/lib/python3.5/dist-packages/ner/extractor/extractor.py", line 11, in <module>
from ner.network import NER
File "/usr/local/lib/python3.5/dist-packages/ner/network.py", line 17, in <module>
import tensorflow as tf
File "/usr/local/lib/python3.5/dist-packages/tensorflow/__init__.py", line 24, in <module>
from tensorflow.python import * # pylint: disable=redefined-builtin
File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/__init__.py", line 52, in <module>
from tensorflow.core.framework.graph_pb2 import *
File "/usr/local/lib/python3.5/dist-packages/tensorflow/core/framework/graph_pb2.py", line 16, in <module>
from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
File "/usr/local/lib/python3.5/dist-packages/tensorflow/core/framework/node_def_pb2.py", line 16, in <module>
from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
File "/usr/local/lib/python3.5/dist-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in <module>
from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
File "/usr/local/lib/python3.5/dist-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in <module>
from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
File "/usr/local/lib/python3.5/dist-packages/tensorflow/core/framework/resource_handle_pb2.py", line 41, in <module>
options=None, file=DESCRIPTOR),
TypeError: __init__() got an unexpected keyword argument 'file'
However, after importing tensorflow first, things do work.
$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow
/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
>>> tensorflow.__version__
'1.7.0'
>>> import ner
>>>
Prior to noticing the previous hack, I had simply installed the older tensorflow version (I forgot which one, sorry), and that fixed it.
Awkward IndexError on specific strings: Сергее-вичу
gives the error while Сергее-вич
works fine.
Code to reproduce:
import ner
extractor = ner.Extractor(model_url='http://lnsigo.mipt.ru/export/models/ner/ner_model_total_rus.tar.gz')
list(extractor('Сергее-вич')) # works!
list(extractor('Сергее-вичу')) # fails
Trace:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/extractor/extractor.py", line 54, in __call__
tags = self.network.predict_for_token_batch([tokens_lemmas])[0]
File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/network.py", line 379, in predict_for_token_batch
batch_x, _ = self.corpus.tokens_batch_to_numpy_batch(tokens_batch)
File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/corpus.py", line 219, in tokens_batch_to_numpy_batch
x['capitalization'][n, :len(utt)] = [tok[0].isupper() for tok in utt]
File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/corpus.py", line 219, in <listcomp>
x['capitalization'][n, :len(utt)] = [tok[0].isupper() for tok in utt]
IndexError: string index out of range
@mu-arkhipov Please add to readme bib reference and link to the paper on arxiv and the table that gives comparison of the model with previous results.
Hello there, great work!
Could you please specify the exact parameters that should be used in order to replicate your experiments from the paper, for both "Bi-LSTM + CRF" and "Bi-LSTM + CRF + external word embedding news" models? The model_params for instantiating the NER network and the learning_params for training.
Thanks!
Just installed NER from Gtihub. Run file training_example.ipynb from https://github.com/deepmipt/ner/blob/master/training_example.ipynb, then the results in example notebook looks bad:
Epoch 4
Eval on valid:
processed 54612 tokens with 5942 phrases; found: 314 phrases; correct: 13.
precision: 4.14%; recall: 0.22%; FB1: 0.42
Eval on train:
processed 217662 tokens with 23499 phrases; found: 1416 phrases; correct: 47.
precision: 3.32%; recall: 0.20%; FB1: 0.38
LOC: precision: 2.36%; recall: 0.21%; F1: 0.39 635
MISC: precision: 0.00%; recall: 0.00%; F1: 0.00 0
ORG: precision: 8.45%; recall: 0.38%; F1: 0.73 284
PER: precision: 1.61%; recall: 0.12%; F1: 0.23 497
Eval on valid:
processed 54612 tokens with 5942 phrases; found: 316 phrases; correct: 13.
precision: 4.11%; recall: 0.22%; FB1: 0.42
LOC: precision: 0.69%; recall: 0.05%; F1: 0.10 144
MISC: precision: 0.00%; recall: 0.00%; F1: 0.00 0
ORG: precision: 4.23%; recall: 0.22%; F1: 0.42 71
PER: precision: 8.91%; recall: 0.49%; F1: 0.93 101
Eval on test:
processed 49888 tokens with 5648 phrases; found: 258 phrases; correct: 15.
precision: 5.81%; recall: 0.27%; FB1: 0.51
LOC: precision: 2.73%; recall: 0.18%; F1: 0.34 110
MISC: precision: 0.00%; recall: 0.00%; F1: 0.00 0
ORG: precision: 9.76%; recall: 0.24%; F1: 0.47 41
PER: precision: 7.48%; recall: 0.49%; F1: 0.93 107
This result is wrong. Please help me!!
Hi there!
I tried running your notebook example as is, just increasing epochs to 20 instead of 5, and I got the error below. Running with 5 epochs worked fine.
I think it has something to do with running the Adam optimizer, as defined the method below, from network.py.
I guess it's running the optimizer code after 10 epochs, and there's something incompatible there with my tensorflow version. I'm running 1.4.1 version.
def get_train_op(self, loss, learning_rate, learnable_scopes=None, lr_decay_rate=None):
global_step = tf.Variable(0, trainable=False)
try:
n_training_samples = len(self.corpus.dataset["train"])
except TypeError:
n_training_samples = 1024
batch_size = tf.shape(self._x_w)[0]
decay_steps = tf.cast(n_training_samples / batch_size, tf.int32)
if lr_decay_rate is not None:
learning_rate = tf.train.exponential_decay(learning_rate,
global_step,
decay_steps=decay_steps,
decay_rate=lr_decay_rate,
staircase=True)
self._learning_rate_decayed = learning_rate
variables = self.get_trainable_variables(learnable_scopes)
# For batch norm it is necessary to update running averages
extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
with tf.control_dependencies(extra_update_ops):
train_op = tf.train.AdamOptimizer(learning_rate).minimize(loss, global_step=global_step, var_list=variables)
return train_op
UnimplementedError Traceback (most recent call last)
~/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1322 try:
-> 1323 return fn(*args)
1324 except errors.OpError as e:
~/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1301 feed_dict, fetch_list, target_list,
-> 1302 status, run_metadata)
1303
~/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in exit(self, type_arg, value_arg, traceback_arg)
472 compat.as_text(c_api.TF_Message(self.status.status)),
--> 473 c_api.TF_GetCode(self.status.status))
474 # Delete the underlying status object from memory otherwise it stays alive
UnimplementedError: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/TensorArrayGradV3, rnn/TensorArrayUnstack/range, gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/gradient_flow)]]
[[Node: Adam/update/_174 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2413_Adam/update", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
During handling of the above exception, another exception occurred:
UnimplementedError Traceback (most recent call last)
in ()
4 'batch_size': 8,
5 'learning_rate_decay': 0.707}
----> 6 results = net.fit(**learning_params)
~/repositorios/russian-ner/ner/network.py in fit(self, batch_gen, batch_size, learning_rate, epochs, dropout_rate, learning_rate_decay)
242 self.train_writer.add_summary(summary)
243
--> 244 self._sess.run(self._train_op, feed_dict=feed_dict)
245 if self.verbouse:
246 self.eval_conll('valid', print_results=True)
~/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
887 try:
888 result = self._run(None, fetches, feed_dict, options_ptr,
--> 889 run_metadata_ptr)
890 if run_metadata:
891 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
~/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1118 if final_fetches or final_targets or (handle and feed_dict_tensor):
1119 results = self._do_run(handle, final_targets, final_fetches,
-> 1120 feed_dict_tensor, options, run_metadata)
1121 else:
1122 results = []
~/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1315 if handle is None:
1316 return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1317 options, run_metadata)
1318 else:
1319 return self._do_call(_prun_fn, self._session, handle, feeds, fetches)
~/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1334 except KeyError:
1335 pass
-> 1336 raise type(e)(node_def, op, message)
1337
1338 def _extend_graph(self):
UnimplementedError: TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/TensorArrayGradV3, rnn/TensorArrayUnstack/range, gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/gradient_flow)]]
[[Node: Adam/update/_174 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2413_Adam/update", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3', defined at:
File "/home/pedro/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/pedro/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in
app.launch_new_instance()
File "/home/pedro/anaconda3/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
app.start()
File "/home/pedro/anaconda3/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 477, in start
ioloop.IOLoop.instance().start()
File "/home/pedro/anaconda3/lib/python3.6/site-packages/zmq/eventloop/ioloop.py", line 177, in start
super(ZMQIOLoop, self).start()
File "/home/pedro/anaconda3/lib/python3.6/site-packages/tornado/ioloop.py", line 888, in start
handler_func(fd_obj, events)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
self._handle_recv()
File "/home/pedro/anaconda3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
self._run_callback(callback, msg)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
callback(*args, **kwargs)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/tornado/stack_context.py", line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 283, in dispatcher
return self.dispatch_shell(stream, msg)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 235, in dispatch_shell
handler(stream, idents, msg)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/ipykernel/kernelbase.py", line 399, in execute_request
user_expressions, allow_stdin)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/ipykernel/ipkernel.py", line 196, in do_execute
res = shell.run_cell(code, store_history=store_history, silent=silent)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/ipykernel/zmqshell.py", line 533, in run_cell
return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2698, in run_cell
interactivity=interactivity, compiler=compiler, result=result)
File "/home/pedro/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2802, in run_ast_nodes
if self.run_code(code, result):
File "/home/pedro/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 16, in
net = NER(corp, **model_params)
File "/home/pedro/repositorios/russian-ner/ner/network.py", line 184, in init
self._train_op = self.get_train_op(loss, learning_rate_ph, lr_decay_rate=learning_rate_decay_ph)
File "/home/pedro/repositorios/russian-ner/ner/network.py", line 375, in get_train_op
train_op = tf.train.AdamOptimizer(learning_rate).minimize(loss, global_step=global_step, var_list=variables)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 343, in minimize
grad_loss=grad_loss)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 414, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 581, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 353, in _MaybeCompile
return grad_fn() # Exit early
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 581, in
grad_scope, op, func_call, lambda: grad_fn(op, *out_grads))
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_grad.py", line 186, in _TensorArrayScatterGrad
grad = g.gather(indices)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 361, in gather
element_shape=element_shape)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4158, in _tensor_array_gather_v3
flow_in=flow_in, dtype=dtype, element_shape=element_shape, name=name)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
...which was originally created as op 'rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3', defined at:
File "/home/pedro/anaconda3/lib/python3.6/runpy.py", line 193, in run_module_as_main
"main", mod_spec)
[elided 19 identical lines from previous traceback]
File "", line 16, in
net = NER(corp, **model_params)
File "/home/pedro/repositorios/russian-ner/ner/network.py", line 138, in init
sequence_lengths)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/contrib/crf/python/ops/crf.py", line 170, in crf_log_likelihood
log_norm = crf_log_norm(inputs, sequence_lengths, transition_params)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/contrib/crf/python/ops/crf.py", line 137, in crf_log_norm
dtype=dtypes.float32)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py", line 614, in dynamic_rnn
dtype=dtype)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py", line 728, in dynamic_rnn_loop
for ta, input in zip(input_ta, flat_input))
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/rnn.py", line 728, in
for ta, input in zip(input_ta, flat_input))
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 107, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 414, in unstack
indices=math_ops.range(0, num_elements), value=value, name=name)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 107, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 442, in scatter
name=name)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4553, in _tensor_array_scatter_v3
flow_in=flow_in, name=name)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/pedro/virtualenv/machine-learning/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
UnimplementedError (see above for traceback): TensorArray has size zero, but element shape is not fully defined. Currently only static shapes are supported when packing zero-size TensorArrays.
[[Node: gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGatherV3 = TensorArrayGatherV3[dtype=DT_FLOAT, element_shape=, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/TensorArrayGradV3, rnn/TensorArrayUnstack/range, gradients/rnn/TensorArrayUnstack/TensorArrayScatter/TensorArrayScatterV3_grad/TensorArrayGrad/gradient_flow)]]
[[Node: Adam/update/_174 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2413_Adam/update", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
On first time init, i had a ReadError after auto-downloading and extracting ner_model_total_rus.tar.gz:
~/speech/lib/python3.5/tarfile.py in open(cls, name, mode, fileobj, bufsize, **kwargs)
1560 fileobj.seek(saved_pos)
1561 continue
-> 1562 raise ReadError("file could not be opened successfully")
1563
1564 elif ":" in mode:
ReadError: file could not be opened successfully
Then, on calling extractor = ner.Extractor()
i haveFileNotFoundError: [Errno 2] No such file or directory: '/home/user/semantic_venv/lib/python3.5/site-packages/ner/extractor/../model/params.json'
When I try to execute the command sudo pip3 install -r ./requirements.txt --user
with dependencies from this repository:
numpy==1.13.1
tensorflow==1.3.0
pymorphy2==0.8
pymorphy2-dicts==2.4.393442.3710985
tqdm==4.19.5
requests==2.18.4
gensim==2.3.0
I get the error:
Could not find a version that satisfies the requirement tensorflow==1.3.0 (from -r ./requirements.txt (line 2)) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1)
No matching distribution found for tensorflow==1.3.0 (from -r ./requirements.txt (line 2))
Just installed NER from Gtihub.
The results in example notebook looks corrupted:
На B-PER
конспирологическом I-PER
саммите O
в B-PER
США I-PER
глава O
Федерального O
Бюро O
Расследований O
сделал O
невероятное O
заявление O
Moreover, re-running example.ipynb
leads to downloading the model again (MD5 sum is wrong). Perhaps, the model was changed on MIPT website?
Code from training_example achieve only 55 F1 in FactRuEval 2016 (82.10 F1 in your report).
What could be the problem?
Please, take a look at this line
model_url='http://lnsigo.mipt.ru/export/ner/ner_model_total_rus.tar.gz'
Probably the correct link is http://lnsigo.mipt.ru/export/models/ner/ner_model_total_rus.tar.gz
http://lnsigo.mipt.ru/export/ner/ner_model_total_rus.tar.gz
Excited to try out your package!
When you pass an empty string or string without normal tokens (for example, ascii emoticons), exception is raised:
>>> import ner
>>> extractor = ner.Extractor()
>>> list(extractor(':)')) # works fine — empty list is returned
>>> list(extractor('')) # fails
>>> list(extractor('|*!*|')) # fails
Maybe these cases should return an empty generator too? Trace:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/extractor/extractor.py", line 54, in __call__
tags = self.network.predict_for_token_batch([tokens_lemmas])[0]
File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/network.py", line 379, in predict_for_token_batch
batch_x, _ = self.corpus.tokens_batch_to_numpy_batch(tokens_batch)
File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/corpus.py", line 205, in tokens_batch_to_numpy_batch
max_token_len = max([len(token) for utt in batch_x for token in utt])
ValueError: max() arg is an empty sequence
I am using tensorflow 1.14.
when I try to run training, i encountered this error.
please help me to resolve this.
mldl@mldlUB1604:/ub16_prj/ner$ python3 ner.py/ub16_prj/ner$
Downloading from http://lnsigo.mipt.ru/export/models/ner/ner_model_total_rus.tar.gz to model/ner_model_total_rus.tar.gz
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 44.2M/44.3M [01:07<00:00, 696kB/s]Extracting model/ner_model_total_rus.tar.gz archive into model/
/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gradients_impl.py:95: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 44.3M/44.3M [01:20<00:00, 696kB/s]
Traceback (most recent call last):
File "ner.py", line 61, in
print_predict(query)
File "ner.py", line 55, in print_predict
tags = network.predict_for_token_batch([tokens_lemmas])[0]
File "/home/mldl/ub16_prj/ner/ner/network.py", line 379, in predict_for_token_batch
batch_x, _ = self.corpus.tokens_batch_to_numpy_batch(tokens_batch)
File "/home/mldl/ub16_prj/ner/ner/corpus.py", line 205, in tokens_batch_to_numpy_batch
max_token_len = max([len(token) for utt in batch_x for token in utt])
ValueError: max() arg is an empty sequence
mldl@mldlUB1604:
How do I use the results to predict on new sentences. Do I save the architecture and trained model file. Want something similar to “example.ipynb”, but on English data.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.