guillaumegenthial / tf_metrics Goto Github PK

I attempted to install the package from Jupyter Notebook, using the code suggested in the raeadme file. Unfortunately, I obtain the following error. I also tried to install it from the terminal but nothing changes. Any clue about it? Any mistake in the code? Am I missing some required packages? Thanks

What about

def has_gpu():
try:
p = Popen(["nvidia-smi"], stdout=PIPE)
stdout, stderror = p.communicate()
if p.returncode != 0:
return False
return True

?

Hi there, thanks for the very useful lib.

As soon as I installed tf_metrics on my azure VM, I started getting the libcublas error for importing tensorflow which never occurred before. What's the installation requirements for tf_metrics? Is there any caution for installing tf_metrics within a specific environment?

I use tensorflow '1.12.0-rc0' gpu with cuda-9.2

/usr/local/lib/python3.6/dist-packages/tf_metrics/init.py in precision(labels, predictions, num_classes, pos_indices, weights, average)
40 """
41 cm, op = _streaming_confusion_matrix(
---> 42 labels, predictions, num_classes, weights)
43 pr, _, _ = metrics_from_confusion_matrix(
44 cm, pos_indices, average=average)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/metrics_impl.py in _streaming_confusion_matrix(labels, predictions, num_classes, weights)
262
263 # Flatten the input if its rank > 1.
--> 264 if predictions.get_shape().ndims > 1:
265 predictions = array_ops.reshape(predictions, [-1])
266

TypeError: '>' not supported between instances of 'NoneType' and 'int'

In python 3, the comparison operators raise an error. Refer to -> https://docs.python.org/3/whatsnew/3.0.html#ordering-comparisons
The ordering comparison operators (<, <=, >=, >) raise a TypeError exception when the operands don’t have a meaningful natural ordering. Thus, expressions like 1 < '', 0 > None or len <= len are no longer valid, and e.g. None < None raises TypeError instead of returning False. A corollary is that sorting a heterogeneous list no longer makes sense – all the elements must be comparable to each other. Note that this does not apply to the == and != operators: objects of different incomparable types always compare unequal to each other.

Hey man, I want to ask the meaning of parameter pos_indices.

Which is the pos_indices in a multi-class task like 'Image Classify'? You can not say that cat or dog is positive if they have no opposite class, like good and bad.

Hello~

i used tf_metrics in https://github.com/dsindex/BERT-BiLSTM-CRF-NER/blob/master/bert_lstm_ner.py

    estimator = tf.contrib.tpu.TPUEstimator(
        use_tpu=FLAGS.use_tpu,
        model_fn=model_fn,
        config=run_config,
        train_batch_size=FLAGS.train_batch_size,
        eval_batch_size=FLAGS.eval_batch_size,
        predict_batch_size=FLAGS.predict_batch_size)

...

        def metric_fn(label_ids, pred_ids, per_example_loss, input_mask):
                    # ['<pad>'] + ["O", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC", "B-MISC", "I-MISC", "X"]
                    indices = [2, 3, 4, 5, 6, 7, 8, 9]
                    precision = tf_metrics.precision(label_ids, pred_ids, num_labels, indices, input_mask)
                    recall = tf_metrics.recall(label_ids, pred_ids, num_labels, indices, input_mask)
                    f = tf_metrics.f1(label_ids, pred_ids, num_labels, indices, input_mask)
                    accuracy = tf.metrics.accuracy(label_ids, pred_ids, input_mask)
                    loss = tf.metrics.mean(per_example_loss)
                    return {
                        'eval_precision': precision,
                        'eval_recall': recall,
                        'eval_f': f,
                        'eval_accuracy': accuracy,
                        'eval_loss': loss,
                    }
                eval_metrics = (metric_fn, [label_ids, pred_ids, per_example_loss, input_mask])
                output_spec = tf.contrib.tpu.TPUEstimatorSpec(
                    mode=mode,
                    loss=total_loss,
                    eval_metrics=eval_metrics,
                    scaffold_fn=scaffold_fn)

...

and the training script https://github.com/dsindex/BERT-BiLSTM-CRF-NER/blob/master/train.sh

python bert_lstm_ner.py   \
        --task_name="NER"  \
        --do_train=True   \
        --use_feature_based=False \
        --do_predict=True \
        --use_crf=True \
        --data_dir=${CDIR}/NERdata  \
        --vocab_file=${bert_model_dir}/vocab.txt  \
        --do_lower_case=${lowercase} \
        --bert_config_file=${bert_model_dir}/bert_config.json \
        --init_checkpoint=${bert_model_dir}/bert_model.ckpt   \
        --max_seq_length=150   \
        --lstm_size=256 \
        --train_batch_size=32   \
        --eval_batch_size=128   \
        --predict_batch_size=128   \
        --bert_dropout_rate=0.1 \
        --bilstm_dropout_rate=0.1 \
        --learning_rate=2e-5   \
        --num_train_epochs=100   \
        --data_config_path=${CDIR}/data.conf \
        --output_dir=${CDIR}/output/result_dir/

the situation is,

when i set the train_batch_size 32 -> 6 and eval_batch_size 128 to 6, eval_f scores are not same.
exactly speaking, the lower batch_size the higher eval_f score we have.
is it normal?

support auc?