nusnlp / nea Goto Github PK

View Code? Open in Web Editor NEW

199.0 9.0 72.0 104 KB

Neural Essay Assessor: An Automated Essay Scoring System Based on Deep Neural Networks

License: GNU General Public License v3.0

Python 100.00%

nea's Introduction

Neural Essay Assessor

An automatic essay scoring system based on convolutional and recurrent neural networks, including GRU and LSTM.

Set Up

Install Keras (with Theano backend)
Prepare data
Run train_nea.py

Data

We have used 5-fold cross validation on ASAP dataset to evaluate our system. This dataset (training_set_rel3.tsv) can be downloaded from here. After downloading the file, put it in the data directory and create training, development and test data using preprocess_asap.py script:

cd data
python preprocess_asap.py -i training_set_rel3.tsv

Options

You can see the list of available options by running:

python train_nea.py -h

Example

The following command trains a model for prompt 1 in the ASAP dataset, using the training and development data from fold 0 and evaluates it.

THEANO_FLAGS="device=gpu0,floatX=float32" python train_nea.py
	-tr data/fold_0/train.tsv
	-tu data/fold_0/dev.tsv
	-ts data/fold_0/test.tsv
	-p 1	# Prompt ID
	--emb embeddings.w2v.txt
	-o output_dir

Frequently Asked Questions

See our FAQ page for a list of frequently asked questions. If the answer to your question is not there, contact me ([email protected]).

License

Neural Essay Assessor is licensed under the GNU General Public License Version 3. Separate commercial licensing is also available. For more information contact:

Kaveh Taghipour ([email protected])
Hwee Tou Ng ([email protected])

Publication

Kaveh Taghipour and Hwee Tou Ng. 2016. A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

nea's People

Contributors

Stargazers

Watchers

nea's Issues

Have anyone reproduced the result?

I have tried to reproduce the result, by got QWK much less than that in the paper. Here is my log for prompt 1, fold_0:

log.txt

Did i do something wrong?

TensorFlow object with the sum method

Hi,

When I try to run the code using the default parameters;

python train_nea.py -tr data/fold_0/train.tsv -tu data/fold_0/dev.tsv -ts data/fold_0/test.tsv -p 1 -o output_dir

I get the following error:

nea/nea/my_layers.py", line 59, in call
    return K.cast(x.sum(axis=1) / mask.sum(axis=1, keepdims=True), K.floatx())
AttributeError: 'Tensor' object has no attribute 'sum'

Any thoughts ?

Thanks !

TypeError: Expected int32, got <tf.Variable 'lstm_1_W_i:0' shape=(50, 300) dtype=float32_ref> of type 'Variable' instead.

hello ~ I got a problem as the following

�[94m[INFO]�[0m (nea.models) Building a REGRESSION model with POOLING
Traceback (most recent call last):
  File "train_nea.py", line 160, in <module>
    model = create_model(args, train_y.mean(axis=0), overal_maxlen, vocab)
  File "/data1/wbxu/AES/nea-master/nea/models.py", line 63, in create_model
    model.add(RNN(args.rnn_dim, return_sequences=True, dropout_W=dropout_W, dropout_U=dropout_U))
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/keras/models.py", line 308, in add
    output_tensor = layer(self.outputs[0])
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/keras/engine/topology.py", line 487, in __call__
    self.build(input_shapes[0])
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/keras/layers/recurrent.py", line 710, in build
    self.W = K.concatenate([self.W_i, self.W_f, self.W_c, self.W_o])
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 716, in concatenate
    return tf.concat(axis, [to_dense(x) for x in tensors])
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1043, in concat
    dtype=dtypes.int32).get_shape(
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 676, in convert_to_tensor
    as_ref=False)
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 741, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 113, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 102, in constant
    tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 374, in make_tensor_proto
    _AssertCompatible(values, dtype)
  File "/home/wbxu/anaconda3/envs/nea-master/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 302, in _AssertCompatible
    (dtype.name, repr(mismatch), type(mismatch).__name__))
TypeError: Expected int32, got <tf.Variable 'lstm_1_W_i:0' shape=(50, 300) dtype=float32_ref> of type 'Variable' instead.

my env is

keras 1.1.0
tensorflow 1.6.0
teano 0.8.2
nltk 3.0.0

How to use the model to predict score for new essays?

I use your embedding.txt ,emb_dim=50,but I will error ,I need to help. THX!

[INFO] (nea.w2vEmbReader) Loading embeddings from: data/embeddings.w2v.txt
Traceback (most recent call last):
File "train_nea.py", line 160, in
model = create_model(args, train_y.mean(axis=0), overal_maxlen, vocab)
File "/home/zetao/nea-master/nea/models.py", line 133, in create_model
emb_reader = EmbReader(args.emb_path, emb_dim=args.emb_dim)
File "/home/zetao/nea-master/nea/w2vEmbReader.py", line 47, in init
assert self.emb_dim == emb_dim, 'The embeddings dimension does not match with the requested dimension'
AssertionError: The embeddings dimension does not match with the requested dimension

Versioning of Keras and Theano

What versions of keras and theano did you use?

I often find the most difficult part of getting a code like this to run is specifying all the unique versions of everything. I wonder if we could do some sort of container in the future (like docker)...

Really looking forward to trying it all out.

What versions of python,theano,keras and tensorflow did you use? I am facing issues with tensorflow.

Originally posted by @nahos in #13 (comment)

ValueError: Error when checking target: expected activation_1 to have 3 dimensions, but got array with shape (356, 1)

When I tried to run the codebase it returns such error.

I guess it's probably because of the updated Keras 2 API, but could you try to build it again?

Thanks a lot ;)

ValueError: Shapes (4, 50, 300) and () are incompatible

Loading the model

Loading a trained model requires the custom objects which were defined in the my_layers.py. Any tips for loading them with the model archs?

example code:
from keras.models import model_from_json

with open('model_arch.json', 'r') as f: model = model_from_json(file.read(f), custom_objects={'MeanOverTime':MeanOverTime}

Some documentation on the issue in others:
loading custom objects in keras

Invalid metric: mean_absolute_error

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1539: invalid start byte

When I run
python preprocess_asap.py -i training_set_rel3.tsv

There are the errors:
Traceback (most recent call last):
File "preprocess_asap.py", line 44, in
dataset = collect_dataset(args.input_file)
File "preprocess_asap.py", line 34, in collect_dataset
for line in f:
File "/home/wyn/anaconda3/envs/py3NLP/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 1539: invalid start byte

I don't know why do I get it?

For help!

AttributeError: 'Dense' object has no attribute 'b'

When I run "python train_nea.py -tr data/fold_0/train.tsv -tu data/fold_0/dev.tsv -ts data/fold_0/test.tsv -p 1 -o output"

I got an error like this:

/Users/yanshengjia/Desktop/aes/nea/nea/models.py:63: UserWarning: Update your LSTM call to the Keras 2 API: LSTM(300, return_sequences=True, dropout=0.5, recurrent_dropout=0.1)
model.add(RNN(args.rnn_dim, return_sequences=True, dropout_W=dropout_W, dropout_U=dropout_U))
/Users/yanshengjia/anaconda/lib/python2.7/site-packages/keras/engine/topology.py:621: UserWarning: Class nea.my_layers.MeanOverTime defines get_output_shape_for but does not override compute_output_shape. If this is a Keras 1 layer, please implement compute_output_shape to support Keras 2.
output_shape = self.compute_output_shape(input_shape)
Traceback (most recent call last):
File "train_nea.py", line 160, in
model = create_model(args, train_y.mean(axis=0), overal_maxlen, vocab)
File "/Users/yanshengjia/Desktop/aes/nea/nea/models.py", line 73, in create_model
model.layers[-1].b.set_value(bias_value)
AttributeError: 'Dense' object has no attribute 'b'

Run for all prompts

when runing train_nea.py -tr /data/fold_1/train.tsv -tu /data/fold_1/dev.tsv -ts /data/fold_1/test.tsv -p 0 -o /output_dir I get an error:

Traceback (most recent call last):
  File "/train_nea.py", line 63, in <module>
    raise NotImplementedError
NotImplementedError

when I run it for an individual prompt it works fine. Any Ideas how to solve this ?

Error while using embeddings from Glove2Vec

File "/home/deepamparmar/Music/nea-master/nea/models.py", line 132, in create_model
emb_reader = EmbReader(args.emb_path, emb_dim=args.emb_dim)
File "/home/deepamparmar/Music/nea-master/nea/w2vEmbReader.py", line 46, in init
assert self.emb_dim == emb_dim, 'The embeddings dimension does not match with the requested dimension'
AssertionError: The embeddings dimension does not match with the requested dimension