robromijnders / lstm_tsc Goto Github PK

View Code? Open in Web Editor NEW

407.0 407.0 148.0 3.36 MB

An LSTM for time-series classification

License: MIT License

Python 100.00%

lstm_tsc's Introduction

Update 10-April-2017

And now it works with Python3 and Tensorflow 1.1.0

Update 02-Jan-2017

I updated this repo. Now it works with Tensorflow 0.12. In this readme I comment on some new benchmarks

LSTM for time-series classification

This post implements a Long Short-term memory for time series classification(LSTM). An LSTM is the extension of the classical Recurrent Neural Network. It has more flexibility and interpretable features such as a memory it can read, write and forget.

Aim

This repo aims to show the minimal Tensorflow code for proper time series classification. The main function loads the data and iterates over training steps. The tsc_model.py scripts contains the actual model. This repo contrasts with another project where I implement a similar script using convolutional neural networks as the model

Data and results

The code generalizes for any of the UCR time series. With the parameter dataset you can run the code on any of their datasets. For your interests, you may compare performances with the nice overview in this paper. They benchmark their CNN and other models on many of the UCR time series datasets This code works amongst others for

Two_Patterns where it achieves state-of-the-art, bein 100% test accuracy
ChlorineConcentration where it achieves state-of-the-art, being 80% test accuracy

Credits

Credits for this project go to Tensorflow for providing a strong example, the UCR archive for the dataset and my friend Ryan for strong feedback.

License is in LICENSE.txt

As always, I am curious to any comments and questions. Reach me at [email protected]

lstm_tsc's People

Contributors

Stargazers

Watchers

Forkers

rwclarity paulhendricks 123mitnik boscong88 lazy404 stitchdeng siddhu95 saifrahmed stevenlol lethienhoa trelsco sourav-sahaji hedgefair akansal1 miguelperalvo ml-lab dejunliu 2dpodcast parrondo ozzie00 zzfnohell garry-pan rajserc guojin118 jayvischeng gdissard dragonkiss81 mrtaste2 azizishekoofeh thinhptran wjssx relue bashir505 allensmile wanjinchang shenxindi benjamesbabala nalsil akash13singh zhonghaoling sirwh sharifelguindi manila800 wangwocg kindpire kedarnath6970 renjithmadhavan tdaseed megamanics cha0s-repo dsadulla hiredd xuetengteng angusg ivanjacobs ioanszilagyi lancerts kartavya96 wangjunji weikang1993 shikarkhane jayden11 tboudreaux pwyang123 superintel akshaydarora knightofdawn iceberg12 gangyueqi zeyu-h simonsleo codeofgod zmm021 m-tian zzd837649498 twinkle123321 erichiller gapper100 gnarziheng varndorfer tzebin fanwei918 amoliu ammaralali gamency liuyemo qinritukou hongminwu skyjiao siqi-z mpqzf weihangchen wsyjwps1983 bekerov lopezfelipe tomfisher mennatareeq arvinganesan gil2abir lidan456

lstm_tsc's Issues

will it work for multivariate time series prediction both regression and classification

great code thanks
may you clarify :
will it work for multivariate time series prediction both regression and classification
1
where all values are continues values
weight height age target
1 56 160 34 1.2
2 77 170 54 3.5
3 87 167 43 0.7
4 55 198 72 0.5
5 88 176 32 2.3

2
or even will it work for multivariate time series where values are mixture of continues and categorical values
for example 2 dimensions have continues values and 3 dimensions are categorical values

color        weight     gender  height  age  target

1 black 56 m 160 34 yes
2 white 77 f 170 54 no
3 yellow 87 m 167 43 yes
4 white 55 m 198 72 no
5 white 88 f 176 32 yes

cost_train_ma value

I wonder why set the initial value of "cost_train_ma" to -np.log(1 / float(num_classes) + 1e-9)

TypeError 'The value of a feed cannot be a tf.Tensor object. '

raise type error with tensorflow 0.11rc0; numpy 1.11.1; python 2.7.12; macOS 10.12;

Train with approximately 60 epochs
tsc_main.py:23: VisibleDeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future
X_batch = X_train[ind_N,ind_start:ind_start+num_steps]
At 0 out of 2000: Cost is TRAIN1.734(1.390) VAL1.599 and val acc is 0.400
At 100 out of 2000: Cost is TRAIN1.380(1.387) VAL1.376 and val acc is 0.300
At 200 out of 2000: Cost is TRAIN1.288(1.339) VAL1.159 and val acc is 0.733
At 300 out of 2000: Cost is TRAIN1.010(1.171) VAL1.106 and val acc is 0.567
At 400 out of 2000: Cost is TRAIN0.935(1.044) VAL1.041 and val acc is 0.533
At 500 out of 2000: Cost is TRAIN1.058(0.982) VAL0.776 and val acc is 0.633
At 600 out of 2000: Cost is TRAIN0.774(0.908) VAL1.153 and val acc is 0.633
At 700 out of 2000: Cost is TRAIN0.823(0.866) VAL0.643 and val acc is 0.833
At 800 out of 2000: Cost is TRAIN0.626(0.814) VAL0.876 and val acc is 0.633
At 900 out of 2000: Cost is TRAIN1.140(0.793) VAL0.792 and val acc is 0.600
At 1000 out of 2000: Cost is TRAIN0.800(0.753) VAL0.835 and val acc is 0.733
At 1100 out of 2000: Cost is TRAIN0.890(0.746) VAL0.571 and val acc is 0.767
At 1200 out of 2000: Cost is TRAIN0.634(0.725) VAL1.061 and val acc is 0.400
At 1300 out of 2000: Cost is TRAIN0.851(0.715) VAL0.550 and val acc is 0.767
At 1400 out of 2000: Cost is TRAIN0.504(0.694) VAL0.682 and val acc is 0.600
At 1500 out of 2000: Cost is TRAIN0.647(0.706) VAL0.870 and val acc is 0.567
At 1600 out of 2000: Cost is TRAIN0.490(0.702) VAL0.430 and val acc is 0.800
At 1700 out of 2000: Cost is TRAIN0.585(0.695) VAL0.548 and val acc is 0.767
At 1800 out of 2000: Cost is TRAIN1.013(0.676) VAL0.638 and val acc is 0.800
At 1900 out of 2000: Cost is TRAIN0.937(0.649) VAL0.817 and val acc is 0.600
tsc_main.py:31: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
test_acc = np.zeros(num_batch)
Traceback (most recent call last):
File "tsc_main.py", line 178, in
acc_test = check_test(X_test,y_test,batch_size,num_steps)
File "tsc_main.py", line 34, in check_test
test_acc[i] = sess.run(accuracy,feed_dict = {input_data: X_batch, targets: y_batch, initial_state:state,keep_prob:1})
File "/usr/local/var/pyenv/versions/2.7.12/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 717, in run
run_metadata_ptr)
File "/usr/local/var/pyenv/versions/2.7.12/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 875, in _run
raise TypeError('The value of a feed cannot be a tf.Tensor object. '
TypeError: The value of a feed cannot be a tf.Tensor object. Acceptable feed values include Python scalars, strings, lists, or numpy ndarrays.

tf.nn.rnn.LSTMcell()

Hi there,

So it looks like in the latest version of TF (1.0?) - tf.nn.rnn.LSTMcell() has been replaced with tf.contrib.rnn.LSTMcell() - I just upgraded TF today and came across this,

tensorflow/tensorflow@b8101a2

AUC calculation

Hi!
Thanks for your code. In addition to accuracy, I am trying to obtain AUC values. I am suspecting that I am doing something wrong in that accuracy results are good but AUC are below 0.5.

This is what I have in the tsc_model.py file:
a = tf.cast(tf.argmax(logits, 1),tf.float32) #AUC
b = tf.cast(self.labels,tf.float32) #AUC
auc = tf.contrib.metrics.streaming_auc(a,b) #AUC
self.auc_value = tf.reduce_mean(tf.cast(auc, "float")) #AUC**

Then, in the main program, I have:
sess.run(tf.initialize_local_variables()) #for AUC
auc_test = sess.run([model.auc_value],feed_dict = {model.input: Xtest_batch, model.labels: ytest_batch, model.keep_prob:1.0})
print('Result of testing: AUC is %5.3f'%(auc_test))

***Please let me know if you spot an incorrect approach. Thanks!

cannot import name 'check_test'

In the commit Make compatible with python3 and TF1.1.0
you deleted the function check_test in the tsc_model.py , but forgot to delete in the code from tsc_model import Model,sample_batch,load_data,check_test in the tsc_main.py

Any chance you could choose a license type for this?

I might recommend MIT license as most likely to lead to more people using it 👍

TypeError: Fetch argument None has invalid type <class 'NoneType'>

Hi, first thing: your project is very interesting
Second: I got an error at the execution at line 164 : result = sess.run([cost,merged,accuracy],feed_dict = {input_data: X_batch, targets: y_batch, keep_prob:1})

Do you know where this could come from ?

Thank you in advance

Multiple TS for the same target variable

Hi there, thank you for this awesome tutorial, it is of great interest for my master research work!

I was trying to adapt the code to my problem, as for each user I want to classify I have three time series of the same length. Firstly I edited load_data and sample_batch in order to have: t1_train,t1_val,t1_test,t2_train,t2_val,t2_test,t3_train,t3_val,t3_test,y_train,y_val,y_test
and
t1_batch,t2_batch,t3_batch,y_batch

In the Model itself is where I'm sure I'm doing something wrong, up to now I've edited your code this way:

self.input1 = tf.placeholder(tf.float32, [None, sl], name = 'input_1')
    self.input2 = tf.placeholder(tf.float32, [None, sl], name = 'input_2')
    self.input3 = tf.placeholder(tf.float32, [None, sl], name = 'input_3')
    self.labels = tf.placeholder(tf.int64, [None], name='labels')
    self.keep_prob = tf.placeholder("float", name = 'Drop_out_keep_prob')

    with tf.name_scope("LSTM_setup") as scope:
      def single_cell1():
        return tf.contrib.rnn.DropoutWrapper(LSTMCell(hidden_size),output_keep_prob=self.keep_prob)

      cell1 = tf.contrib.rnn.MultiRNNCell([single_cell1() for _ in range(num_layers)])
      initial_state1 = cell1.zero_state(self.batch_size, tf.float32)
      
      def single_cell2():
        return tf.contrib.rnn.DropoutWrapper(LSTMCell(hidden_size),output_keep_prob=self.keep_prob)

      cell2 = tf.contrib.rnn.MultiRNNCell([single_cell2() for _ in range(num_layers)])
      initial_state2 = cell2.zero_state(self.batch_size, tf.float32)

      def single_cell3():
        return tf.contrib.rnn.DropoutWrapper(LSTMCell(hidden_size),output_keep_prob=self.keep_prob)

      cell3 = tf.contrib.rnn.MultiRNNCell([single_cell3() for _ in range(num_layers)])
      initial_state3 = cell3.zero_state(self.batch_size, tf.float32)
    
    input_list1 = tf.unstack(tf.expand_dims(self.input1,axis=2),axis=1)
    input_list2 = tf.unstack(tf.expand_dims(self.input2,axis=2),axis=1)    
    input_list3 = tf.unstack(tf.expand_dims(self.input3,axis=2),axis=1)

    outputs1,_ = core_rnn.static_rnn(cell1, input_list1, dtype=tf.float32)
    outputs2,_ = core_rnn.static_rnn(cell2, input_list2, dtype=tf.float32)
    outputs3,_ = core_rnn.static_rnn(cell3, input_list3, dtype=tf.float32)

    output = tf.concat([outputs1[-1], outputs2[-1], outputs3[-1]], 0)

When I try to train the network this is the error that I get:
ValueError: Attempt to have a second RNNCell use the weights of a variable scope that already has weights: 'rnn/multi_rnn_cell/cell_0/lstm_cell'; and the cell was not constructed as LSTMCell(..., reuse=True). To share the weights of an RNNCell, simply reuse it in your second calculation, or create a new one with the argument reuse=True.

Would you have any suggestion or tip for me to get it working? Thank you in advance!

Tensor Flow 1.5 needs the following code modifications to work.

in tsc_model.py

from::

from tensorflow.contrib.rnn.python.ops import core_rnn

to::

import tensorflow.contrib.rnn as core_rnn

No check_test in model.py

In tsc_main .py ,
from::

from tsc_model import Model,sample_batch,load_data,check_test

to::

from tsc_model import Model,sample_batch,load_data

Some Datasets like ECG200 have Negative class labels which makes the softmax classifier fail.

TypeError: Fetch argument None has invalid type <class 'NoneType'>

Hi, first thing: your project is very interesting.
Second: I got an error at the execution at

summary = sess.run( merged , {input_data: nextBatch, labels: nextBatchLabels})

Do you have any ideas how to fix it ?

Thank you in advance

Traceback (most recent call last):
File "Oriole+LSTM.py", line 220, in
summary = sess.run( merged , {input_data: nextBatch, labels: nextBatchLabels})
File "C:\Anaconda\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 789, in run
run_metadata_ptr)
File "C:\Anaconda\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 984, in _run
self._graph, fetches, feed_dict_string, feed_handles=feed_handles)
File "C:\Anaconda\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 410, in init
self._fetch_mapper = _FetchMapper.for_fetch(fetches)
File "C:\Anaconda\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 227, in for_fetch
(fetch, type(fetch)))
TypeError: Fetch argument None has invalid type <class 'NoneType'>

Deprecation Warnings

I am running into some deprecation warnings when running this code in a notebook.

WARNING:tensorflow:<tensorflow.python.ops.rnn_cell.BasicLSTMCell object at 0x7f8745d1eef0>: Using a concatenated state is slower and will soon be deprecated.  Use state_is_tuple=True.

/home/condauser/anaconda3/lib/python3.5/site-packages/ipykernel/__main__.py:7: DeprecationWarning: converting an array with ndim > 0 to an index will result in an error in the future

I think these are causing the script to stop, as I am getting an InvalidArgument error later on.

I am on tensorflow 0.9.0rc0.

download UCR

thank you for this awesome tutorial,now,I try to run it,however,when I read the full pdf provided by UCRArchive ,still can't find the password to unzip data.would you give me an obvious hint?

Accuracy vs Cost

Hi,
Thanks for the nice example. I have a question about your check_test() function. When you're running the session and generating the output, it's [model.cost, model.accuracy] but after append and tuple you have [accuracy, cost]. I'm afraid to say that it's a swap between accuracy and cost. Even, when I ran it on another time-series dataset, it generates the accuracy of greater than 1, which is definitely incorrect.

Thanks,
Shekoofeh

loadtxt “could not convert string to float”

in the 'tsc_model',
data_train = np.loadtxt(datadir+'_TRAIN.txt',delimiter=',')

The result is 'could not convert string to float:'
I don't know what is the reason. Thank you

Any destriptions for those datasets?

Could not understand the actual meaning for each dataset and where it comes from.