Comments (4)
Hi!
Thanks for your interest. Could you please specify when do you get this error (training or after) because the code is normally running on my workstation? Maybe, you could try to reshape the output tensor in the generator (line 90: batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.sequence_length,1) )
from image-nilm.
@BHafsa
Hi,
Thank you for your reply!
I get these errors in training
the model !
I don't find generator.py
, and I find the batchgenerator.py
in folder disaggregate
.
I change the code in batchgenerator.py
in line 81
:
original code:
69 def __getitem__(self, idx) :
70 """Generates the image representation for each sample
71
72 Args:
73 idx (int): the index of first element in the batch
74
75 Returns:
76 (batch_x, batch_y): the augmented data for the current batch
77 """
78 batch_x = self.x_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:]
79 batch_x = self.ts_imaging(batch_x).reshape(-1,self.img_size,self.img_size,1)
80 if self.y_sequence is not None:
81 batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:]
82 else:
83 batch_y = None
84
85 return batch_x, batch_y
change the code:
69 def __getitem__(self, idx) :
70 """Generates the image representation for each sample
71
72 Args:
73 idx (int): the index of first element in the batch
74
75 Returns:
76 (batch_x, batch_y): the augmented data for the current batch
77 """
78 batch_x = self.x_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:]
79 batch_x = self.ts_imaging(batch_x).reshape(-1,self.img_size,self.img_size,1)
80 if self.y_sequence is not None:
81 batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.img_size,self.img_size,1) ####reshape the output
82 else:
83 batch_y = None
84
85 return batch_x, batch_y
error:
File "D:\code\image-nilm-main\image-nilm-main\src\disaggregate\batchgenerator.py", line 81, in __getitem__
batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.img_size,self.img_size,1)
ValueError: cannot reshape array of size 7680 into shape (120,120,1)
I use the exepriments.py
in the folder src
. Because I don't have other datasets like SynD
and REFIT
, I just comment these line in experiments.py
. And this is my test code.
import warnings, os, pickle, yaml
warnings.filterwarnings("ignore")
import tensorflow as tf
tf.config.optimizer.set_jit(True)
# u_name = os.uname()[1]
#
# if u_name == 'user-Legion-T530-28ICB':
# os.environ['CUDA_VISIBLE_DEVICES'] = '0'
#
# if u_name == 'ultron':
# os.environ['CUDA_VISIBLE_DEVICES'] = '1'
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from sys import argv
from time import time
from copy import deepcopy
# NILMTK Imports
from nilmtk.api import API
from disaggregate.Im2Seq import Im2Seq
################### SETTINGS ###################
DATA_DIR = './data/'
settings = {
'metrics': ['mae', 'nde', 'f1score', 'r2score'],
'batch_size': 64,
'epochs': 10,
'sampling_interval': 10,
'sequence_length': 120,
'chunksize': 200,
'DROP_ALL_NANS': True,
'experiment': {}
}
try:
with open("out/I2S-settings.yaml", "w") as file:
documents = yaml.dump(settings, file)
except FileNotFoundError:
os.mkdir('out/')
with open("out/I2S-settings.yaml", "w") as file:
documents = yaml.dump(settings, file)
disaggregation_methods = {
# OLD-SCHOOL NILM Algorithms
# 'FHMMExact': FHMMExact({}),
# baseline DNN Approaches
# 'DAE': DAE({
# 'n_epochs': settings['epochs'],
# 'batch_size': settings['batch_size'],
# 'sequence_length': settings['sequence_length']
# }
# ),
# Im2Seq DNN Approaches
'CNN_MTF': Im2Seq({
'n_epochs': settings['epochs'],
'batch_size': settings['batch_size'],
'img_method': 'mtf',
'sequence_length': settings['sequence_length'],
}),
'CNN_GASF': Im2Seq({
'n_epochs': settings['epochs'],
'batch_size': settings['batch_size'],
'img_method': 'gasf',
'sequence_length': settings['sequence_length'],
}),
'CNN_RP': Im2Seq({
'n_epochs': settings['epochs'],
'batch_size': settings['batch_size'],
'img_method': 'rp',
'sequence_length': settings['sequence_length'],
})
}
################### DEFINE EXPERIMENTS ###################
basic_experiments = {
# 'SynD-1': {
# 'data_set': 'SynD',
# 'house': 1,
# 'ac_type': ['active'],
# 'f1': {
# 'train_dates': ['2019-10-01', '2020-01-01'],
# 'test_dates': ['2020-02-01', '2020-02-15'],
# },
# 'appliances': [
# 'fridge',
# 'dish washer',
# 'washing machine',
# 'microwave'
# ]
# },
'UK-2': {
'data_set': 'ukdale2',
'house': 1,
'ac_type': 'active',
'f1': {
'train_dates': ['2013-07-25', '2013-10-25'],
'test_dates': ['2013-10-15', '2013-11-30'],
},
'appliances': [
'fridge',
'dish washer',
'washing machine',
'microwave'
]
},
# 'REFIT-5': {
# 'data_set': 'REFIT',
# 'house': 6,
# 'ac_type': 'active',
# 'f1': {
# 'train_dates': ['2014-07-25', '2014-10-25'],
# 'test_dates': ['2014-11-15', '2014-11-30'],
# },
# 'appliances': [
# 'fridge',
# 'dish washer',
# 'washing machine',
# 'microwave'
# ]
# }
}
################### CONDUCT EXPERIMENTS ###################
for ex_name, experiment in basic_experiments.items():
print('Doing {} now...'.format(ex_name))
experiment_f1 = {
'power': {
'mains': experiment['ac_type'],
'appliance': experiment['ac_type']
},
'sample_rate': settings['sampling_interval'],
'appliances': experiment['appliances'],
'chunksize': settings['chunksize'],
'DROP_ALL_NANS': settings['DROP_ALL_NANS'],
'methods': disaggregation_methods,
'train': {
'datasets': {
'{}'.format(experiment['data_set']): {
'path': '{}{}.h5'.format(DATA_DIR, experiment['data_set']),
'buildings': {
experiment['house']: {
'start_time': experiment['f1']['train_dates'][0],
'end_time': experiment['f1']['train_dates'][1]
}
}
}
}
},
'test': {
'datasets': {
'{}'.format(experiment['data_set']): {
'path': '{}{}.h5'.format(DATA_DIR, experiment['data_set']),
'buildings': {
experiment['house']: {
'start_time': experiment['f1']['test_dates'][0],
'end_time': experiment['f1']['test_dates'][1]
}
}
}
},
'metrics': settings['metrics']
}
}
################### START ###################
start = time()
# Conduct experiment in NILMTK
api_results_f1 = API(experiment_f1)
error_df_f1 = api_results_f1.errors
error_keys_df_f1 = api_results_f1.errors_keys
# Save results in Pickle file.
df_dict = {
'error_keys': api_results_f1.errors_keys,
'errors': api_results_f1.errors,
'train_mains': api_results_f1.train_mains,
'train_submeters': api_results_f1.train_submeters,
'test_mains': api_results_f1.test_mains,
'test_submeters': api_results_f1.test_submeters,
'gt': api_results_f1.gt_overall,
'predictions': api_results_f1.pred_overall,
}
pickle.dump(df_dict, open("out/i2s_{}-{}-df_dict.p".format(experiment['data_set'], experiment['house']), "wb"))
################### RESULTS ###################
print('Experiment took: {} minutes'.format(round((time()-start)/60,1)))
for metric, f1_errors in zip(error_keys_df_f1, error_df_f1):
ff_errors = round(f1_errors, 3)
ff_errors.to_csv('out/i2s_{}.csv'.format(metric), sep='\t')
exit()
error:
runfile('D:/code/image-nilm-main/image-nilm-main/src/experiments.py')
Reloaded modules: disaggregate.Im2Seq, transformations.mtf, transformations.descritizer, disaggregate.batchgenerator
Doing UK-2 now...
Started training for Im2Seq_mtf
Joint training for Im2Seq_mtf
............... Loading Data for training ...................
Loading data for ukdale2 dataset
Loading building ... 1
Dropping missing values
...............Im2Seq_mtf partial_fit running...............
First model training for fridge
training with GPU
Epoch 1/10
Traceback (most recent call last):
File "D:\code\image-nilm-main\image-nilm-main\src\experiments.py", line 196, in <module>
api_results_f1 = API(experiment_f1)
File "E:\Software\Anaconda3\envs\nilmtk\lib\nilmtk\api.py", line 62, in __init__
self.experiment(params)
File "E:\Software\Anaconda3\envs\nilmtk\lib\nilmtk\api.py", line 107, in experiment
self.train_jointly(clf,d)
File "E:\Software\Anaconda3\envs\nilmtk\lib\nilmtk\api.py", line 260, in train_jointly
clf.partial_fit(self.train_mains,self.train_submeters)
File "D:\code\image-nilm-main\image-nilm-main\src\disaggregate\Im2Seq.py", line 144, in partial_fit
validation_steps=int(len(v_y) // self.batch_size))
File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1433, in fit_generator
steps_name='steps_per_epoch')
File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\engine\training_generator.py", line 220, in model_iteration
batch_data = _get_next_batch(generator, mode)
File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\engine\training_generator.py", line 362, in _get_next_batch
generator_output = next(generator)
File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\utils\data_utils.py", line 785, in get
six.reraise(*sys.exc_info())
File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\six.py", line 703, in reraise
raise value
File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\utils\data_utils.py", line 779, in get
inputs = self.queue.get(block=True).get()
File "E:\Software\Anaconda3\envs\nilmtk\lib\multiprocessing\pool.py", line 657, in get
raise self._value
File "E:\Software\Anaconda3\envs\nilmtk\lib\multiprocessing\pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\utils\data_utils.py", line 571, in get_index
return _SHARED_SEQUENCES[uid][i]
File "D:\code\image-nilm-main\image-nilm-main\src\disaggregate\batchgenerator.py", line 81, in __getitem__
batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.img_size,self.img_size,1)
ValueError: cannot reshape array of size 7680 into shape (120,120,1)
Could you help me ?
Thank you very much!
Best wishes!
from image-nilm.
Hi again!
You should reshape it this way self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.sequence_length,1) and not this way (-1,self.img_size,self.img_size,1)
from image-nilm.
Hey,
Thank you again!
It works! Cool!
from image-nilm.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from image-nilm.