Giter Club home page Giter Club logo

Comments (4)

BHafsa avatar BHafsa commented on September 2, 2024

Hi!
Thanks for your interest. Could you please specify when do you get this error (training or after) because the code is normally running on my workstation? Maybe, you could try to reshape the output tensor in the generator (line 90: batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.sequence_length,1) )

from image-nilm.

Wusir2018 avatar Wusir2018 commented on September 2, 2024

@BHafsa
Hi,

Thank you for your reply!

I get these errors in training the model !

I don't find generator.py, and I find the batchgenerator.py in folder disaggregate.

I change the code in batchgenerator.py in line 81:

original code:

69    def __getitem__(self, idx) :
70       """Generates the image representation for each sample
71    
72        Args:
73            idx (int): the index of first element in the batch
74    
75        Returns:
76            (batch_x, batch_y): the augmented data for the current batch
77        """
78        batch_x = self.x_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:]
79        batch_x = self.ts_imaging(batch_x).reshape(-1,self.img_size,self.img_size,1)
80        if self.y_sequence is not None:
81            batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:]
82        else:
83            batch_y = None
84    
85        return batch_x, batch_y

change the code:

69    def __getitem__(self, idx) :
70       """Generates the image representation for each sample
71    
72        Args:
73            idx (int): the index of first element in the batch
74    
75        Returns:
76            (batch_x, batch_y): the augmented data for the current batch
77        """
78        batch_x = self.x_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:]
79        batch_x = self.ts_imaging(batch_x).reshape(-1,self.img_size,self.img_size,1)
80        if self.y_sequence is not None:
81            batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.img_size,self.img_size,1)  ####reshape the output
82        else:
83            batch_y = None
84    
85        return batch_x, batch_y

error:

File "D:\code\image-nilm-main\image-nilm-main\src\disaggregate\batchgenerator.py", line 81, in __getitem__
    batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.img_size,self.img_size,1)

ValueError: cannot reshape array of size 7680 into shape (120,120,1)

I use the exepriments.py in the folder src . Because I don't have other datasets like SynD and REFIT, I just comment these line in experiments.py. And this is my test code.

import warnings, os, pickle, yaml
warnings.filterwarnings("ignore")
import tensorflow as tf
tf.config.optimizer.set_jit(True)



# u_name = os.uname()[1]
#
# if u_name == 'user-Legion-T530-28ICB':
#     os.environ['CUDA_VISIBLE_DEVICES'] = '0'
#
# if u_name == 'ultron':
#     os.environ['CUDA_VISIBLE_DEVICES'] = '1'
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from sys import argv
from time import time
from copy import deepcopy

# NILMTK Imports
from nilmtk.api import API
from disaggregate.Im2Seq import Im2Seq


################### SETTINGS ###################

DATA_DIR = './data/'

settings = {
    'metrics': ['mae', 'nde', 'f1score', 'r2score'],
    'batch_size': 64,
    'epochs': 10,
    'sampling_interval': 10,
    'sequence_length': 120, 
    'chunksize': 200,
    'DROP_ALL_NANS': True,
    'experiment': {}
}

try:
    with open("out/I2S-settings.yaml", "w") as file:
        documents = yaml.dump(settings, file)

except FileNotFoundError:
    os.mkdir('out/')
    with open("out/I2S-settings.yaml", "w") as file:
        documents = yaml.dump(settings, file)

disaggregation_methods = {
    # OLD-SCHOOL NILM Algorithms
    # 'FHMMExact': FHMMExact({}),
    # baseline DNN Approaches
    # 'DAE': DAE({
    #         'n_epochs': settings['epochs'], 
    #         'batch_size': settings['batch_size'], 
    #         'sequence_length': settings['sequence_length']
    #     }
    #     ),
    # Im2Seq DNN Approaches
    'CNN_MTF': Im2Seq({
        'n_epochs': settings['epochs'],
        'batch_size': settings['batch_size'],
        'img_method': 'mtf',
        'sequence_length': settings['sequence_length'],
    }),

    'CNN_GASF': Im2Seq({
        'n_epochs': settings['epochs'],
        'batch_size': settings['batch_size'],
        'img_method': 'gasf',
        'sequence_length': settings['sequence_length'],
    }),

    'CNN_RP': Im2Seq({
        'n_epochs': settings['epochs'],
        'batch_size': settings['batch_size'],
        'img_method': 'rp',
        'sequence_length': settings['sequence_length'],
    })

}

################### DEFINE EXPERIMENTS ###################


basic_experiments = {


    # 'SynD-1': {
    #     'data_set': 'SynD',
    #     'house': 1,
    #     'ac_type': ['active'],

    #     'f1': {
    #         'train_dates': ['2019-10-01', '2020-01-01'],
    #         'test_dates': ['2020-02-01', '2020-02-15'],
    #     },

    #     'appliances': [
    #         'fridge',
    #         'dish washer',
    #         'washing machine',
    #         'microwave'
    #     ]
    # },


     'UK-2': {
        'data_set': 'ukdale2',
        'house': 1,
        'ac_type': 'active',

        'f1': {
            'train_dates': ['2013-07-25', '2013-10-25'],
            'test_dates': ['2013-10-15', '2013-11-30'],
        },

        'appliances': [
            'fridge',
            'dish washer',
            'washing machine',
            'microwave'
        ]
    },
    
    # 'REFIT-5': {
    #     'data_set': 'REFIT',
    #     'house': 6,
    #     'ac_type': 'active',

    #     'f1': {
    #         'train_dates': ['2014-07-25', '2014-10-25'],
    #         'test_dates': ['2014-11-15', '2014-11-30'],
    #     },

    #     'appliances': [
    #         'fridge',
    #         'dish washer',
    #         'washing machine',
    #         'microwave'
    #     ]
    # }
}

################### CONDUCT EXPERIMENTS ###################

for ex_name, experiment in basic_experiments.items():

    print('Doing {} now...'.format(ex_name))

    experiment_f1 = {
        'power': {
            'mains': experiment['ac_type'],
            'appliance': experiment['ac_type']
        },
        'sample_rate': settings['sampling_interval'],
        'appliances': experiment['appliances'],

        'chunksize': settings['chunksize'],
        'DROP_ALL_NANS': settings['DROP_ALL_NANS'],

        'methods': disaggregation_methods,
        'train': {
            'datasets': {
                '{}'.format(experiment['data_set']): {
                    'path': '{}{}.h5'.format(DATA_DIR, experiment['data_set']),
                    'buildings': {
                        experiment['house']: {
                            'start_time': experiment['f1']['train_dates'][0],
                            'end_time': experiment['f1']['train_dates'][1]
                        }
                    }
                }
            }
        },
        'test': {
            'datasets': {
                '{}'.format(experiment['data_set']): {
                    'path': '{}{}.h5'.format(DATA_DIR, experiment['data_set']),
                    'buildings': {
                        experiment['house']: {
                            'start_time': experiment['f1']['test_dates'][0],
                            'end_time': experiment['f1']['test_dates'][1]
                        }
                    }
                }
            },
            'metrics': settings['metrics']
        }
    }

    ################### START ###################

    start = time()
    # Conduct experiment in NILMTK
    api_results_f1 = API(experiment_f1)

    error_df_f1 = api_results_f1.errors
    error_keys_df_f1 = api_results_f1.errors_keys

    # Save results in Pickle file.
    df_dict = {
        'error_keys': api_results_f1.errors_keys,
        'errors': api_results_f1.errors,

        'train_mains': api_results_f1.train_mains,
        'train_submeters': api_results_f1.train_submeters,

        'test_mains': api_results_f1.test_mains,
        'test_submeters': api_results_f1.test_submeters,

        'gt': api_results_f1.gt_overall,
        'predictions': api_results_f1.pred_overall,
    }
    pickle.dump(df_dict, open("out/i2s_{}-{}-df_dict.p".format(experiment['data_set'], experiment['house']), "wb"))

    ################### RESULTS ###################
    print('Experiment took: {} minutes'.format(round((time()-start)/60,1)))

    for metric, f1_errors in zip(error_keys_df_f1, error_df_f1):
        ff_errors = round(f1_errors, 3)
        ff_errors.to_csv('out/i2s_{}.csv'.format(metric), sep='\t')

exit()

error:

runfile('D:/code/image-nilm-main/image-nilm-main/src/experiments.py')
Reloaded modules: disaggregate.Im2Seq, transformations.mtf, transformations.descritizer, disaggregate.batchgenerator
Doing UK-2 now...
Started training for  Im2Seq_mtf
Joint training for  Im2Seq_mtf
............... Loading Data for training ...................
Loading data for  ukdale2  dataset
Loading building ...  1
Dropping missing values
...............Im2Seq_mtf partial_fit running...............
First model training for  fridge
training with GPU
Epoch 1/10
Traceback (most recent call last):

  File "D:\code\image-nilm-main\image-nilm-main\src\experiments.py", line 196, in <module>
    api_results_f1 = API(experiment_f1)

  File "E:\Software\Anaconda3\envs\nilmtk\lib\nilmtk\api.py", line 62, in __init__
    self.experiment(params)

  File "E:\Software\Anaconda3\envs\nilmtk\lib\nilmtk\api.py", line 107, in experiment
    self.train_jointly(clf,d)

  File "E:\Software\Anaconda3\envs\nilmtk\lib\nilmtk\api.py", line 260, in train_jointly
    clf.partial_fit(self.train_mains,self.train_submeters)

  File "D:\code\image-nilm-main\image-nilm-main\src\disaggregate\Im2Seq.py", line 144, in partial_fit
    validation_steps=int(len(v_y) // self.batch_size))

  File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1433, in fit_generator
    steps_name='steps_per_epoch')

  File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\engine\training_generator.py", line 220, in model_iteration
    batch_data = _get_next_batch(generator, mode)

  File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\engine\training_generator.py", line 362, in _get_next_batch
    generator_output = next(generator)

  File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\utils\data_utils.py", line 785, in get
    six.reraise(*sys.exc_info())

  File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\six.py", line 703, in reraise
    raise value

  File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\utils\data_utils.py", line 779, in get
    inputs = self.queue.get(block=True).get()

  File "E:\Software\Anaconda3\envs\nilmtk\lib\multiprocessing\pool.py", line 657, in get
    raise self._value

  File "E:\Software\Anaconda3\envs\nilmtk\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))

  File "E:\Software\Anaconda3\envs\nilmtk\lib\site-packages\tensorflow\python\keras\utils\data_utils.py", line 571, in get_index
    return _SHARED_SEQUENCES[uid][i]

  File "D:\code\image-nilm-main\image-nilm-main\src\disaggregate\batchgenerator.py", line 81, in __getitem__
    batch_y = self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.img_size,self.img_size,1)

ValueError: cannot reshape array of size 7680 into shape (120,120,1)

Could you help me ?

Thank you very much!

Best wishes!

from image-nilm.

BHafsa avatar BHafsa commented on September 2, 2024

Hi again!
You should reshape it this way self.y_sequence[idx * self.batch_size : (idx+1) * self.batch_size,:].reshape(-1,self.sequence_length,1) and not this way (-1,self.img_size,self.img_size,1)

from image-nilm.

Wusir2018 avatar Wusir2018 commented on September 2, 2024

Hey,
Thank you again!
It works! Cool!

from image-nilm.

Related Issues (2)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.