Giter Club home page Giter Club logo

Comments (1)

dryglicki avatar dryglicki commented on June 30, 2024 1

I have nothing to provide you but solidarity. I am running into this same problem with a TFRecords data pipeline:

def _parse_function(example_proto):
    feature_description = {
            'ny'  : tf.io.FixedLenFeature([], tf.int64, default_value = 0),
            'nx'  : tf.io.FixedLenFeature([], tf.int64, default_value = 0),
            'ntp' : tf.io.FixedLenFeature([], tf.int64, default_value = 0),
            'ntf' : tf.io.FixedLenFeature([], tf.int64, default_value = 0),
            'ncp' : tf.io.FixedLenFeature([], tf.int64, default_value = 0),
            'ncf' : tf.io.FixedLenFeature([], tf.int64, default_value = 0),
            'priors' : tf.io.FixedLenFeature([], tf.string, default_value = ''),
            'forecasts' : tf.io.FixedLenFeature([], tf.string, default_value = ''),
            }
    features = tf.io.parse_example(example_proto, feature_description)
    priors = tf.io.parse_tensor(features['priors'], tf.float32)
    forecasts = tf.io.parse_tensor(features['forecasts'], tf.float32)

    ny  = features['ny']
    nx  = features['nx']
    ntp = features['ntp']
    ntf = features['ntf']
    ncp = features['ncp']
    ncf = features['ncf']

    priors = tf.reshape(priors, shape = [ntp, ny, nx, ncp])
    forecasts = tf.reshape(forecasts, shape = [ntf, ny, nx, ncf])

    return priors, forecasts

...
def create_dataset_onr_tfrecords(path,
                                 glob,
                                 batch_size = 32,
                                 compression = 'GZIP',
                                 shuffle = True,
                                 deterministic = False):
    return tf.data.Dataset.list_files(str(path / glob), shuffle = shuffle).interleave(
            lambda x: tf.data.TFRecordDataset(x, compression_type = compression),
                    cycle_length = tf.data.AUTOTUNE,
                    num_parallel_calls = tf.data.AUTOTUNE,
                    deterministic = deterministic
                    ).map(
                        _parse_function,
                        num_parallel_calls = tf.data.AUTOTUNE
                    ).batch(
                        batch_size, drop_remainder = True
                    ).prefetch(tf.data.AUTOTUNE)

I'll spare you the plot, but I am having the same issue with a vanilla TF dataset. I've tried removing interleave, removing GZIP compression, calling TFRecordDataset directly, removed batching, removed prefetching... nothing.

I believe this is a Tensorflow problem and (in particular) a TF Dataset problem: tensorflow/tensorflow#65675

This TF 2.16 + K3 era has been a disaster. Not the Keras part -- just some growing pains. But TF, man...

from keras.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.