hypergan / hypergan Goto Github PK

Composable GAN framework with api and user interface

License: MIT License

Python 98.70% Shell 0.03% C++ 0.12% Cuda 1.14% Batchfile 0.01%

gan unsupervised-learning generative-adversarial-network artificial-intelligence machine-learning machine-learning-api python online-learning computer-vision pytorch

hypergan's Introduction

README

HyperGAN 1.0

A composable GAN built for developers, researchers, and artists.

HyperGAN is in pre-release and open beta.

Logos generated with examples/colorizer

See more on the hypergan youtube

About
Documentation
Changelog
Quick start
API
The pip package hypergan
Datasets
Features
Showcase
Sponsors
Contributing
Versioning
Citation

About

HyperGAN builds generative adversarial networks in pytorch and makes them easy to train and share.

For a general introduction to GANs see http://blog.aylien.com/introduction-generative-adversarial-networks-code-tensorflow/

Join the community discord.

Documentation

Gitbook documentation

Changelog

See the full changelog here: Changelog.md

Quick start

Requirements

OS: Windows, OSX, Linux

For training:

GPU: Nvidia, GTX 1080+ recommended

Install

Install HyperGAN For users: pip3 install hypergan

For developers: Download this repo and run python3 setup.py develop

Test it out

hypergan train preset:celeba -s 128x128x3

Join the community

Once you've made something cool, be sure to share it on the Discord (https://discord.gg/t4WWBPF).

Create a new model

  hypergan new mymodel

This will create a mymodel.json based off the default configuration. You can change configuration templates with the -c flag.

List configuration templates

  hypergan new mymodel -l

See all configuration templates with --list-templates or -l.

Train

  hypergan train folder/ -s 32x32x3 -c mymodel --resize

API

import hypergan as hg

Note this API is currently under work in 1.0. If you are reading this before 1.0 is released check the examples.

See the gitbook documentation for more details.

Using a trained hypergan model

my_gan = hg.GAN('model.hypergan')
batch_sample = my_gan.sample()

Training a gan

gan = hg.GAN("default.json", inputs=hg.inputs.ImageLoader(...))
trainable_gan = hg.TrainableGAN(gan)
for step in trainable_gan.train():
    print("I'm on step ", step)

Examples

See the examples https://github.com/hypergan/HyperGAN/tree/master/examples

Tutorials

See the tutorials https://hypergan.gitbook.io/hypergan/tutorials

The pip package hypergan

pip install hypergan

Training

  # Train a 32x32 gan with batch size 32 on a folder of pngs
  hypergan train [folder] -s 32x32x3 -b 32 --config [name]

Sampling

  hypergan sample [folder] -s 32x32x3 -b 32 --config [name] --sampler batch_walk --save_samples

By default hypergan will not save training samples to disk. To change this, use --save_samples.

Additional Arguments

To see a detailed list, run

  hypergan -h

Running on CPU

You can switch the backend with:

  hypergan [...] -B cpu

Don't train on CPU! It's too slow.

Troubleshooting

Make sure that your cuda, nvidia drivers, pillow, pytorch, and pytorch vision are the latest version.

Check the discord for help.

Development mode

If you wish to modify hypergan

git clone https://github.com/hypergan/hypergan
cd hypergan
python3 setup.py develop

Make sure to pip3 uninstall hypergan to avoid version conflicts.

Datasets

To build a new network you need a dataset.

Creating a Dataset

Datasets in HyperGAN are meant to be simple to create. Just use a folder of images. Nested folders work too.

Cleaning up data

HyperGAN is built to be resilient to all types of unclean data. By default images are resized then cropped if necessary.

See --nocrop, --random_crop and --resize for additional image scaling options.

Features

A list of features in the 1.0 release:

API
CLI
Viewer - an electron app to explore and create models
Cross platform - Windows, OSX, Linux
Inference - Add AI content generation to your project
Training - Train custom models using accelerated parallel training backends
Sharing - Share built models with each other. Use them in python projects as hypergan models, or in any project as onxx models
Customizable - Define custom architectures in the json, or replace any component with your own pytorch creation
Data - Built to work on unclean data and multiple data types
Unsupervised learning
Unsupervised alignment - Align one distribution to another or discover new novel distributions.
Transfer learning
Online learning

Showcase

1.0 models are still training

Submit your showcase with a pull request!

For more, see the #showcase room in

Contributing

Contributions are welcome and appreciated! We have many open issues in the Issues tab. Join the discord.

See how to contribute.

Versioning

HyperGAN uses semantic versioning. http://semver.org/

TLDR: x.y.z

x is incremented on stable public releases.
y is incremented on API breaking changes. This includes configuration file changes and graph construction changes.
z is incremented on non-API breaking changes. z changes will be able to reload a saved graph.

Citation

  HyperGAN Community
  HyperGAN, (2016-2020+), 
  GitHub repository, 
  https://github.com/HyperGAN/HyperGAN

HyperGAN comes with no warranty or support.

hypergan's People

Contributors

Stargazers

Watchers

Forkers

arasharchor zdx3578 collector-m wanjinchang ml-lab drzhanying benjamesbabala yifanw90 leezqcst lyk125 walkoncross jmrinaldi vyraun caomw george-wu509 v-italy rhythm92 raj347 johndpope dillondoyle karimpedia xiangyongcao xuanhan863 qqgeogor codeaudit solertis chunniunai220ml geniusgeek ieee820 linpingchuan wangg12 joyhuang9473 albertlzg shatu xjwxjw wmengti phecy xuecaihu techscientist bodokaiser qinghuizhao sher-ali ruitonghuang kim3366 zhixinshu snakeroot91 mtyka simonsleo ifarhankhan dippedpretzels 0xqq jungel2star liupeng89 nanfengpo createamind tzukit122 archenroot gdseller hkcaesar sabotag3d vg3095 bobvanluijt likeyu21 andrewyang96 sbuckleybonanno jpvolt zachluo elivasquezhdz turtletheturt sammyloto matthewstidham oyshoboy boosting cthpw103 tenorbert ultimatepea matthewnetwork ml-ai-nlp-ir truong225 m00nwtchr akshayjh wampiflampi iduwei mattbilik winwinjjiang logbie alexanderjacobsen yassinelaadraoui c-connors c1a1o1 aaik98 pavelzx deanvangreunen iamsyedzaidi25 neurai slipknottn riffsircar kawaiipotato117 reiisky dreadchild

hypergan's Issues

Invalid Syntax error

Hey I want to use the HyperGAN for an artistic project.
But I am struggling here. I succeed in importing hypergan and with as hg.
But when I try hypergan new mymodel
I receive a syntax error.
What should I do? Can you help me out to run it properly.

ps: I am on Windows 64bit and no hands on experience with python

Weird colored corners when starting training

When i start training, the images always start with weird corners. Ive tried 3 differnet sizes so far: 32x32 ,64x64 and 64x80. All of them start with theese weird colored corners and they stay around for a while which messes things up for me. Here are 3 starting images: imgur.com/a/hdzTt

Awesome papers and resources

beta-VAE is also very good ref : http://openreview.net/forum?id=Sy2fzU9gl

Learning an interpretable factorised representation of the independent data gen- erative factors of the world without supervision is an important precursor for the development of artificial intelligence that is able to learn and reason in the same way that humans do. We introduce -VAE, a new state-of-the-art framework for automated discovery of interpretable factorised latent representations from raw image data in a completely unsupervised manner. Our approach is a modification of the variational autoencoder (VAE) framework. We introduce an adjustable hy- perparameter that balances latent channel capacity and independence constraints with reconstruction accuracy. We demonstrate that -VAE with appropriately tuned > 1 qualitatively outperforms VAE ( = 1), as well as state of the art unsu- pervised (InfoGAN) and semi-supervised (DC-IGN) approaches to disentangled factor learning on a variety of datasets (celebA, faces and chairs). Furthermore, we devise a protocol to quantitatively compare the degree of disentanglement learnt by different models, and show that our approach also significantly outperforms all baselines quantitatively. Unlike InfoGAN, -VAE is stable to train, makes few assumptions about the data and relies on tuning a single hyperparameter , which can be directly optimised through a hyperparameter search using weakly labelled data or through heuristic visual inspection for purely unsupervised data.

and

https://arxiv.org/abs/1606.05579

Early Visual Concept Learning with Unsupervised Deep Learning

Automated discovery of early visual concepts from raw image data is a major open challenge in AI research. Addressing this problem, we propose an unsupervised approach for learning disentangled representations of the underlying factors of variation. We draw inspiration from neuroscience, and show how this can be achieved in an unsupervised generative model by applying the same learning pressures as have been suggested to act in the ventral visual stream in the brain. By enforcing redundancy reduction, encouraging statistical independence, and exposure to data with transform continuities analogous to those to which human infants are exposed, we obtain a variational autoencoder (VAE) framework capable of learning disentangled factors. Our approach makes few assumptions and works well across a wide variety of datasets. Furthermore, our solution has useful emergent properties, such as zero-shot inference and an intuitive understanding of "objectness".

Default configurations may be unstable

Default configurations may diverge.

When a configuration is created it is set to random values within a predefined range. That range has some bad areas at the moment.

Until this is fixed see examples/configs for stable configs

Documentations on Command Line Usage

Are there any detailed documentations on how to use HyperGAN from command line?

Specifically,

how to continue training using a saved file
how to load saved model for deployment
what are the arguments to "build" "sample"

The default argument prompt is unclear about those.

Script for using a pre-trained model?

Looking at the examples, there are many different scripts for training models: https://github.com/255BITS/HyperGAN/tree/master/examples, but there doesn't appear to be any scripts for only using pre-trained model as they all require data sets as well? Am I looking in the wrong place?

Pause and Continue Trainning

Everything works like a charm. The only problem i have though is that i do not know how to properly stop and continue training from a point. Tried a couple of times and i had to start from scratch.
hypergan train [folder] -s 32x32x3 -f png -b 32 --config [name] --sampler static_batch --sample_every 5 --save_samples
Do i have to make a custom config? Thanks in advance, VK

ps: No hands on experience sorry if this something i should know. And the same dataset with unsupervised learning.

Pre-trained models for fine-tuning on similar data sets?

Would it be possible to release the pre-trained models used for the examples in the ReadMe (And possibly other models), so that individuals can fine tune them on similar data sets?

Training a model completely from scratch is not practical for everyone that wants to experiment with GANs, because of the time and computing resources required. Being able to fine tune models would help those who are limited to cloud computing solutions, those who have weaker hardware, etc..., be able to experiment with creating higher resolution outputs.

Image Loader finds the class labels, but not the images.

Here is the command I try to run:

    hypergan serve '/hyper/thumbnails/'  -s 64x64x3 -f jpg -b 32 --config test --sampler static_batch --sample_every 5

But as you can see, it finds the class labels, but not the image. I am sure the files are 64x64.

[hypergan] Welcome. This is one of 1 possible configurations.
[hypergan] Config file /home/will/.hypergan/configs/test.json
[loader] ImageLoader found 0 images with 194 different class labels
Traceback (most recent call last):
File "/usr/local/bin/hypergan", line 8, in
cli.CLI()
File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 22, in init
self.run()
File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 259, in run
crop=crop
File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 191, in setup_input_graph
channels=channels)
File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 220, in setup_input_loader
height=height)
File "/usr/local/lib/python3.5/dist-packages/hypergan/loaders/image_loader.py", line 21, in labelled_image_tensors_from_directory
assert len(filenames)!=0, "No images found in "+directory
AssertionError: No images found in /hyper/thumbnails

--crop does not work in 0.8

> hypergan train . -c 64 --device '/cpu:0' --crop                                                                                                                      
[hypergan] Welcome.  This is one of  1  possible configurations.
[hypergan] Config file /Users/martyn/.hypergan/configs/64.json
[loader] ImageLoader found 19 images with 28 different class labels
Traceback (most recent call last):
  File "/usr/local/bin/hypergan", line 8, in <module>
    cli.CLI()
  File "/usr/local/lib/python3.5/site-packages/hypergan/cli.py", line 22, in __init__
    self.run()
  File "/usr/local/lib/python3.5/site-packages/hypergan/cli.py", line 259, in run
    crop=crop
  File "/usr/local/lib/python3.5/site-packages/hypergan/cli.py", line 191, in setup_input_graph
    channels=channels)
  File "/usr/local/lib/python3.5/site-packages/hypergan/cli.py", line 220, in setup_input_loader
    height=height)
  File "/usr/local/lib/python3.5/site-packages/hypergan/loaders/image_loader.py", line 55, in labelled_image_tensors_from_directory
    height, width, dynamic_shape=True)
  File "/usr/local/lib/python3.5/site-packages/hypergan/loaders/resize_image_patch.py", line 163, in resize_image_with_crop_or_pad
    original_height, original_width, _ =     _ImageDimensions(image, dynamic_shape=dynamic_shape)
  File "/usr/local/lib/python3.5/site-packages/hypergan/loaders/resize_image_patch.py", line 222, in _ImageDimensions
    return array_ops.unpack(array_ops.shape(images))
AttributeError: module 'tensorflow.python.ops.array_ops' has no attribute 'unpack'

Originally reported in #55

Windows hypergan.

I installed hypergan through pip but it cant find the hypergan command HELP!!!!

Error in init.py

I get this output:

hypergan train ~/hgan_dataset/ -f jpg -b 8 --config hej --device '/cpu:0'
[hypergan] Welcome back. You are one of 27000000000000 possible configurations.
Traceback (most recent call last):
File "/Users/goransandstrom/anaconda/bin/hypergan", line 5, in
run(args)
File "/Users/goransandstrom/anaconda/lib/python3.5/site-packages/hypergan/init.py", line 311, in run
other_config = copy.copy(config)
File "/Users/goransandstrom/anaconda/lib/python3.5/copy.py", line 97, in copy
rv = reductor(4)
TypeError: 'NoneType' object is not callable

Error when importing hypergan in Python 3.6: "SyntaxError: name 'iteration' is used prior to global declaration"

Hey thanks for the extensive Readme and the whole project! Really looks amazing what you can do with it!

Hope you can help me out here. I've installed hypergan via
pip3 install hypergan --upgrade

on OSX 10.12, python 3.6 with Tensorflow 1.0.0

My Tensorflow install is working, checked everything and even reinstalled it before upgrading hypergan – but i still get this error when trying to import it:

>>> import hypergan
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/hypergan/__init__.py", line 2, in <module>
    from .gan import GAN
  File "/usr/local/lib/python3.6/site-packages/hypergan/gan.py", line 17, in <module>
    from hypergan.trainers import *
  File "/usr/local/lib/python3.6/site-packages/hypergan/trainers/adam_trainer.py", line 90
    global iteration
SyntaxError: name 'iteration' is used prior to global declaration

Do you know what causes this error and how i could fix it?

Partial mode collapse

An area to discuss issues related to the bug in hypergan - partial mode collapse. This is the technical term for the broken parts of the manifold.

0.8 Documentation/examples

Examples and documentation are ongoing.

Right now all the examples have been updated to 0.8 but there are no new ones(yet).

hypergan does not support single channel images

hypergan train folder -s 32x32x1 -f png -b 32 --sampler static_batch --sample_every 5 --config wgan          4940ms  Mi  8 Mär 09:24:39 2017
[hypergan] Welcome.  This is one of  1  possible configurations.
[hypergan] Config file /Users/bodokaiser/.hypergan/configs/wgan.json
[loader] ImageLoader found 3684 images with 1 different class labels
[discriminator] Class loss is off.  Unsupervised learning mode activated.
[generator] layer Tensor("generator/g_layers_0/BiasAdd:0", shape=(32, 8, 8, 256), dtype=float32, device=/device:GPU:0) 16384
[generator] layer Tensor("generator/g_layers_1/BiasAdd:0", shape=(32, 16, 16, 128), dtype=float32, device=/device:GPU:0) 32768
[generator] layer Tensor("generator/g_layers_2/BiasAdd:0", shape=(32, 32, 32, 1), dtype=float32, device=/device:GPU:0) 1024
[discriminator] layer Tensor("discriminator/AvgPool:0", shape=(64, 16, 16, 32), dtype=float32, device=/device:GPU:0)
Traceback (most recent call last):
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 671, in _call_cpp_shape_fn_impl
    input_tensors_as_shapes, status)
  File "/usr/local/Cellar/python3/3.6.0/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 89, in __exit__
    next(self.gen)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 2 in both shapes must be equal, but are 1 and 3 for 'discriminator/concat_1' (op: 'ConcatV2') with input shapes: [32,16,16,1], [32,16,16,3], [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bodokaiser/.virtualenvs/hypergan/bin/hypergan", line 8, in <module>
    cli.CLI()
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/hypergan/cli.py", line 22, in __init__
    self.run()
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/hypergan/cli.py", line 271, in run
    self.gan = GAN(config, graph, device=args.device)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/hypergan/gan.py", line 39, in __init__
    self.create_graph(graph_type, device)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/hypergan/gan.py", line 54, in create_graph
    tf_graph.create(graph)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/hypergan/graph/graph.py", line 118, in create
    d_real, d_fake, d_reals, d_fakes = self.discriminator(x, f, None, g, z)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/hypergan/graph/graph.py", line 57, in discriminator
    discriminators.append(discriminator.create(self.gan, discriminator, x, g, xs, gs,prefix="d_"+str(i)))
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/hypergan/discriminators/pyramid_discriminator.py", line 103, in discriminator
    xg = tf.concat(axis=0, values=[xs[index], gs[index]])
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1034, in concat
    name=name)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 519, in _concat_v2
    name=name)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
    op_def=op_def)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2329, in create_op
    set_shapes_for_outputs(ret)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1667, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "/Users/bodokaiser/.virtualenvs/hypergan/lib/python3.6/site-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Dimension 2 in both shapes must be equal, but are 1 and 3 for 'discriminator/concat_1' (op: 'ConcatV2') with input shapes: [32,16,16,1], [32,16,16,3], [].

The generator nets created here are [<tf.Tensor 'generator/Tanh:0' shape=(32, 8, 8, 3) dtype=float32>, <tf.Tensor 'generator/Tanh_1:0' shape=(32, 16, 16, 3) dtype=float32>, <tf.Tensor 'generator/Tanh_2:0' shape=(32, 32, 32, 1) dtype=float32>] but should have last dimension 1.

Not Converging for 2D Test

So I have pulled from the GitHub master and I have been running
python 2d-distribution.py train --sample_every 10000
and after waiting a good 10 min or so, I get results like this:

While my system is not exactly top of the line right now, I have also been running this on my desktop with a GTX 980, to a similar effect. Am I just not giving it enough time, or is there something up here?

Adding a second category to a model previously trained on a single category, results in an error

ubuntu@ip-Address:~$ hypergan train faceGAN/data/faces -s 128x128x3 -                                                                                        f png -b 32 --config rmsprop-lsgan --sampler static_batch --sample_every 5 --sav                                                                                        e_every 1000
[hypergan] Welcome.  This is one of  1  possible configurations.
[hypergan] Config file /home/ubuntu/.hypergan/configs/rmsprop-lsgan.json
[loader] ImageLoader found 2390 images with 2 different class labels
[discriminator] Class loss is on.  Semi-supervised learning mode activated.
2017-06-20 23:10:39.845772: W tensorflow/core/platform/cpu_feature_guard.cc:45]                                                                                         The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are                                                                                         available on your machine and could speed up CPU computations.
2017-06-20 23:10:39.845812: W tensorflow/core/platform/cpu_feature_guard.cc:45]                                                                                         The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are                                                                                         available on your machine and could speed up CPU computations.
2017-06-20 23:10:39.845832: W tensorflow/core/platform/cpu_feature_guard.cc:45]                                                                                         The TensorFlow library wasn't compiled to use AVX instructions, but these are av                                                                                        ailable on your machine and could speed up CPU computations.
2017-06-20 23:10:39.845850: W tensorflow/core/platform/cpu_feature_guard.cc:45]                                                                                         The TensorFlow library wasn't compiled to use AVX2 instructions, but these are a                                                                                        vailable on your machine and could speed up CPU computations.
2017-06-20 23:10:39.845858: W tensorflow/core/platform/cpu_feature_guard.cc:45]                                                                                         The TensorFlow library wasn't compiled to use FMA instructions, but these are av                                                                                        ailable on your machine and could speed up CPU computations.
2017-06-20 23:10:42.736813: I tensorflow/stream_executor/cuda/cuda_gpu_executor.                                                                                        cc:893] successful NUMA node read from SysFS had negative value (-1), but there                                                                                         must be at least one NUMA node, so returning NUMA node zero
2017-06-20 23:10:42.737303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:9                                                                                        40] Found device 0 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:1e.0
Total memory: 11.17GiB
Free memory: 11.11GiB
2017-06-20 23:10:42.737328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:9                                                                                        61] DMA: 0
2017-06-20 23:10:42.737338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:9                                                                                        71] 0:   Y
2017-06-20 23:10:42.737350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1                                                                                        030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus                                                                                         id: 0000:00:1e.0)
[generator] layer Tensor("generator/g_layers_0/BiasAdd:0", shape=(32, 8, 8, 256)                                                                                        , dtype=float32, device=/device:GPU:0) 16384
[generator] layer Tensor("generator/g_layers_1/BiasAdd:0", shape=(32, 16, 16, 12                                                                                        8), dtype=float32, device=/device:GPU:0) 32768
[generator] layer Tensor("generator/g_layers_2/BiasAdd:0", shape=(32, 32, 32, 64                                                                                        ), dtype=float32, device=/device:GPU:0) 65536
[generator] layer Tensor("generator/g_layers_3/BiasAdd:0", shape=(32, 64, 64, 32                                                                                        ), dtype=float32, device=/device:GPU:0) 131072
[generator] layer Tensor("generator/g_layers_4/BiasAdd:0", shape=(32, 128, 128,                                                                                         3), dtype=float32, device=/device:GPU:0) 49152
[discriminator] layer Tensor("discriminator/AvgPool:0", shape=(64, 64, 64, 32),                                                                                         dtype=float32, device=/device:GPU:0)
[discriminator] layer Tensor("discriminator/AvgPool_1:0", shape=(64, 32, 32, 70)                                                                                        , dtype=float32, device=/device:GPU:0)
[discriminator] layer Tensor("discriminator/AvgPool_2:0", shape=(64, 16, 16, 146                                                                                        ), dtype=float32, device=/device:GPU:0)
[discriminator] layer Tensor("discriminator/AvgPool_3:0", shape=(64, 8, 8, 298),                                                                                         dtype=float32, device=/device:GPU:0)
[discriminator] layer Tensor("discriminator/AvgPool_4:0", shape=(64, 4, 4, 596),                                                                                         dtype=float32, device=/device:GPU:0)
 |= Loading network from /home/ubuntu/.hypergan/saves/rmsprop-lsgan/model.ckpt
 |= Loading checkpoint from /home/ubuntu/.hypergan/saves/rmsprop-lsgan
2017-06-20 23:10:48.065156: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
2017-06-20 23:10:48.067161: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp not found in checkpoint
2017-06-20 23:10:48.067649: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias not found in checkpoint
2017-06-20 23:10:48.068152: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/Matrix/RMSProp_1 not found in checkpoint
2017-06-20 23:10:48.068688: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/Matrix/RMSProp not found in checkpoint
2017-06-20 23:10:48.069157: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/Matrix not found in checkpoint
2017-06-20 23:10:48.069595: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_bn_end/gamma/RMSProp_1 not found in checkpoint
2017-06-20 23:10:48.069988: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_bn_end/gamma/RMSProp not found in checkpoint
2017-06-20 23:10:48.070526: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_bn_end/gamma not found in checkpoint
2017-06-20 23:10:48.070988: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_bn_end/beta/RMSProp_1 not found in checkpoint
2017-06-20 23:10:48.071449: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_bn_end/beta/RMSProp not found in checkpoint
2017-06-20 23:10:48.071905: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_bn_end/beta not found in checkpoint
2017-06-20 23:10:48.073409: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.074667: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.075862: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.076847: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.077741: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.078564: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.079292: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.080059: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.080733: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.081318: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.081953: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.082587: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.083176: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.083717: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.084277: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.084771: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.085277: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.085754: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.086265: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.086785: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.087337: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.088670: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.089747: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.090759: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.091298: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.091806: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.092321: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.092764: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.093206: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.093670: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.107526: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.108418: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.108530: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.108416: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.109344: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.109348: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.109801: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.110606: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.111109: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.111588: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.112176: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.112450: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.112830: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.113359: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.113684: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.114048: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.114153: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.114354: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.114754: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.115065: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.118289: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.118527: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.118541: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.122209: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.122573: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.122674: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.123201: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.123593: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.123882: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.123938: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.124323: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.124703: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.125271: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.125402: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.125629: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.125647: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.127066: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.127814: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.127999: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.128390: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.128547: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.128600: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.129028: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.129336: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.129536: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.129736: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.130232: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.130404: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.130550: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.131092: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.131264: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.131517: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.131558: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.131986: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.132172: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.132628: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.132698: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.133313: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.133537: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.133854: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.133941: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.134211: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.134504: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.134698: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.134899: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.135359: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.135534: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.135801: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.136052: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.136306: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.136488: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.136740: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.137349: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.137448: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.137611: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.137821: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.138156: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.138507: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.138759: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.138918: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.139166: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
2017-06-20 23:10:48.139315: W tensorflow/core/framework/op_kernel.cc:1158] Not f                                                                                        ound: Key d_fc_end/bias/RMSProp_1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.                                                                                        py", line 1139, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.                                                                                        py", line 1121, in _run_fn
    status, run_metadata)
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/error                                                                                        s_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Key d_fc_end/bias/RMSProp                                                                                        _1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
         [[Node: save/RestoreV2_166/_129 = _Recv[client_terminated=false, recv_d                                                                                        evice="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/repli                                                                                        ca:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_544_save/Restor                                                                                        eV2_166", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]                                                                                        ()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/hypergan", line 8, in <module>
    cli.CLI()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 22, in __i                                                                                        nit__
    self.run()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 277, in ru                                                                                        n
    self.gan.load_or_initialize_graph(self.save_file)
  File "/usr/local/lib/python3.5/dist-packages/hypergan/gan.py", line 82, in loa                                                                                        d_or_initialize_graph
    saver.restore(self.sess, save_file)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.                                                                                        py", line 1548, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.                                                                                        py", line 789, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.                                                                                        py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.                                                                                        py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.                                                                                        py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key d_fc_end/bias/RMSProp                                                                                        _1 not found in checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
         [[Node: save/RestoreV2_166/_129 = _Recv[client_terminated=false, recv_d                                                                                        evice="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/repli                                                                                        ca:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_544_save/Restor                                                                                        eV2_166", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]                                                                                        ()]]

Caused by op 'save/RestoreV2_11', defined at:
  File "/usr/local/bin/hypergan", line 8, in <module>
    cli.CLI()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 22, in __i                                                                                        nit__
    self.run()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 277, in ru                                                                                        n
    self.gan.load_or_initialize_graph(self.save_file)
  File "/usr/local/lib/python3.5/dist-packages/hypergan/gan.py", line 81, in loa                                                                                        d_or_initialize_graph
    saver = tf.train.Saver()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.                                                                                        py", line 1139, in __init__
    self.build()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.                                                                                        py", line 1170, in build
    restore_sequentially=self._restore_sequentially)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.                                                                                        py", line 691, in build
    restore_sequentially, reshape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.                                                                                        py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.                                                                                        py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_io_ops.                                                                                        py", line 640, in restore_v2
    dtypes=dtypes, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_de                                                                                        f_library.py", line 767, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.p                                                                                        y", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.p                                                                                        y", line 1269, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key d_fc_end/bias/RMSProp_1 not found i                                                                                        n checkpoint
         [[Node: save/RestoreV2_11 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:                                                                                        localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_11/tensor                                                                                        _names, save/RestoreV2_11/shape_and_slices)]]
         [[Node: save/RestoreV2_166/_129 = _Recv[client_terminated=false, recv_d                                                                                        evice="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/repli                                                                                        ca:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_544_save/Restor                                                                                        eV2_166", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]                                                                                        ()]]

ubuntu@ip-Address:~$

Adversarial encoding not working

Adversarial encoding is a method for building a function encode(X) -> z. This can be seen as the opposite of a typical Generator.

This should probably be implemented as an additional loss in hypergan/losses

Might need #38 fixed first, depending on implementation.

No module named 'hyperchamber'

While trying to run the toy example the program unfortunately crashes immediately with this error:

Traceback (most recent call last):
  File "/usr/local/bin/hypergan", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/home/garbade/CloudStation/libs/dcgan/09_hypergan/hypergan/bin/hypergan", line 5, in <module>
    import hypergan as hg
  File "/usr/local/lib/python3.4/dist-packages/hypergan/__init__.py", line 2, in <module>
    from .gan import GAN
  File "/usr/local/lib/python3.4/dist-packages/hypergan/gan.py", line 4, in <module>
    from hyperchamber import Config
ImportError: No module named 'hyperchamber'

I tried to install hyperchamber using

pip3 install hyperchamber

But the error persists. Any ideas?
Is the installation of hyperchamber more evolved?
When I just start python3 from terminal, I can import hyperchamber so I'm not sure what is going wrong...

I'm using TF 1.0 - GPU version, python 3.4, Ubuntu 14.04

Densenet D/G

Need some samples. Does it converge?

Configuration `dtype` does not work

dtype is always tf.float32

pygame.error: No available video device

After training I'm getting the below output, is it possible to disable pygame? I'm training in the cloud so I don't have a video device available. Thx

\ 0:  d_loss 0.22 g_loss 0.50
Traceback (most recent call last):
  File "/usr/local/bin/hypergan", line 122, in <module>
    gancli.run()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 198, in run
    self.train()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 143, in train
    self.step()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 102, in step
    sample_list = self.sample(sample_file)
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 87, in sample
    sample_list = self.sampler.sample(sample_file, self.args.save_samples)
  File "/usr/local/lib/python3.5/dist-packages/hypergan/samplers/base_sampler.py", line 29, in sample
    return [{'image':path, 'label':'sample'} for sample_data, sample_filename in samples]
  File "/usr/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 3812, in get_controller
    yield default
  File "/usr/local/lib/python3.5/dist-packages/hypergan/samplers/base_sampler.py", line 25, in sample
    self.plot(sample_data, path, save_samples)
  File "/usr/local/lib/python3.5/dist-packages/hypergan/samplers/base_sampler.py", line 46, in plot
    GlobalViewer.update(image)
  File "/usr/local/lib/python3.5/dist-packages/hypergan/pygame_viewer.py", line 26, in update
    self.screen = self.pg.display.set_mode(size)
pygame.error: No available video device

how do i train

whenever i try to train in hypergan i doesn't work so how do i do it the instructions aren't clear to me
I am very new to hypergan

How to set sizes to 640x480?

I'm not sure I understand how to set the sizes. I have 640x480 PNGs so I would expect to set: -s 640x480x3

But the examples states:

# Train a 256x256 gan with batch size 32 on a folder of pngs
hypergan train [folder] -s 32x32x3 -f png -b 32 --config [name]

I would expect that 32x32x3 means: width 32, height 32, channels 3.

What is the relation between the 256x256 images and the -s 32x32x3 flag?

Model not saving

I've had a model training for a few days set to periodically save, but to my disappointment, there's no model.ckpt anywhere on disc.

I'm using the tensorflow-1.0 branch on an AWS p2.xlarge instance. I did have to make a single modification in order to get it to not immediately break -- I believe I had to copy several of the regularizers from the main branch into the tensorflow-1.0 branch, but otherwise it's unmodified.

The issue is of course primarily that it's not actually saving, but I have a second question: is there anything at all that I can do to save the model presently running? I'd really rather not waste so much time and money if possible (though of course if there's nothing to be done I understand; I should have verified earlier that it was saving).

small update to readme: --save_samples

Hi,

Just wondering whether the the following addition is necessary for the readme? Currently the sampling section (with the animated gif of people's faces) says:

# Train a 256x256 gan with batch size 32 on a folder of pngs hypergan train [folder] -s 32x32x3 -f png -b 32 --config [name] --sampler static_batch --sample_every 5

I was doing this and wondering why my sample folder remained empty until I used hypergan -h to reveal more CLI commands. There I found --save_samples which when added to the above command now populates the samples folder as expected every n samples (in this case 5).

Also as a very small remark, there is a tiny typo in the hypergan -h output

--sample_every SAMPLE_EVERY Saves a sample ever X steps.
*ever

Neither are a big deal at all, really great work!

How to continue training from a checkpoint?

Checkpoint files for the models seem to be saved, but how do I continue training from a checkpoint? And how do I string different resolution models together to create bigger and better outputs?

Failed to load the native TensorFlow runtime.

Hi all,

After installing hypergan (CUDA + pip3 install hypergan numpy tensorflow-gpu hyperchamber pillow pygame) I'm getting the following error. Any idea what is happening here? I'm using Ubuntu 16.04LTS

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <modul
e>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_i
mport_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/usr/lib/python3.5/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: libcudnn.so.5: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/bin/hypergan", line 7, in <module>
    import hypergan as hg
  File "/usr/local/lib/python3.5/dist-packages/hypergan/__init__.py", line 2, in <module>
    from .gan import GAN
  File "/usr/local/lib/python3.5/dist-packages/hypergan/gan.py", line 1, in <module>
    from hypergan.gans.standard_gan import StandardGAN
  File "/usr/local/lib/python3.5/dist-packages/hypergan/gans/standard_gan.py", line 10, in <module>
    from hypergan.discriminators import *
  File "/usr/local/lib/python3.5/dist-packages/hypergan/discriminators/dcgan_discriminator.py", line 1, in <module>
    import tensorflow as tf
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 52, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/usr/lib/python3.5/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/lib/python3.5/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: libcudnn.so.5: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

Having trouble sampling a trained model

Copied from a PR comment: "
Im a grad student in Media Design Practices at Art Center College of Design. A colleague and I are working with HyperGAN for a media piece we are working on. We are having a hard time figuring out the bash for building or generating hallucinations after we successfully trained our GAN. Any help would be greatly appreciated, we are new to this. Thanks so much!
"

L2 Regularization

This is something that was added in 0.6. Need to confirm that it's better or make it a hyperparam.

No images found in directory

When using the command "hypergan serve img_align_celeba_png/ -s 32x32x3 -f png -b 32 --config 123456" no images are found in my directory "img_align_celeba_png", although there are images.

Traceback (most recent call last):
File "/usr/local/bin/hypergan", line 8, in
cli.CLI()
File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 22, in init
self.run()
File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 259, in run
crop=crop
File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 191, in setup_input_graph
channels=channels)
File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 220, in setup_input_loader
height=height)
File "/usr/local/lib/python3.5/dist-packages/hypergan/loaders/image_loader.py", line 21, in labelled_image_tensors_from_directory
assert len(filenames)!=0, "No images found in "+directory
AssertionError: No images found in img_align_celeba_png/

Encountering a broken input file kills tensorflow worker

To reproduce:

cp real_img.jpg imgs/ # a valid image
touch imgs/broken.jpg # create 0 byte image
hypergan train imgs -f jpg

Expected:

a network trained on a single image (real_img)

Actual:

Workers die leaving hg stalled

Image resize should preserve aspect ratio

Problem:
Preserving aspect ratios is important in some cases. Neither --crop or --resize will do this.

Proposed solution:
Make --resize do this by default. If we want the original behavior do --scale. If --resize and the image is under resolution, just draw a black border around it.

TypeError: invalid file: None. Upon trying any of the quick start commands.

OS: Ubuntu 16.04
Python: 3.5.3
Cuda: 8.0
CuDNN: 5.0

Traceback (most recent call last):
  File "/home/valerie/.local/bin/hypergan", line 67, in <module>
    config = hc.Selector().load(config_filename)
  File "/home/valerie/.local/lib/python3.5/site-packages/hyperchamber/selector.py", line 112, in load
    content = open(filename)
TypeError: invalid file: None

Everything seems to have installed correctly, however, hypergan appears to be passing None for a file. I've downloaded one of the default configs and tried using the --config switch, but the error prevails.

hypergan cli doesn't load examples/*

Decoupling this issue from #55

To reproduce:

Train/save colorizer model
Load model with hypergan cli
Expected: loads
Actual: error

Expand sampling methods

Right now we support very simple sampling methods.

http://arxiv.org/abs/1609.04468 describes several awesome ones.

To add a new sampler, just add it to the hypergan/samplers directory.

This is in the 0.8 branch.

Deprecation warnings when starting hypergan

WARNING:tensorflow:From /home/martyn/dev/hypergan/hypergan/gan.py:87 in load_or_
initialize_graph.: initialize_all_variables (from tensorflow.python.ops.variable
s) is deprecated and will be removed after 2017-03-02.

Loading from checkpoint fails

ubuntu@ip-Address:~$ sudo hypergan train faceGAN/data/images -s 64x64x3 -f png -b 32 --config rmsprop-lsgan-64x64 --sampler static_batch --sample_every 5
[hypergan] Welcome.  This is one of  1  possible configurations.
[hypergan] Config file /home/ubuntu/.hypergan/configs/rmsprop-lsgan-64x64.json
[loader] ImageLoader found 971 images with 1 different class labels
[discriminator] Class loss is off.  Unsupervised learning mode activated.
2017-06-11 01:19:56.354298: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 01:19:56.354333: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 01:19:56.354364: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 01:19:56.354389: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 01:19:56.354396: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-06-11 01:19:59.217952: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-06-11 01:19:59.218458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: Tesla K80
major: 3 minor: 7 memoryClockRate (GHz) 0.8235
pciBusID 0000:00:1e.0
Total memory: 11.17GiB
Free memory: 11.11GiB
2017-06-11 01:19:59.218487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0
2017-06-11 01:19:59.218500: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0:   Y
2017-06-11 01:19:59.218514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0)
[generator] layer Tensor("generator/g_layers_0/BiasAdd:0", shape=(32, 8, 8, 256), dtype=float32, device=/device:GPU:0) 16384
[generator] layer Tensor("generator/g_layers_1/BiasAdd:0", shape=(32, 16, 16, 128), dtype=float32, device=/device:GPU:0) 32768
[generator] layer Tensor("generator/g_layers_2/BiasAdd:0", shape=(32, 32, 32, 64), dtype=float32, device=/device:GPU:0) 65536
[generator] layer Tensor("generator/g_layers_3/BiasAdd:0", shape=(32, 64, 64, 3), dtype=float32, device=/device:GPU:0) 12288
[discriminator] layer Tensor("discriminator/AvgPool:0", shape=(64, 32, 32, 32), dtype=float32, device=/device:GPU:0)
[discriminator] layer Tensor("discriminator/AvgPool_1:0", shape=(64, 16, 16, 70), dtype=float32, device=/device:GPU:0)
[discriminator] layer Tensor("discriminator/AvgPool_2:0", shape=(64, 8, 8, 146), dtype=float32, device=/device:GPU:0)
[discriminator] layer Tensor("discriminator/AvgPool_3:0", shape=(64, 4, 4, 292), dtype=float32, device=/device:GPU:0)
[discriminator] layer Tensor("discriminator/AvgPool_4:0", shape=(64, 2, 2, 584), dtype=float32, device=/device:GPU:0)
 |= Loading network from /home/ubuntu/.hypergan/saves/rmsprop-lsgan-64x64/model.ckpt
 |= Loading checkpoint from /home/ubuntu/.hypergan/saves/rmsprop-lsgan-64x64
2017-06-11 01:20:03.577352: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.577420: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.578469: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.578512: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.579677: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.580614: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.580642: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.581405: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.581719: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.581844: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.582102: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.582844: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.585071: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.590412: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.590430: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.591622: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.591944: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.592430: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.592430: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.592901: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.593068: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.593264: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.593860: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.594773: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.595318: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.595443: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.595700: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.596263: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.596596: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.596779: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.597124: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.597467: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.598250: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.598769: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.598915: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.598967: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.599345: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.599449: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.600127: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.600614: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.601069: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.601516: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.602565: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.602694: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.603069: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.603710: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.604843: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.606392: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.606888: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.619423: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.621206: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.621290: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.621563: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.621629: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.622095: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.622610: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.622775: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.623205: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.623348: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.623539: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.624446: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.624729: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.625249: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.625385: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.625741: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.626624: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.626793: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.627038: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.627134: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.627199: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.627881: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.628142: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.628573: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.628866: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.629018: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.629435: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.629801: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.629902: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.630197: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.630420: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.630512: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.630771: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.631301: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
2017-06-11 01:20:03.631478: W tensorflow/core/framework/op_kernel.cc:1152] Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1039, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1021, in _run_fn
    status, run_metadata)
  File "/usr/lib/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/hypergan", line 8, in <module>
    cli.CLI()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 22, in __init__
    self.run()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 277, in run
    self.gan.load_or_initialize_graph(self.save_file)
  File "/usr/local/lib/python3.5/dist-packages/hypergan/gan.py", line 82, in load_or_initialize_graph
    saver.restore(self.sess, save_file)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1457, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 778, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 982, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1032, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1052, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]

Caused by op 'save/Assign_154', defined at:
  File "/usr/local/bin/hypergan", line 8, in <module>
    cli.CLI()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 22, in __init__
    self.run()
  File "/usr/local/lib/python3.5/dist-packages/hypergan/cli.py", line 277, in run
    self.gan.load_or_initialize_graph(self.save_file)
  File "/usr/local/lib/python3.5/dist-packages/hypergan/gan.py", line 81, in load_or_initialize_graph
    saver = tf.train.Saver()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1056, in __init__
    self.build()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 1086, in build
    restore_sequentially=self._restore_sequentially)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 691, in build
    restore_sequentially, reshape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 419, in _AddRestoreOps
    assign_ops.append(saveable.restore(tensors, shapes))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py", line 155, in restore
    self.op.get_shape().is_fully_defined())
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py", line 270, in assign
    validate_shape=validate_shape)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
    use_locking=use_locking, name=name)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [49,8192] rhs shape= [97,8192]
         [[Node: save/Assign_154 = Assign[T=DT_FLOAT, _class=["loc:@generator/g_lin_proj/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/gpu:0"](generator/g_lin_proj/Matrix/RMSProp_1, save/RestoreV2_154/_225)]]

ubuntu@ip-Address:~$

Windows compatibility

I know Windows support probably isn't much of a priority, but it seems like there are a very small number of barriers to Windows compatibility (out of the box) for HyperGAN.

tf.contrib.ffmpeg - Not supported, but only necessary for audio tasks. I propose something like the following in hypergan/loaders/audio_loaders.py:

try:
    from tensorflow.contrib import ffmpeg
except ImportError:
    import warnings
    warnings.warn("ffmpeg support not found, audio loaders will not work!")```

fcntl - I'll be honest I have no idea what this does, but it seems to only be used once in cli.py for ipython support.

I propose something like:

        if(self.args.ipython):
            import fcntl # Move the import into the if statement (remove from top)
            fd = sys.stdin.fileno()
            fl = fcntl.fcntl(fd, fcntl.F_GETFL)
            fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

This way there would only be an error thrown if someone is trying to use a feature their OS doesn't support, rather than "import hypergan" failing.

Thanks.

Variational encoder broken

The vae was working in the past, but in the newest architecture it's not working.

This would be a fun thing for a contributor to work on. You would first need to add a new type of encoder(copy the linear_encoder)

Should also support adversarial encoding eventually - although it's fine if it doesn't right away.

Modifications to work in Windows?

I'm just curious how feasible you think it would be to get this running in Windows. tf.contrib.ffmpeg ops aren't currently supported in Windows (is this only used for animation?), and neither is fcntl. flask, flask-cors, hyperchamber, tensorflow, scipy, and Pillow already work in Windows.

Great work by the way!

hypergan: command not found

I did everything correctly, but hypergan command does not work

daniel@daniel-pc:$ pip3 install hypergan --upgrade
Collecting hypergan
Downloading hypergan-0.9.4.tar.gz
Building wheels for collected packages: hypergan
Running setup.py bdist_wheel for hypergan ... done
Stored in directory: /home/daniel/.cache/pip/wheels/e4/c8/6b/f5111f93fa7cd698aa9d67e99f04bcdda76f6cd5139a90f50f
Successfully built hypergan
Installing collected packages: hypergan
Successfully installed hypergan-0.9.4
daniel@daniel-pc:$ pip3 install numpy tensorflow-gpu hyperchamber pillow pygame
Collecting numpy
Downloading numpy-1.13.1-cp35-cp35m-manylinux1_x86_64.whl (16.9MB)
100% |████████████████████████████████| 16.9MB 101kB/s
Collecting tensorflow-gpu
Downloading tensorflow_gpu-1.2.1-cp35-cp35m-manylinux1_x86_64.whl (89.2MB)
100% |████████████████████████████████| 89.2MB 17kB/s
Collecting hyperchamber
Collecting pillow
Downloading Pillow-4.2.1-cp35-cp35m-manylinux1_x86_64.whl (5.8MB)
100% |████████████████████████████████| 5.8MB 232kB/s
Collecting pygame
Using cached pygame-1.9.3-cp35-cp35m-manylinux1_x86_64.whl
Collecting markdown>=2.6.8 (from tensorflow-gpu)
Downloading Markdown-2.6.8.tar.gz (307kB)
100% |████████████████████████████████| 317kB 441kB/s
Collecting six>=1.10.0 (from tensorflow-gpu)
Using cached six-1.10.0-py2.py3-none-any.whl
Collecting html5lib==0.9999999 (from tensorflow-gpu)
Collecting bleach==1.5.0 (from tensorflow-gpu)
Using cached bleach-1.5.0-py2.py3-none-any.whl
Collecting protobuf>=3.2.0 (from tensorflow-gpu)
Using cached protobuf-3.3.0-cp35-cp35m-manylinux1_x86_64.whl
Collecting werkzeug>=0.11.10 (from tensorflow-gpu)
Using cached Werkzeug-0.12.2-py2.py3-none-any.whl
Collecting wheel>=0.26 (from tensorflow-gpu)
Using cached wheel-0.29.0-py2.py3-none-any.whl
Collecting backports.weakref==1.0rc1 (from tensorflow-gpu)
Using cached backports.weakref-1.0rc1-py3-none-any.whl
Collecting requests>=2.4.2 (from hyperchamber)
Using cached requests-2.18.1-py2.py3-none-any.whl
Collecting olefile (from pillow)
Collecting setuptools (from protobuf>=3.2.0->tensorflow-gpu)
Using cached setuptools-36.0.1-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.4.2->hyperchamber)
Using cached chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.6,>=2.5 (from requests>=2.4.2->hyperchamber)
Using cached idna-2.5-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests>=2.4.2->hyperchamber)
Using cached certifi-2017.4.17-py2.py3-none-any.whl
Collecting urllib3<1.22,>=1.21.1 (from requests>=2.4.2->hyperchamber)
Using cached urllib3-1.21.1-py2.py3-none-any.whl
Building wheels for collected packages: markdown
Running setup.py bdist_wheel for markdown ... done
Stored in directory: /home/daniel/.cache/pip/wheels/85/a7/08/33ee5cd488d0365d8bed79d1d4e5c28dd3fbfc7f6d0ad4bb09
Successfully built markdown
Installing collected packages: numpy, markdown, six, html5lib, bleach, setuptools, protobuf, werkzeug, wheel, backports.weakref, tensorflow-gpu, chardet, idna, certifi, urllib3, requests, hyperchamber, olefile, pillow, pygame
Successfully installed backports.weakref-1.0rc1 bleach-1.5.0 certifi-2017.4.17 chardet-3.0.4 html5lib-0.9999999 hyperchamber-0.2.8 idna-2.5 markdown-2.6.8 numpy-1.13.1 olefile-0.44 pillow-4.2.1 protobuf-3.3.0 pygame-1.9.3 requests-2.18.1 setuptools-36.0.1 six-1.10.0 tensorflow-gpu-1.2.1 urllib3-1.21.1 werkzeug-0.12.2 wheel-0.29.0

But every hypergan command I write shows this console output. hypergan: command not found
When I try to use virtualenv it doesent work eather
daniel@daniel-pc:$ sudo virtualenv --system-site-packages -p python3 hypergan
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/daniel/hypergan/bin/python3
Not overwriting existing python script /home/daniel/hypergan/bin/python (you must use /home/daniel/hypergan/bin/python3)
Installing setuptools, pkg_resources, pip, wheel...done.
daniel@daniel-pc:$ hypergan
hypergan: příkaz nebyl nalezen
daniel@daniel-pc:$ source hypergan/bin/activate
(hypergan) daniel@daniel-pc:$ hypergan
hypergan: příkaz nebyl nalezen

Support for changing the size of the training data and thus the output images on an already trained model?

In an effort to speed up training times, I would like to train a model first on 32x32px images, and then on 64x64px, 128x128px, 256x256px, etc...

Currently when running:

hypergan train faceGAN/data/images -s 64x64x3 -f png -b 32 --config rmsprop-lsgan-64x64 --sampler static_batch --sample_every 5

Or:

hypergan train faceGAN/data/images -s 64x64x3 -f png -b 32 --config rmsprop-lsgan --sampler static_batch --sample_every 5

The models saved in .hypergan can only continue their training if you use the same size value. This sort of scaling up technique that I want to try seems to be successful for style transfer system, and it seems that logically the model can exploit it's already learned knowledge to speed up training on larger images.

Multiple discriminators

I think we may need additional loss terms or steps, I have not been able to get this network to converge - tho the blurry samples seemed promising. I've disabled this in the 0.6 default configuration for now.

'z' should be in 'encoder'

'z' is on the main scope of the config file, move it to the encoder section.

ValueError: Dimension 0 in both shapes must be equal, but are 160 and 256 for 'discriminator/concat' (op: 'ConcatV2') with input shapes: [64,160,320,3], [64,256,512,3], [].

I have a dataset of images of size [320x160x3] (WxHxC) .
I'm trying train a 'dcgan-like' generator on it.

hypergan train ~/path/to/my/data/ -b 64 -f png --config examples/configs/improved-gan.json --sampler static_batch --sample_every 50 -s 320x160x3

However it gives me a dimension mismatch error:

ValueError: Dimension 0 in both shapes must be equal, but are 160 and 256 for 'discriminator/concat' (op: 'ConcatV2') with input shapes: [64,160,320,3], [64,256,512,3], [].

Is it not possible to train the GAN with those image dimensions?
If I leave out the -s 320x160x3 parameter the training is at least starting (not converging apparantly though), but it seems that the images are downsampled to 64x64x3 internally which I'd like to avoid...
Any ideas how to do that?

Broken with newer versions of Tensorflow?

I'm attempting to use vanilla HyperGAN, and I got hit by an error which seemed to stem from tf.concat -- "TypeError: Expected int32, got list containing Tensors of type '_Message' instead."

A bit more digging indicated that the Tensorflow API changed regarding the ordering of parameters to tf.concat, and I was able to suppress those errors by regexing through the code and reversing the order of parameters. To me, that seems to indicate that the issue is that I'm using a newer Tensorflow version than the one for which HyperGAN was written.

I've now got another set of errors that I suspect have the same root cause (Tensorflow version) -- "ValueError: Only call softmax_cross_entropy_with_logits with named arguments (labels=..., logits=..., ...)"

I'm presently on Tensorflow version '0.12.head'

I have no idea how much work is required to make HyperGAN work with newer versions of Tensorflow, nor am I advocating for that to happen. I do think, however, that it would be very helpful to users to indicate with which versions of Tensorflow HyperGAN is confirmed to be compatible.

Thanks!

Errors on "import hypergan"

It is probably something I did wrong, but I thought I let you know.

I just installed HyperGAN from source using python3 setup.py develop

The first error I got when trying "import hypergan" in a Python console was that the HyperChamber module was missing. After installing that one separated this error went away.

But now I am stuck with this. It seems like the hypergan.regularizers classes do not get installed properly (at least over here):

import hypergan
Traceback (most recent call last):
File "", line 1, in
File "hypergan/init.py", line 2, in
from .gan import GAN
File "hypergan/gan.py", line 13, in
from hypergan.discriminators import *
File "hypergan/discriminators/densenet_discriminator.py", line 5, in
import hypergan.regularizers.minibatch_regularizer as minibatch_regularizer
ImportError: No module named regularizers.minibatch_regularizer

Any ideas?

progressive enhancement

Could anyone share precisely how to use the 'progressive enhancement' feature and how it works?