Giter Club home page Giter Club logo

disentanglement_lib's Introduction

disentanglement_lib

Sample visualization

disentanglement_lib is an open-source library for research on learning disentangled representation. It supports a variety of different models, metrics and data sets:

  • Models: BetaVAE, FactorVAE, BetaTCVAE, DIP-VAE
  • Metrics: BetaVAE score, FactorVAE score, Mutual Information Gap, SAP score, DCI, MCE, IRS, UDR
  • Data sets: dSprites, Color/Noisy/Scream-dSprites, SmallNORB, Cars3D, and Shapes3D
  • It also includes 10'800 pretrained disentanglement models (see below for details).

disentanglement_lib was created by Olivier Bachem and Francesco Locatello at Google Brain Zurich for the large-scale empirical study

Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem. ICML (Best Paper Award), 2019.

The code is tested with Python 3 and is meant to be run on Linux systems (such as a Google Cloud Deep Learning VM). It uses TensorFlow, Scipy, Numpy, Scikit-Learn, TFHub and Gin.

How does it work?

disentanglement_lib consists of several different steps:

  • Model training: Trains a TensorFlow model and saves trained model in a TFHub module.
  • Postprocessing: Takes a trained model, extracts a representation (e.g. by using the mean of the Gaussian encoder) and saves the representation function in a TFHub module.
  • Evaluation: Takes a representation function and computes a disentanglement metric.
  • Visualization: Takes a trained model and visualizes it.

All configuration details and experimental results of the different steps are saved and propagated along the steps (see below for a description). At the end, they can be aggregated in a single JSON file and analyzed with Pandas.

Usage

Installing disentanglement_lib

First, clone this repository with

git clone https://github.com/google-research/disentanglement_lib.git

Then, navigate to the repository (with cd disentanglement_lib) and run

pip install .[tf_gpu]

(or pip install .[tf] for TensorFlow without GPU support). This should install the package and all the required dependencies. To verify that everything works, simply run the test suite with

dlib_tests

Downloading the data sets

To download the data required for training the models, navigate to any folder and run

dlib_download_data

which will install all the required data files (except for Shapes3D which is not publicly released) in the current working directory. For convenience, we recommend to set the environment variable DISENTANGLEMENT_LIB_DATA to this path, for example by adding

export DISENTANGLEMENT_LIB_DATA=<path to the data directory>

to your .bashrc file. If you choose not to set the environment variable DISENTANGLEMENT_LIB_DATA, disentanglement_lib will always look for the data in your current folder.

Reproducing prior experiments

To fully train and evaluate one of the 12'600 models in the paper Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, simply run

dlib_reproduce --model_num=<?>

where <?> should be replaced with a model index between 0 and 12'599 which corresponds to the ID of which model to train. This will take a couple of hours and add a folder output/<?> which contains the trained model (including checkpoints and TFHub modules), the experimental results (in JSON format) and visualizations (including GIFs). To only print the configuration of that model instead of training, add the flag --only_print.

After having trained several of these models, you can aggregate the results by running the following command (in the same folder)

dlib_aggregate_results

which creates a results.json file with all the aggregated results.

Running different configurations

Internally, disentanglement_lib uses gin to configure hyperparameters and other settings. To train one of the provided models but with different hyperparameters, you need to write a gin config such as examples/model.gin. Then, you may use the following command

dlib_train --gin_config=examples/model.gin --model_dir=<model_output_directory>

to train the model where --model_dir specifies where the results should be saved.

To evaluate the newly trained model consistent with the evaluation protocol in the paper Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations, simply run

dlib_reproduce --model_dir=<model_output_directory> --output_directory=<output>

Similarly, you might also want to look at dlib_postprocess and dlib_evaluate if you want to customize how representations are extracted and evaluated.

Starting your own research

disentanglement_lib is easily extendible and can be used to implement new models and metrics related to disentangled representations. To get started, simply go through examples/example.py which shows you how to create your own disentanglement model and metric and how to benchmark them against existing models and metrics.

Pretrained disentanglement_lib modules

Reproducing all the 12'600 models in the study Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations requires a substantial computational effort. To foster further research, disentanglement_lib includes 10'800 pretrained disentanglement_lib modules that correspond to the results of running dlib_reproduce with --model_num=<?> between 0 and 10'799 (the other models correspond to Shapes3D which is not publicly available). Each disentanglement_lib module contains the trained model (in the form of a TFHub module), the extracted representations (also as TFHub modules) and the recorded experimental results such as the different disentanglement scores (in JSON format). This makes it easy to compare new models to the pretrained ones and to compute new disentanglement metrics on the set of pretrained models.

To access the 10'800 pretrained disentanglement_lib modules, you may download individual ones using the following link:

https://storage.googleapis.com/disentanglement_lib/unsupervised_study_v1/<?>.zip

where <?> corresponds to a model index between 0 and 10'799 (example).

Each ZIP file in the bucket corresponds to one run of dlib_reproduce with that model number. To learn more about the used configuration settings, look at the code in disentanglement_lib/config/unsupervised_study_v1/sweep.py or run:

dlib_reproduce --model_num=<?> --only_print

Frequently asked questions

How do I make pretty GIFs of my models?

If you run dlib_reproduce, they are automatically saved to the visualizations subfolder in your output directory. Otherwise, you can use the script dlib_visualize_dataset to generate them or call the function visualize(...) in disentanglement_lib/visualize/visualize_model.py.

How are results and models saved?

After each of the main steps (training/postprocessing/evaluation), an output directory is created. For all steps, there is a results folder which contains all the configuration settings and experimental results up to that step. The gin subfolder contains the operative gin config for each step in the gin format. The json subfolder contains files with the operative gin config and the experimental results of that step but in JSON format. Finally, the aggregate subfolder contains aggregated JSON files where each file contains both the configs and results from all preceding steps.

The training step further saves the TensorFlow checkpoint (in a tf_checkpoint subfolder) and the trained model as a TFHub module (in a tfhub subfolder). Similarly, the postprocessing step saves the representation function as a TFHub module (in a tfhub subfolder). If you run dlib_reproduce, it will create subfolders for all the different substeps that you ran. In particular, it will create an output directory for each metric that you computed.

How do I access the results?

To access the results, first aggregate all the results using dlib_aggregate_results by specifying a glob pattern that captures all the results files. For example, after training a couple of different models with dlib_reproduce, you would specify

dlib_aggregate --output_path=<...>.json \
  --result_file_pattern=<...>/*/metrics/*/*/results/aggregate/evaluation.json

The first * in the glob pattern would capture the different models, the second * different representations and the last * the different metrics. Finally, you may access the aggregated results with:

from disentanglement_lib.utils import aggregate_results
df = aggregate_results.load_aggregated_json_results(output_path)

Where to look in the code?

The following provides a guide to the overall code structure:

(1) Training step:

  • disentanglement_lib/methods/unsupervised: Contains the training protocol (train.py) and all the model functions for training the methods (vae.py). The methods all inherit from the GaussianEncoderModel class.
  • disentanglement_lib/methods/shared: Contains shared architectures, losses, and optimizers used in the different models.

(2) Postprocessing step:

  • disentanglement_lib/postprocess: Contains the postprocessing pipeline (postprocess.py) and the two extraction methods (methods.py).

(3) Evaluation step:

  • disentanglement_lib/evaluation: Contains the evaluation protocol (evaluate.py).

  • disentanglement_lib/evaluation/metrics: Contains implementation of the different disentanglement metrics.

Hyperparameters and configuration files:

  • disentanglement_lib/config/unsupervised_study_v1: Contains the gin configuration files (*.gin) for the different steps as well as the hyperparameter sweep (sweep.py) for the experiments in the paper Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations.

Shared functionality:

  • bin: Scripts to run the different pipelines, visualize the data sets as well as the models and aggregate the results.

  • disentanglement_lib/data/ground_truth: Contains all the scripts used to generate the data. All the datasets (in named_data.py) are instances of the class GroundTruthData}.

  • disentanglement_lib/utils: Contains helper functions to aggregate and save the results of the pipeline as well as the trained models.

  • disentanglement_lib/visualize: Contains visualization functions for the datasets and the trained models.

NeurIPS 2019 Disentanglement Challenge

The library is also used for the NeurIPS 2019 Disentanglement challenge. The challenge consists of three different datasets.

  1. Simplistic rendered images (mpi3d_toy)
  2. Realistic rendered images (mpi3d_realistic): not yet published
  3. Real world images (mpi3d_real): not yet published

Currently, only the simplistic rendered dataset is publicly available and will be automatically downloaded by running the following command.

dlib_download_data

Other datasets will be made available at the later stages of the competition. For more information on the competition kindly visit the competition website. More information about the dataset can be found here or in the arXiv preprint On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset.

Abstract reasoning experiments

The library also includes the code used for the experiments of the following paper in the disentanglement_lib/evaluation/abstract_reasoning subdirectory:

Are Disentangled Representations Helpful for Abstract Visual Reasoning? Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, Olivier Bachem. NeurIPS, 2019.

The experimental protocol consists of two parts: First, to train the disentanglement models, one may use the the standard replication pipeline (dlib_reproduce), for example via the following command:

dlib_reproduce --model_num=<?> --study=abstract_reasoning_study_v1

where <?> should be replaced with a model index between 0 and 359 which corresponds to the ID of which model to train.

Second, to train the abstract reasoning models, one can use the automatically installed pipeline dlib_reason. To configure the model, copy and modify disentanglement_lib/config/abstract_reasoning_study_v1/stage2/example.gin as needed. Then, use the following command to train and evaluate an abstract reasoning model:

dlib_reason --gin_config=<?> --input_dir=<?> --output_dir=<?>

The results can then be found in the results subdirectory of the output directory.

Fairness experiments

The library also includes the code used for the experiments of the following paper in disentanglement_lib/evaluation/metrics/fairness.py:

On the Fairness of Disentangled Representations Francesco Locatello, Gabriele Abbati, Tom Rainforth, Stefan Bauer, Bernhard Schoelkopf, Olivier Bachem. NeurIPS, 2019.

To train and evaluate all the models, simply use the following command:

dlib_reproduce --model_num=<?> --study=fairness_study_v1

where <?> should be replaced with a model index between 0 and 12'599 which corresponds to the ID of which model to train.

If you only want to reevaluate an already trained model using the evaluation protocol of the paper, you may use the following command:

dlib_reproduce --model_dir=<model_output_directory> --output_directory=<output> --study=fairness_study_v1

UDR experiments

The library also includes the code for the Unsupervised Disentanglement Ranking (UDR) method proposed in the following paper in disentanglement_lib/bin/dlib_udr:

Unsupervised Model Selection for Variational Disentangled Representation Learning Sunny Duan, Loic Matthey, Andre Saraiva, Nicholas Watters, Christopher P. Burgess, Alexander Lerchner, Irina Higgins.

UDR can be applied to newly trained models (e.g. obtained by running dlib_reproduce) or to the existing pretrained models. After the models have been trained, their UDR scores can be computed by running:

dlib_udr --model_dirs=<model_output_directory1>,<model_output_directory2> \
  --output_directory=<output>

The scores will be exported to <output>/results/aggregate/evaluation.json under the model_scores attribute. The scores will be presented in the order of the input model directories.

Weakly-Supervised experiments

The library also includes the code for the weakly-supervised disentanglement methods proposed in the following paper in disentanglement_lib/bin/dlib_reproduce_weakly_supervised:

Weakly-Supervised Disentanglement Without Compromises Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, Michael Tschannen.

dlib_reproduce_weakly_supervised --output_directory=<output> \
   --gin_model_config_dir=<dir> \
   --gin_model_config_name=<name> \
   --gin_postprocess_config_glob=<postprocess_configs> \
   --gin_evaluation_config_glob=<eval_configs> \
   --pipeline_seed=<seed>

Semi-Supervised experiments

The library also includes the code for the semi-supervised disentanglement methods proposed in the following paper in disentanglement_lib/bin/dlib_reproduce_semi_supervised:

Disentangling Factors of Variation Using Few Labels Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem.

dlib_reproduce_weakly_supervised --output_directory=<output> \
   --gin_model_config_dir=<dir> \
   --gin_model_config_name=<name> \
   --gin_postprocess_config_glob=<postprocess_configs> \
   --gin_evaluation_config_glob=<eval_configs> \
   --gin_validation_config_glob=<val_configs> \
   --pipeline_seed=<seed> \
   --eval_seed=<seed> \
   --supervised_seed=<seed> \
   --num_labelled_samples=<num> \
   --train_percentage=0.9 \
   --labeller_fn="@perfect_labeller"

Feedback

Please send any feedback to [email protected] and [email protected].

Citation

If you use disentanglement_lib, please consider citing:

@inproceedings{locatello2019challenging,
  title={Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations},
  author={Locatello, Francesco and Bauer, Stefan and Lucic, Mario and Raetsch, Gunnar and Gelly, Sylvain and Sch{\"o}lkopf, Bernhard and Bachem, Olivier},
  booktitle={International Conference on Machine Learning},
  pages={4114--4124},
  year={2019}
}

This is not an officially supported Google product.

disentanglement_lib's People

Contributors

obachem avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

disentanglement_lib's Issues

Feature request: Please replace tf.layers with tf.keras.layers

In the official TensorFlow blog posts, it was said that tf.layers would be depreciated and tf.keras.layers would be the preferred method. However in architectures.py tf.layers has been used everywhere to create the encoder and decoder models. It would be great if these could be modifed to use tf.keras.layers since others would be able to reuse the code without worry.

#feature request

Lookup index for the trained models

Does a lookup index exist for the 10'800 pretrained disentanglement_lib modules? I wanted to quickly reproduce specific experiments (e.g. btcvae, beta=4, dsprites), but I am not sure how to find that pretrained model.

Thanks!

Disentanglement metrics don't seem to work with continuous factors

I might be misunderstanding (e.g. perhaps all the ground-truth factors are meant to be discretized), but it seems like all of the metrics require latent factors to be discrete, e.g. using a classifier rather than a regressor for DCI. This is a bit inconvenient for users who would like to use your library to evaluate disentanglement solely given matrices of continuous latents and ground truth factors (without using the representation_function API). I know it's a design choice rather than a bug, but I'd still like to make the suggestion that it would be helpful for future users if you supported this use case!

Resource issue when evaluating the factor-vae metric

When models are evaluated with factor-vae metric, num_variance_estimate (default 10000) samples are fed to representation_function (https://github.com/google-research/disentanglement_lib/blob/master/disentanglement_lib/evaluation/metrics/factor_vae.py#L57). So, users having ready-made NVIDIA GPUs cannot deal with 10000x64x64x3 inputs as a single batch. I tested with GTX TITAN Xp with 12GB memory and Tensorflow 1.13.1. Please modify the code to extract representations with batch_size and concatenate them.

Bug during ScreamDSprites initialization

I have noticed a bug when trying to use ScreamDsprites:

from disentanglement_lib.data.ground_truth.dsprites import ScreamDSprites
ScreamDSprites([1,2,3,4,5])

Raises an error:
Traceback (most recent call last): File "<input>", line 1, in <module> File "disentanglement_lib/disentanglement_lib/data/ground_truth/dsprites.py", line 183, in __init__ scream.thumbnail((350, 274, 3)) File ".venv/disentanglement_lib/lib/python3.6/site-packages/PIL/Image.py", line 2299, in thumbnail x, y = map(math.floor, size) ValueError: too many values to unpack (expected 2)

The issue seems to be that thumbnail size is taking a tuple of 3 values instead of 2.
I have modified l.182 of dsprites.py from scream.thumbnail((350, 274, 3)) to scream.thumbnail((350, 274)) and this is working properly.

I am using Pillow 8.0.1, but the thumbnail size was already 2D in Pillow 5.0.0 so this is not a compatibility issue.

I should add that the bug is also present in cars3D

Weak Dataset Sampling - Possible Discrepancy Between Implementation and Arxiv Paper?

I have been examining the implementation of generating the weakly supervised dataset from the paper Weakly-Supervised Disentanglement Without Compromises, however I think there is a discrepancy between the function simple_dynamics in disentanglement_lib/methods/weak/train_weak_lib and what is described in Section 5 from version 3 the Arxiv paper.

Examined Function

def simple_dynamics(z, ground_truth_data, random_state,
return_index=False, k=gin.REQUIRED):
"""Create the pairs."""
if k == -1:
k_observed = random_state.randint(1, ground_truth_data.num_factors)
else:
k_observed = k
index_list = random_state.choice(
z.shape[1], random_state.choice([1, k_observed]), replace=False)
idx = -1
for index in index_list:
z[:, index] = np.random.choice(
range(ground_truth_data.factors_num_values[index]))
idx = index
if return_index:
return z, idx
return z, k_observed

Excerpt & Corresponding Code

The following is an excerpt from the Experimental Setup subsection from Section 5 of the paper Weakly-Supervised Disentanglement Without Compromises, split into sections that I assume should correspond to the above code:

  1. To create data sets with weak supervision from the existing disentanglement data sets, we first sample from the discrete z according to the ground-truth generative model (1)–(2).

    def simple_dynamics(z, ground_truth_data, random_state,
    return_index=False, k=gin.REQUIRED):

  2. Then, we sample k factors of variation that should not be shared by the two images...

    if k == -1:
    k_observed = random_state.randint(1, ground_truth_data.num_factors)
    else:
    k_observed = k
    index_list = random_state.choice(
    z.shape[1], random_state.choice([1, k_observed]), replace=False)

  3. ... and re-sample those coordinates to obtain z˜. This ensures that each image pair differs in at most k factors of variation.

    idx = -1
    for index in index_list:
    z[:, index] = np.random.choice(
    range(ground_truth_data.factors_num_values[index]))
    idx = index

  4. For k we consider the range from 1 to d − 1. This last setting corresponds to the case where all but one factor of variation are re-sampled. We study both the case where k is constant across all pairs in the data set and where k is sampled uniformly in the range [d − 1] for every training pair (k = Rnd in the following). Unless specified otherwise, we aggregate the results for all values of k.

Problem With 2?

From the above excerpt there seems to be a problem with line:

index_list = random_state.choice(
z.shape[1], random_state.choice([1, k_observed]), replace=False)

In particular the expression random_state.choice([1, k_observed]). Instead of keeping k fixed half of the time k will be set to 1.

I may be misunderstanding things from the excerpt, but to me this seems odd that this is happening.

Fix?

Based on this, should lines 48 and 49 not be the following?

index_list = random_state.choice(z.shape[1], k_observed, replace=False)

Problem With 4?

Based on the following excerpt it seems as though factors in the sampled pairs should always differ.

...We study both the case where k is constant across all pairs in the data set...

z[:, index] = np.random.choice(
range(ground_truth_data.factors_num_values[index]))

However, based on lines 52-53 this is not the case. There is a chance for the re-sampled factor to be the same. It is not guaranteed to be different.

This probability of being the same will only increase if the ground truth dimensionality/size of that factor is small.

Fix?

Sampling with the original value for the particular differing z removed from the range.

Untested possible code for 1 input factor:

choices = set(range(ground_truth_data.factors_num_values[index])) - {z[0, index]}
z[0, index] = np.random.choice(choices) 

warning during pip install .[tf_gpu]

I got a error in runningpip install .[tf_gpu]:
tensorboard 1.14.0 has requirement setuptools>=41.0.0, but you'll have setuptools 40.8.0 which is incompatible.
what should I do?

conv_encoder function Differs from Beta-VAE Paper?

Based on row 3 of Table 1 on page 13 of "beta-VAE: Learning Basic Visual
Concepts with a Constrained Variational Framework"
(https://openreview.net/forum?id=Sy2fzU9gl)

The Beta-VAE paper states that all the convolutional kernels are of size 4x4. I am not sure if it is intended that the 3rd and 4th convolutional layers in the conv_encoder function reduce these kernel sizes to 2x2?

e3 = tf.layers.conv2d(
inputs=e2,
filters=64,
kernel_size=2,
strides=2,
activation=tf.nn.relu,
padding="same",
name="e3",
)
e4 = tf.layers.conv2d(
inputs=e3,
filters=64,
kernel_size=2,
strides=2,
activation=tf.nn.relu,
padding="same",
name="e4",
)

Visualization tool outputs GIF files which are just grey

Running dlib_reproduce - model dir and output dir outputs all stationary visualizations, but any animations/GIFS are just grey boxes. The same problem also occurs when running directly using the visualize function.

TensorFlow: 1.14.0

Encoder architecture does not match paper

Is the encoder architecture ( conv_encoder ) from here the one used in the experiments in the paper?
It seems to be a discrepancy in terms of the size of conv. kernels used in the 3rd and 4th layers.
The code uses 2x2 convolutions there, while, according to Table 2 from the Appendix of the paper 4x4 convolutions are used all the way.

DCI metric: different length of latent codes

Dear Google research group:

Thanks for the great implement, I have a related question.

In DCI metric, dose the dimention D of code C (in 2 THEORETICAL FRAMEWORK of the paper) affect the output number? If so, how it affect disentanglement, completeness and informativeness? Can I use this metric to compare two latent codes with different length?

Thank you very much for your help.

Best Wishes,

Alex

Not tf 1.14 compatible

The newest tf 1.14 release removes tensorflow.python.ops.linalg.linear_operator_util.matmul_with_boradcast.
This causes the tests to fail and I can't run the example code.

Bug: Factor-VAE metric can only be evaluated if representation size equals number of latent factors

Hi,

the latest changes in v1.1 introduced an assertion that prevents Factor-VAE to be evaluated if the number of dimensions of the representation is not equal to the number of latent variables of the dataset:

assert representations.shape[1] == ground_truth_data.num_factors

I am not 100% sure, but I think the metric should also be computable in the other case.

Again, this is problematic for the disentanglement challenge. As it ends this Friday, a quick fix would be appreciated :-)

Requirements.txt

I'm having trouble installing everything. Using setup.py installs tf2.0, which gives me several errors when I run dlib_tests. Can the authors please share a requirements.txt file / fix the setup.py file?

Strange score distribution in the paper.

image
This is Figure 7 MIG distribution on dSprites. The MIG score should be non-negative, and the scores are low. I analyze the pre-trained models and get the following results.
image

I don't know why they are so different, and the other metrics are different too.

Why do animations of the visualized model depend on ground truth data for means?

It seems that the latent traversal animations depend on the ground truth data means for sampling — why is that? Why not sample latents directly and then use the mean representation (from postprocessing) without relying on ground truth statistics?

I might not be completely understanding that portion. Thanks!

https://github.com/google-research/disentanglement_lib/blob/master/disentanglement_lib/visualize/visualize_model.py#L120

confused about 2020ICML-weakly supervised disentanglement

Hi, weakly-supervised disentanglement is a nice work. But I am still confused about some concept. Looking forward to your explanation.

  • in the paper at page 5, you compare the work with prior work. "Our approach critically differs in the sense that S is not known and needs to be estimated for every pair of images." May I ask you about the impletation of S estimation? It seems that I have ignore some crucial details.
  • What is the meaning of Rnd? Is it an abbreviations?

I cant run test command after pip install

Hi guys, I am having this kind or error when I try the test command

dlib_tests

after the pip install command.
pip install .[tf_gpu]

======================================================================
ERROR: test_train_model(['/home/phong/data/Code_hay/disentanglement_lib/disentanglement_lib/config/tests/methods/unsupervised/train_test.gin'], ['model.model = @Vae()', 'vae.beta = 10.', 'reconstruction_loss.loss_fn = @l2_loss', "reconstruction_loss.activation = 'logits'", "reconstruction_loss.activation = 'tanh'", 'encoder.encoder_fn = @conv_encoder', 'decoder.decoder_fn = @deconv_decoder', 'encoder.num_latent = 10']) (main.TrainTest)
test_train_model(['/home/phong/data/Code_hay/disentanglement_lib/disentanglement_lib/config/tests/methods/unsupervised/train_test.gin'], ['model.model = @Vae()', 'vae.beta = 10.', 'reconstruction_loss.loss_fn = @l2_loss', "reconstruction_loss.activation = 'logits'", "reconstruction_loss.activation = 'tanh'", 'encoder.encoder_fn = @conv_encoder', 'decoder.decoder_fn = @deconv_decoder', 'encoder.num_latent = 10']) (main.TrainTest)
test_train_model(['/home/phong/data/Code_hay/disentanglement_lib/disentanglement_lib/config/tests/methods/unsupervised/train_test.gin'], ['model.model = @Vae()', 'vae.beta = 10.', 'reconstruction_loss.loss_fn = @l2_loss', "reconstruction_loss.activation = 'logits'", "reconstruction_loss.activation = 'tanh'", 'encoder.encoder_fn = @conv_encoder', 'decoder.decoder_fn = @deconv_decoder', 'encoder.num_latent = 10'])

Traceback (most recent call last):
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/absl/testing/parameterized.py", line 265, in bound_param_test
test_method(self, *testcase_params)
File "/home/phong/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train_test.py", line 92, in test_train_model
gin_bindings)
File "/home/phong/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train.py", line 53, in train_with_gin
train(model_dir, overwrite)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/gin/config.py", line 1073, in gin_wrapper
utils.augment_exception_message_and_reraise(e, err_str)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/gin/utils.py", line 49, in augment_exception_message_and_reraise
six.raise_from(proxy.with_traceback(exception.traceback), None)
File "", line 3, in raise_from
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/gin/config.py", line 1050, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/home/phong/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train.py", line 123, in train
steps=training_steps)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2876, in train
rendezvous.raise_errors()
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/error_handling.py", line 131, in raise_errors
six.reraise(typ, value, traceback)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2871, in train
saving_listeners=saving_listeners)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1192, in _train_model_default
saving_listeners)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1484, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1252, in run
run_metadata=run_metadata)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1353, in run
raise six.reraise(*original_exc_info)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/six.py", line 693, in reraise
raise value
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1338, in run
return self._sess.run(*args, **kwargs)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1411, in run
run_metadata=run_metadata)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/training/monitored_session.py", line 1169, in run
return self._sess.run(*args, **kwargs)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/phong/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node encoder/e1/Conv2D (defined at /data/Code_hay/disentanglement_lib/disentanglement_lib/methods/shared/architectures.py:173) ]]
[[loss/_497]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node encoder/e1/Conv2D (defined at /data/Code_hay/disentanglement_lib/disentanglement_lib/methods/shared/architectures.py:173) ]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node encoder/e1/Conv2D:
IteratorGetNext (defined at /data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train.py:155)

Input Source operations connected to node encoder/e1/Conv2D:
IteratorGetNext (defined at /data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train.py:155)

Original stack trace for 'encoder/e1/Conv2D':
File "/miniconda3/envs/deep/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/miniconda3/envs/deep/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train_test.py", line 96, in
absltest.main()
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/testing/absltest.py", line 1909, in main
_run_in_app(run_tests, args, kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/testing/absltest.py", line 2026, in _run_in_app
app.run(main=main_function)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/testing/absltest.py", line 2024, in main_function
function(argv, args, kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/testing/absltest.py", line 2295, in run_tests
argv, args, kwargs, xml_reporter.TextAndXMLTestRunner)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/testing/absltest.py", line 2265, in _run_and_get_tests_result
test_program = unittest.TestProgram(*args, **kwargs)
File "/miniconda3/envs/deep/lib/python3.7/unittest/main.py", line 101, in init
self.runTests()
File "/miniconda3/envs/deep/lib/python3.7/unittest/main.py", line 271, in runTests
self.result = testRunner.run(self.test)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/testing/_pretty_print_reporter.py", line 87, in run
return super(TextTestRunner, self).run(test)
File "/miniconda3/envs/deep/lib/python3.7/unittest/runner.py", line 176, in run
test(result)
File "/miniconda3/envs/deep/lib/python3.7/unittest/suite.py", line 84, in call
return self.run(*args, **kwds)
File "/miniconda3/envs/deep/lib/python3.7/unittest/suite.py", line 122, in run
test(result)
File "/miniconda3/envs/deep/lib/python3.7/unittest/suite.py", line 84, in call
return self.run(*args, **kwds)
File "/miniconda3/envs/deep/lib/python3.7/unittest/suite.py", line 122, in run
test(result)
File "/miniconda3/envs/deep/lib/python3.7/unittest/case.py", line 676, in call
return self.run(*args, **kwds)
File "/miniconda3/envs/deep/lib/python3.7/unittest/case.py", line 628, in run
testMethod()
File "/miniconda3/envs/deep/lib/python3.7/site-packages/absl/testing/parameterized.py", line 265, in bound_param_test
test_method(self, *testcase_params)
File "/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train_test.py", line 92, in test_train_model
gin_bindings)
File "/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train.py", line 53, in train_with_gin
train(model_dir, overwrite)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/gin/config.py", line 1050, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/train.py", line 123, in train
steps=training_steps)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2871, in train
saving_listeners=saving_listeners)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1188, in _train_model_default
features, labels, ModeKeys.TRAIN, self.config)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2709, in _call_model_fn
config)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1146, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 2967, in _model_fn
features, labels, is_export_mode=is_export_mode)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 1549, in call_without_tpu
return self._call_model_fn(features, labels, is_export_mode=is_export_mode)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/tpu/tpu_estimator.py", line 1867, in _call_model_fn
estimator_spec = self._model_fn(features=features, **kwargs)
File "/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/vae.py", line 43, in model_fn
z_mean, z_logvar = self.gaussian_encoder(features, is_training=is_training)
File "/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/unsupervised/vae.py", line 93, in gaussian_encoder
input_tensor, is_training=is_training)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/gin/config.py", line 1050, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/shared/architectures.py", line 53, in make_gaussian_encoder
is_training=is_training)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/gin/config.py", line 1050, in gin_wrapper
return fn(*new_args, **new_kwargs)
File "/data/Code_hay/disentanglement_lib/disentanglement_lib/methods/shared/architectures.py", line 173, in conv_encoder
name="e1",
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
return func(*args, **kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/layers/convolutional.py", line 424, in conv2d
return layer.apply(inputs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 1479, in apply
return self.call(inputs, *args, **kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/layers/base.py", line 537, in call
outputs = super(Layer, self).call(inputs, *args, **kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 634, in call
outputs = call_fn(inputs, *args, **kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 146, in wrapper
), args, kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 446, in converted_call
return _call_unconverted(f, args, kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 253, in _call_unconverted
return f(*args, **kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/keras/layers/convolutional.py", line 196, in call
outputs = self._convolution_op(inputs, self.kernel)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1079, in call
return self.conv_op(inp, filter)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 635, in call
return self.call(inp, filter)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 234, in call
name=self.name)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1953, in conv2d
name=name)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 1071, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/miniconda3/envs/deep/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

In call to configurable 'model' (<function train at 0x7f41779bb8c0>)


Ran 13 tests in 52.660s

FAILED (errors=4)

running dlib_postprocess: what values should the flags gin_config and gin_bindings take?

I downloaded the pretrained model with index 0. To start playing with the model, I want to run dlib_postprocess. I need to give values to the following four flags:

  1. model_dir: "Directory to take trained model from."
  2. output_dir: "Directory to save extracted representation to."
  3. gin_config: "List of paths to the config files."
  4. gin_bindings: "Newline separated list of Gin parameter bindings."

The first two are obvious:

--model_dir=../0/model/tfhub
--output_dir=wherever_I_want_to_save_the_output_of_postprocessing

The last two are less clear to me. For gin_config I tried:

--gin_config= ../0/postprocessed/mean/results/gin/postprocess.gin

And I got the following error:

Traceback (most recent call last): File "/home/mencia/Internship/Projects/Disentanglement_Locatello/env/bin/dlib_postprocess", line 45, in <module> app.run(main) File "/home/mencia/Internship/Projects/Disentanglement_Locatello/env/lib/python3.5/site-packages/absl/app.py", line 299, in run _run_main(main, args) File "/home/mencia/Internship/Projects/Disentanglement_Locatello/env/lib/python3.5/site-packages/absl/app.py", line 250, in _run_main sys.exit(main(argv)) File "/home/mencia/Internship/Projects/Disentanglement_Locatello/env/bin/dlib_postprocess", line 41, in main FLAGS.gin_bindings) File "/home/mencia/Internship/Projects/Disentanglement_Locatello/env/lib/python3.5/site-packages/disentanglement_lib/postprocessing/postprocess.py", line 55, in postprocess_with_gin gin.parse_config_files_and_bindings(gin_config_files, gin_bindings) File "/home/mencia/Internship/Projects/Disentanglement_Locatello/env/lib/python3.5/site-packages/gin/config.py", line 1634, in parse_config_files_and_bindings parse_config_file(config_file, skip_unknown) File "/home/mencia/Internship/Projects/Disentanglement_Locatello/env/lib/python3.5/site-packages/gin/config.py", line 1602, in parse_config_file raise IOError(err_str.format(config_file, prefixes)) OSError: Unable to open file: ~/Internship/Projects/Disentanglement_Locatello/pretrained_models/0/postprocessed/sampled/results/gin/postprocess.gin. Searched config paths: [''].

I also tried importing sweep to dlib_postprocess and defining:

gin_bindings, gin_config = get_model_config(model_num=0)

But this did not work either.

How can I run dlib_postprocess?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.