# A Heuristic for Unsupervised Model Selection for Variational Disentangled Representation Learning

@article{Duan2020AHF, title={A Heuristic for Unsupervised Model Selection for Variational Disentangled Representation Learning}, author={Sunny Duan and Nicholas Watters and Lo{\"i}c Matthey and Christopher P. Burgess and Alexander Lerchner and Irina Higgins}, journal={ArXiv}, year={2020}, volume={abs/1905.12614} }

Disentangled representations have recently been shown to improve fairness, data efficiency and generalisation in simple supervised and reinforcement learning tasks. To extend the benefits of disentangled representations to more complex domains and practical applications, it is important to enable hyperparameter tuning and model selection of existing unsupervised approaches without requiring access to ground truth attribute labels, which are not available for most datasets. This paper addresses… Expand

#### Figures, Tables, and Topics from this paper

#### 30 Citations

Robust Disentanglement of a Few Factors at a Time

- Computer Science, Mathematics
- NeurIPS
- 2020

This work introduces population-based training (PBT) for improving consistency in training variational autoencoders (VAEs) and demonstrates the validity of this approach in a supervised setting and introduces the recursive rPU-VAE approach, which shows striking improvement in state-of-the-art unsupervised disentanglement performance and robustness across multiple datasets and metrics. Expand

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation

- Computer Science, Mathematics
- J. Mach. Learn. Res.
- 2020

Theoretically show that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data, and investigate concrete benefits of enforcing disentanglement of the learned representations and consider a reproducible experimental setup covering several data sets. Expand

Odd-One-Out Representation Learning

- Computer Science
- ArXiv
- 2020

A weakly-supervised downstream task based on odd-one-out observations is suitable for model selection by observing high correlation on a difficult downstream abstract visual reasoning task and is empirically show that a bespoke metric-learning VAE model which performs highly on this task also out-performs other standard unsupervised and a weakly -supervised disentanglement model across several metrics. Expand

Demystifying Inductive Biases for β-VAE Based Architectures

- Computer Science
- ArXiv
- 2021

Light is shed on the inductive bias responsible for the success of VAE-based architectures and it is shown that in classical datasets the structure of variance, induced by the generating factors, is conveniently aligned with the latent directions fostered by the VAE objective. Expand

GL-DISEN: GLOBAL-LOCAL DISENTANGLEMENT FOR UNSUPERVISED LEARNING OF GRAPH-LEVEL REPRE- SENTATIONS

- 2020

Graph-level representation learning plays a crucial role in a variety of tasks such as molecular property prediction and community analysis. Currently, several models based on mutual information… Expand

Demystifying Inductive Biases for (Beta-)VAE Based Architectures

- Computer Science
- ICML
- 2021

Light is shed on the inductive bias responsible for the success of VAE-based architectures and it is shown that in classical datasets the structure of variance induced by the generating factors is conveniently aligned with the latent directions fostered by the VAE objective. Expand

Disentanglement and Local Directions of Variance

- Computer Science
- ECML/PKDD
- 2021

This work quantifies the effects of global and local directions of variance in the data on disentanglement performance using proposed measures and seems to find empirical evidence of a negative effect of local variance directions on disENTanglement. Expand

A Commentary on the Unsupervised Learning of Disentangled Representations

- Computer Science, Mathematics
- AAAI
- 2020

The theoretical result showing that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases and the practical challenges it entails is discussed. Expand

Learning Disentangled Representations in the Imaging Domain

- Computer Science
- 2021

The need for disentangled representations is motivated, key theory, and detail practical building blocks and criteria for learning such representations are presented, and applications in medical imaging and computer vision are discussed emphasising choices made in exemplar key works. Expand

An Empirical Study of Uncertainty Gap for Disentangling Factors

- Computer Science
- Trustworthy AI @ ACM Multimedia
- 2021

It is empirically found that the significant factor with the largest Uncertainty Gap should be disentangled before insignificant factors, which indicates that a suitable order of disentangling factors facilities the performance. Expand

#### References

SHOWING 1-10 OF 59 REFERENCES

Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations

- Computer Science, Mathematics
- ICML
- 2019

This paper theoretically shows that the unsupervised learning of disentangled representations is fundamentally impossible without inductive biases on both the models and the data, and trains more than 12000 models covering most prominent methods and evaluation metrics on seven different data sets. Expand

Learning Deep Disentangled Embeddings with the F-Statistic Loss

- Computer Science, Mathematics
- NeurIPS
- 2018

A new paradigm for discovering disentangled representations of class structure is proposed and a novel loss function based on the $F$ statistic is proposed, which describes the separation of two or more distributions. Expand

A Framework for the Quantitative Evaluation of Disentangled Representations

- Computer Science
- ICLR
- 2018

A framework for the quantitative evaluation of disentangled representations when the ground-truth latent structure is available is proposed and three criteria are explicitly defined and quantified to elucidate the quality of learnt representations and thus compare models on an equal basis. Expand

Variational Inference of Disentangled Latent Concepts from Unlabeled Observations

- Computer Science, Mathematics
- ICLR
- 2018

This work considers the problem of unsupervised learning of disentangled representations from large pool of unlabeled observations, and proposes a variational inference based approach to inferdisentangled latent factors. Expand

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

- Computer Science
- ICLR
- 2017

Learning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificial… Expand

Life-Long Disentangled Representation Learning with Cross-Domain Latent Homologies

- Computer Science, Mathematics
- NeurIPS
- 2018

This work proposes a novel algorithm for unsupervised representation learning from piece-wise stationary visual data: Variational Autoencoder with Shared Embeddings (VASE), which automatically detects shifts in the data distribution and allocates spare representational capacity to new knowledge, while simultaneously protecting previously learnt representations from catastrophic forgetting. Expand

Interventional Robustness of Deep Latent Variable Models

- Computer Science, Mathematics
- ArXiv
- 2018

The interventional robustness score is introduced, which provides a quantitative evaluation of robustness in learned representations with respect to interventions on generative factors and changing nuisance factors, and how this score can be estimated from labeled observational data, that may be confounded, and further provide an efficient algorithm that scales linearly in the dataset size. Expand

Disentangling Disentanglement in Variational Autoencoders

- Computer Science, Mathematics
- ICML
- 2019

We develop a generalisation of disentanglement in VAEs---decomposition of the latent representation---characterising it as the fulfilment of two factors: a) the latent encodings of the data having an… Expand

Towards a Definition of Disentangled Representations

- Mathematics, Computer Science
- ArXiv
- 2018

It is suggested that those transformations that change only some properties of the underlying world state, while leaving all other properties invariant are what gives exploitable structure to any kind of data. Expand

Isolating Sources of Disentanglement in Variational Autoencoders

- Computer Science, Mathematics
- NeurIPS
- 2018

We decompose the evidence lower bound to show the existence of a term measuring the total correlation between latent variables. We use this to motivate our $\beta$-TCVAE (Total Correlation… Expand