---
author:
- |
    AAAI Press\
    Association for the Advancement of Artificial Intelligence\
    2275 East Bayshore Road, Suite 160\
    Palo Alto, California 94303\
bibliography:
- 'references.bib'
title: 'Supplementary material for learning generative networks from off-target samples '
---

 Proof for theorems
===================

We first have the following lemma, that readily follows from the Proposition 1 and Theorem 1 of Goodfellow et al [@GoodfellowNIPS2014].

For a fixed generator $G$, the optimal discriminator $\hat{\D}$ that achieves the minimum of equation (6) is $$\hat{\D}^*(\x) = \frac{\hatw(\x)q(\x)}{\hatw(\x)q(\x) + \pg(\x)}$$ and the $\pg$ that achieves the maximum of of $\hat{D}^*$ is $\pg = \hatw(\x)q(\x)$

Recall the definition of $\rho$, $$\rho = \sup_{x \in \suppp} \frac{q(\x)}{p(\x)}$$

We first prove the upper-bound on KL divergence between $p$ and $\pg$.

If $w(\x) \geq \epsilon\ \forall x\in \suppq$, and $J(\hatw) \leq \epsilon^2$, then $$\KL(p||\pg) \leq \log \Big(\frac{1}{1-\epsilon \rho}\Big)$$

$$\begin{aligned}
\KL(p||\pg) &= \int_\x p(\x) \log \frac{p(\x)}{\hatw(\x)q(\x)}d\x\\\end{aligned}$$

Since $J(\hatw)\leq \epsilon^2$, we have $|\hatw(\x) - w(\x)|\leq \epsilon$, and $\forall \x\in \suppq$, $\hat{w}(\x) \geq w(\x) -
\epsilon$. Decreasing the denominator increases the value of the whole expression, we may take $ w(\x)-\epsilon$ to upper bound the $\KL(p||\pg)$. $$\begin{aligned}
\KL(p||\pg) &\leq \int_\x p(\x) \log \frac{p(\x)}{q(\x)(w(\x)-\epsilon)}\\
&= -\int_\x p(\x) \log \frac{q(\x)w(\x) - q(\x)\epsilon}{p(\x)}\\
&= -\int_\x p(\x) \log \Big( 1 - \frac{q(\x)}{p(\x)}\epsilon \Big)\\
&\leq  -\int_\x p(\x) \log \Big( 1 - \rho\epsilon \Big)\end{aligned}$$ where the last line follows from the definition of $\rho$, and decreasing the quantity inside log increases the value of the whole expression. Now this is equivalent to $$\begin{aligned}
\KL(p||\pg) \leq -log(1-\rho\epsilon)\end{aligned}$$ Notice that $0\leq\rho \epsilon\leq 1$ because $\rho = \sup_{ x \in \suppp} \frac{1}{w(\x)}$ and $\epsilon < w(\x)$.

We now prove the upper-bound on reverse-KL divergence between $p$ and $\pg$.

If $J(\hatw) \leq \epsilon^2$, then $$KL(p_G || p) \leq (1 + \epsilon) \log (1 + \epsilon\rho)$$

By Lemma 1, $p_G(\x) = \hat{w}(\x) q(\x)$, so $$KL(p_G || p) = \int_\x \hat{w}(\x) q(\x) \log \frac{\hat{w}(\x) q(\x)}{p(\x)} d\x$$ Since $J(\hatw)\leq \epsilon^2$, we have $|\hatw(\x) - w(\x)|\leq \epsilon$, and $\forall x\in \suppq$, $\hat{w}(\x) \leq w(\x) +
\epsilon$. So, $$\begin{aligned}
KL(p_G \mid p) & \leq \int_\x (w(\x) + \epsilon)q(\x) \log \frac{(w(\x) + \epsilon)
    q(\x)}{p(\x)} d\x \\
& \leq \int_\x (w(\x)q(\x) + \epsilon q(\x)) \log \frac{w(\x)q(\x) + \epsilon
    q(\x)}{p(\x)} d\x \\
& \leq \int_\x (p(\x) + \epsilon q(\x)) \log \frac{p(\x) + \epsilon q(\x)}{p(\x)} d\x \\
& \leq \int_\x (p(\x) + \epsilon q(\x)) \log \left(1 +
  \epsilon\frac{q(\x)}{p(\x)}\right) d\x\\ 
& \leq \int_\x (p(\x) + \epsilon q(\x)) \log (1 + \epsilon\rho) d\x\\
&  \leq \log (1 + \epsilon\rho) \int_\x) (p(\x) + \epsilon q(\x)) d\x\\
&  \leq  (1 + \epsilon) \log (1 + \epsilon\rho)\\\end{aligned}$$

Details of experiments
======================

We have used Keras[^1], a deep learning software package, to implement our NNs.

 Architecture for experiments
-----------------------------

For the first domain, $D$, $G$, and $\hatw$ takes in as input the number of objects to be placed and the shape of the object. They do not conditioned on a state. For the second domain, they take poses of objects as an input. They do not condition on the objects already placed. For the last domain, they take poses of all the objects on the table as input.

For the first domain, we use 3 layers of dense network for $D$,$G$, and $\hatw$. For the second and last domain, we use convolutional layers, as the location of each pose in the vector is unimportant. Specifically, for $D$, we have a input convolutional layer that has filter size 2 by 2, with stride of 2 by 1, and 256 filters. This is followed by a max pooling layer whose size is 2 by 1, and then this conv-max pooling is repeated one more time. Then, it is followed by two more conv layers whose number of filters is 256, and the filter size is 3 by 1. Then, the output gets merged with the action input, which is then followed by three dense layers each with size 32, 256, and 32. Lastly, It uses sigmoid activation in its output layer.

The generator has exactly the same architecture, except that the last three dense layers have 32, 32, and 32 as its number of nodes. It has linear output layer. $\hatw$ has exactly the same architecture as the generator

The maximum number of epochs for training $D$ and $G$ is 500, with batch size of 32. We used Adam with learning rate 0.001 for $D$ and $G$, and Adadelta with learning rate 1.0 for $\hatw$.

[^1]: <https://github.com/fchollet/keras>
