Q: Could we drop the hidden layer in a skip-gram word2vec and train only a square weight matrix? After pondering on the (skip-gram) word2vec algorithm and the fact that its single hidden layer is linearly activated, I am not 100% sure that I understand the significance of everything that is happening.
The skip-gram word2vec algorithm optimizes (Mikolov et al. (2013)) 
$$p(w_O | w_I) = \frac{exp({v'_{w_O}}^T v_{w_I})}{\sum_{w=1}^{W} exp({v'_w}^T v_{w_I})}$$
$v_w$ ($v'_w$) is the input (output) embeddings of the word $w_I$ (input word) and $w_O$ (context word) we are interested in. This algorithm is often represented as a neural network with an $n$-dimensional input layer that takes the sparse vector representing a word, an $m$-dimensional hidden layer (where $m < n$) with linear activation, and an $n$-dimensional softmax activated output layer.
The fact the word2vec uses linear activation on the hidden layer suggests that we can reduce the neural network to an $n$-dimensional input layer that connects directly to an $n$-dimensional softmax activated output layer (no hidden layer), and optimize a single square weight matrix ($V$).
If we take this approach of optimizing a single matrix, we could then perform some type of matrix factorization or decomposition, and decompose $V$ into 2 matrices of dimension  $n \times m$ and $m \times n$. I originally thought that this way would allow us to test different embedding dimensions ($m$) while ensuring that the embeddings are optimal (given that $V$ is optimal) (no retraining needed). 
However, it turns out that the math is not that simple. Such a decomposition of a square matrix ($V$) into 2 non-square matrices is only possible if $V$ is not full rank. Furthermore, if $rank(V)=r < n$, then it is only possible to decompose $V$ into an $n \times r$ and an $r \times n$ matrix.
Question: Could you please comment on the following conclusions:


*

*When we design our word2vec embedding, we can choose the embedding dimensionality. So, are we forcing $rank({{V'}_O}^T V_I) = m$ by training a word2vec with $m < n$ hidden nodes?

*By a training a network with one $n \times n$ weight matrix (i.e. no hidden layer) and calculating its rank, we could find the optimal dimension of the embedding layer (optimal in terms of lowest loss, not computational efficiency).


I hope I made my thoughts clear enough but would be happy to clarify.

EDIT:
I ran a small experiment and it seems to me that including the hidden layer does ensure a defined rank. I trained a model without the hidden layer and the resulting weight matrix was full rank. 
