Yellow Box → Convolutional LayerBlue Box → Principle Component Analysis LayerRed Box → Independent component Analysis Layer

Now from the above graph we can already know what I wanted to try out, just have couple of convolution layer to extract higher level of features from image and then perform PCA to reduce dimension, and use ICA to statistically make those principle components independent from one another.

The reason why I am using PCA before ICA, is from my past experiments, using ICA as a dimensionality reduction did not converge so well. Also please note that there are multiple method of ICA we can choose, such as Gradient ascent. I am just going to use FastICA.

Before moving on lets first implement PCA layer, as seen above, also one another thing to note here is the fact that I am going to use Adam to optimize the weighs for Fast ICA rather than direct assignment.