Neural Network classification with pretraining

User 978 | 11/24/2014, 3:14:23 AM

Hi, Does the deeplearning module currently support pre-training? In my case I have a large set of unclassified data that I'd like to learn structure from, followed by fine tuning with a much smaller classified set.

I was hoping to do something similar to this stacked autoencoder pre-training http://nbviewer.ipython.org/github/lisa-lab/pylearn2/blob/master/pylearn2/scripts/tutorials/stackedautoencoders/stackedautoencoders.ipynb

It appears that currently all networks must be passed into neuralnet_classifier- which only supports a single column of classifications as output- making this kind of pre-training not currently possible.

Comments

User 940 | 11/24/2014, 7:18:08 PM

Hi knaps,

Unfortunately, our deep learning module does not support un-supervised pre-training via an auto-encoder. However, we will support feature extraction in our upcoming 1.1 release. This will allow you to pre-train on a related classified set, and then use features extracted by that deep neural network to train a different/simpler classifier on the original task. This was originally explored by the DeCaF paper: http://arxiv.org/pdf/1310.1531v1.pdf

In my experience this is extremely effective for image classification of small datasets, and is much simpler to tune than auto-encoders. What kind of data are you working with? It's possible that this functionality will work for you as well.

Otherwise, we are currently thinking about feature requests for future development. We'll certainly put unsupervised pre-training on that list.

Cheers! -Piotr


User 978 | 11/26/2014, 6:24:56 AM

Piotr, Thanks for the quick reply. I'm looking at a user history dataset with many millions of observations to predict a success event that occurs extremely rarely. Since the DeCAF feature extraction requires classified observations in both stages, and focuses on convolutional networks, probably not good for my use case. But looking forward to what else you all come up with.