User 3125 | 1/25/2016, 7:12:31 PM
Hi, I wanted to experiment a bit with the learning rate schedule and the different parameters for exponential decay. However, I got confused by the usage of the words iteration and epoch in the documentation and the output of the neuralnet_classifier.create function.
My understanding (I could be wrong) from the literature is that an epoch is a full pass of all training examples, while an iteration is a full pass of a batch (whatever it's size). However, the output I got from the neuralnet_classifier.create function in the progress table suggests that iteration and epoch are used as synonyms, as the value in iteration column only changes once the number in the examples column has reached the size of my training set.
What does that mean for the learningrateschedule, learningratestep and learningrategamma parameters? The documentation for exponentialdecay says that the learning rate is decreased over iterations. I guess this means a decrease happens each time the iteration number increases? The other two parameters and the formula given (newlr = lr * lrgamma^(epoch/lrstep)) to calculate the updated learning rate refer to epochs. Would it be correct to assume that epoch and iteration are used as synonyms? Thus, reducing the step size of the imagenet net from 100000 to 100 and leaving the gamma at 0.1 would give me a learning rate of approx. 0.000977 at epoch (iteration) 2 and 0.000955 at epoch 3 when starting with a learning rate of 0.001?