Module leaf::solvers::sgd::momentum

A Stochastic Gradient Descent with Momentum

Momentum in solving neural networks works similar to they way it does in physics. If you travel into a a direction with a high velocity, it becomes very hard to change (or reverse) the direction in which you are moving.

Similarly when adjusting gradients during solving, keeping a part of the previous gradient update can make solving faster, since if you keep adjusting the gradients into the same direction you will reach the optimum faster. It also makes solving more stable.

Structs

Momentum

Stochastic Gradient Descent with Momentum.