rmsprop.txt

from:
https://github.com/tensorflow/tensorflow/blob/9e8a4938c06301cc3afa4cd3a3c6c2279e81c98f/tensorflow/core/ops/training_ops.cc#L443-L465

Update '*var' according to the RMSProp algorithm.
Note that in dense implement of this algorithm, ms and mom will 
update even if the grad is zero, but in this sparse implement, ms 
and mom will not update in iterations the grad is zero.

mean_square = decay * mean_square + (1-decay) * gradient ** 2
Delta = learning_rate * gradient / sqrt(mean_square + epsilon)
ms <- rho * ms_{t-1} + (1-rho) * grad * grad
mom <- momentum * mom_{t-1} + lr * grad / sqrt(ms + epsilon)
var <- var - mom

var: Should be from a Variable().
ms: Should be from a Variable().
mom: Should be from a Variable().
lr: Scaling factor. Must be a scalar.
epsilon: Ridge term. Must be a scalar.
rho: Decay rate. Must be a scalar.
grad: The gradient.
out: Same as "var".
use_locking: If `True`, updating of the var, m, and v tensors will be protected
  by a lock; otherwise the behavior is undefined, but may exhibit less
  contention.