Apollo optimizer in tensorflow

Nov 09, 2021 1 min read

Apollo Optimizer in Tensorflow 2.x

Notes:

Warmup is important with Apollo optimizer, so be sure to pass in a learning rate schedule vs. a constant learning rate for learning_rate. One cycle scheduler is given as an example in one_cycle_lr_schedule.py
To clip gradient norms as in paper, add either clipnorm (parameter-wise clipping by norm) or global_clipnorm to the arguments (for example clipnorm=0.1).
Decoupled weight decay is used by default.

GitHub

John was the first writer to have joined pythonawesome.com. He has since then inculcated very effective writing and reviewing culture at pythonawesome which rivals have found impossible to imitate.