## Gradient Centralization TensorFlow

Instantly improve your training performance of TensorFlow models with just 2 lines of code!

This Python package implements Gradient Centralization in TensorFlow, a simple and effective optimization technique for Deep Neural Networks as suggested by Yong et al. in the paper Gradient Centralization: A New Optimization Technique for Deep Neural Networks. It can both speedup training process and improve the final generalization performance of DNNs.

## Installation

Run the following to install:

```
pip install gradient-centralization-tf
```

## About the Examples

`gctf_mnist.ipynb`

This notebook shows the the process of using the `gradient-centralization-tf`

Python package to train on the Fashion MNIST dataset availaible from `tf.keras.datasets`

. It further also compares using `gctf`

and performance without using `gctf`

.

`gctf_horses_v_humans.ipynb`

This notebook shows the the process of using the `gradient-centralization-tf`

Python package to train on the Horses vs Humans dataset by Laurence Moroney. It further also compares using `gctf`

and performance without using `gctf`

.

## Usage

`gctf.centralized_gradients_for_optimizer`

Create a centralized gradients functions for a specified optimizer.

#### Arguments:

`optimizer`

: a`tf.keras.optimizers.Optimizer`

object. The optimizer you are using.

#### Example:

```
>>> opt = tf.keras.optimizers.Adam(learning_rate=0.1)
>>> opt.get_gradients = gctf.centralized_gradients_for_optimizer(opt)
>>> model.compile(optimizer = opt, ...)
```

#### Returns:

A `tf.keras.optimizers.Optimizer`

object.

`gctf.get_centralized_gradients`

Computes the centralized gradients.

This function is ideally not meant to be used directly unless you are building a custom optimizer, in which case you could point `get_gradients`

to this function. This is a modified version of `tf.keras.optimizers.Optimizer.get_gradients`

.

#### Arguments:

`optimizer`

: a`tf.keras.optimizers.Optimizer`

object. The optimizer you are using.`loss`

: Scalar tensor to minimize.`params`

: List of variables.

#### Returns:

A gradients tensor.

`gctf.optimizers`

Pre built updated optimizers implementing GC.

This module is speciially built for testing out GC and in most cases you would be using `gctf.centralized_gradients_for_optimizer`

though this module implements `gctf.centralized_gradients_for_optimizer`

. You can directly use all optimizers with `tf.keras.optimizers`

updated for GC.

#### Example:

```
>>> model.compile(optimizer = gctf.optimizers.adam(learning_rate = 0.01), ...)
>>> model.compile(optimizer = gctf.optimizers.rmsprop(learning_rate = 0.01, rho = 0.91), ...)
>>> model.compile(optimizer = gctf.optimizers.sgd(), ...)
```

#### Returns:

A `tf.keras.optimizers.Optimizer`

object.

## Developing `gctf`

To install `gradient-centralization-tf`

, along with tools you need to develop and test, run the following in your virtualenv:

```
git clone https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow
# or clone your own fork
pip install -e .[dev]
```

## GitHub

https://github.com/Rishit-dagli/Gradient-Centralization-TensorFlow