This is an implementation of the cyclemoid activation function for PyTorch.
The cyclemoid function achieved state-of-the-art results in a recent benchmark with other popular activation functions as shown below:
Note that this is a figure from the paper submitted on April 1st, 2022. An arxiv preprint will be uploaded soon.
You can install the cyclemoid-pytorch package via
pip install cyclemoid_pytorch
This package implements a
CycleMoid class and a
cyclemoid function. You can use these are drop-in replacements for any activation in PyTorch. For example
from cyclemoid_pytorch import CycleMoid torch.nn.Sequential( # ..., CycleMoid(), # instead of torch.nn.ReLU() # ... )
from cyclemoid_pytorch import cyclemoid # ... def forward(self, x): # ... x = cyclemoid(x) # instead of torch.sigmoid(x)
import matplotlib.pyplot as plt import torch from cyclemoid_pytorch import cyclemoid x = torch.arange(-5, 5, 0.01) y = cyclemoid(x) plt.plot(x, y)
For a concrete usage, check out the demo notebook.
You can now also use the cyclemoid activation in Keras.
import tensorflow as tf from cyclemoid_pytorch.easteregg import CycleMoid tf.keras.utils.get_custom_objects()['cyclemoid'] = CycleMoid model = tf.keras.Sequential( [ tf.keras.Input(...), tf.keras.layers.Conv2D(..., activation="cyclemoid"), # ... ] )