Differentiable Morphing

Image morphing without reference points by applying warp maps and optimizing over them.
Differentiable Morphing is machine learning algorithm that can morph any two images without reference points. It called "differentiable morphing" because neural network here is not used in traditional data to label mapping sence, but as an easy way to solve optimisation problem where one image is mapped to another via warp maps that are found by gradient descent. So after maps are found there is no need for the network itself.

Results

example_1

example_2

example_3

Dependencies

Tensorflow 2.1.3 and above.

Usage

morph.py -s images/img_1.jpg -t images/img_2.jpg

-s Source file
-t Target file

Unnecessery parameters:
-e Number of epochs to train maps on traning stage
-a Addition map multiplyer
-m Multiplication map multiplyer
-w Warp map multiplyer
-add_first If true add map would be applyed to the source image before mult map. (might work better in some cases)

Idea

Suppose we want to produce one image from another in a way that we use as much useful information as possible, so if two given images share any similarities between them we make use of these similarities.

toy_example

After several trials I found out that the best way to achieve such effect is to use following formula.

formula

Here "Mult map" removes unnesecessery parts of an image and shifts color balance, "Add map" creates new colors that are not present in original image and "Warp map" distort an image in some way to reproduce shifting, rotation and scaling of objects. W operation is dense_image_warp method that present in tensorflow and usually used for optical flow estimation tasks.

All maps are found by gradient descent using very simple convolution network. Now, by applying alpha scaling parameter to every map we will get smooth transition from one image to another without any loss of useful data (at least for the given toy example).

transition

GitHub