Author: Hui-Yu Huang (purelyvivid@gmail.com, NCTU) 2020/06
First-Order-Model[1] (NIPS 19), an advanced image animation approach by unsupervised key point detection, has shown extraordinary capability in sparse motion transformation. However, number of key points is a preset hyperparameter and key points learn independently on each heatmaps.
To leverage the relation between key points, I propose a method which regards key points as a Graph, modeled by Attention-Based Graph Neural Network. Graph constraints the key points and shows how key points related to each other, providing informative prior for model training, and can be visualized for human understanding.
- Original Model
- Left to right: ( source image - driving video - predictions - driving video with Key Point - predictions with Key Point )
- Proposed Model
- Left to right: ( source image - driving video - predictions - driving video with Key Point Graph - predictions with Key Point Graph )
- Original Model
- Find the most motion frame: 04
- Left to right: ( source image - driving video - predictions - driving video with Heatmap and Key Point - predictions with with Heatmap and Key Point )
- Up to down: (key point NO. : background + NO. 1~10 )
- Proposed Model
- Find the most motion frame: 10
- Left to right: ( source image - driving video - predictions - driving video with Heatmap and Key Point Graph - predictions with with Heatmap and Key Point Graph )
- Up to down: (key point NO. : background + NO. 1~10 )
[1] This repository is modefied from the source code for the paper First Order Motion Model for Image Animation