AniUnFlow-SamT: Unsupervised Segment-Guided Multi-Frame Optical Flow for 2D Animation with Hybrid Temporal Transformers
Abstract
Optical flow estimation in 2D animation remains challenging due to flat-color regions, weak textures, sharp boundaries, and large non-rigid motion.We propose AniUnFlow-SamT, an unsupervised segment-guided multi-frame optical flow framework with hybrid temporal transformers for anime-style videos. Instead of relying on two-frame estimation, the proposed model leverages short clips and combines temporal slot memory, segment-guided object correspondence, layered affine motion priors, and two-stage dense correlation refinement, followed by boundary-aware residual correction. To train without ground-truth flow, we design a unified objective integrating photometric reconstruction, smoothness regularization, forward-backward consistency, segment-level coherence, boundary-aware constraints, and multi-frame cycle consistency. This design improves motion boundary sharpness, regional coherence, and temporal stability under large motion and occlusion heavy scenarios. Overall, AniUnFlow-SamT provides a practical and robust unsupervised solution for animation-oriented optical flow and supports downstream tasks such as interpolation and temporally consistent stylized video processing.
References
L. Siyao, S. Zhao, W. Yu, W. Sun, D. Metaxas, C. C. Loy,
and Z. Liu, “ Deep Animation Video Interpolation in the
Wild ,” in 2021 IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR), 2021, pp. 6583–6591.
H. Jiang, D. Sun, V. Jampani, M.-H. Yang,
E. G. Learned-Miller, and J. Kautz, “Super slomo:
High quality estimation of multiple intermediate
frames for video interpolation,” 2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition,
pp. 9000–9008, 2017. [Online]. Available:
https://api.semanticscholar.org/CorpusID:10817557
S. Niklaus and F. Liu, “Softmax splatting
for video frame interpolation,” 2020 IEEE/CVF
Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 5436–5445, 2020. [Online]. Available:
https://api.semanticscholar.org/CorpusID:212675709
R. Narita, K. Hirakawa, and K. Aizawa, “Optical
flow based line drawing frame interpolation using
distance transform to support inbetweenings,” 2019
IEEE International Conference on Image Processing
(ICIP), pp. 4200–4204, 2019. [Online]. Available:
https://api.semanticscholar.org/CorpusID:202777808
H. Zhu, X. Liu, T.-T. Wong, and P.-A. Heng,
“Globally optimal toon tracking,” ACM Trans. Graph.,
vol. 35, no. 4, Jul. 2016. [Online]. Available:
https://doi.org/10.1145/2897824.2925872
D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “ PWC-Net:
CNNs for Optical Flow Using Pyramid, Warping, and
Cost Volume ,” in 2018 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), 2018, pp. 8934–
Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in Computer Vision – ECCV 2020, A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Cham: Springer International Publishing, 2020, pp.402–419.
S. Jiang, D. Campbell, Y. Lu, H. Li, and R. I. Hartley, “Learning to estimate hidden motions with global motion aggregation,” 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9752–9761, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:233033368
H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao, “Gmflow: Learning optical flow via global matching,” 2022 IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR), pp. 8111–8120, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:244709323
L. Siyao, Y. Li, B. Li, C. Dong, Z. Liu, and C. C. Loy, “Animerun: 2d animation visual correspondence from open source 3d movies,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 18 996–19 007.
Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai, and H. Li, “Flowformer: A transformer architecture for optical flow,” in Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVII. Berlin, Heidelberg: Springer-Verlag, 2022, p. 668–685.
J. Wang and E. Adelson, “Representing moving images with layers,” IEEE Transactions on Image Processing, vol. 3, no. 5, pp. 625–638, 1994.
L. Sevilla-Lara, D. Sun, V. Jampani, and M. J. Black, “Optical flow with semantic segmentation and localized layers,” 2016 IEEE Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 3889–3898, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:12168077
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick, “Segment anything,” arXiv:2304.02643, 2023.
N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer, “Sam
: Segment anything in images and videos,” arXiv preprint arXiv:2408.00714, 2024. [Online]. Available: https://arxiv.org/abs/2408.00714
A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox, “ FlowNet: Learning Optical Flow with Convolutional Networks ,” in 2015 IEEE International Conference on Computer Vision (ICCV). Los Alamitos, CA, USA: IEEE Computer Society, Dec. 2015, pp. 2758–2766. [Online]. Available:
https://doi.ieeecomputersociety.org/10.1109/ICCV.2015.316
E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, “Flownet 2.0: Evolution of optical flow estimation with deep networks,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1647–1655, 2016. [Online]. Available: https://api.semanticscholar.org/CorpusID:3759573
T.-W. Hui, X. Tang, and C. C. Loy, “ LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation ,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8981–8989.
X. Shi, Z. Huang, D. Li, M. Zhang, K. C. Cheung, S. See, H. Qin, J. Dai, and H. Li, “Flowformer++: Masked cost volume autoencoding
for pretraining optical flow estimation,” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1599–1610, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:257280104
X. Shi, Z. Huang, W. Bian, D. Li, M. Zhang, K. Cheung, S. See, H. Qin, J. Dai, and H. Li, “Videoflow: Exploiting temporal cues for multi-frame optical flow estimation,” 10 2023, pp. 12 435–12 446.
S. Meister, J. Hur, and S. Roth, “Unflow: Unsupervised learning of optical flow with a bidirectional census loss,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 11 2017.
P. Liu, I. King, M. R. Lyu, and J. Xu, “Ddflow: learning optical flow with unlabeled data distillation,” in Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, ser. AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019. [Online]. Available:
https://doi.org/10.1609/aaai.v33i01.33018770
K. Luo, C. Wang, S. Liu, H. Fan, J. Wang, and J. Sun, “Upflow: Upsampling pyramid for unsupervised optical flow learning,” 06 2021, pp. 1045–1054.
S. Liu, K. Luo, A. Luo, C. Wang, F. Meng, and B. Zeng, “Asflow: Unsupervised optical flow learning with adaptive pyramid sampling,” IEEE Trans. Cir. and Sys. for Video Technol., vol. 32, no. 7, p. 4282–4295, Jul. 2022. [Online]. Available:https://doi.org/10.1109/TCSVT.2021.3130281
A. Stone, D. Maurer, A. Ayvaci, A. Angelova, and R. Jonschkowski, “ SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping ,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 3886–3895.
L. Kong and J. Yang, “Mdflow: Unsupervised optical flow learning by reliable mutual knowledge distillation,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, pp. 677–688, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:252253907
M. Feng, L. Liu, H. Jia, G. Xu, and X. Yang, “Flowda: Unsupervised domain adaptive framework for optical flow estimation,” 2023. [Online]. Available: https://arxiv.org/abs/2312.16995
S. Yuan, L. Luo, Z. Hui, C. Pu, X. Xiang, R. Ranjan, and D. Demandolx, “Unsamflow: Unsupervised optical flow guided by segment anything model,” 2024. [Online]. Available: https://arxiv.org/abs/2405.02608
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), 2017.
DOI: http://dx.doi.org/10.21553/rev-jec.456
Copyright (c) 2026 REV Journal on Electronics and Communications
ISSN: 1859-378X Copyright © 2011-2026 |
|
