UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing

Jianhong Bai1*, Tianyu He2† Yuchi Wang3, Junliang Guo2, Haoji Hu1, Zuozhu Liu1, Jiang Bian2
1Zhejiang University, 2Microsoft Research Asia, 3Peking University.
*Work done during an internship at Microsoft Research Asia.
†Corresponding Author.
arXiv Code
Your browser does not support the image tag.

UniEdit supports both video motion editing in the time axis (i.e., from playing guitar to eating or waving) and various video appearance editing scenarios (i.e., stylization, rigid/non-rigid object replacement, background modification).


Abstract

Recent advances in text-guided video editing have showcased promising results in appearance editing (e.g., stylization). However, video motion editing in the temporal dimension (e.g., from eating to waving), which distinguishes video editing from image editing, is underexplored. In this work, we present UniEdit, a tuning-free framework that supports both video motion and appearance editing by harnessing the power of a pre-trained text-to-video generator within an inversion-then-generation framework. To realize motion editing while preserving source video content, based on the insights that temporal and spatial self-attention layers encode inter-frame and intra-frame dependency respectively, we introduce auxiliary motion-reference and reconstruction branches to produce text-guided motion and source features respectively. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers. Extensive experiments demonstrate that UniEdit covers video motion editing and various appearance editing scenarios, and surpasses the state-of-the-art methods.


Method

Overview of UniEdit. It follows an inversion-then-generation pipeline and consists of a main editing path, an auxiliary reconstruction branch and an auxiliary motion-reference branch. The reconstruction branch produces source features for content preservation, and the motion-reference branch yields text-guided motion features for motion injection. The source features and motion features are injected into the main editing path through spatial self-attention (SA-S) and temporal self-attention (SA-T) modules respectively. We further introduce spatial structure control to retain the coarse structure of the source video.

UniEdit + VideoCrafter2




UniEdit + LaVie (Motion Editing)







UniEdit + LaVie (Appearance Editing)




















Comparison with State-of-the-Art Methods

MasaCtrl* indicates we adapt MasaCtrl to the text-to-video model.



Responsible AI Considerations

UniEdit is a tuning-free approach and is intended for advancing AI/ML research on video editing. We encourage users to use the model responsibly. We discourage users from using the codes to generate intentionally deceptive or untrue content or for inauthentic activities. It is suggested to add watermarks to prevent misuse.

BibTeX

@article{bai2024uniedit,
            title={UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing},
            author={Bai, Jianhong and He, Tianyu and Wang, Yuchi and Guo, Junliang and Hu, Haoji and Liu, Zuozhu and Bian, Jiang},
            journal={arXiv preprint arXiv:2402.13185},
            year={2024}
          }