ComfyUI-IF_MemoAvatar

Memory-Guided Diffusion for Expressive Talking Video Generation

#ORIGINAL REPO MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation <br> Longtao Zheng*, Yifan Zhang*, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, Shuicheng Yan <br> Project Page | arXiv | Model

This repository contains the example inference script for the MEMO-preview model. The gif demo below is compressed. See our project page for full videos.

ComfyUI-IF_MemoAvatar

Memory-Guided Diffusion for Expressive Talking Video Generation

Overview

This is a ComfyUI implementation of MEMO (Memory-Guided Diffusion for Expressive Talking Video Generation), which enables the creation of expressive talking avatar videos from a single image and audio input.

Features

• Generate expressive talking head videos from a single image
• Audio-driven facial animation
• Emotional expression transfer
• High-quality video output

Installation

git clone the repo to your custom_nodes folder and then

cd ComfyUI-IF_MemoAvatar
pip install -r requirements.txt

I removed xformers from the file because it needs a particular combination of pytorch on windows to work

if you are on linux you can just run

pip install xformers 

for windows users if you don't have xformers on your env

pip show xformers 

follow this guide to install a good comfyui environment if you don't see any version install the latest following this free guide

https://ko-fi.com/post/Installing-Triton-and-Sage-Attention-Flash-Attenti-P5P8175434

Model Files

The models will automatically download to the following locations in your ComfyUI installation:

models/checkpoints/memo/
├── audio_proj/
├── diffusion_net/
├── image_proj/
├── misc/
│ ├── audio_emotion_classifier/
│ ├── face_analysis/
│ └── vocal_separator/
└── reference_net/
models/wav2vec/
models/vae/sd-vae-ft-mse/
models/emotion2vec/emotion2vec_plus_large/

Copy the faceanalisys/models models from the folder directly into faceanalisys just until I make sure don't just move then duplicate them cos HF will detect empty and download them every time