Upload folder using huggingface_hub
Browse files- .msc +0 -0
- .mv +1 -0
- README.md +45 -48
- configuration.json +1 -0
- face_det.onnx +3 -0
- face_landmark.onnx +3 -0
- fantasyportrait_model.ckpt +3 -0
- pd_fpg.pth +3 -0
- upload.py +12 -0
.msc
ADDED
|
Binary file (843 Bytes). View file
|
|
|
.mv
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
Revision:master,CreatedAt:1754971377
|
README.md
CHANGED
|
@@ -1,103 +1,100 @@
|
|
| 1 |
-
---
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
-
|
| 7 |
-
|
| 8 |
-
|
|
|
|
| 9 |
|
| 10 |
[](https://fantasy-amap.github.io/fantasy-portrait/)
|
| 11 |
[](https://arxiv.org/abs/2507.12956)
|
| 12 |
-
[](https://huggingface.co/datasets/acvlab/FantasyPortrait
|
| 13 |
[](https://huggingface.co/papers/2507.12956)
|
| 14 |
|
| 15 |
-
## 🔥
|
| 16 |
-
*
|
| 17 |
|
| 18 |
-
##
|
| 19 |
-
|
| 20 |
|
| 21 |
|  |  |
|
| 22 |
| :---: | :---: |
|
| 23 |
|  |  |
|
| 24 |
|  |  |
|
| 25 |
|
|
|
|
|
|
|
| 26 |
|
| 27 |
-
|
| 28 |
-
### 🛠️Installation
|
| 29 |
-
|
| 30 |
-
Clone the repo:
|
| 31 |
|
| 32 |
```
|
| 33 |
git clone https://github.com/Fantasy-AMAP/fantasy-portrait.git
|
| 34 |
cd fantasy-portrait
|
| 35 |
```
|
| 36 |
|
| 37 |
-
|
| 38 |
```
|
| 39 |
apt-get install ffmpeg
|
| 40 |
-
#
|
| 41 |
pip install -r requirements.txt
|
| 42 |
-
#
|
| 43 |
pip install flash_attn
|
| 44 |
```
|
| 45 |
|
| 46 |
-
### 📦Multi-Expr
|
| 47 |
-
|
| 48 |
|
| 49 |
-
|
| 50 |
-
|
|
|
|
| 51 |
| --------------|-------------------------------------------------------------------------------|-------------------------------|
|
| 52 |
-
| Wan2.1-I2V-14B-720P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) |
|
| 53 |
-
| FantasyPortrait | 🤗 [Huggingface](https://huggingface.co/acvlab/FantasyPortrait/) 🤖 [ModelScope](https://www.modelscope.cn/models/amap_cvlab/FantasyPortrait/) |
|
| 54 |
|
| 55 |
-
|
| 56 |
``` sh
|
| 57 |
pip install "huggingface_hub[cli]"
|
| 58 |
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P --local-dir ./models/Wan2.1-I2V-14B-720P
|
| 59 |
huggingface-cli download acvlab/FantasyPortrait --local-dir ./models
|
| 60 |
```
|
| 61 |
|
| 62 |
-
|
| 63 |
``` sh
|
| 64 |
pip install modelscope
|
| 65 |
modelscope download Wan-AI/Wan2.1-I2V-14B-720P --local_dir ./models/Wan2.1-I2V-14B-720P
|
| 66 |
modelscope download amap_cvlab/FantasyPortrait --local_dir ./models
|
| 67 |
```
|
| 68 |
|
| 69 |
-
### 🔑
|
| 70 |
``` sh
|
| 71 |
bash infer_single.sh
|
| 72 |
```
|
| 73 |
|
| 74 |
-
### 🔑
|
| 75 |
-
If you use input image and drive videos with multiple people, you can run as follows:
|
| 76 |
``` sh
|
| 77 |
bash infer_multi.sh
|
| 78 |
```
|
| 79 |
-
If you use input image with multiple people and different multiple single-human driven videos, you can run as follows:
|
| 80 |
-
```sh
|
| 81 |
-
bash infer_multi_diff.sh
|
| 82 |
-
```
|
| 83 |
|
| 84 |
-
### 📦Speed and VRAM Usage
|
| 85 |
-
We present a detailed table here. The model is tested on a single A100.
|
| 86 |
|
| 87 |
-
|
|
|
|
|
|
|
|
|
|
| 88 |
|-|-|-|-|
|
| 89 |
-
|torch.bfloat16|None (
|
| 90 |
-
|torch.bfloat16|7*10**9 (7B)|32.
|
| 91 |
-
|torch.bfloat16|0|42.
|
| 92 |
|
| 93 |
|
| 94 |
|
| 95 |
-
## 🧩
|
| 96 |
-
|
| 97 |
-
|
| 98 |
|
| 99 |
-
## 🔗
|
| 100 |
-
|
| 101 |
```
|
| 102 |
@article{wang2025fantasyportrait,
|
| 103 |
title={FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers},
|
|
@@ -107,5 +104,5 @@ If you find this repository useful, please consider giving a star ⭐ and citati
|
|
| 107 |
}
|
| 108 |
```
|
| 109 |
|
| 110 |
-
##
|
| 111 |
-
|
|
|
|
| 1 |
+
---
|
| 2 |
+
frameworks:
|
| 3 |
+
- Pytorch
|
| 4 |
+
license: Apache License 2.0
|
| 5 |
+
tasks:
|
| 6 |
+
- text-to-video-synthesis
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
# FantasyPortrait:基于表情增强扩散变换器的多角色肖像动画生成
|
| 10 |
|
| 11 |
[](https://fantasy-amap.github.io/fantasy-portrait/)
|
| 12 |
[](https://arxiv.org/abs/2507.12956)
|
| 13 |
+
[](https://huggingface.co/datasets/acvlab/FantasyPortrait)
|
| 14 |
[](https://huggingface.co/papers/2507.12956)
|
| 15 |
|
| 16 |
+
## 🔥 最新动态!!
|
| 17 |
+
* 2025年8月10日:我们已发布推理代码、模型权重和数据集。
|
| 18 |
|
| 19 |
+
## 演示
|
| 20 |
+
更多有趣的结果,请访问我们的[网站](https://fantasy-amap.github.io/fantasy-portrait/)。
|
| 21 |
|
| 22 |
|  |  |
|
| 23 |
| :---: | :---: |
|
| 24 |
|  |  |
|
| 25 |
|  |  |
|
| 26 |
|
| 27 |
+
## 快速开始
|
| 28 |
+
### 🛠️ 安装
|
| 29 |
|
| 30 |
+
克隆仓库:
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
```
|
| 33 |
git clone https://github.com/Fantasy-AMAP/fantasy-portrait.git
|
| 34 |
cd fantasy-portrait
|
| 35 |
```
|
| 36 |
|
| 37 |
+
安装依赖:
|
| 38 |
```
|
| 39 |
apt-get install ffmpeg
|
| 40 |
+
# 确保 torch >= 2.0.0
|
| 41 |
pip install -r requirements.txt
|
| 42 |
+
# 注意:必须安装 flash attention
|
| 43 |
pip install flash_attn
|
| 44 |
```
|
| 45 |
|
| 46 |
+
### 📦 Multi-Expr 数据集
|
| 47 |
+
我们公开了首个多人肖像面部表情视频数据集 **Multi-Expr Dataset**,请通过以下链接下载:
|
| 48 |
|
| 49 |
+
|
| 50 |
+
### 🧱 模型下载
|
| 51 |
+
| 模型 | 下载链接 | 说明 |
|
| 52 |
| --------------|-------------------------------------------------------------------------------|-------------------------------|
|
| 53 |
+
| Wan2.1-I2V-14B-720P | 🤗 [Huggingface](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-720P) 🤖 [ModelScope](https://www.modelscope.cn/models/Wan-AI/Wan2.1-I2V-14B-720P) | 基础模型
|
| 54 |
+
| FantasyPortrait | 🤗 [Huggingface](https://huggingface.co/acvlab/FantasyPortrait/) 🤖 [ModelScope](https://www.modelscope.cn/models/amap_cvlab/FantasyPortrait/) | 我们的表情条件权重
|
| 55 |
|
| 56 |
+
使用 huggingface-cli 下载模型:
|
| 57 |
``` sh
|
| 58 |
pip install "huggingface_hub[cli]"
|
| 59 |
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P --local-dir ./models/Wan2.1-I2V-14B-720P
|
| 60 |
huggingface-cli download acvlab/FantasyPortrait --local-dir ./models
|
| 61 |
```
|
| 62 |
|
| 63 |
+
使用 modelscope-cli 下载模型:
|
| 64 |
``` sh
|
| 65 |
pip install modelscope
|
| 66 |
modelscope download Wan-AI/Wan2.1-I2V-14B-720P --local_dir ./models/Wan2.1-I2V-14B-720P
|
| 67 |
modelscope download amap_cvlab/FantasyPortrait --local_dir ./models
|
| 68 |
```
|
| 69 |
|
| 70 |
+
### 🔑 单人肖像推理
|
| 71 |
``` sh
|
| 72 |
bash infer_single.sh
|
| 73 |
```
|
| 74 |
|
| 75 |
+
### 🔑 多人肖像推理
|
|
|
|
| 76 |
``` sh
|
| 77 |
bash infer_multi.sh
|
| 78 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
|
|
|
|
|
|
|
| 80 |
|
| 81 |
+
### 📦 速度与显存占用
|
| 82 |
+
我们在��提供详细表格。模型在单张A100上进行测试。
|
| 83 |
+
|
| 84 |
+
|`torch_dtype`|`num_persistent_param_in_dit`|速度|所需显存|
|
| 85 |
|-|-|-|-|
|
| 86 |
+
|torch.bfloat16|None (无限制)|15.5秒/迭代|40G|
|
| 87 |
+
|torch.bfloat16|7*10**9 (7B)|32.8秒/迭代|20G|
|
| 88 |
+
|torch.bfloat16|0|42.6秒/迭代|5G|
|
| 89 |
|
| 90 |
|
| 91 |
|
| 92 |
+
## 🧩 社区贡献
|
| 93 |
+
我们 ❤️ 来自开源社区的贡献!如果您的工作改进了 FantasyPortrait,请告知我们。
|
| 94 |
+
您也可以直接发送邮件至 [[email protected]](mailto://[email protected])。我们很乐意引用您的项目,方便大家使用。
|
| 95 |
|
| 96 |
+
## 🔗 引用
|
| 97 |
+
如果本仓库对您有帮助,请考虑给我们一个 star ⭐ 并引用以下论文:
|
| 98 |
```
|
| 99 |
@article{wang2025fantasyportrait,
|
| 100 |
title={FantasyPortrait: Enhancing Multi-Character Portrait Animation with Expression-Augmented Diffusion Transformers},
|
|
|
|
| 104 |
}
|
| 105 |
```
|
| 106 |
|
| 107 |
+
## 致谢
|
| 108 |
+
感谢 [Wan2.1](https://github.com/Wan-Video/Wan2.1)、[PD-FGC](https://github.com/Dorniwang/PD-FGC-inference) 和 [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) 开源他们的模型和代码,为本项目提供了宝贵的参考和支持。我们非常感谢他们对开源社区的贡献。
|
configuration.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"framework":"Pytorch","task":"text-to-video-synthesis"}
|
face_det.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7ea8de1da304c1459a11f637798bb1140805365aeb3cf6637ca6d61909720aec
|
| 3 |
+
size 370619
|
face_landmark.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:463aaa48dd24b9d3b966c76accf9a27f7c590a73a3827d5002a975e46bd54995
|
| 3 |
+
size 2865751
|
fantasyportrait_model.ckpt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:95e1b17849ba13766ea21bd9a9a160b115e80f5dc9adf810da600a9b606c81df
|
| 3 |
+
size 4013364090
|
pd_fpg.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b34f69d8e6070d1f1e9986a31bc954a8998d45ae74436aa7de128825b3606007
|
| 3 |
+
size 39542043
|
upload.py
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from huggingface_hub import HfApi
|
| 2 |
+
|
| 3 |
+
api = HfApi()
|
| 4 |
+
local_dir = "/Users/zuojing/wangqiang/FantasyPortrait"
|
| 5 |
+
repo_id = "acvlab/FantasyPortrait"
|
| 6 |
+
|
| 7 |
+
# 上传文件夹
|
| 8 |
+
api.upload_folder(
|
| 9 |
+
folder_path=local_dir,
|
| 10 |
+
repo_id=repo_id,
|
| 11 |
+
repo_type="model"
|
| 12 |
+
)
|