ComfyUI-全民舞王-MusePose

已于 2024-06-14 17:01:59 修改
阅读量1k 收藏 2
点赞数 8
分类专栏： Ai 文章标签： comfyui musepose sd
于 2024-06-14 16:48:54 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/xzpdxz/article/details/139685397
版权

musepose-demo

项目介绍

  • https://github.com/TMElyralab/MusePose
  • 发布的模型能够根据给定的姿势序列，生成参考图中人物的舞蹈视频，结果质量超越了同一主题中几乎所有当前开源的模型。
  • 发布pose align算法，以便用户可以将任意舞蹈视频与任意参考图像对齐，这显著提高了推理性能并增强了模型的可用性

部署

cd custom_nodes
git clone https://github.com/TMElyralab/MusePose.git
pip install -r requirements.txt
pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0"

模型下载

cd MusePose

import os
import wget
from tqdm import tqdm

os.makedirs('pretrained_weights', exist_ok=True)

urls = ['https://download.openmmlab.com/mmdetection/v2.0/yolox/yolox_l_8x8_300e_coco/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth',
        'https://huggingface.co/yzd-v/DWPose/resolve/main/dw-ll_ucoco_384.pth',
        'https://huggingface.co/TMElyralab/MusePose/resolve/main/MusePose/denoising_unet.pth',
        'https://huggingface.co/TMElyralab/MusePose/resolve/main/MusePose/motion_module.pth',
        'https://huggingface.co/TMElyralab/MusePose/resolve/main/MusePose/pose_guider.pth',
        'https://huggingface.co/TMElyralab/MusePose/resolve/main/MusePose/reference_unet.pth',
        'https://huggingface.co/lambdalabs/sd-image-variations-diffusers/resolve/main/unet/diffusion_pytorch_model.bin',
        'https://huggingface.co/lambdalabs/sd-image-variations-diffusers/resolve/main/image_encoder/pytorch_model.bin',
        'https://huggingface.co/stabilityai/sd-vae-ft-mse/resolve/main/diffusion_pytorch_model.bin'
        ]

paths = ['dwpose', 'dwpose', 'MusePose', 'MusePose', 'MusePose', 'MusePose', 'sd-image-variations-diffusers/unet', 'image_encoder', 'sd-vae-ft-mse']

for path in paths:
  os.makedirs(f'pretrained_weights/{path}', exist_ok=True)

# saving weights 
for url, path in tqdm(zip(urls, paths)):
    filename = wget.download(url, f'pretrained_weights/{path}')

config_urls = ['https://huggingface.co/lambdalabs/sd-image-variations-diffusers/resolve/main/unet/config.json',
           'https://huggingface.co/lambdalabs/sd-image-variations-diffusers/resolve/main/image_encoder/config.json',
           'https://huggingface.co/stabilityai/sd-vae-ft-mse/resolve/main/config.json']

config_paths = ['sd-image-variations-diffusers/unet', 'image_encoder', 'sd-vae-ft-mse']

# saving config files 
for url, path in tqdm(zip(config_urls, config_paths)):
    filename = wget.download(url, f'pretrained_weights/{path}')

# renaming model name as given in readme
os.rename('pretrained_weights/dwpose/yolox_l_8x8_300e_coco_20211126_140236-d3bd2b23.pth', 'pretrained_weights/dwpose/yolox_l_8x8_300e_coco.pth')

运行效果

在这里插入图片描述

工作流

{
  "last_node_id": 16,
  "last_link_id": 11,
  "nodes": [
    {
      "id": 5,
      "type": "filenamestring",
      "pos": [
        983,
        765
      ],
      "size": {
        "0": 210,
        "1": 26
      },
      "flags": {},
      "order": 3,
      "mode": 0,
      "inputs": [
        {
          "name": "filenames",
          "type": "VHS_FILENAMES",
          "link": 6,
          "slot_index": 0,
          "label": "filenames"
        }
      ],
      "outputs": [
        {
          "name": "STRING",
          "type": "STRING",
          "links": [
            3
          ],
          "shape": 3,
          "slot_index": 0,
          "label": "STRING"
        }
      ],
      "properties": {
        "Node name for S&R": "filenamestring"
      }
    },
    {
      "id": 13,
      "type": "filenamestring",
      "pos": [
        1241,
        768
      ],
      "size": {
        "0": 210,
        "1": 26
      },
      "flags": {},
      "order": 6,
      "mode": 0,
      "inputs": [
        {
          "name": "filenames",
          "type": "VHS_FILENAMES",
          "link": 8,
          "slot_index": 0,
          "label": "filenames"
        }
      ],
      "outputs": [
        {
          "name": "STRING",
          "type": "STRING",
          "links": [
            9
          ],
          "shape": 3,
          "slot_index": 0,
          "label": "STRING"
        }
      ],
      "properties": {
        "Node name for S&R": "filenamestring"
      }
    },
    {
      "id": 4,
      "type": "museposealign",
      "pos": [
        1077,
        542
      ],
      "size": {
        "0": 315,
        "1": 150
      },
      "flags": {},
      "order": 4,
      "mode": 0,
      "inputs": [
        {
          "name": "image",
          "type": "IMAGE",
          "link": 4,
          "slot_index": 0,
          "label": "image"
        },
        {
          "name": "video",
          "type": "STRING",
          "link": 3,
          "widget": {
            "name": "video"
          },
          "slot_index": 1,
          "label": "video"
        }
      ],
      "outputs": [
        {
          "name": "IMAGE",
          "type": "IMAGE",
          "links": [
            7
          ],
          "shape": 3,
          "slot_index": 0,
          "label": "IMAGE"
        }
      ],
      "properties": {
        "Node name for S&R": "museposealign"
      },
      "widgets_values": [
        "",
        512,
        700,
        300,
        0
      ]
    },
    {
      "id": 10,
      "type": "VHS_VideoCombine",
      "pos": [
        371,
        810
      ],
      "size": [
        210,
        641.7777777777778
      ],
      "flags": {},
      "order": 2,
      "mode": 0,
      "inputs": [
        {
          "name": "images",
          "type": "IMAGE",
          "link": 5,
          "label": "images"
        },
        {
          "name": "audio",
          "type": "VHS_AUDIO",
          "link": null,
          "label": "audio"
        },
        {
          "name": "meta_batch",
          "type": "VHS_BatchManager",
          "link": null,
          "label": "batch_manager"
        }
      ],
      "outputs": [
        {
          "name": "Filenames",
          "type": "VHS_FILENAMES",
          "links": [
            6
          ],
          "shape": 3,
          "slot_index": 0,
          "label": "Filenames"
        }
      ],
      "properties": {
        "Node name for S&R": "VHS_VideoCombine"
      },
      "widgets_values": {
        "frame_rate": 24,
        "loop_count": 0,
        "filename_prefix": "AnimateDiff",
        "format": "video/h264-mp4",
        "pix_fmt": "yuv420p",
        "crf": 19,
        "save_metadata": true,
        "pingpong": false,
        "save_output": false,
        "videopreview": {
          "hidden": false,
          "paused": false,
          "params": {
            "filename": "AnimateDiff_00028.mp4",
            "subfolder": "",
            "type": "temp",
            "format": "video/h264-mp4"
          }
        }
      }
    },
    {
      "id": 11,
      "type": "VHS_VideoCombine",
      "pos": [
        973,
        807
      ],
      "size": [
        390,
        896
      ],
      "flags": {},
      "order": 5,
      "mode": 0,
      "inputs": [
        {
          "name": "images",
          "type": "IMAGE",
          "link": 7,
          "label": "images"
        },
        {
          "name": "audio",
          "type": "VHS_AUDIO",
          "link": null,
          "label": "audio"
        },
        {
          "name": "meta_batch",
          "type": "VHS_BatchManager",
          "link": null,
          "label": "batch_manager"
        }
      ],
      "outputs": [
        {
          "name": "Filenames",
          "type": "VHS_FILENAMES",
          "links": [
            8
          ],
          "shape": 3,
          "slot_index": 0,
          "label": "Filenames"
        }
      ],
      "properties": {
        "Node name for S&R": "VHS_VideoCombine"
      },
      "widgets_values": {
        "frame_rate": 24,
        "loop_count": 0,
        "filename_prefix": "AnimateDiff",
        "format": "video/h264-mp4",
        "pix_fmt": "yuv420p",
        "crf": 19,
        "save_metadata": true,
        "pingpong": false,
        "save_output": false,
        "videopreview": {
          "hidden": false,
          "paused": false,
          "params": {
            "filename": "AnimateDiff_00029.mp4",
            "subfolder": "",
            "type": "temp",
            "format": "video/h264-mp4"
          }
        }
      }
    },
    {
      "id": 6,
      "type": "LoadImage",
      "pos": [
        592,
        1014
      ],
      "size": {
        "0": 368.78228759765625,
        "1": 444.1121826171875
      },
      "flags": {},
      "order": 0,
      "mode": 0,
      "outputs": [
        {
          "name": "IMAGE",
          "type": "IMAGE",
          "links": [
            4,
            10
          ],
          "shape": 3,
          "slot_index": 0,
          "label": "IMAGE"
        },
        {
          "name": "MASK",
          "type": "MASK",
          "links": null,
          "shape": 3,
          "label": "MASK"
        }
      ],
      "properties": {
        "Node name for S&R": "LoadImage"
      },
      "widgets_values": [
        "pic (4).png",
        "image"
      ]
    },
    {
      "id": 9,
      "type": "VHS_LoadVideo",
      "pos": [
        7,
        837
      ],
      "size": [
        357.83404541015625,
        876.5938585069445
      ],
      "flags": {},
      "order": 1,
      "mode": 0,
      "inputs": [
        {
          "name": "meta_batch",
          "type": "VHS_BatchManager",
          "link": null,
          "label": "batch_manager"
        }
      ],
      "outputs": [
        {
          "name": "IMAGE",
          "type": "IMAGE",
          "links": [
            5
          ],
          "shape": 3,
          "slot_index": 0,
          "label": "IMAGE"
        },
        {
          "name": "frame_count",
          "type": "INT",
          "links": null,
          "shape": 3,
          "label": "frame_count"
        },
        {
          "name": "audio",
          "type": "VHS_AUDIO",
          "links": null,
          "shape": 3,
          "label": "audio"
        },
        {
          "name": "video_info",
          "type": "VHS_VIDEOINFO",
          "links": null,
          "shape": 3,
          "label": "video_info"
        }
      ],
      "properties": {
        "Node name for S&R": "VHS_LoadVideo"
      },
      "widgets_values": {
        "video": "dance.mp4",
        "force_rate": 40,
        "force_size": "Disabled",
        "custom_width": 512,
        "custom_height": 512,
        "frame_load_cap": 0,
        "skip_first_frames": 100,
        "select_every_nth": 1,
        "choose video to upload": "image",
        "videopreview": {
          "hidden": false,
          "paused": false,
          "params": {
            "frame_load_cap": 0,
            "skip_first_frames": 100,
            "force_rate": 40,
            "filename": "dance.mp4",
            "type": "input",
            "format": "video/mp4",
            "select_every_nth": 1
          }
        }
      }
    },
    {
      "id": 14,
      "type": "VHS_VideoCombine",
      "pos": [
        1376,
        803
      ],
      "size": [
        390,
        896
      ],
      "flags": {},
      "order": 8,
      "mode": 0,
      "inputs": [
        {
          "name": "images",
          "type": "IMAGE",
          "link": 11,
          "label": "images"
        },
        {
          "name": "audio",
          "type": "VHS_AUDIO",
          "link": null,
          "label": "audio"
        },
        {
          "name": "meta_batch",
          "type": "VHS_BatchManager",
          "link": null,
          "label": "batch_manager"
        }
      ],
      "outputs": [
        {
          "name": "Filenames",
          "type": "VHS_FILENAMES",
          "links": [],
          "shape": 3,
          "slot_index": 0,
          "label": "Filenames"
        }
      ],
      "properties": {
        "Node name for S&R": "VHS_VideoCombine"
      },
      "widgets_values": {
        "frame_rate": 60,
        "loop_count": 0,
        "filename_prefix": "AnimateDiff",
        "format": "video/h264-mp4",
        "pix_fmt": "yuv420p",
        "crf": 19,
        "save_metadata": true,
        "pingpong": false,
        "save_output": false,
        "videopreview": {
          "hidden": false,
          "paused": false,
          "params": {
            "filename": "AnimateDiff_00034.mp4",
            "subfolder": "",
            "type": "temp",
            "format": "video/h264-mp4"
          }
        }
      }
    },
    {
      "id": 12,
      "type": "musepose",
      "pos": [
        1465,
        509
      ],
      "size": {
        "0": 315,
        "1": 246
      },
      "flags": {},
      "order": 7,
      "mode": 0,
      "inputs": [
        {
          "name": "image",
          "type": "IMAGE",
          "link": 10,
          "label": "image"
        },
        {
          "name": "video",
          "type": "STRING",
          "link": 9,
          "widget": {
            "name": "video"
          },
          "label": "video"
        }
      ],
      "outputs": [
        {
          "name": "IMAGE",
          "type": "IMAGE",
          "links": [
            11
          ],
          "shape": 3,
          "slot_index": 0,
          "label": "IMAGE"
        }
      ],
      "properties": {
        "Node name for S&R": "musepose"
      },
      "widgets_values": [
        "",
        512,
        512,
        1000,
        48,
        4,
        3,
        10,
        12
      ]
    }
  ],
  "links": [
    [
      3,
      5,
      0,
      4,
      1,
      "STRING"
    ],
    [
      4,
      6,
      0,
      4,
      0,
      "IMAGE"
    ],
    [
      5,
      9,
      0,
      10,
      0,
      "IMAGE"
    ],
    [
      6,
      10,
      0,
      5,
      0,
      "VHS_FILENAMES"
    ],
    [
      7,
      4,
      0,
      11,
      0,
      "IMAGE"
    ],
    [
      8,
      11,
      0,
      13,
      0,
      "VHS_FILENAMES"
    ],
    [
      9,
      13,
      0,
      12,
      1,
      "STRING"
    ],
    [
      10,
      6,
      0,
      12,
      0,
      "IMAGE"
    ],
    [
      11,
      12,
      0,
      14,
      0,
      "IMAGE"
    ]
  ],
  "groups": [],
  "config": {},
  "extra": {
    "ds": {
      "scale": 0.6303940863128529,
      "offset": {
        "0": 746.0277438571095,
        "1": 34.665619475590915
      }
    }
  },
  "version": 0.4
}
ComfyUI 基础教程（六）秋叶启动器管理ComfyUI
细节控-AI先行者的博客
12-15 1万+
秋葉aaaki大佬的的绘世启动器，支持管理comfyui了，这对comfyui的使用者来说，是个好消息。
Comfyui-MusePose 开源项目使用教程
gitblog_00006的博客
10-11 654
Comfyui-MusePose 开源项目使用教程 Comfyui-MusePose 项目地址: https://gitcode.com/gh_mirrors/co/Comfyui-MusePose ...
ComfyUI神级工作流大揭秘！手把手教你用WAN2.1节点炼成AI视频生产线
最新发布
m0_71745754的博客
03-08 2909
（附节点配置全流程图解+避坑指南）刚入坑ComfyUI却被节点连线逼疯？这份全网独家的WAN2.1万相视频工作流配置手册，将用16个核心节点解锁AI视频工业级流水线！从零搭建只需三步，小白也能做出动态分镜+双语字幕+无损压缩的爆款视频，一、工作流全景图：16个节点如何化身AI视频工厂Mermaid二、核心节点拆解：6大必装神器详解（⚠这些节点配置错1个，整个流水线直接崩！\1. 双文本核弹：Clip+T5双编码器。
MusePose：用于虚拟人类生成的姿势驱动的图像到视频框架。
hjx_dou的专栏
05-30 754
这个东西有个好处，就是你可以生成一个虚拟人物，然后自己做一个视频，将你的动作和语音迁移到虚拟人物上，变成一个更可控的数字人。刚发布的新项目，看演示的话，应该可以根据你的肢体动作，直接驱动图片进行同样的动作。我觉得还是有很大潜力的，有兴趣的同学可以跟进一下。
探索3D姿态编辑的新纪元：ComfyUI 3D Pose Editor
gitblog_00099的博客
06-24 804
探索3D姿态编辑的新纪元：ComfyUI 3D Pose Editor 去发现同类优质开源项目:https://gitcode.com/ 在数字创意领域，每一个细微的创新都能引领全新的艺术表达方式。今天，我们要向您隆重介绍ComfyUI 3D Pose Editor——一个旨在扩展ComfyUI潜能的自定义节点工具集，让3D姿势编辑变得前所未有的直观与强大。 项目介绍 ComfyUI 3D Pos...
20210819-全民所有自然资源资产清查技术指南（试行稿）.pdf
11-15
自然资源资产清查是国家对全民所有土地、矿产、森林、草原、湿地和海洋等自然资源进行系统性盘点和评估的过程。清查旨在摸清资源家底，明确自然资源的存量、质量和空间分布，为国家宏观管理和科学决策提供基础数据...
安卓应用-健康医疗-全民健身软件 v1.0.zip
09-15
"安卓应用-健康医疗-全民健身软件 v1.0.zip" 这个标题揭示了几个关键信息。首先，它表明这是一个专为Android操作系统设计的应用程序，这意味着它将能够在各种Android设备上运行。其次，应用属于“健康医疗”类别，这...
狂暴版-Python-全民飞机大战/素材+源码
09-20
Python飞机大战源码是一个典型的基于Python及其pygame库实现的射击类游戏项目。下面将从项目概述、主要组成部分、核心功能实现以及开发流程等几个方面进行详细介绍。 一、项目概述 飞机大战游戏是一款经典的射击类...
2020淘宝+京东+支付宝双十一-双11全民养猫-全民营业自动化脚本
11-13
在2020年的双十一购物狂欢节期间，淘宝、京东和支付宝等电商平台推出了"全民养猫"活动，吸引大量用户参与。这个活动本质上是一种互动游戏，用户可以通过完成各种任务来"养猫"，最终获得红包或者优惠券等奖励。为了...
远离慢性病------全民健康保障系统学习教案.pptx
11-28
全民健康保障系统不仅是一个科技产品，它更是现代医疗、信息技术与公共卫生管理相结合的产物，旨在通过综合性的信息化手段，提升慢性病的预防、管理和治疗效果。 首先，我们要认识到慢性病的定义与特点。慢性病，...
【Comfyui便携版第三方库环境安装踩坑案列】
08-22
python3.11.8_include
【comfyui教程】一键更换模特姿势，ComfyUI工作流分享！
2401_84760719的博客
12-13 1213
comfyui教程：一键更换模特姿势，ComfyUI工作流分享在创意图像生成领域，ComfyUI正成为一款不可忽视的神器。作为一款专为 Stable Diffusion 设计的基于节点的图形用户界面（GUI），它通过链接不同的模块（称为“节点”）来实现图像生成的高自由度和高度定制化。今天，我们就来分享一个精彩的ComfyUI 工作流：一键更换模特姿势，让你的创作效率直线飙升！温馨提示：篇幅有限，已打包文件夹，获取方式在：文末。
AI魔法: 静态瞬间变身动感舞影-MusePose终于来了
xuxu96
06-28 953
众聚GPU：解锁训练平台的力量，让每一点算力都物超所值
探索虚拟世界的桥梁：MusePose深度解析与应用探索
gitblog_00443的博客
10-10 913
探索虚拟世界的桥梁：MusePose深度解析与应用探索 Comfyui-MusePose 项目地址: https://gitcode.com/gh_mirrors/co/Comfyui-MusePose ...
探秘未来虚拟现实：MusePose引领的图像到视频生成新时代
gitblog_00089的博客
06-08 903
探秘未来虚拟现实：MusePose引领的图像到视频生成新时代 Comfyui-MusePose 项目地址: https://gitcode.com/gh_mirrors/co/Comfyui-MusePose ...
ComfyUI-MuseTalk部署错误解决日志
ruoxuer的博客
05-27 923
1、提示缺 No module named 'aliyunsdkcore'库；2、执行mim install mmengine 命令时，遇到了报错；
comfyUI-MuseTalk用户交互体验的参数设计和设置
pulledup的博客
08-04 1786
comfyUI-MuseTalk的参数设置......该视频的原始帧率会导致输出到缓存中的帧图片的数量的多少，即总批次数batch-sizes（这里简称bss）；最后，参考视频的帧率ref-fps，经验值，英伟达显存等于16G的显卡芯片，1帧
ComfyUI姿态自适应调整工作流
m0_53162279的博客
05-14 1302
Comfyui姿态自适应姿态调整工作流
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值