【文生视频系列】MoneyPrinterTurbo项目初探

1. 背景介绍   

        一直有在关注文生视频方面的进展信息。比较知名的可能还是sora这类基于Diffusion model的文生视频模型。但今天要讲的是另外的文生视频项目,也是偶然间在git上看到的,项目名称叫MoneyPrinterTurbo,是基于MoneyPrinter项目增加了中文支持,虽然使用方式和sora这类可能类似,通过大模型生成视频的脚本信息,然后通过模型进行生成。但又有其不同点,它是通过视频描述信息生成关键词,然后基于关键词对图像视频网站进行检索出相关视频片段后,进行视频合并,并配音,生成最终的视频,原理类似于视频剪辑,并不复杂。

        实际操作下来,发现虽然MoneyPrinterTurbo支持了中文,但是由于视频素材网站还是以外国网站为主,所以基于中文关键词翻译成英文,再去检索的效果比较差。实际体验下来用英文生成视频的效果远好于中文,但质量也不要太报过高的期待,感觉可用但还不够,字幕描述和视频画面一致性还有待提升。由于网络质量问题,执行过程中,从视频素材网站下载内容,可能会出现多次错误,需要频繁尝试才行。假如能利用国内的视频素材,这个项目的生成质量还是可以的。

2. 实验案例展示

2.1 案例1

页面配置:

视频主题:

health and fitness

视频提示语:

Incorporating a healthy lifestyle is essential for overall well-being. Regular exercise, a balanced diet, and adequate sleep are the cornerstones of a healthy routine. Exercise not only helps in maintaining a healthy weight but also improves cardiovascular health, boosts mood, and strengthens muscles and bones. A balanced diet rich in fruits, vegetables, lean proteins, and whole grains provides essential nutrients for the body to function optimally. Additionally, getting enough sleep is crucial for physical and mental health, as it allows the body to recover and rejuvenate. By making these small changes, one can significantly improve their health and fitness.

关键词:

Health and Fitness Lifestyle, Exercise and Nutrition, Cardiovascular Health Benefits, Balanced Diet Importance, Muscle and Bone Strength

执行日志:

2024-07-28 15:06:36.803 | INFO     | __main__:<module>:783 - 开始生成视频
2024-07-28 15:06:36.804 | INFO     | __main__:<module>:784 - {
    "video_subject": "health and fitness",
    "video_script": "Incorporating a healthy lifestyle is essential for overall well-being. Regular exercise, a balanced diet, and adequate sleep are the cornerstones of a healthy routine. Exercise not only helps in maintaining a healthy weight but also improves cardiovascular health, boosts mood, and strengthens muscles and bones. A balanced diet rich in fruits, vegetables, lean proteins, and whole grains provides essential nutrients for the body to function optimally. Additionally, getting enough sleep is crucial for physical and mental health, as it allows the body to recover and rejuvenate. By making these small changes, one can significantly improve their health and fitness.",
    "video_terms": "Health and Fitness Lifestyle, Exercise and Nutrition, Cardiovascular Health Benefits, Balanced Diet Importance, Muscle and Bone Strength",
    "video_aspect": "9:16",
    "video_concat_mode": "random",
    "video_clip_duration": 5,
    "video_count": 1,
    "video_source": "pexels",
    "video_materials": null,
    "video_language": "",
    "voice_name": "en-US-BrianNeural-Male",
    "voice_volume": 1.0,
    "voice_rate": 1.0,
    "bgm_type": "random",
    "bgm_file": "",
    "bgm_volume": 0.2,
    "subtitle_enabled": true,
    "subtitle_position": "bottom",
    "custom_position": 70.0,
    "font_name": "MicrosoftYaHeiBold.ttc",
    "text_fore_color": "#FFFFFF",
    "text_background_color": "transparent",
    "font_size": 60,
    "stroke_color": "#000000",
    "stroke_width": 1.5,
    "n_threads": 2,
    "paragraph_number": 1
}
2024-07-28 15:06:36.805 | INFO     | app.services.task:start:210 - start task: 9e3da794-f3b0-446f-8c9e-576774525370, stop_at: video
2024-07-28 15:06:36.805 | INFO     | app.services.task:generate_script:18 - 
## generating video script
2024-07-28 15:06:36.806 | DEBUG    | app.services.task:generate_script:27 - video script: 
Incorporating a healthy lifestyle is essential for overall well-being. Regular exercise, a balanced diet, and adequate sleep are the cornerstones of a healthy routine. Exercise not only helps in maintaining a healthy weight but also improves cardiovascular health, boosts mood, and strengthens muscles and bones. A balanced diet rich in fruits, vegetables, lean proteins, and whole grains provides essential nutrients for the body to function optimally. Additionally, getting enough sleep is crucial for physical and mental health, as it allows the body to recover and rejuvenate. By making these small changes, one can significantly improve their health and fitness.
2024-07-28 15:06:36.806 | INFO     | app.services.task:generate_terms:38 - 
## generating video terms
2024-07-28 15:06:36.807 | DEBUG    | app.services.task:generate_terms:52 - video terms: [
    "Health and Fitness Lifestyle",
    "Exercise and Nutrition",
    "Cardiovascular Health Benefits",
    "Balanced Diet Importance",
    "Muscle and Bone Strength"
]
2024-07-28 15:06:36.814 | INFO     | app.services.task:generate_audio:75 - 
## generating audio
2024-07-28 15:06:36.816 | INFO     | app.services.voice:azure_tts_v1:1057 - start, voice name: en-US-BrianNeural, try: 1
2024-07-28 15:06:47.004 | INFO     | app.services.voice:azure_tts_v1:1077 - completed, output file: ./storage/tasks/9e3da794-f3b0-446f-8c9e-576774525370/audio.mp3
2024-07-28 15:06:47.008 | INFO     | app.services.task:generate_subtitle:103 - 
## generating subtitle, provider: edge
2024-07-28 15:06:47.015 | INFO     | app.services.voice:create_subtitle:1262 - completed, subtitle file created: ./storage/tasks/9e3da794-f3b0-446f-8c9e-576774525370/subtitle.srt, duration: 41.712
2024-07-28 15:06:47.017 | INFO     | app.services.task:get_video_materials:141 - 
## downloading videos from pexels
2024-07-28 15:06:47.018 | INFO     | app.services.material:search_videos_pexels:47 - searching videos: https://api.pexels.com/videos/search?query=Health+and+Fitness+Lifestyle&per_page=20&orientation=portrait, with proxies: {}
2024-07-28 15:06:52.041 | ERROR    | app.services.material:search_videos_pexels:83 - search videos failed: HTTPSConnectionPool(host='api.pexels.com', port=443): Max retries exceeded with url: /videos/search?query=Health+and+Fitness+Lifestyle&per_page=20&orientation=portrait (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-07-28 15:06:52.044 | INFO     | app.services.material:download_videos:208 - found 0 videos for 'Health and Fitness Lifestyle'
2024-07-28 15:06:52.046 | INFO     | app.services.material:search_videos_pexels:47 - searching videos: https://api.pexels.com/videos/search?query=Exercise+and+Nutrition&per_page=20&orientation=portrait, with proxies: {}
2024-07-28 15:06:53.399 | INFO     | app.services.material:download_videos:208 - found 18 videos for 'Exercise and Nutrition'
2024-07-28 15:06:53.401 | INFO     | app.services.material:search_videos_pexels:47 - searching videos: https://api.pexels.com/videos/search?query=Cardiovascular+Health+Benefits&per_page=20&orientation=portrait, with proxies: {}
2024-07-28 15:06:54.544 | INFO     | app.services.material:download_videos:208 - found 18 videos for 'Cardiovascular Health Benefits'
2024-07-28 15:06:54.547 | INFO     | app.services.material:search_videos_pexels:47 - searching videos: https://api.pexels.com/videos/search?query=Balanced+Diet+Importance&per_page=20&orientation=portrait, with proxies: {}
2024-07-28 15:06:59.567 | ERROR    | app.services.material:search_videos_pexels:83 - search videos failed: HTTPSConnectionPool(host='api.pexels.com', port=443): Max retries exceeded with url: /videos/search?query=Balanced+Diet+Importance&per_page=20&orientation=portrait (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-07-28 15:06:59.576 | INFO     | app.services.material:download_videos:208 - found 0 videos for 'Balanced Diet Importance'
2024-07-28 15:06:59.578 | INFO     | app.services.material:search_videos_pexels:47 - searching videos: https://api.pexels.com/videos/search?query=Muscle+and+Bone+Strength&per_page=20&orientation=portrait, with proxies: {}
2024-07-28 15:07:04.596 | ERROR    | app.services.material:search_videos_pexels:83 - search videos failed: HTTPSConnectionPool(host='api.pexels.com', port=443): Max retries exceeded with url: /videos/search?query=Muscle+and+Bone+Strength&per_page=20&orientation=portrait (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-07-28 15:07:04.600 | INFO     | app.services.material:download_videos:208 - found 0 videos for 'Muscle and Bone Strength'
2024-07-28 15:07:04.601 | INFO     | app.services.material:download_videos:216 - found total videos: 25, required duration: 42 seconds, found duration: 479.0 seconds
2024-07-28 15:07:04.603 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/5528012/5528012-hd_1080_1920_25fps.mp4
2024-07-28 15:07:09.299 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-cfd843b3fd33e3532131e7a7e78116d4.mp4
2024-07-28 15:07:09.300 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/5310962/5310962-hd_1080_1920_25fps.mp4
2024-07-28 15:07:09.301 | INFO     | app.services.material:save_video:158 - video already exists: ./storage/cache_videos/vid-d0a022f903340954ed6723349c4d7218.mp4
2024-07-28 15:07:09.302 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-d0a022f903340954ed6723349c4d7218.mp4
2024-07-28 15:07:09.302 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/6740718/6740718-hd_1080_1920_25fps.mp4
2024-07-28 15:07:15.488 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-d491e5f68e92df5d1c82df611fee789a.mp4
2024-07-28 15:07:15.492 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/5310966/5310966-hd_1080_1920_25fps.mp4
2024-07-28 15:07:15.495 | INFO     | app.services.material:save_video:158 - video already exists: ./storage/cache_videos/vid-d1fd2830013c4c68eea71fc1bcb71325.mp4
2024-07-28 15:07:15.495 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-d1fd2830013c4c68eea71fc1bcb71325.mp4
2024-07-28 15:07:15.496 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/6893205/6893205-hd_1080_1920_25fps.mp4
2024-07-28 15:07:19.030 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-37eb1326ec50872f829b98a80159a201.mp4
2024-07-28 15:07:19.031 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/6707359/6707359-hd_1080_1920_25fps.mp4
2024-07-28 15:07:23.727 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-49bf808f0b13a74e93673a230c2d40da.mp4
2024-07-28 15:07:23.728 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/7017803/7017803-hd_1080_1920_30fps.mp4
2024-07-28 15:07:23.729 | INFO     | app.services.material:save_video:158 - video already exists: ./storage/cache_videos/vid-dde20b6d271f06194124dd94bd6f24ff.mp4
2024-07-28 15:07:23.729 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-dde20b6d271f06194124dd94bd6f24ff.mp4
2024-07-28 15:07:23.730 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/5319426/5319426-hd_1080_1920_25fps.mp4
2024-07-28 15:07:28.742 | ERROR    | app.services.material:download_videos:248 - failed to download video: {
    "provider": "pexels",
    "url": "https://videos.pexels.com/video-files/5319426/5319426-hd_1080_1920_25fps.mp4",
    "duration": 19
} => HTTPSConnectionPool(host='videos.pexels.com', port=443): Max retries exceeded with url: /video-files/5319426/5319426-hd_1080_1920_25fps.mp4 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-07-28 15:07:28.748 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/5645107/5645107-hd_1080_1920_25fps.mp4
2024-07-28 15:07:33.356 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-25a6036bfe6839e5378354e84b63ae62.mp4
2024-07-28 15:07:33.359 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/3694919/3694919-hd_1080_1920_30fps.mp4
2024-07-28 15:07:38.374 | ERROR    | app.services.material:download_videos:248 - failed to download video: {
    "provider": "pexels",
    "url": "https://videos.pexels.com/video-files/3694919/3694919-hd_1080_1920_30fps.mp4",
    "duration": 23
} => HTTPSConnectionPool(host='videos.pexels.com', port=443): Max retries exceeded with url: /video-files/3694919/3694919-hd_1080_1920_30fps.mp4 (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
2024-07-28 15:07:38.377 | INFO     | app.services.material:download_videos:233 - downloading video: https://videos.pexels.com/video-files/5320011/5320011-hd_1080_1920_25fps.mp4
2024-07-28 15:07:38.380 | INFO     | app.services.material:save_video:158 - video already exists: ./storage/cache_videos/vid-81294f94e5ac183e7de4a0582ff48144.mp4
2024-07-28 15:07:38.382 | INFO     | app.services.material:download_videos:238 - video saved: ./storage/cache_videos/vid-81294f94e5ac183e7de4a0582ff48144.mp4
2024-07-28 15:07:38.383 | INFO     | app.services.material:download_videos:243 - total duration of downloaded videos: 45.0 seconds, skip downloading more
2024-07-28 15:07:38.385 | SUCCESS  | app.services.material:download_videos:249 - downloaded 9 videos
2024-07-28 15:07:38.387 | INFO     | app.services.task:generate_final_videos:175 - 
## combining video: 1 => ./storage/tasks/9e3da794-f3b0-446f-8c9e-576774525370/combined-1.mp4
2024-07-28 15:07:38.423 | INFO     | app.services.video:combine_videos:42 - max duration of audio: 42.1 seconds
2024-07-28 15:07:38.424 | INFO     | app.services.video:combine_videos:46 - each clip will be maximum 5 seconds long
2024-07-28 15:07:45.101 | INFO     | app.services.video:combine_videos:117 - resizing video to 1080 x 1920, clip size: 720 x 1280
2024-07-28 15:07:45.152 | INFO     | app.services.video:combine_videos:117 - resizing video to 1080 x 1920, clip size: 720 x 1280
2024-07-28 15:07:48.307 | INFO     | app.services.video:combine_videos:117 - resizing video to 1080 x 1920, clip size: 720 x 1280
2024-07-28 15:07:49.312 | INFO     | app.services.video:combine_videos:117 - resizing video to 1080 x 1920, clip size: 720 x 1280
2024-07-28 15:07:49.817 | INFO     | app.services.video:combine_videos:129 - writing
2024-07-28 15:08:54.208 | SUCCESS  | app.services.video:combine_videos:140 - completed
2024-07-28 15:08:54.213 | INFO     | app.services.task:generate_final_videos:191 - 
## generating video: 1 => ./storage/tasks/9e3da794-f3b0-446f-8c9e-576774525370/final-1.mp4
2024-07-28 15:08:54.213 | INFO     | app.services.video:generate_video:212 - start, video size: 1080 x 1920
2024-07-28 15:08:54.214 | INFO     | app.services.video:generate_video:213 -   ① video: ./storage/tasks/9e3da794-f3b0-446f-8c9e-576774525370/combined-1.mp4
2024-07-28 15:08:54.214 | INFO     | app.services.video:generate_video:214 -   ② audio: ./storage/tasks/9e3da794-f3b0-446f-8c9e-576774525370/audio.mp3
2024-07-28 15:08:54.215 | INFO     | app.services.video:generate_video:215 -   ③ subtitle: ./storage/tasks/9e3da794-f3b0-446f-8c9e-576774525370/subtitle.srt
2024-07-28 15:08:54.215 | INFO     | app.services.video:generate_video:216 -   ④ output: ./storage/tasks/9e3da794-f3b0-446f-8c9e-576774525370/final-1.mp4
2024-07-28 15:08:54.216 | INFO     | app.services.video:generate_video:231 - using font: ./resource/fonts/MicrosoftYaHeiBold.ttc
2024-07-28 15:09:56.414 | SUCCESS  | app.services.video:generate_video:302 - completed
2024-07-28 15:09:56.432 | SUCCESS  | app.services.task:start:307 - task 9e3da794-f3b0-446f-8c9e-576774525370 finished, generated 1 videos.
2024-07-28 15:09:56.455 | INFO     | __main__:<module>:805 - 视频生成完成

日志主要步骤:

## generating video script

## generating video terms

## generating audio

## generating subtitle, provider: edge

## downloading videos from pexels

## combining video

## generating video

生成视频结果:

AIGC文生视频-health and fitness

3. 项目安装流程

项目地址:

MoneyPrinterTurbo

项目功能:

提供一个视频主题,就可以全自动生成视频文案、视频素材、视频字幕、视频背景音乐,然后合成一个高清的短视频。

资源要求:

  • 建议最低 CPU 4核或以上,内存 8G 或以上,显卡非必须
  • Windows 10 或 MacOS 11.0 以上系统

安装流程:

1. 下载项目
git clone https://github.com/harry0703/MoneyPrinterTurbo.git        

2.修改配置文件

  • 将 config.example.toml 文件复制一份,命名为 config.toml
  • 按照 config.toml 文件中的说明,配置好 pexels_api_keys 和 llm_provider,并根据 llm_provider 对应的服务商,配置相关的 API Key

这里我使用的llm_provider是月之暗面,其实这种中文的生成,对于现有绝大多数大模型来说,都不是什么大问题。

月之暗面的api链接:Moonshot AI - 开放平台

3.安装docker并启动
cd MoneyPrinterTurbo

docker compose up

4. 访问Web界面

打开浏览器,访问 http://0.0.0.0:8501

就可以看到刚才示例中的编辑页面了

### 开源项目用于文本生成视频 为了实现从文本生成视频的任务,通常需要结合多个技术组件来完成整个流程。一方面,存在专门针对文本到图像或文本到视频生成的应用程序,这些应用程序利用了高性能的Diffusion Transformer(DiT),能够有效地将自然语言描述转化为视觉内容[^1]。 ```python from transformers import pipeline text_to_video_generator = pipeline('text-to-video-generation') video_output = text_to_video_generator("A beautiful sunset over the ocean.") ``` 然而,在许多情况下,直接的文字生成视频功能并不是现成可用的。因此,一种常见的做法是先使用文本转语音(TTS)工具创建音频文件,再借助视频编辑软件或其他专用APIs合成最终的产品。例如,可以采用开源AI模型如GPT-2生成详细的场景描述作为中间步骤的一部分。 ### 开源项目用于图像生成视频 对于由静态图像序列构建动态影像的需求而言,同样存在着多种解决方案可供选择。某些框架允许开发者定义一系列变换规则,从而让计算机自动地把单张或多张静止的画面组合起来形成连贯的动作片段。值得注意的是,MetalCamera这样的iOS平台上的GPU加速库也提供了强大的能力来进行实时渲染以及复杂的后期特效处理操作[^3]。 ```bash # 安装 MetalCamera 及其依赖项 pip install metalcamera opencv-python numpy ``` 尽管如此,具体实施过程中还需要考虑诸如帧率同步、过渡效果设计等因素以确保产出质量达到预期标准。此外,也可以探索其他专注于多媒体创作领域的第三方插件和服务提供商所推出的特色功能模块。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包

打赏作者

源泉的小广场

感谢大佬的支持和鼓励!

¥1 ¥2 ¥4 ¥6 ¥10 ¥20
扫码支付:¥1
获取中
扫码支付

您的余额不足,请更换扫码支付或充值

打赏作者

实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值