lipku / metahuman-stream Goto Github PK

Real time streaming digital human based on nerf

Home Page: https://zhuanlan.zhihu.com/p/675131165

License: MIT License

Python 58.81% HTML 4.30% C++ 0.45% Cuda 19.49% C 1.26% Shell 0.06% Dockerfile 0.50% JavaScript 15.13%

digital-human metahuman nerf realtime streaming aigc digihuman

metahuman-stream's Introduction

A streaming digital human based on the Ernerf model， realize audio video synchronous dialogue. It can basically achieve commercial effects.
基于ernerf模型的流式数字人，实现音视频同步对话。基本可以达到商用效果

![Watch the video]

Features

支持声音克隆
支持大模型对话
支持多种音频特征驱动：wav2vec、hubert
支持全身视频拼接
支持rtmp和webrtc
支持视频编排：不说话时播放自定义视频

1. Installation

Tested on Ubuntu 20.04, Python3.10, Pytorch 1.12 and CUDA 11.3

1.1 Install dependency

conda create -n nerfstream python=3.10
conda install pytorch==1.12.1 torchvision==0.13.1 cudatoolkit=11.3 -c pytorch
conda activate nerfstream
pip install -r requirements.txt
pip install "git+https://github.com/facebookresearch/pytorch3d.git"
pip install tensorflow-gpu==2.8.0

linux cuda环境搭建可以参考这篇文章 https://zhuanlan.zhihu.com/p/674972886

2. Quick Start

默认采用webrtc推流到srs

2.1 运行rtmpserver (srs)

export CANDIDATE='<服务器外网ip>'
docker run --rm --env CANDIDATE=$CANDIDATE \
  -p 1935:1935 -p 8080:8080 -p 1985:1985 -p 8000:8000/udp \
  registry.cn-hangzhou.aliyuncs.com/ossrs/srs:5 \
  objs/srs -c conf/rtc.conf

2.2 启动数字人：

python app.py

如果访问不了huggingface，在运行前

export HF_ENDPOINT=https://hf-mirror.com

用浏览器打开http://serverip:8010/rtcpush.html, 在文本框输入任意文字，提交。数字人播报该段文字
备注：服务端需要开放端口 tcp:8000,8010,1985; udp:8000

3. More Usage

3.1 使用LLM模型进行数字人对话

目前借鉴数字人对话系统LinlyTalker的方式，LLM模型支持Chatgpt,Qwen和GeminiPro。需要在app.py中填入自己的api_key。

用浏览器打开http://serverip:8010/rtcpushchat.html

3.2 声音克隆

可以任意选用下面两种服务，推荐用gpt-sovits

3.2.1 gpt-sovits

服务部署参照gpt-sovits
运行

python app.py --tts gpt-sovits --TTS_SERVER http://127.0.0.1:5000 --CHARACTER test --EMOTION default

3.2.2 xtts

运行xtts服务，参照 https://github.com/coqui-ai/xtts-streaming-server

docker run --gpus=all -e COQUI_TOS_AGREED=1 --rm -p 9000:80 ghcr.io/coqui-ai/xtts-streaming-server:latest

然后运行，其中ref.wav为需要克隆的声音文件

python app.py --tts xtts --REF_FILE data/ref.wav --TTS_SERVER http://localhost:9000

3.3 音频特征用hubert

如果训练模型时用的hubert提取音频特征，用如下命令启动数字人

python app.py --asr_model facebook/hubert-large-ls960-ft

3.4 设置背景图片

python app.py --bg_img bg.jpg

3.5 全身视频拼接

3.5.1 切割训练用的视频

ffmpeg -i fullbody.mp4 -vf crop="400:400:100:5" train.mp4

用train.mp4训练模型

3.5.2 提取全身图片

ffmpeg -i fullbody.mp4 -vf fps=25 -qmin 1 -q:v 1 -start_number 0 data/fullbody/img/%d.jpg

3.5.2 启动数字人

python app.py --fullbody --fullbody_img data/fullbody/img --fullbody_offset_x 100 --fullbody_offset_y 5 --fullbody_width 580 --fullbody_height 1080 --W 400 --H 400

--fullbody_width、--fullbody_height 全身视频的宽、高
--W、--H 训练视频的宽、高
ernerf训练第三步torso如果训练的不好，在拼接处会有接缝。可以在上面的命令加上--torso_imgs data/xxx/torso_imgs，torso不用模型推理，直接用训练数据集里的torso图片。这种方式可能头颈处会有些人工痕迹。

3.6 不说话时用自定义视频替代

提取自定义视频图片

ffmpeg -i silence.mp4 -vf fps=25 -qmin 1 -q:v 1 -start_number 0 data/customvideo/img/%d.png

运行数字人

python app.py --customvideo --customvideo_img data/customvideo/img --customvideo_imgnum 100

3.7 webrtc p2p

此种模式不需要srs

python app.py --transport webrtc

用浏览器打开http://serverip:8010/webrtc.html

3.8 rtmp推送到srs

安装rtmpstream库
参照 https://github.com/lipku/python_rtmpstream
启动srs

docker run --rm -it -p 1935:1935 -p 1985:1985 -p 8080:8080 registry.cn-hangzhou.aliyuncs.com/ossrs/srs:5

运行数字人

python app.py --transport rtmp --push_url 'rtmp://localhost/live/livestream'

用浏览器打开http://serverip:8010/echo.html

4. Docker Run

不需要第1步的安装，直接运行。

docker run --gpus all -it --network=host --rm  registry.cn-hangzhou.aliyuncs.com/lipku/nerfstream:v1.3

docker版本已经不是最新代码，可以作为一个空环境，把最新代码拷进去运行。

5. Data flow

6. 数字人模型文件

可以替换成自己训练的模型(https://github.com/Fictionarry/ER-NeRF)

.
├── data
│   ├── data_kf.json
│   ├── au.csv			
│   ├── pretrained
│   └── └── ngp_kf.pth

7. 性能分析

帧率
在Tesla T4显卡上测试整体fps为18左右，如果去掉音视频编码推流，帧率在20左右。用4090显卡可以达到40多帧/秒。
优化：新开一个线程运行音视频编码推流
延时
整体延时3s左右
（1）tts延时1.7s左右，目前用的edgetts，需要将每句话转完后一次性输入，可以优化tts改成流式输入
（2）wav2vec延时0.4s，需要缓存18帧音频做计算（3）srs转发延时，设置srs服务器减少缓冲延时。具体配置可看 https://ossrs.net/lts/zh-cn/docs/v5/doc/low-latency

8. TODO

添加chatgpt实现数字人对话
声音克隆
数字人静音时用一段视频代替

如果本项目对你有帮助，帮忙点个star。也欢迎感兴趣的朋友一起来完善该项目。
Email: [email protected]
微信公众号：数字人技术

metahuman-stream's People

Contributors

Stargazers

Watchers

Forkers

yyheart xuyangcao kiminh kill136 xzhdx urgecode freespy thetargo liuqqwwe kedreamix cylonspace anselorville qrnkg jackstephen iamleon121 fanxi1992 qoo8888 hadesnull123 yangke1981 samsgates yogipargaien bhswallow dengmin adofsauron liyilungit yslion liuzhongh redstarxz zcfrank1st maxmax2016 amorjnyh yuting-li angive orangels wosiwo songsa1 zhuxiu1234 chulung jeromecn bguozili kieojk yonzhao foldingverse srymurphy datahome73 bosen365 oupula yuanzhongqiao jivaklong ivoidcat zhangziliang04 bchengwang delayless songfang damonpkl skstudying adambear yanyuxiyangzk happyuncle super-alex 1846659840 hushi55 yuantao1880 opensorceycw nypyp bobwong89757 huangjianyi0701 xxsuper meng-x renfengyi chenliqiang1106 goodluckluyan hongwei90 havocera fugui0000 chrismii omanhom kuun993 xunnew barryzm richard28039 zuodh okiahooo lyyilin typhonium jsfgit yelban shuiniu86 milusheep oldtan2020 jackzhousz alexpopo assassindesign fingerx zuomingcai steven-yan unravel-xu 731why ylz201 nixiaodi

metahuman-stream's Issues

推流后如何添加背景图

大佬，想问下在推理视频完成后，如何在绿幕视频上添加背景图。（应该推流前还是后，类比ffmpeg的filter_complex）
提前感谢

环境： wsl2 ubuntu20
操作步骤：按文档安装完，启动nginx, 访问echo页面，输入文字，发送后，无反应。
flask_sockets 已改，websocket正常。
rtmp-server有错误提示：
[2024-02-19 12:13:27.941][ERROR][1][hsbl0o71][4] serve error code=1011(SocketTimeout)(Socket io timeout) : service cycle : rtmp: stream service : rtmp: publish timeout 20000ms, nb_msgs=0
thread [1][hsbl0o71]: do_cycle() [./src/app/srs_app_rtmp_conn.cpp:262][errno=4]
thread [1][hsbl0o71]: service_cycle() [./src/app/srs_app_rtmp_conn.cpp:456][errno=4]
thread [1][hsbl0o71]: do_publishing() [./src/app/srs_app_rtmp_conn.cpp:1035][errno=62](Interrupted system call)

大佬，请问推流怎么加上身子呢？

全身视频贴图后，切割线处对应不上

694_1711808558.mp4

那个领带处切割感很明显。用的是最新代码。

延时问题

项目正常跑通，4090上延时还是很大，而且跑的越久延时越大，一开始大概延时20秒往上，几分钟后延时会达到几分钟，请问这是什么原因？

[aac @ 0x953bb00] Input contains (near) NaN/+-Inf

我们完成了搭建，但是出现了这个错误，貌似是音频解码的错误。
这个错误是随机的，有些音频可以推送，但是推送失败的音频在本地可以整成播放。
请问作者知道这样的原因吗？

[libopenh264 @ 0xace9e00] Incorrect library version loaded

出现这个问题可能是什么原因呢？谢谢作者

环境配置好以后，运行python app.py，进程卡住，没有任何显示，也没有打印start websocket server

手动启动报错，docker启动也报错

启动时报错，并且web页面没有显示人物视频只是一直转圈加载

dcoker启动也报错，无语了

nginx

root@sd-server-01:~/avatar# nginx
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Unknown error)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Unknown error)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Unknown error)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Unknown error)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Unknown error)
nginx: [emerg] still could not bind()

如何解决呢

希望能够添加一个演示视频，可以直接看到本项目的演示效果

如题

输入文本后数字人没有变化

1、docker启动
2、能看到页面和奥巴马动态视频
3、浏览器network查看websocket连接状态正常

大佬请教下，下一步该怎么排查问题？

搭建完成，在网页调用tts合成的时候报错

请教作者，这个可能是什么原因？谢谢

修改python/site-packages/werkzeug/routing/matcher.py，注释websocket_mismatch = True

          修改python/site-packages/werkzeug/routing/matcher.py，注释websocket_mismatch = True

Originally posted by @lipku in #12 (comment)
您能细致描述一下吗，我看是两个条件判断都设置为True, 但是我尝试均不成功。

Cant get any video stream on Vlc with rtmp://localhost/live/livestream, err with Requested output format 'flv' is not a suitable output format

python 3.10
pytorch 1.12.1 cuda 11.3
ubuntu 18.04
ffmpeg 4.4.2

after the install guide, run docker and app.py, cant get the video on vlc and get a err in app.py that said Requested output format 'flv' is not a suitable output format, did there are any advise to fix this issue? if you know, please help me, thx

[INFO] warm up ASR live model, expected latency = 1.400000s
start websocket server
[INFO] frame_to_text... 
[INFO] warm-up done, actual latency = 2.698054s
[NULL @ 0x7f91eacabea0] Requested output format 'flv' is not a suitable output format
[INFO] frame_to_text... 
[INFO] frame_to_text... 
[INFO] frame_to_text... 
[INFO] frame_to_text... 
------actual avg fps:28.9061
[INFO] frame_to_text... 
[INFO] frame_to_text... 
[INFO] frame_to_text... 
[INFO] frame_to_text... 
------actual avg fps:39.0283

win10+Anaconda环境下如何搭建

已经使用Anaconda搭建好第一步。第二部的2.1 运行rtmpserver (srs)该如何实现？

请问生成的音频数据，如何保存为wav文件呢

srs镜像从哪儿获取呢？

作者您好，按照readme基本已经完成了所有的配置，但是这一步 docker run --rm -it -p 1935:1935 -p 1985:1985 -p 8080:8080 registry.cn-hangzhou.aliyuncs.com/ossrs/srs:5 中的镜像是从哪儿下载呢？还是说这些操作只能在阿里云服务器上执行？谢谢

readme启动步骤问题

readme的意思是本地要装一份环境，然后还要用docker。还是本地和docker二选一即可，能否标号完整的启动步骤。启动失败中。

换自己的训练的模型出错，用了wav2vec训练，结果还是出错

训练时，换了wav2vec还是报以上同样的错误，我训练出来的模型是38.4M，我看作者大佬是的ngp_kf.pth文件大小是38M，哪里有问题呢，错误信息：
RuntimeError: Error(s) in loading state_dict for NeRFNetwork:
size mismatch for individual_codes: copying a param with shape torch.Size([12000, 4]) from checkpoint, the shape in current model is torch.Size([10000, 4]).
size mismatch for individual_codes_torso: copying a param with shape torch.Size([12000, 8]) from checkpoint, the shape in current model is torch.Size([10000, 8]).

换自己训练的模型运行就报错，大佬帮看看，感谢！

换自己的训练模型时，声音我是用的Hubert，启动app.py报错，大佬帮看看，感谢！
trainer = Trainer('ngp', opt, model, device=device, workspace=opt.workspace, criterion=criterion, fp16=opt.fp16, metrics=metrics, use_checkpoint=opt.ckpt)
File "/root/nerf/nerf_triplane/utils.py", line 724, in init
self.load_checkpoint(self.use_checkpoint)
File "/root/nerf/nerf_triplane/utils.py", line 1824, in load_checkpoint
missing_keys, unexpected_keys = self.model.load_state_dict(checkpoint_dict['model'], strict=False)
File "/root/miniconda3/envs/er/lib/python3.9/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for NeRFNetwork:
size mismatch for individual_codes: copying a param with shape torch.Size([12000, 4]) from checkpoint, the shape in current model is torch.Size([10000, 4]).
size mismatch for individual_codes_torso: copying a param with shape torch.Size([12000, 8]) from checkpoint, the shape in current model is torch.Size([10000, 8]).

博主，一些憨憨拿你的代码在B站上割韭菜

告他丫的，真不要脸

在WSL2的docker环境下跑通，4090Ti

但还有有点小问题：播放速度会很快，不管是视频的播放速度还是语音的播放速度都像是2倍速：

20240126-235527.mp4

替换成自己训练的模型报错

GPU RAM最低要求是多少，6G可用吗

docker正常启动后，输入文本没有反应没有相应的输出。

如上

nerfreal.py中的warm_up()是必要的嗎

如題，請問有辦法不要做warm_up()嗎? 我發現做了warm_up()後，聲音會比較慢出來，會有播放延遲的感覺，缺乏了實時的互動效果。

效果怎么样？有训练代码吗？

请问可以将背景透明吗？

有什么方法将背景设置透明吗？

执行python app.py报错

RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
详细报错信息如下：
[INFO] warm up ASR live model, expected latency = 1.400000s
start websocket server
[INFO] frame_to_text...
Exception in thread Thread-2 (render):
Traceback (most recent call last):
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/ycw/project/company/metahuman-stream/app.py", line 155, in render
nerfreal.render()
File "/home/ycw/project/company/metahuman-stream/nerfreal.py", line 155, in render
self.asr.warm_up()
File "/home/ycw/project/company/metahuman-stream/asrreal.py", line 445, in warm_up
self.run_step()
File "/home/ycw/project/company/metahuman-stream/asrreal.py", line 220, in run_step
logits, labels, text = self.frame_to_text(inputs)
File "/home/ycw/project/company/metahuman-stream/asrreal.py", line 341, in frame_to_text
result = self.model(inputs.input_values.to(self.device))
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1969, in forward
outputs = self.wav2vec2(
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 1563, in forward
hidden_states, extract_features = self.feature_projection(extract_features)
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 487, in forward
hidden_states = self.projection(norm_hidden_states)
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ycw/anaconda3/envs/nerfstream/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

vlc拉流，播放延迟的问题

老师您好，我这边尝试使用系统的pipe配合ffmpeg去推流，参考这个方法：https://zhuanlan.zhihu.com/p/656815080
我看您代码里有类似的方法。我使用该方法将音视频对齐了，现在唯一的问题就是代码已经推流，但是vlc播放器总是在推了100多帧（大概3-4s）时候才开始播放。我怀疑是缓存机制，但是设置了vlc播放器的缓存，仍然有这个延迟问题。请问您当时试错时候遇到过吗

OSError: CUDA_HOME environment variable is not set

ModuleNotFoundError: No module named '_raymarching_face'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/zhihui/work/metahuman-stream/app.py", line 19, in
from nerf_triplane.network import NeRFNetwork
File "/home/zhihui/work/metahuman-stream/nerf_triplane/network.py", line 6, in
from .renderer import NeRFRenderer
File "/home/zhihui/work/metahuman-stream/nerf_triplane/renderer.py", line 10, in
import raymarching
File "/home/zhihui/work/metahuman-stream/raymarching/init.py", line 1, in
from .raymarching import *
File "/home/zhihui/work/metahuman-stream/raymarching/raymarching.py", line 12, in
from .backend import _backend
File "/home/zhihui/work/metahuman-stream/raymarching/backend.py", line 31, in
_backend = load(name='_raymarching_face',
File "/home/zhihui/anaconda3/envs/metahuman/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1202, in load
return _jit_compile(
File "/home/zhihui/anaconda3/envs/metahuman/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1425, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/zhihui/anaconda3/envs/metahuman/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1514, in _write_ninja_file_and_build_library
extra_ldflags = _prepare_ldflags(
File "/home/zhihui/anaconda3/envs/metahuman/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1622, in _prepare_ldflags
extra_ldflags.append(f'-L{_join_cuda_home("lib64")}')
File "/home/zhihui/anaconda3/envs/metahuman/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 2125, in _join_cuda_home
raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
运行app.py时出现这个问题

本地网页端输入内容，生成数字人区域无反应？

如标题，如何解决这个问题，其他配置一切正常！

推流语速问题

使用现有奥巴马模型,现有语音模型,跑起来推流的时候,发现语速很快,有没有控制参数可用?使用 rtmp_streaming 有参数可以控制吗?

运行`python app.py`出现一大堆报错！

(py39) root@vultr:~/AIGC/metahuman-stream# python app.py 
Traceback (most recent call last):
  File "/root/AIGC/metahuman-stream/raymarching/raymarching.py", line 10, in <module>
    import _raymarching_face as _backend
ModuleNotFoundError: No module named '_raymarching_face'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 2100, in _run_ninja_build
    subprocess.run(
  File "/root/miniconda3/envs/py39/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/root/AIGC/metahuman-stream/app.py", line 19, in <module>
    from nerf_triplane.network import NeRFNetwork
  File "/root/AIGC/metahuman-stream/nerf_triplane/network.py", line 6, in <module>
    from .renderer import NeRFRenderer
  File "/root/AIGC/metahuman-stream/nerf_triplane/renderer.py", line 10, in <module>
    import raymarching
  File "/root/AIGC/metahuman-stream/raymarching/__init__.py", line 1, in <module>
    from .raymarching import *
  File "/root/AIGC/metahuman-stream/raymarching/raymarching.py", line 12, in <module>
    from .backend import _backend
  File "/root/AIGC/metahuman-stream/raymarching/backend.py", line 31, in <module>
    _backend = load(name='_raymarching_face',
  File "/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1308, in load
    return _jit_compile(
  File "/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1710, in _jit_compile
    _write_ninja_file_and_build_library(
  File "/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1823, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 2116, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension '_raymarching_face': [1/3] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=_raymarching_face -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/TH -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/miniconda3/envs/py39/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options '-fPIC' -O3 -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -c /root/AIGC/metahuman-stream/raymarching/src/raymarching.cu -o raymarching.cuda.o 
FAILED: raymarching.cuda.o 
/usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=_raymarching_face -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/TH -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/miniconda3/envs/py39/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options '-fPIC' -O3 -std=c++14 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -c /root/AIGC/metahuman-stream/raymarching/src/raymarching.cu -o raymarching.cuda.o 
In file included from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/util/string_view.h:4,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/util/StringUtil.h:6,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/util/Exception.h:5,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/Generator.h:11,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/CPUGeneratorImpl.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/Context.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/cuda/CUDAContext.h:18,
                 from /root/AIGC/metahuman-stream/raymarching/src/raymarching.cu:5:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/util/C++17.h:27:2: error: #error You need C++17 to compile PyTorch
   27 | #error You need C++17 to compile PyTorch
      |  ^~~~~
In file included from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /root/AIGC/metahuman-stream/raymarching/src/raymarching.cu:6:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4:2: error: #error C++17 or later compatible compiler is required to use PyTorch.
    4 | #error C++17 or later compatible compiler is required to use PyTorch.
      |  ^~~~~
In file included from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:4,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:9,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/torch.h:3,
                 from /root/AIGC/metahuman-stream/raymarching/src/raymarching.cu:6:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/ATen.h:4:2: error: #error C++17 or later compatible compiler is required to use ATen.
    4 | #error C++17 or later compatible compiler is required to use ATen.
      |  ^~~~~
[2/3] c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=_raymarching_face -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/TH -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/miniconda3/envs/py39/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++14 -c /root/AIGC/metahuman-stream/raymarching/src/bindings.cpp -o bindings.o 
FAILED: bindings.o 
c++ -MMD -MF bindings.o.d -DTORCH_EXTENSION_NAME=_raymarching_face -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/TH -isystem /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /root/miniconda3/envs/py39/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++14 -c /root/AIGC/metahuman-stream/raymarching/src/bindings.cpp -o bindings.o 
In file included from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/extension.h:5,
                 from /root/AIGC/metahuman-stream/raymarching/src/bindings.cpp:1:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4:2: error: #error C++17 or later compatible compiler is required to use PyTorch.
    4 | #error C++17 or later compatible compiler is required to use PyTorch.
      |  ^~~~~
In file included from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/util/string_view.h:4,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/util/StringUtil.h:6,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/util/Exception.h:5,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/core/Device.h:5,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/TensorBody.h:11,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/Tensor.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/Tensor.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/autograd/function_hook.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/autograd/cpp_hook.h:2,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/autograd/variable.h:6,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/autograd/autograd.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/autograd.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/extension.h:5,
                 from /root/AIGC/metahuman-stream/raymarching/src/bindings.cpp:1:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/c10/util/C++17.h:27:2: error: #error You need C++17 to compile PyTorch
   27 | #error You need C++17 to compile PyTorch
      |  ^~~~~
In file included from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:4,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:9,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/extension.h:5,
                 from /root/AIGC/metahuman-stream/raymarching/src/bindings.cpp:1:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/ATen.h:4:2: error: #error C++17 or later compatible compiler is required to use ATen.
    4 | #error C++17 or later compatible compiler is required to use ATen.
      |  ^~~~~
In file included from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/ivalue.h:1499,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/List_inl.h:4,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/List.h:490,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/IListRef_inl.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/IListRef.h:632,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/WrapDimUtils.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/TensorNames.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/NamedTensorUtils.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/autograd/variable.h:11,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/autograd/autograd.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/autograd.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/extension.h:5,
                 from /root/AIGC/metahuman-stream/raymarching/src/bindings.cpp:1:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/ivalue_inl.h: In lambda function:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/ivalue_inl.h:1061:30: error: ‘is_convertible_v’ is not a member of ‘std’; did you mean ‘is_convertible’?
 1061 |         if constexpr (::std::is_convertible_v<typename c10::invoke_result_t<T &&, Future&>, IValueWithStorages>) {
      |                              ^~~~~~~~~~~~~~~~
      |                              is_convertible
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/ivalue_inl.h:1061:91: error: expected ‘(’ before ‘,’ token
 1061 |         if constexpr (::std::is_convertible_v<typename c10::invoke_result_t<T &&, Future&>, IValueWithStorages>) {
      |                                                                                           ^
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/ivalue_inl.h:1061:111: error: expected primary-expression before ‘>’ token
 1061 |         if constexpr (::std::is_convertible_v<typename c10::invoke_result_t<T &&, Future&>, IValueWithStorages>) {
      |                                                                                                               ^
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/ivalue_inl.h:1061:112: error: expected primary-expression before ‘)’ token
 1061 |         if constexpr (::std::is_convertible_v<typename c10::invoke_result_t<T &&, Future&>, IValueWithStorages>) {
      |                                                                                                                ^
In file included from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/boxing/KernelFunction_impl.h:1,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/boxing/KernelFunction.h:251,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/op_registration/op_registration.h:11,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/library.h:68,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/autograd/autograd_not_implemented_fallback.h:3,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/autograd.h:4,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
                 from /root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/torch/extension.h:5,
                 from /root/AIGC/metahuman-stream/raymarching/src/bindings.cpp:1:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h: In static member function ‘static Result c10::impl::BoxedKernelWrapper<Result(Args ...), typename std::enable_if<((c10::guts::conjunction<c10::guts::disjunction<std::is_constructible<c10::IValue, typename std::decay<Args>::type>, std::is_same<c10::TensorOptions, typename std::decay<Args>::type> >...>::value && c10::guts::conjunction<c10::guts::disjunction<c10::impl::has_ivalue_to<T, void>, std::is_same<void, ReturnType> >, c10::guts::negation<std::is_lvalue_reference<_Tp> > >::value) && (! c10::impl::is_tuple_of_mutable_tensor_refs<Result>::value)), void>::type>::call(const c10::BoxedKernel&, const c10::OperatorHandle&, c10::DispatchKeySet, Args ...)’:
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h:229:25: error: ‘is_same_v’ is not a member of ‘std’; did you mean ‘is_same’?
  229 |     if constexpr (!std::is_same_v<void, Result>) {
      |                         ^~~~~~~~~
      |                         is_same
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h:229:35: error: expected primary-expression before ‘void’
  229 |     if constexpr (!std::is_same_v<void, Result>) {
      |                                   ^~~~
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h:229:35: error: expected ‘)’ before ‘void’
/root/miniconda3/envs/py39/lib/python3.9/site-packages/torch/include/ATen/core/boxing/impl/boxing.h:229:18: note: to match this ‘(’
  229 |     if constexpr (!std::is_same_v<void, Result>) {
      |                  ^
ninja: build stopped: subcommand failed.

主要的pip依赖版本如下：

pytorch3d                    0.7.5
torch                        2.1.1+cu118
torchaudio                   2.1.1+cu118
torchvision                  0.16.1+cu118

CUDA版本：

$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

Step:

clone了这个仓库
运行

docker run --rm -it -p 1935:1935 -p 1985:1985 -p 8080:8080 registry.cn-hangzhou.aliyuncs.com/ossrs/srs:5

...
[2024-01-04 07:50:56.319][INFO][1][92y51g1q] Process: cpu=1.00%,23MB, threads=2
[2024-01-04 07:51:00.704][INFO][1][9r6618wj] Hybrid cpu=1.00%,23MB, cid=1,0, timer=62,0,0, clock=0,49,0,0,0,0,0,0,0
[2024-01-04 07:51:01.324][INFO][1][92y51g1q] Process: cpu=1.00%,23MB, threads=2
[2024-01-04 07:51:05.705][INFO][1][9r6618wj] Hybrid cpu=1.00%,23MB, cid=1,0, timer=63,0,0, clock=0,48,1,0,0,0,0,0,0
[2024-01-04 07:51:06.328][INFO][1][92y51g1q] Process: cpu=0.00%,23MB, threads=2
[2024-01-04 07:51:10.705][INFO][1][9r6618wj] Hybrid cpu=1.00%,23MB, cid=1,0, timer=63,0,0, clock=0,48,1,0,0,0,0,0,0
[2024-01-04 07:51:11.333][INFO][1][92y51g1q] Process: cpu=0.00%,23MB, threads=2
[2024-01-04 07:51:15.706][INFO][1][9r6618wj] Hybrid cpu=0.00%,23MB, cid=1,0, timer=63,0,0, clock=0,48,1,0,0,0,0,0,0
[2024-01-04 07:51:16.338][INFO][1][92y51g1q] Process: cpu=0.00%,23MB, threads=2
[2024-01-04 07:51:20.706][INFO][1][9r6618wj] Hybrid cpu=0.00%,23MB, cid=1,0, timer=62,0,0, clock=0,47,1,1,0,0,0,0,0
[2024-01-04 07:51:21.343][INFO][1][92y51g1q] Process: cpu=1.00%,23MB, threads=2

运行python app.py

推流延迟问题

大佬,你Demo中依赖的ffmpeg版本是多少?

我们现在用你的Demo推流,到云端,在本地观看,从发文字到观看,其中延迟有30s左右.我们的ffmpeg是4.4.2.

大佬，为什么模特不眨眼了，是哪个参数需要设置？

训练数据是正常的，眨眼的，然后用main.py跑出来的也是正常的，用这个项目流式跑出来就完全不眨眼了，一直是一个眼神，盯住了。。

SRS镜像

请问使用更快地SRS需要更改echo.html吗，我用第一个SRS可以有画面，第二个更快地没有画面出现

OSError: CUDA_HOME environment variable is not set

ModuleNotFoundError: No module named '_raymarching_face'

During handling of the above exception, another exception occurred:

报错

[WARN] Failed to load optimizer.
[INFO] loaded scheduler.
[INFO] loaded scaler.
[INFO] load 7272 frames.
Loading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7272/7272 [00:00<00:00, 87434.94it/s]
[INFO] eye_area: 0.25 - 0.25
[INFO] loading ASR model cpierse/wav2vec2-large-xlsr-53-esperanto...
/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py:381: UserWarning: Passing gradient_checkpointing to a config initialization is deprecated and will be removed in v5 Transformers. Using model.gradient_checkpointing_enable() instead, or if you are using the Trainer API, pass gradient_checkpointing=True in your TrainingArguments.
warnings.warn(
Ignored unknown kwarg option normalize
Ignored unknown kwarg option normalize
Ignored unknown kwarg option normalize
Ignored unknown kwarg option normalize
[tcp @ 0x558b1aefc640] Connection to tcp://localhost:1935 failed: Connection refused
[rtmp @ 0x558b1aefc240] Cannot open connection tcp://localhost:1935
failed to open stream output context, stream will not work
[INFO] warm up ASR live model, expected latency = 1.400000s
start websocket server
[INFO] frame_to_text...
[INFO] warm-up done, actual latency = 1.121296s
[INFO] frame_to_text...
[INFO] frame_to_text...
[INFO] frame_to_text...
[INFO] frame_to_text...
------actual avg fps:39.8501
[INFO] frame_to_text...
[INFO] frame_to_text...
[INFO] frame_to_text...
[INFO] frame_to_text...
------actual avg fps:44.1976
[INFO] frame_to_text...
[INFO] frame_to_text...
[INFO] frame_to_text...