zed+jetson nano+gstreamer+nginx实现播放网络摄像头视频和保存图片

目录

目录

1. 简介

2. nginx配置

2.1安装nginx

2.2安装nginx-http-flv-module，与nginx链接

2.2.1编译nginx-http-flv-module

2.3nginx完整配置

3. gstreamer

3.1ubuntu18.04安装gstreamer

3.2gstreamer的zedsrc

4. gstreamer推流

4.1jetson nano端两个gstreamer命令：（一个发双目rtmp视频流，一个把双目转单目rtmp视频流）

4.2服务器端gstreamer命令（用于保存双目摄像头的左右目视频）：

5. 过程出现的问题

---

1. 简介

zed相机的数据采集需要一台带有gpu的计算机设备连接，经过调研，选用jetson nano边缘设备来连接zed相机，我们采用的方案是将zed相机采集数据传输到jetson nano，然后使用gstreamer结合zedsrc，将zed相机双目图像推流成rtmp数据流。一方面，在jetson nano段执行另一个gstreamer流，将zed双目图像数据流裁剪出单目rtmp流，以此来提供监控需求，另一方面，原先推出的双目图像rtmp流用于双目图像数据保存，对后续双目图像的处理提供条件。

具体架构如下图所示

2. nginx配置

2.1安装nginx

nginx需要安装nginx-http-flv-module以支持flv格式视频流，需要下载该模块，并从源码重新编译nginx，在编译nginx前还需要下载一些必要的依赖，以下参考了使用ffmpeg+nginx+flvjs实现web播放rtsp视频流

wget https://nginx.org/download/nginx-1.18.0.tar.gz
tar zxf nginx-1.18.0.tar.gz

2.2安装nginx-http-flv-module，与nginx链接

git clone https://github.com/winshining/nginx-http-flv-module.git

2.2.1编译nginx-http-flv-module

首先进入刚刚解压的nginx文件夹目录下，用configure文件配置nginx-http-flv-module（如果出现找不到configure的错误，一般是/path/to/nginx-http-flv-module这个路径写错了，而且这个命令建议手打）

udo apt-get install libssl-dev

/path/to/nginx-http-flv-module为刚刚git clone的路径下的文件路径，如/home/nginx-http-flv-module

cd nginx-1.18.0
./configure --add-module=/path/to/nginx-http-flv-module
make
sudo make install

2.3nginx完整配置

sudo vi /usr/local/nginx/conf/nginx.conf 打开nginx配置文件

在这个配置里面，1935是监听rtmp流的，发送到该端口的rtmp流被接受，然后做成hls切片，做成m3u8后发到这个地址，然后前端就可以播放该地址的视频流

http://192.168.94.141:4399/hls/mystream.m3u8

#worker_processes  1; # Windows下应设置为1，因为不支持Unix域套接字
worker_processes  auto; # 从1.3.8和1.2.5版本开始支持自动配置
#daemon off;
#worker_cpu_affinity  0001 0010 0100 1000; # 仅在FreeBSD和Linux上可用
#worker_cpu_affinity  auto; # 从1.9.10版本开始支持自动配置

error_log logs/error.log error;

# 如果模块被编译为动态模块并且需要与RTMP相关的功能，必须指定以下命令，并且必须在events指令之前，否则模块将无法加载或在启动NGINX时加载失败

#load_module modules/ngx_http_flv_live_module.so;

events {
    worker_connections  4096;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    keepalive_timeout  65;

    server {
        listen       4399;

        location / {
            root   /var/www;
            index  index.html index.htm;
        }

        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }

        location /live {
            flv_live on; # 开启flv实时流（订阅）
            chunked_transfer_encoding  on; # 开启'Transfer-Encoding: chunked'响应

            add_header 'Access-Control-Allow-Origin' '*'; # 添加额外的HTTP标头
            add_header 'Access-Control-Allow-Credentials' 'true'; # 添加额外的HTTP标头
        }

        location /hls {
            types {
                application/vnd.apple.mpegurl m3u8;
                video/mp2t ts;
            }

            root /tmp;
            expires -1;
            add_header Cache-Control no-cache;
            add_header 'Access-Control-Allow-Origin' '*';
            add_header Access-Control-Allow-Credentials 'true';
            add_header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept";
            add_header Access-Control-Allow-Methods "GET, POST, OPTIONS";
        }


        location /stat {
            # 流媒体和录制统计的配置

            rtmp_stat all;
            rtmp_stat_stylesheet stat.xsl;
        }

        location /stat.xsl {
            root /var/www/rtmp; # 指定stat.xsl所在的位置
        }

        # 如果需要JSON样式的统计，无需指定stat.xsl，而是使用新的指令rtmp_stat_format

        #location /stat {
        #    rtmp_stat all;
        #    rtmp_stat_format json;
        #}

        location /control {
            rtmp_control all; # rtmp控制模块的配置
        }
    }
}

rtmp_auto_push on;
rtmp_auto_push_reconnect 1s;
rtmp_socket_dir /tmp;

rtmp {
    out_queue           4096;
    out_cork            8;
    max_streams         128;
    timeout             15s;
    drop_idle_publisher 15s;

    log_interval 5s; # 日志模块用于记录在access.log中的时间间隔，对于调试非常有用
    log_size     1m; # 日志模块用于记录access.log中的缓冲区大小
  
    server {
        listen 1935;
        server_name www.test.*; # 用于后缀通配符匹配虚拟主机名

        application myapp {
            live on;
            gop_cache on; # 开启GOP缓存，减少视频第一帧的等待时间
        }

        application hls {

            live on;
            hls on;
            hls_path /tmp/hls;
            hls_fragment 3s;
            #hls_max_fragment 3s;
       	    hls_cleanup on;
   	        hls_playlist_length 9s; 

        }

    }
}

3. gstreamer

3.1ubuntu18.04安装gstreamer

（也可以忽略此章，直接看下面zedsrc的安装，里面就有安装gstreamer）

安装

apt-get install libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudio

验证安装

dpkg -l | grep gstreamer

（什么都没显示就是没安上了）

3.2gstreamer的zedsrc

我们的目标包括要获取zed的双目图像，其中可以尝试v4l2src，直接读取硬件端口的zed相机，但这种模式下，没办法使用zed相机自带的一些sdk的功能，而且也只能默认读取双目的，要想进一步拿单目的，也可以像下面做的一样，把rtmp裁剪

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,width=2560,height=720,framerate=30/1 ! videoconvert ! x264enc tune=zerolatency bitrate=5000 speed-preset=superfast ! h264parse ! mpegtsmux ! hlssink target-duration=5 playlist-root=http://192.168.94.141:5959/zedstream/

安装zedsrc只需要安装官网的方法，首先正确安装zed sdk，然后安装gstreamer，然后再下载zedsrc的包，编译即可。具体操作流程如下链接。

https://github.com/stereolabs/zed-gstreamer

4. gstreamer推流

4.1jetson nano端两个gstreamer命令：（一个发双目rtmp视频流，一个把双目转单目rtmp视频流）

gst-launch-1.0 zedsrc camera-resolution=1 camera-fps=15 stream-type=2  !  videoconvert !   x264enc key-int-max=1   bitrate=20000 ! flvmux ! rtmpsink location="rtmp://172.17.75.4:1935/hls/mystream"

gst-launch-1.0 rtmpsrc location="rtmp://172.17.75.4:1935/hls/mystream" ! flvdemux name=demux     demux.video ! queue ! h264parse ! avdec_h264 ! videoconvert ! videocrop top=0 bottom=1242 left=0 right=0 ! x264enc key-int-max=1 bitrate=20000 ! flvmux ! rtmpsink location="rtmp://172.17.75.4:1935/hls/mystream2"

其中比较关键的参数在于key-int-max=1的设置，设置关键帧间隔为1，这样才能让nginx的hls在裁剪视频流时，找到关键帧，如果不设置关键帧，在nginx端接收到rtmp流时，nginx会报错，说在将rtmp流做成hls切片时，就算再nginx端设置了hls切片最大长度，也因为没有关键帧的原因导致hls切片没法确定切片，最后超过限制nginx强制切片为50s左右一个切片，这就会导致m3u8文件非常巨大，并且没有关键帧，非常卡顿。在设置key-int-max=1后效果提升显著。

其中videocrop top=0 bottom=1242 left=0 right=0 为将原本穿出的上下布局的左右目图片，保留一个作为监控视频流，原本的双目左右目视频流用于源双目视频数据获取。

其中camera-resolution=0为指定2k分辨率，streame-type=2为指定双目图像均采集，以上参数均可在zed gstreamer github观看对应参数。bitrate=20000为指定比特率，较高画质需指定高点的比特率，

4.2服务器端gstreamer命令（用于保存双目摄像头的左右目视频）：

然后这个gstream的命令是写入rtmp流的，所以先得配置nginx在一个服务器上，往那个服务器上推

gst-launch-1.0 rtmpsrc location="rtmp://172.17.75.4/hls/mystream" ! flvdemux name=demux demux.video ! queue ! h264parse ! avdec_h264 ! videoconvert ! video/x-raw,format=I420 ! videorate ! video/x-raw,framerate=1/1 ! tee name=tee tee. ! queue ! videocrop top=0 bottom=720 ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! multifilesink location="/opt/jupyter/zxg/pwcnet/images/testimage/left_frame_%04d.jpg" tee. ! queue ! videocrop top=720 bottom=0 ! videoconvert ! video/x-raw,format=I420 ! jpegenc ! multifilesink location="/opt/jupyter/zxg/pwcnet/images/testimage/right_frame_%04d.jpg"

由于在我们的场景中，jetson nano和服务器在同一个局域网下，所以直接从jetson nano端发出的rtmp流用multifilesink保存图片，根据当前时间戳保存，用supervisor挂载在指定数据集文件夹下。

但由于zed相机fps设置限制，每秒帧数较多，导致保存的图片数量较大，目前尚未解决，加了一个定时删除文件夹下图片的脚本，但该脚本会和gstreamer命令中，按保存个数的形式命名文件相冲突left_frame_%04d.jpg。

5. 过程出现的问题

暂未解决问题：

1.如果摄像头选择2k、1080p的分辨率，真实帧率只能达到每秒5帧，最后的监控效果比较差，经常卡顿。所以目前使用的是720p的分辨率。目前像素较差。

2.zed相机和jetson nano，用supervisor的方式挂载zed相机发送gstreamer流，现在运行5-8个小时左右，会掉线报错the camera not detected。并且在硬件列表看不到zed的硬件，需要重启等待或拔插相机。

3.当前从gstreamer的rtmp流保存的图片数量较多，需5分钟清除一次保存图片，大概10小时就能存满200g的内存（可用脚本控制保存图片，暂未实现）

已解决问题的部分链接（只记得比较麻烦的问题）：

https://forums.developer.nvidia.com/t/zed-cam/75003   

a  website about zed python   important


https://github.com/stereolabs/zed-sdk/issues/574

https://github.com/stereolabs/zed-sdk/issues/333

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
uff 0.6.9 requires protobuf>=3.3.0, but you have protobuf 3.0.0 which is incompatible.





problem:
running python.py
Illegal instruction (core dumped)
s:
export OPENBLAS_CORETYPE=ARMV8
https://github.com/stereolabs/zed-sdk/issues/333

一个安装numpy出现的问题，重新安装也没用，需要升级pip

 Running setup.py bdist_wheel for numpy ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-qyl8lkuw/numpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp56_aqkjdpip-wheel- --python-tag cp36:
  Running from numpy source directory.
  Cythonizing sources
  performance hint: _generator.pyx:707:41: Exception check after calling '_shuffle_int' will always require the GIL to be acquired.
  Possible solutions:
  	1. Declare '_shuffle_int' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  	2. Use an 'int' return type on '_shuffle_int' to allow an error code to be returned.
  performance hint: _generator.pyx:736:45: Exception check after calling '_shuffle_int' will always require the GIL to be acquired.
  Possible solutions:
  	1. Declare '_shuffle_int' as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
  	2. Use an 'int' return type on '_shuffle_int' to allow an error code to be returned.
  
  Error compiling Cython file:
  ------------------------------------------------------------
  ...
      cdef sfc64_state rng_state
  
      def __init__(self, seed=None):
          BitGenerator.__init__(self, seed)
          self._bitgen.state = <void *>&self.rng_state
          self._bitgen.next_uint64 = &sfc64_uint64
                                     ^
  ------------------------------------------------------------
  
  _sfc64.pyx:90:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to the type of the value being assigned.
  Processing numpy/random/_bounded_integers.pxd.in
  Processing numpy/random/_generator.pyx
  Processing numpy/random/_sfc64.pyx
  Traceback (most recent call last):
    File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 235, in <module>
      main()
    File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 231, in main
      find_process_files(root_dir)
    File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 222, in find_process_files
      process(root_dir, fromfile, tofile, function, hash_db)
    File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 188, in process
      processor_function(fromfile, tofile)
    File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 78, in process_pyx
      [sys.executable, '-m', 'cython'] + flags + ["-o", tofile, fromfile])
    File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'cython', '-3', '--fast-fail', '-o', '_sfc64.c', '_sfc64.pyx']' returned non-zero exit status 1.
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-qyl8lkuw/numpy/setup.py", line 508, in <module>
      setup_package()
    File "/tmp/pip-build-qyl8lkuw/numpy/setup.py", line 488, in setup_package
      generate_cython()
    File "/tmp/pip-build-qyl8lkuw/numpy/setup.py", line 285, in generate_cython
      raise RuntimeError("Running cythonize failed!")
  RuntimeError: Running cythonize failed!
  
  ----------------------------------------
  Failed building wheel for numpy
  Running setup.py clean for numpy
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-qyl8lkuw/numpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Running from numpy source directory.
  
  `setup.py clean` is not supported, use one of the following instead:
  
    - `git clean -xdf` (cleans all files)
    - `git clean -Xdf` (cleans all versioned files, doesn't touch
                        files that aren't checked into the git repo)
  
  Add `--force` to your command to use it anyway if you must (unsupported).
  
  
  ----------------------------------------
  Failed cleaning build dir for numpy
Failed to build numpy
Installing collected packages: numpy
  Running setup.py install for numpy ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-qyl8lkuw/numpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-bo0k6la5-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    Running from numpy source directory.
    
    Note: if you need reliable uninstall behavior, then install
    with pip instead of using `setup.py install`:
    
      - `pip install .`       (from a git repo or downloaded source
                               release)
      - `pip install numpy`   (last NumPy release on PyPi)
    
    
    Cythonizing sources
    
    Error compiling Cython file:
    ------------------------------------------------------------
    ...
        cdef sfc64_state rng_state
    
        def __init__(self, seed=None):
            BitGenerator.__init__(self, seed)
            self._bitgen.state = <void *>&self.rng_state
            self._bitgen.next_uint64 = &sfc64_uint64
                                       ^
    ------------------------------------------------------------
    
    _sfc64.pyx:90:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to the type of the value being assigned.
    numpy/random/_bounded_integers.pxd.in has not changed
    numpy/random/_generator.pyx has not changed
    Processing numpy/random/_sfc64.pyx
    Traceback (most recent call last):
      File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 235, in <module>
        main()
      File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 231, in main
        find_process_files(root_dir)
      File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 222, in find_process_files
        process(root_dir, fromfile, tofile, function, hash_db)
      File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 188, in process
        processor_function(fromfile, tofile)
      File "/tmp/pip-build-qyl8lkuw/numpy/tools/cythonize.py", line 78, in process_pyx
        [sys.executable, '-m', 'cython'] + flags + ["-o", tofile, fromfile])
      File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/usr/bin/python3', '-m', 'cython', '-3', '--fast-fail', '-o', '_sfc64.c', '_sfc64.pyx']' returned non-zero exit status 1.
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-qyl8lkuw/numpy/setup.py", line 508, in <module>
        setup_package()
      File "/tmp/pip-build-qyl8lkuw/numpy/setup.py", line 488, in setup_package
        generate_cython()
      File "/tmp/pip-build-qyl8lkuw/numpy/setup.py", line 285, in generate_cython
        raise RuntimeError("Running cythonize failed!")
    RuntimeError: Running cythonize failed!
    
    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-qyl8lkuw/numpy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-bo0k6la5-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-qyl8lkuw/numpy/

解决方法升级一下pip就可以了，

python3 -m pip install --upgrade pip

问题：zed的numpy好像是1.19.5,会报错，安装1.19.4就可以了

Illegal instruction (core dumped)

python3 --m pip install numpy==1.19.4