h264 html5,ffmpeg - HTML5 H264 video sometimes not displaying - Stack Overflow

Given this stream from an RTSP camera which produce H264 stream:

Input #0, rtsp, from 'rtsp://admin:admin@192.168.0.15:554':

Metadata:

title : LIVE555 Streaming Media v2017.10.28

comment : LIVE555 Streaming Media v2017.10.28

Duration: N/A, start: 0.881956, bitrate: N/A

Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1600x900, 25 fps, 25 tbr, 90k tbn, 50 tbc

I want to run ffmpeg and pipe its output to a HTML5 video component with MSE.

Everything is fine and smooth as long I run this ffmpeg command (piping is removed!):

$ ffmpeg -i 'rtsp://admin:admin@192.168.0.15:554' -c:v copy -an -movflags frag_keyframe+empty_moov -f mp4

However it takes a bit time at the beginning.

I realized that the function avformat_find_stream_info makes about 15-20 seconds of delay on my system. Here is the docs.

Now I have also realized that if I add -probesize 32, avformat_find_stream_info will return almost immediately, but it cause some warnings:

$ ffmpeg -probesize 32 -i 'rtsp://admin:admin@192.168.0.15:554' -c:v copy -an -movflags frag_keyframe+empty_moov -f mp4

[rtsp @ 0x1b2b300] Stream #0: not enough frames to estimate rate; consider increasing probesize

[rtsp @ 0x1b2b300] decoding for stream 0 failed

Input #0, rtsp, from 'rtsp://admin:admin@192.168.0.15:554':

Metadata:

title : LIVE555 Streaming Media v2017.10.28

comment : LIVE555 Streaming Media v2017.10.28

Duration: N/A, start: 0.000000, bitrate: N/A

Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1600x900, 25 tbr, 90k tbn, 50 tbc

If I dump out this stream (into a file, test.mp4), all mediaplayers can play it perfectly.

However if I pipe this output into the HTML5 video with MSE, the stream sometimes displayed correctly, sometimes it just doesn't. No warnings or error messages are printed on the console in the browser.

From the second output I can see the fps is missing. I tried to set it up manually, but was not succeed (it seemed I could not change it manually).

How can I avoid avformat_find_stream_info and have the HTML5 MSE playback if I know everything of the stream beforehand?

Update

According to @szatmary's comments and answers I have search for a h264 bitstream parser.

This is what I found. I did also save the mp4 file which is not playable by HTML5 video, but by VLC it does, and I dropped into this analyser.

Here is a screenshot of my analysis:

Some facts here:

until #66 there is no type7 (SPS) unit in the stream.

62 is the last PPS before the first SPS arrived.

there are a lot of PPS even before 62.

bitstream ends at #103.

playing in VLC the stream is 20 seconds long.

I have several things to clear:

the #62 and #66 sps/pps units (or whatever) are holding metadata only for the next coming frames, or they can even refer to previous frames?

VLC plays 20 seconds, is it possible that it scans the whole file before, then play the frames from #1 based on #62 and #66 infos? - if VLC would get the file as stream, in this case it might play only a few seconds (#66 - #103).

most important: what shall I do with the bitstream parser to make HTML5 video playing this data? Shall I drop all the units before #62? Or before #66?

Now I'm really lost in this topic. I have created a video, with FFMPEG but this time I allowed it to finish its avformat_find_stream_info function.

Saved the video with the same methods as previously. VLC now plays 18 seconds (this is okay, I have a 1000 frame limitation in ffmpeg command).

However let's see now the bitstream information:

Now PPS and SPS are 130 and 133 respectively. This resulted a stream which is 2 sec shorter than before. (I guess)

Now I have learned that in a correct parsed h264 there can still be a lot of units before the first SPS(/PPS).

SO I would finetune my question above: what shall I do with the bitstream parser to make HTML5 video playing this data?

Also the bitstream parser I have found is not good, because it uses a binary wrapper => it can not be run purely on the client side.

I'm looking at mp4box now.