SS之发展史和Technical Overview

 IIS Smooth Streaming Technical Overview

 http://learn.iis.net/page.aspx/626/smooth-streaming-technical-overview/

主要讲了SS的发展史。这篇是文档的记录笔记，很多东西总是看的时候很清楚，过后一段时间不接触就能忘的一干二净。

Smooth Streaming: HTTP-based adaptive streaming extension of IIS 7.0.

1. Multiple Bit Rate Streaming

    Microsoft 最初与2000年引入MBRT这个概念，使用用ASF文件格式，allows storage of multiple video and audio tracks inside a single file, and the WIndows Media streaming protocols support swtiching streams during a broadcast.

2. Intelligent Streaming

    改进了MBR技术，combined bandwidth detection, stream thinning等，但是仍然需要media encode成MBR ASP 文件。

3. From streaming protocols（RTSP, MMS, RTMP） back to plain HTTP download

4. Media delivery methods:

1) Traditional streaming

    RTSP(Real-Time Streaming Protocol) 就是传统流协议的好例子，a stateful protocol. Client 连到streaming server以后，client通过发送像PLAY, PAUSE, TEARDOWN这些命令给server，这样server一直知道client的状态直到断开。client和server之间建立session以后，the server begins sending the media as a steady stream of small packets(The format of these packets is known as RTP).一般RTP包的大小是1452bytes。如果视频流的bit rate是1Mbps的话，每个packet负载 (1452*8) / (1024&1024) = 11 (ms)的视频信息。

    而HTTP是一个stateless的协议，Windows Media Services 使用修改过的HTTP，MS-WMSP（Windows Media HTTP Streaming Protocol）来做streaming。MS-WMSP 使用标准HTTP来传输数据和消息，同时维护session状态。Silverlight只支持WMS HTTP-based delivery.

    向RTSP和MS-WMSP这样的传统流协议有2个重要特点。一是，The server sends the data packets to the client as a real-time rate only. Meida endcode成多少比特率，streaming的时候就是多少。二是，The server only sends ahead enough data packets to fill the client buffer. Client buffer 一般是1-10秒（WMP和Silverlight都是5秒），server最多只会多穿1-10秒的数据过来，就算暂停了半个小时，也就缓冲了1-10秒。

2) Progressive download

    Progressive：Most player clients allow the media file to be played back while the downloads is still in progress. 连接建立以后，HTTP web server向client发送数据，其实就是把整个媒体文件都下载到web browser的cache里头。如果用户暂停半个小时，可能视频已经下载完了就可以直接观看了。现在的视频网站比如YouTube，MySpace都是使用这种技术。但是如果用户又看了1分钟觉得不好看，那就白白多下载了那么多。IIS7 提供了一个叫做Bit Rate Throttling 的extension来缓解这种浪费。

3) HTTP based Adaptive streaming

    看起来像streaming但其实是HTTP progressive download。In a typical adaptive streaming implementation ,the video/audio source is cut into many short segments(chunks) and encoded. Chunks are typically 2-4 seconds long. At the video codec level, this typically means that each chunk is cut along video GOP (Group of Pictures) boundaries (each chunk starts with a key frame) and has no dependencies on past or future chunks/GOPs. 这样每一个chunk都是独立的，client收到以后就有全部的信息进行解码。

    The encoded chunks are hosted on a HTTP Web server. A client requests the chunks from the web server in a linear fashion and downloads them using plain HTTP progressive download. As the chunks are downloaded to the client, the client plays back the sequence of chunks in linear order. Adaptive 的部分体现在，如果video/audio被编码成很多不同的比特率，每种比特率的流都被分成很多2-4秒的chunk，那么client就可以根据网络带宽，CPU处理速度等情况，请求不同比特率的chunk。

    NBC 2008 年北京奥运会网站就使用了这种技术并且获得了成功。整个开发过程分为3大步：

    a）Use encoders to produce multiple WMV files of different bit rates/resolutions for each source.

    b) Split each WMV files into thousands of 2s chunks(actually small files)

    c) Upload the chunks to the CDN'S web server and then build a Silverlight player that would download the chunks and play them in sequence.

    这种技术的缺点在于，每一个chunk最后都成一个小文件，可以想象最后要维护多少个文件！

5. IIS Smooth Streaming

    跟上面的adaptive streaming差不多，但是优化了其缺点，重新设计了content creation 和delivery部分，one-file-per-chunk 演变成a single contiguous file for each encoded bit rate. IIS SS uses the MPEG-4 Part 14 (ISO/IEC 14496-12) file format as its disk(storage) and wire(transport) format. SS spec defines each chunk/GOP as an MPEG-4 Movie Fragment and stores it within a contiguous MP4 file for easy random access. 这样每一个bit rate 流对应一个MP4文件，用户请求特定的source time segment时，server动态的在MP4文件中找合适的Movie Fragment box （chunk是按需生成的），单独发送出去。而怎么根据用户的网络等情况请求不同bit rate的chunk，这个重任就落到了client player的身上，silverlight会帮我们搞定。server是不管的，只是按url参数办事。

    MP4的几个优点：支持H.264 codec；支持payload fragmentation within the file。MP4 file specification: ISO/IEC 14492-12 ISO Base Media File Format specification.

    SS 有两种格式：Live （wire formate）和VOD (disk file format)。

    MP4文件的基本单元是box，里面可以有data和metadata。一般情况下，metadata会写在data前面，这样client在播放的时候就能先获得关于data的信息，但是不适用于live streaming的情况。MP4 file spec allow MP4 boxes to be organized in a fragmented manner, where the file can be written "as you go" as a series of short metadata/data box pairs, rather than one long metadata/data pair. 所以Smooth Streaming files 也被称为Fragmented MP4 files or fMp4.

    High-level overview of SS file:

    The file starts with file-level metadata ('moov') that generically describes the file, but the bulk of the payload is actually contained in the fragment boxes that also carry more accurate fragment-level metadata ('moof') and media data ('mdat'). A typical SS file has a fragment for each 2 seconds of video/audio. Closing the file is an 'mfra' index box that allows easy and accurate seeking within the file.

    MP4 fragment:

6. SS Media Assets

    上一篇SS之创建SS Web Server中下载了Big Buck Bunny的SS资源，解压后的文件列表：

1) *.ismv

如果MP4文件既有视频又有音频，或者只有视频的话，每个比特率的文件被编码成一个ismv文件。

2) *.isma

如果MP4文件只有音频的话，每个比特率的文件被编码成一个isma文件。

3) *.ism

Server manifest file （XML文件）. 只被IIS SS server使用，describes the relationships between the media tracks, bit rates and files on disk. 这个manifest 格式基于SMIL 2.0 XML format specification.

4) *.ismc

Client manifest file （XML文件）. 这是server传给client的第一个文件，描述了server可以提供的流信息，包括codec, bit rates encoded, video resolutions, markers, captions, etc.

    Support VC-1, WMA, H.264 and AAC codecs.

7. SS playback

1) The server must translate URL requests into exact byte range offsets within the MP4 file.

2) The client can request chunks in a more developer-friendly manner.

    client 向 server请求的第一个东西就是ismc manifest文件，client就可以知道视频是用什么样的codec来编码的，有哪些bit rate可选，可选的chunks列表。接着client向server请求SS chunks：http://video.foo.com/NBA.ism/QualityLevels(400000)/Fragments(video=610275114)

这个URL包含的意思是，请给我bit rate是400000，fragment偏移为610275114的chunk。The fragment start offset is expressed in an agreed-upon time unit (usually 100 ns).

    收到这个URL以后，server在 *.ism server manifest 文件里找QualityLevel对应哪个*.ismv/*.isma文件，然后读取这个*.ismv/*.isma文件，根据 tfra index box，找到URL请求的fragment box （‘moof’ + 'mdat'）。之后把这个fragment box 拿出来，以一个单独文件的形式发送给client。The sent fragment/file can now be automatically cached further down the network, 这样另外一个client请求同样的fragment时，可以直接从cache发过去。