A Quick Note on Error Recovery for Streaming Applications

There’re generally 3 ways to recover error caused by Internet transmission, including Retransmission, Redundant Data, and Error Concealment.

Retransmission

Retransmission is easy to understand. Just resend the packet that is lost or has error. It has the following pros and cons,

Pros:

  • Retransmission resends the entire packet, so the data error/lost is repaired accurately.
  • It has low overhead when there’s enough bandwidth, as it doesn’t computation.

Cons:

  • Retransmission increases the delay, as the sender needs to wait for some signal from receiver side about the packets needs to be retransmitted, or an timeout event has occurred. Also the retransmission itself take some time.
  • Retransmission takes bandwidth. If there’s congestion at the network, retransmission can make the congestion worse.

In video streaming application context, there’re other aspects that matters, and retransmission can be selective based on a few factors,

  • Important/Urgent packets are retransmitted first. For example, I frame packets are more important than B, P frame packets.
  • Packets are only retransmitted when there’s enough time. For example, if a packet has missed its play time, there’s no need to retransmit the packet.

Redundant Data

Redundant Data is well known as Forward Error Correction, a technique that sends additional information which can be used to recover lost/error packet at the receiver side.

Pros:

  • Bandwidth requirement doesn’t change when there’s loss, so it won’t make the congestion worse if there’s one.
  • Receiver doesn’t needs sender’s action to recover.
  • Delay is better if the computation is fast. This is also related to the dependency of the FEC, as the recovery may requires waiting of several packets to arrive first.

Const:

  • It has constant overhead in terms of network bandwidth. Even when there’s no lost/error, redundant error still has to be sent.
  • In terms of burst data loss, this method could fail.

To improve FEC against burst data loss, there’s two methods can be used,

  1. Arrange packets in multiple dimensions.
  2. Interleaving, rearrange the packet order when doing FEC.

Interleaving itself has some pros and cons

Pros:

  • It transfer burst data loss to random data loss, as the consecutive packets in the network transmission are not consecutive packets in FEC.
  • It doesn’t add any overhead to bandwidth or computation, only rearrangement is done.

Cons:

  • It could cause delay, as the recovery needs to wait for packets in a longer distance in network transmission.

Error Concealment

Error concealment refers to the technique of trying to coneal the error instead of recover it accurately.

Pros:

  • no overhead in terms of bandwidth, no matter there’s a congestion or not.
  • Delay can be small if the concealment computation is fast.

Cons:

  • It may not always give a good result
  • The computation can be complicated if complex concealment algorithms are used.

The concealment techniques include splice, noise substitution, repetition, interpolation, regeneration etc.

Dynamic Adaptive Streaming over HTTP

This is the third method of the http video delivery. Unlike the first two methods: HTTP Progressive Download and Play, and HTTP Pseudo-streaming, this is a real streaming technology and it’s applicable for both live video and video on demand (VOD).

The basic idea of Dynamic Adaptive Streaming over HTTP (DASH) is to divide the video clip/stream into small chunks (called streamlets). These small chunks will be downloaded to the browsers’ cache and combined by the client (browser or browser plug in) for play out.

This technology is new and implemented by several companies in different ways using different names.

  • HTTP Live Streaming (Apple)
  • Smooth Streaming (Microsoft)
  • HTTP Dynamic Streaming (Adobe)
  • Adaptive Bitrate (Octoshape)

All methods are based on the basic idea mentioned above, but unfortunately they’re not compatible. HTTP Dynamic Streaming is supported on Adobe Flash Platform with Adobe Flash Media Server (server side) and Adobe Flash Player 10.1 and above (client side). Smooth Streaming is supported by Microsoft IIS Server (server side) and Silverlight (client side). HTTP Live Streaming is supported by a list of servers (e.g. Wowza Media Server) and clients (e.g. QuickTime Player) can be found here.

The reason that there’re more servers and clients available for Apple’s HTTP Live Streaming is that Apple’s iPhone and iPad devices support HTTP Live Streaming play out and Apple has made its HTTP Live Streaming specification available as RFC here.

Here we illustrate the basic idea using HTTP Dynamic Streaming from Adobe as an example. One can find demo videos here, or here.

Once I started to view the video, I checked my browser (I’m using Chrome on Ubuntu) cache at ~/.cache/google-chrome/Default/Cache/, I see a list of files (using ls –l command). When the video is being played, I kept refreshing the list (by typing ls –l repeatedly). I found there’s a new file created at about every 10 seconds.

Note that if you’re Windows 7, the browser cache should be at

C:Usersroman10AppDataLocalGoogleChromeUser DataDefaultCache
Where roman10 is my username.

Then I paused the video, n new files are created any more. If I resume the play back, new files started to appear again. These files are actually small video chunks with meta data. And the Flash player is combining them dynamically to a video stream as the video is playing out. I tried to play a single chunk using VLC player, it won’t work.

For advantages and disadvantages of DASH in comparison with other two HTTP methods, please refer http video delivery.

Update: Scheduling Logic is in Client (Player)

One important fact about DASH is the scheduling logic is implemented at the client/player side as HTTP is stateless and the server doesn’t keep track of a video session. Therefore, the player needs to measure the network condition and dynamically switch between video chunks of different size and quality.

HTTP Video Delivery — HTTP Pseudo Streaming

HTTP Pseudo Streaming is the second method in HTTP Video Delivery. The method is also based on HTTP progressive download as the first method does, and it makes use of the partial download functionality of HTTP to allow user to seek the video to part that has not been downloaded yet.

HTTP Pseudo Streaming requires support from both the client side and server side. For the server side, plug-ins are available for Apache, litghttpd etc. At client side, custom players are required to resynchronize the video, read metadata etc. Two examples of players that supports HTTP Pseudo Streaming ar eJWPlayer and FlowPlayer.

For an example of HTTP Pseudo Streaming, open up any Youtube video. Youtube actually uses lighttpd for server side and its own customized player based on Flash technology for client side (Flash player and Flash based media player are different. Please refer here. )

Below is a screenshot of the Wireshark network traffic capture when I was watching a Youtube video,

http_pseudo_streaming

Figure 1. Wireshark Capture of HTTP Request for HTTP Pseudo Streaming

As there’re lots of segmented packets, I selected a video packet, right clicked it and selected follow TCP stream to get the screen above. It’s a single HTTP request followed by a response from the Youtube HTTP server.

I tried to seek to a part that is not downloaded yet, then do “follow TCP stream” again. I found the HTTP header contains the following strings for the initial request and the seek,

=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor%2Coc
%3AU0hQSllNVF9FSkNOMF9RR1hB&
fexp=911106%2C910206&algorithm=throttle-factor&itag=34&ip=0.0.0.0&
burst=40&sver=3&signature=B3B26708552F1C9FE68 7AAB13EFE6D73F294624F.0F9EEB822A5CF4AE5443CC5798B2F415C16B75E4&
expire=1315926000&key=yt1&ipbits=0&factor=1.25&id=d46a2f20e9b7b525&
redirect_counter=1 HTTP/1.1

=id%2Cexpire%2Cip%2Cipbits%2Citag%2Calgorithm%2Cburst%2Cfactor%2Coc
%3AU0hQSllNVF9FSkNOMF9RR1hB&fexp=911106%2C910206&algorithm=throttle-
factor&itag=34&ip=0.0.0.0&burst=40&sver=3&signature=B3B26708552F1C9FE68
7AAB13EFE6D73F294624F.0F9EEB822A5CF4AE5443CC5798B2F415C16B75E4&
expire=1315926000&key=yt1&ipbits=0&factor=1.25&id=d46a2f20e9b7b525&begin
=1032070
HTTP/1.1

The seek HTTP request contains a string “begin=1032070”, which should be used at the HTTP server to jump to the corresponding portion of the video.

Same as the first method, HTTP Pseudo Streaming download the video clip to browser cache. For Google Chrome, one can find the video clip at,

Ubuntu Linux: /home/roman10/.cache/google-chrome/Default/Cache/

Windows: C:Usersroman10AppDataLocalGoogleChromeUser DataDefaultCache

Where roman10 is my username for both OS.

For the benefits and drawbacks of this method, and its comparison with other HTTP method, please refer to HTTP Video Delivery. Note that as HTTP psuedo streaming is not real streaming, it doesn’t support live video streaming.

HTTP Video Delivery–HTTP Progressive Download and Play

Progressive Download and Play is the first of the three methods in HTTP Video Delivery. The basic idea of this method is to embed the video through media player (e.g. JWPlayer, FlowPlayer etc.). When user request to play the video, a HTTP Get request will send to the web server, and the video will be downloaded through HTTP for play out.

This method is supported by web servers by default, as it treats a video as a normal file like an image, or a CSS file. The play out, buffering and other video play specifics are handled by the media player which usually consists of some javascript and flash objects. More info about these players be found here.

Below is an example of HTTP Progressive Download and Play Out video, it’s delivered using JWPlayer as client.

 

Once you click the button to start play, the video download is triggered. Below is a screenshot of the network traffic captured using Wireshark.

network_traffic

Figure 1. Wireshark Capture Screenshot

The video is actually downloaded to browser cache, as I used Google Chrome at Ubuntu Linux to watch the video, the file is saved to

/home/roman10/.cache/google-chrome/Default/Cache/ folder. Below is a screenshot of the files under the directory,

video_at_cache_2

Figure 2. List of Files under Google Chrome Cache

I can either locate the video clip by time or by file size, and play out the video using vlc as shown below,

video_at_cache_1

Figure 3. VLC Play Out of the Cached Video

If you’re using Windows, the video file can be found under,

C:Usersroman10AppDataLocalGoogleChromeUser DataDefaultCache, where roman10 is the username.

For the benefits and drawbacks of this method, and its comparison with other HTTP video delivery method, you can refer here.

You may also want to check out,

HTTP Pseudo Streaming

Dynamic Adaptive Streaming over HTTP

JWPlayer and FlowPlayer vs. Adobe Flash Player

JWPlayer and FlowPlayer are video players that support Flash video. What confusing is they’re referred as Flash Video Players sometimes. (e.g. FlowPlayer claims itself as Flash Video Player for the Web.) Why do I need another Flash Video Player if I already have Adobe Flash Player installed?

Adobe Flash Player is actually a virtual machine that runs Flash files, which end with an extension of .swf. SWF files can contain animations, video/audio and web UI. Both JWPlayer and FlowPlayer consist SWF files which is downloaded to browsers’ cache and played by Adobe Flash Player. In other words, JWPlayer and FlowPlayer are “played” by Adobe Flash Player. It’s like Adobe Flash is the JVM (Java Virtual Machine), and JWPlayer and FlowPlayer are two Java programs.

JWPlayer actually supports more than just Flash, it also supports HTML5 video for iPhone and iPad device. In essence, JWPlayer and FlowPlayer are just a collection of javascript and SWF files that allow a website publisher to embed video into a web page, customize the outlook of the video display area, and control the behavior of the video play out etc. And Flash video is one (and probably most important one) of the technologies/platforms they support.

Video Delivery in HTTP

As Internet becomes faster and faster, videos become important content of many websites. There’re many methods to publish video on Internet. This blog focuses on technologies around HTTP.

There’re 3 ways to deliver video content in HTTP, progressive download and play, HTTP Pseudostreaming, and Dynamic Adaptive Streaming over HTTP.

0. Why HTTP?

When people choose their video delivery method, HTTP offers a few advantages,

  • HTTP doesn’t require a dedicated media server like RTSP/RTP/RTCP and RTMP/RTMPT/RTMPS streaming technologies require. Web Servers (Some of HTTP method requires Plug-in/server-side scripting to support)
  • HTTP network packets can reach almost any devices that connected to internet, while many other protocols (e.g. RTMP on port 1935) might be blocked by firewalls.
  • HTTP is supported by existing server and caching infrastructure.
  • HTTP is stateless, it doesn’t require a consistent one-to-one connection between client and server when video is deliveried and played. So the number of clients a server can serve is higher than connection-oriented method. In other words, HTTP scales better.

1. Progressive Download and Play

This is the most basic method, supported by almost all web servers. (Apache, lighthttpd, IIS, etc.) The basic principle is to download the entire video file and play the file. If the video file is long, the client can download and play the downloaded part at the same time.

The main advantages of this method are,

  • Easy to configure (from publisher’s POV)
  • Video play can be stopped and buffered (from user’s POV)

The main disadvantages of this method are,

  • Entire video is downloaded to the client, unless user close the browser page. It wastes a lot of bandwidth sometimes. Suppose a user watches the first 4 minutes of an one-hour long video, but the entire video is downloaded if the network is fast enough to download in 5 minutes or the user keeps the page open after watching. (From the publisher’s POV and user’s POV when user pay by usage)
  • Entire video is downloaded to client, the content is not protected. (From publisher’s POV)
  • Video is only seekable for the downloaded part. In other words, if a user want to watch 40~45 minutes in a one-hour long video, he/she needs to wait the video to download the first 40 minutes. (From the user’s POV)

If you watch a unseekable video on a website, and the video can be found on your browser’s cache, most likely the website is using HTTP Progressive download and play.

2. HTTP Pseudostreaming (Pseudo-streaming)

HTTP Pseudo-streaming is, as its name suggests, not real streaming. It is based on progressive download method, but it mimic Video on Demand (VOD) streaming.

It’s not supported on web servers by default, but extensions and plug-ins can be added to Apache, Tomcat, IIS, or lighttpd to support pseudo-streaming.

Besides the advantages metioned for HTTP progressive download and play method, HTTP pseduo-streaming adds the seekable feature to the video. Also as the video is seekable, the skipped part is not downloaded, so it reduces the bandwidth waste at some cases.

If you watch a seekable video delivered through HTTP on a website, and the video can be found on your browser’s cache, then the website is using HTTP Pseudo-streaming to deliver the content to you. As an example, Youtube actually uses this technique with lighthttpd web servers.

3. Dynamic Adaptive Streaming over HTTP (DASH)

This is the latest technology. The basic idea is to divide a video into many small parts and deliver them over HTTP. Those small parts are then combined at the client side and played out. This method supports both video on demand and live streaming.

There are several different implementations of this method with different names, all based on the basic idea mentioned above,

  • HTTP Live Streaming (Apple)
  • Smooth Streaming (Microsoft)
  • HTTP Dynamic Streaming (Adobe)
  • Adaptive Bitrate (Octoshape)

The advantages of DASH are,

  • not dependent on specifized streaming servers or proprietary transfer protocols
  • Better protection for streaming content as the video clip/stream are divided into small chunks

The disadvantages of DASH are,

  • Requires client support. e.g. Adobe Flash Player only supports HTTP Dynamic Streaming after 10.1)
  • Different implementations are not compatible completely (There’re standardization work on-going and the comptability is on the way…)
  • The Live Streaming has longer delay compared with traditional Live streaming technologies (e.g. RTP, RTMP)
  • The Live Streaming quality adaption is not as fast as traditional Live streaming technologies as the quality switch can usually only occurs at streamlet boundaries. If the streamlet is 5 seconds, then the quality adaption usually occurs every 5 seconds.

For examples, some video streaming to iPhone Safari are actually using HTTP Live Streaming.

How to Stress Test RTMP Server using Flazr

Flazr is a collection of multimedia streaming tools implemented in Java, including a RTMP server, a RTMP client and some other utilities. It runs on both Linux and Windows.

One of the things that Flazr can do is to stress test your RTMP server. In a testing environment, your bandwidth (hundreds of clients receiving video streams simultaneously requires high bandwidth) or computing power (hundreds of clients playing/decoding video requires a lot of CPU power) are normally limited so that you won’t have enough resources to stress test your RTMP server exactly as it’s used in production.

Flazr can create lots of connections with a third-party RTMP streaming server and constantly receive video data without playing it, so it doesn’t have the CPU power issue. If you place the Flazr client at the same machine as your RTMP server, you won’t have the bandwidth issue also.

To use Flazr for RTMP stress testing is quite simple. Suppose your RTMP url is rtmp://173.23.24.25:80/live/test.stream, you can simply go Flazr directory and type the following command,

./client.sh -version 00000000 -load 20 -threads 15 -host 173.23.24.25 -port 80     -app live test.stream

This command will create 20 connections (-load 20) to receive data from the RTMP server.

If you want more options, you can just type ./client.sh to view a list of supported configurations.

References:

Flazr Official Website: http://flazr.com/

Youtube Video Tutorial: http://www.youtube.com/watch?v=Wa2bDQLKf8s