Real-time Video Streaming And How It’s Done On iOS!
When you are watching a video on Youtube or enjoying a show on Netflix, you are using the video streaming technology. Since the beginning of the 20th century, the greatest minds have kept on developing new ways to deliver the media from a producer to an end-user. Both Youtube and Netflix are examples of the modern technology that requires uploaded video files in order to allow end-users to pause, stop or play the video at will. With this another technology co-exists one that we call “live-streaming”.
I heard that compression is important…
One of the core differences between these two types of media streaming is that during “live-streaming” everything happens in real-time. There are many examples out there: live broadcasts of events, concerts, and shows, Skype video calls, and, obviously, twitch.tv streams. Usually the video is being “streamed” from one device to another.
To ensure great viewer experience during live broadcasts, one thing must be reduced dramatically – latency. Latency is a delay that occurs between two users, sending and receiving. It is imperative that the end-user sees exactly what happens right instead of what happened a while ago. If the latency was too big during a Skype call, we would have had to wait for the data to reach your collocutor. The whole conversation would look more like a “video-mailing service”.
Live-streaming demands developers to be creative and use the latest technology in order to make viewer experience as rich as possible. It is important to reduce the delay, even if two users are separated by oceans. Generally, data flows through the Internet and that is why the size of data is important. Small files are delivered nearly instantly; big files travel a long time before they reach an end-user.
Media compression is very important. There is no need in streaming with the highest quality, modern compression methods allow us to compress the size of media multiple times and still get a high quality video. To put it simply, the better compression, the more viewers will be able to see the stream without delays and technical issues.
To compress a video, we need to use codecs. H.264 is a codec that is widely used to compress a video. There are software and hardware encoders, the latter is more convenient to use on handheld devices. Lesser CPU usage allows the device to work longer.
Modern iPhones and iPads support both video and audio coding on a hardware level. Up until recently, Apple has not allowed developers to use fully the potential of the hardware video encoder. The ways to implement live-streaming features in applications for iOS were limited. We, developers, had only two methods.
The first method is to use a software encoder from an external library; an encoder like ffmpeg, for example. This method has multiple downsides:
- Huge CPU load;
- Extensive power usage;
- High hardware requirements.
Here is a simple example: to compress a video in 360p with such a method, you need 100% of the CPU resources on iPhone4.
The second method is based on accessing a hardware encoder. One of the biggest downsides of this method was that developers had no direct access to the encoded frames. In order to use this method, we have to work with several small encoded pieces. The whole process requires splitting and saving files as well as reading them right after. Such a complicated compression takes up for a while before streaming!.. It is unacceptable for live-streaming.
Glory to Apple!
After the release of iOS8, Apple presented Video Toolbox Framework, a set of tools that allows developers to directly access hardware video encoder. This instrument can encode video in real-time. This library allows us to improve video compression while lowering energy consumption and prolonging the battery life.
This library is easy-to-use yet creates some obstacles for a developer. Firstly, there is not enough information about the instrument. Secondly, video compression requires additional coding. Overall, making a live-streaming feature is still a tricky task.
One of the most important tasks of our team was to simplify the whole process of creating media streaming features for iOS applications. We wrapped together a couple of software solutions that will prepare videostreams for broadcasting and we want to take Video Toolbox Framework to its limits!
Read more about Video Streaming here:
