Speed Thrills - Tackling the YouTube video processing challenge

Few things can be more frustrating than waiting for what feels like years for that amazing video you just shot to finish uploading to the site so you can publish it and share it with your friends. Whether it’s the speed a video plays back, or the speed a video uploads, at YouTube speed matters. Why do we care so much about speed? Because we know that speed can mean the difference between you having a good experience, or you leaving YouTube in search of a better, faster one.

Today, as we look at 2011, we’d like to review our progress in improving the processing and publishing speed for videos. By processing and publishing, we mean what happens behind the scenes between the moment you press the YouTube upload button to the moment that video appears on the site for your friends to see.

Here is the challenge: 35 hours of video are uploaded to YouTube every minute. Over time, the average file size is going up, the average resolution is going up, as people switch to HD cameras and the average duration of the videos is going up. Moreover, an increase in resolution results in a quadratic increase in the number of pixels to be processed - a 1080p HD video has 2M pixels compared to 300k pixels in a 480p (DVD quality) video.

In the last few years the YouTube infrastructure team has tackled this challenge in a variety of different ways. We highlight two of them:

• The first is to overlap uploads and video processing without waiting for the upload to finish. This results in a base quality version of your video going live very quickly after the upload completes, making the link to your video active and shareable.
• The second technique, which we’ve internally codenamed Hydra (after Greek mythology’s nine-headed monster) tackles this problem by leveraging Google’s massive cloud computing capabilities. We split a single video into small chunks and process each chunk simultaneously on different machines. Hydra then reassembles all the processed chunks so that you see a seamless video, processed and published in a fraction of the time it would have taken to do it previously.

As you can see from the above chart, we’re making big strides when it comes to our battle to get videos processed and live faster. Here are some additional stats:

We know that there remains a lot for us to do when it comes to improving processing time and reliability and we certainly don’t think we have come close to solving the problem entirely. We have made big strides over the years and our speed is accelerating. When we make another big leap, we’ll post another blog to let you know. Until then, keep uploading!

• We now process videos seven times faster than we did in 2008.
• We’ve gotten four times faster in the last six months
• 60% of all YouTube videos go live in under a minute as compared to none last year.