E x e c u t i v e S u m m a r y The need to reduce the bandwidth of online video, while maintaining the quality, is being driven by two factors: ? Internet video bandwidth is forecast to grow 4 times its current size in the next 4 years. Ultra HD video is liable to be a major reason why video bandwidth will grow so much because it takes 5 times as much bandwidth as regular HD. ? Mobile video bandwidth is forecast to increase tenfold within the next 5 years. This growth is stretching the ability of mobile networks to deliver a satisfying experience. There are two main approaches to compressing video: ? Traditional Compression ? attacking the general abstract problem of video data reduction ? Environmental Pre-Process ? optimizing the experience of a video for a specific set of viewers, devices, and viewing locales The traditional approach to compression focuses on removing redundant information, such as parts of the image that don?t change between video frames. Advances in the technology often require all the components in the video chain of delivery to change. For example, in switching from h.264 (AVC) to h.265 (HEVC), most mobile devices will need to change to support the newer standard. The environmental pre-process approach seeks to remove details in the image that cannot be perceived by a viewer in a specific viewing situation. For example, seated at normal viewing distances from a big screen television some of the detail in an Ultra HD image cannot be perceived by the average viewer with 20/20 vision, and even by the lucky 1% with 20/10 vision. Environmental approaches work with traditional compression technology and do not require substantial changes to the video chain of delivery. When a new standard in traditional compression is adopted it can often achieve substantial bandwidth savings over its predecessor. H.264 halved the amount of bandwidth required to deliver video at the same quality as MPEG2. Many expect HEVC to achieve similar savings over h.264. Environmental pre-process approaches can considerably improve the ability of any traditional compression technique. For example, if a video is destined for a big screen television with viewers seated at normal viewing distances from the screen, pre-processing can provide an additional 30% reduction in video bandwidth over traditional compression alone with no degradation of the viewing experience. Working together, traditional and environmental approaches to video compression can achieve extraordinary bandwidth savings. This will broaden the reach of Ultra HD on broadband connections, and improve the reliability and quality of mobile video experiences. OPTIMIZ ING VIDEO DEL IVERY: WHY THE VIEWING ENVIRONMENT MAT TERS Author: Colin Dixon, Founder and Chief Analyst, nScreenMedia | Date: 2015Q3 This paper is made possible by the sponsorship of www.InterDigital.com Optimizing Video Delivery: Why the Viewing Environment Matters Page 1 I n t r o d u c t i o n With major online video providers such as Netflix, Amazon and Comcast, all introducing Ultra HD streaming, the race is on to capture the emerging audience for high quality video online. Unfortunately, that audience is much smaller than providers would like, since many broadband connections simply cannot provide enough bandwidth to deliver an Ultra HD movie. According to Akamai, just 10% of broadband connections can achieve a speed of 15 mbps, the bandwidth the company considers a minimum to receive Ultra HD video. 1 Of the 711 million homes worldwide that have fixed broadband connects, just 85 million could actually stream an Ultra HD show or movie. 2 If video providers could shave an additional 10% from the bandwidth required to deliver UHD content, they could reach an additional 26 million customers. The opportunity for bandwidth savings in mobile video delivery is just as compelling. There are three times as many mobile broadband connections worldwide as there are fixed connections. However, not only will shrinking the bandwidth increase the number of people who can watch mobile video, the overall experience will also be improved. The variability in mobile bandwidth, as users move around within and between cell towers, can be extreme. One minute a user can have 5 mbps of bandwidth available and the next one tenth of that. Shrinking the bandwidth required to stream video by 10% will reduce buffering and stream failures, thus improving the average experience level enjoyed by all viewers. Reducing the amount of bandwidth necessary to deliver video over mobile and fixed broadband networks is becoming an imperative for the entire industry. According to Cisco, worldwide Internet traffic will grow from 42.4 Exabytes i per month in 2014 to 136.1 Exabytes per month in 2019 3. Internet video will grow from 25 Exabytes per month to 104.9 Exabytes per month over the same period. In other words, video will be over three-quarters of Internet bandwidth by 2019! i An Exabyte is one million Terabytes, or 1,000,000,000,000,000,000 bytes 0 10 20 30 40 2014 2019 Total Internet Traffic & Video Internet Traffic: 2014 and 2019 Internet Video Traffic Total Internet Traffic Ex ab yt es /M on th Source: Cisco VNI, 2015 A 10% reduction in the bandwidth required to deliver an Ultra HD movie would increase the number of people able to stream it by 26 million. By 2019, worldwide Internet video traffic will quadruple to 105 Exabytes per month. ~ Cisco VNI, 2015 Optimizing Video Delivery: Why the Viewing Environment Matters Page 2 Realizing bandwidth savings in the delivery of Ultra HD is critical for online video providers. The aggregate bandwidth needed to deliver an Ultra HD movie is also much higher than for an HD version. A two hour Ultra HD movie consumes 5 times more bandwidth to deliver than the same movie at HD 720p resolution. ii For a company delivering 1 million Ultra HD movie streams per month, a likely scenario by 2020, a 10% bandwidth decrease would save nearly $600,000 per year in delivery costs. The growth in mobile bandwidth is also forecast to be driven by video consumption in the coming months and years. According to Ericsson, worldwide mobile data consumption will grow ten-fold between 2014 and 2020, to 25 Exabytes per month. 4 Video will occupy 55% of mobile data in 2020. ii Assumes Ultra HD requires 15 mbps stream bandwidth and 720p HD requires 3 mbps For a company delivering 1 million UHD movies a month a 10% reduction in streaming bandwidth will save $600,000 a year in delivery costs. Optimizing Video Delivery: Why the Viewing Environment Matters Page 3 O p p o r t u n i t i e s t o i m p r o v e t h e p r o c e s s The battle to reduce the bandwidth needed to deliver video means compression is an area of almost constant innovation. Over the last 20 years there has been a stream of companies contributing to the evolution of the technology. And the techniques each company has used have become ever more sophisticated, employing increasingly esoteric mathematical algorithms. However, all of the work to date can be broadly categorized into two main approaches: ? Traditional Compression ? attacking the general abstract problem of video data reduction ? Environmental Pre-Process ? optimizing the experience of a video for a specific set of viewers, devices and viewing locales The two approaches are far from mutually exclusive. Working together, the two can achieve extraordinary bandwidth savings over one approach alone. To understand why, let?s look at how each works. The Traditional Compression Approach Fundamentally, the objective of traditional approaches to video bandwidth reduction is to remove redundant information from video data, and to compress the result to a target bitrate iii for delivery. What does removing redundant video information mean? When it is removed, a viewer is unaware of any changes to the video image. And there is normally plenty of redundant information to be removed! For example, in a newscast often only the head and mouth of the newscaster are moving for extended periods of time; everything else in the video frame is unchanging. Today?s codecs iv are able isolate the parts of the video that change from those that don?t. Rather than storing or streaming the full video image for every video frame, the traditional approach stores the first frame and then just stores the parts that change. For a newscast, and many other types of video, this can result in dramatic reductions in bandwidth. Codecs use mathematical algorithms to remove the unchanging, and therefore redundant, data. Over the years improvements in these algorithms have resulted in big decreases in the bandwidth required to deliver video. For example, h.264 has now widely replaced its predecessor MPEG2 because in many circumstances it delivered the same or better quality using half the bandwidth. 5 Many expect HEVC v, one of the most sophisticated codecs to date, to achieve similar savings over h.264. iii Bitrate is the amount of data delivered per second. iv Codecs are devices or software programs that compress video data, so that it can be delivered faster and stored in less space, and decompress it for viewing. Codec is short for enCOde/DECode. v High Efficiency Video Codec The two main approaches to video compression, traditional and environmental, work together to achieve more bandwidth savings than either does alone. Traditional compression removes redundant information from video data and compresses the result using specific bitrate profiles for optimal delivery on various devices. Optimizing Video Delivery: Why the Viewing Environment Matters Page 4 Once the codec has removed the redundant information in the video it can be compressed further into a number of bitrate profiles for delivery. These profiles are necessary to optimize delivery to the many screens consumers use to view the video. For example, there is no point in sending a full 1080p video image to a tiny smartphone screen since the viewer won?t be able to see the fine detail in the image. In many cases a 480p (standard definition television) image will look the same on a 7? screen as 1080p. Once again, the bandwidth savings can be huge if the correct bitrate profile is selected. Using h.264, one of the most popular codecs today, a 1080p video might require an 8?10 mbps bitrate to look good on a 65" television screen. The same video will look great on a smartphone with a bitrate of 1?2 mbps. However, one of the disadvantages of traditional approaches is that new codecs are frequently incompatible with all existing equipment. That means devices like smartphones, tablets and smart TVs must be replaced with devices that support the new codec. For this reason, it can take many years for the market to adapt to new codecs. Advances in the traditional approach to video compression often require all existing equipment to be replaced or upgraded. Optimizing Video Delivery: Why the Viewing Environment Matters Page 5 The Environmental Pre-Process Approach In some ways, the objective of the environmental pre-process approach is quite similar to the traditional one. Both seek to remove video information that the viewer cannot perceive. However, the traditional approach does not take into account the specific viewing conditions of an individual viewer. For example, when viewing a movie on a 50" television from normal distances, the average person can?t tell the difference between a 1080p HD and Ultra HD video. Yet the Ultra HD image contains 4 times as much data as the 1080p. Environmental approaches seek to remove the data that cannot be perceived by the viewer in the specific viewing situation they are in. Environmental approaches do not replace traditional codecs, they work in tandem with them. A video can be pre-processed to remove the detail that will not be perceived by the viewer. The pre-processed video is then handed to the encoder vi, which then compresses it in the normal way and formats it for delivery. Thus, environmental approaches will yield improvements regardless of the specific encoding technology used. Figure 1. How traditional and environmental compression techniques work together vi An encoder is a device (or piece of software) which compresses the video and formats it ready for delivery to consumer devices. A decoder is a device (or piece of software) that reverses what the encoder did to the video so that it can be played on a specific device. Environmental approaches to compression remove details that cannot be perceived by viewers in specific viewing situations Optimizing Video Delivery: Why the Viewing Environment Matters Page 6 H o w t h e E n v i r o n m e n t a l A p p r o a c h W o r k s Many environmental approaches take advantage of the abilities, and limitations, of human vision to figure out what can and cannot be perceived by the normal viewer. Eliminating what cannot be perceived can result in big reductions in the amount of video data needed to represent an image. The most obvious feature of vision to take advantage of is visual acuity, or sharpness of vision. That is what an optician is measuring when he asks a patient to read a row of letters from a Snellen eye test chart. Patients with 20/20 vision can read a row on the Snellen chart from 20 feet away that the average person can read at 20 feet. Someone with 20/10 vision can read a row from the chart at 20 feet that an average person would need to stand at 10 feet to read. However, only 1% of the population has 20/10 vision. 6 Clearly, there is little point in leaving more detail in a video image than someone with 20/10 vision can perceive at a normal viewing distance from a screen. Removing the imperceptible data can shrink the amount of video image data required. However, the way the eye perceives images is more complex than just acuity, and this too can be used to eliminate imperceptible data. The Snellen test chart used at the optician?s is an ideal test for the eye because it uses high contrast black letters on white background. This gives the eye the best chance to see the difference between the edge of the letter and the white paper. The eye?s acuity is very dependent on contrast to make out fine detail. Simply put, the less contrast there is between the edge of an object and its background, the tougher it is for the eye to see details. Another factor affecting the ability of the eye to perceive detail is an object?s brightness (or luminance). If an object is very bright or very dim it is much harder for the eye to see fine detail in it. Also, very bright objects make it hard to see details in objects close by. There are other factors which affect the eye?s ability to perceive detail, including the size of an object, background brightness, and peripheral vision effects. All of these factors can be modelled in so-called ?contrast sensitivity functions? (CSF). This mathematical model can predict what fine detail in an image can be perceived, and what cannot. Using the CSF, environmental approaches can identify what details can or cannot be perceived in specific viewing conditions, such as from normal viewing distance from a screen, or in a brightly lit room. The ability of the eye to perceive detail is affected by distance from an object, its brightness, and the brightness of the background. Figure 2. The Snellen Chart gives the eye the best chance to see image details Optimizing Video Delivery: Why the Viewing Environment Matters Page 7 Eliminating the imperceptible image data can yield dramatic savings. nScreenMedia contracted with an independent auditor to validate the savings measured by InterDigital (this paper?s sponsor) for its pre-processing. 7 The audited results for the company?s pre-processing techniques show impressive bandwidth savings over traditional compression technology alone. For example, encoding 4K video for a normal viewing distance of 5 times the TV screen height (5H) allowed 4K video showing a sweeping aerial view of a city (old_town_cross_2160) to be compressed 64% more than with an HEVC encoder alone. A video of a running crowd, among the hardest content to compress, achieved 15% savings. Across all types of content the average savings seen were 30%. 15mbps Target Bitrate Distance from screen (in display-heights) Sequence 3H 4H 5H % savings % savings % savings crowd_run_2160 1.45 4.01 15.21 park_joy_2160 2.02 5.85 20.06 ducks_take_off_2160 0.86 4.49 17.59 in_to_tree_2160 11.91 22.14 35.38 old_town_cross_2160 44.59 51.52 63.75 Minimal savings 0.86 4.01 15.21 Maximal savings 44.59 51.52 63.75 Average savings 12.17 17.60 30.40 Figure 4. Environmental techniques improve compression ratios for a variety of video content Figure 3. Contrast Sensitivity Functions help environmental approaches remove imperceptible image detail Optimizing Video Delivery: Why the Viewing Environment Matters Page 8 C o n c l u s i o n Both the traditional compression and environmental pre-processing approaches will continue to evolve. Companies focused on the traditional compression approach will be wrestling with the implementation of HEVC for the next several years. As with its predecessor h.264, engineers will continue to optimize the implementations of HEVC so that, over time, its efficiency will continue to improve. Hardware will also be revised and improved to support the technology, and will take several years to deploy into the market. Beyond that there is already discussion of new approaches that push the math techniques employed in compression ever further, though many of these will see implementation in the successor to HEVC. 8 Similarly, environmental pre-processing will continue to evolve and improve, though a major upgrade of the hardware and infrastructure of video delivery should not be required to enjoy the benefits. For example, the pre-process can work with adaptive bitrate streaming techniques to deliver content optimized for the different viewing distances used by smartphone and tablet users. The pre-process approach could use the camera in the tablet or smartphone to sense how bright or dark the viewing area is. It could then create a version of the video optimized for those conditions. Of course, there are limits to how far the traditional and environmental approaches can go. However, for now at least we do not seem to be close to them. And if the evolution of computer processors is any guideline, when a limit is reached scientists will develop entirely new approaches to get around them. References 1 Akamai, State of The Internet, Q4 2014, Akamai, p21 www.akamai.com/stateoftheinternet/ (accessed on 5/22/15) 2 ITU, Statistics: ICT Facts and Figures for 2014, ITU, 2014, www.itu.int/en/ITU-D/Statistics/Pages/stat/default.aspx (accessed on 5/22/15) 3 Cisco, VNI Forecast Highlights, Cisco Visual Networking Index, www.cisco.com/web/solutions/sp/vni/vni_forecast_highlights/index.html (accessed on 5/26/15) 4 Ericsson, Ericsson Mobility Report, Ericsson, February 2015, p7 5 Jan Ozer, H264 vs MPEG-2 Quality, Streaming Learning Center, May 26 2011, www.streaminglearningcenter.com/articles/h264-vs-mpeg-2-quality.html (accessed on 7/24/15) 6 The Eye Care Institute, What does 20/20 vision mean?, The Eye Care Institute Newsletter, July 2008, www.eyecareinstitute.com/email/7-16-08.htm (accessed on 5/26/15) 7 The audit was performed at InterDigital?s San Diego office on July 9th 2015 by Steve Hawley of TVStrategies. He verified the test setup, process, execution and results. TVStrategies was contracted by nScreenMedia to perform the audit, and was entirely independent from InterDigital. 8 Yun He, Joern Ostermann, Oscar C. Au, Nam Ling, Introduction to the Issue on Video Coding: HEVC and Beyond, IEEE Journal of Selected Topics in Signal Processing, Vol. 7, No. 6, December 2013 This paper is made possible by the generous contribution of: www.InterDigital.com www.nScreenMedia.com For more information contact: Info@nScreenMedia.com Optimizing Video Delivery: Why the Viewing Environment Matters Page 9 Executive Summary Introduction Opportunities to improve the process The Traditional Compression Approach The Environmental Pre-Process Approach How the Environmental Approach Works Conclusion References