The Vault

MWC19: 360 Video Standards

View the slides from our demonstration of 360 Video Standards from MWC19!

© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved. 5G Video Standards On Video, VR/AR, Point Clouds, and Beyond © 2019 InterDigital, Inc. All Rights Reserved. Social Experience Sharing 5G Video Experience 360° Immersive MediaUHD Video Streaming VR & AR 2 Video Standards Landscape © 2019 InterDigital, Inc. All Rights Reserved.3 Video compression standards define video bitstream formats at bit-level to ensure video information can be exchanged using a common format Video system standards define file formats, media presentation formats, transport stream formats, A/V sync information, etc. to guarantee media presentation Video application standards select the core video compression and system standards (e.g. profiles and levels) based on specific application requirements Other related SDOs 0H.625 MPEG HEVC MPEG DASH ISO MBFF ITU-T-SG-16 WP3 Q.6 ISO/IEC SC29 WG11 MPEG ISO/IEC SC29 WG1 JPEG Video Technology Roadmap – Labs © 2019 InterDigital, Inc. All Rights Reserved.4 2018 2019 2020 2021 2022 2023 2024 Point Cloud Coding MPEG-I Systems Stds. Compression Standards Point Cloud Coding ext. H.266 version 1 H.266 version v2, v3, … Iterations of Video Platforms, 360-degree video (3DoF), AR, 3DoF+, 6DoF, … Light Field codec Light Field codec v2 OMAF v2 Immersive Media AF Augmented Media AF Long-term non-standards research Light Field exploration Non-stds. research InterDigital Video Technologies © 2019 InterDigital, Inc. All Rights Reserved.5 Bitstream Spatial Prediction Motion Compensated Prediction + Reconstructed Video Out Viewport Video Out Residual Coefficients C oding M ode, Prediction M ode M otion Prediction Block Projection Format Conversion Entropy Decoding Quantization-1 Transform-1 Loop Filter Reference Picture Store At a Glance InterDigital Recent Key Technologies in Video • Efficient advanced temporal motion vector prediction • Optimal bi-directional optical flow with low complexity and bit width control © 2019 InterDigital, Inc. All Rights Reserved.6 • Bi-predictive weighted averaging (BPWA) for prediction • Horizontal geometry padding for 360-degree video coding VVC (VTM-3.0) vs. HEVC (HM-16.19), Oct 2018 50% 100% AI (All Intra) RA (Random Access) LD (Low Delay) HEVC VVC HIGH EFFICIENCY$ VR Related SDOs and SIGs © 2019 InterDigital, Inc. All Rights Reserved.7 I n t e r n a t i o n a l T e l e c omm u n i c a t i o n U n i o n © 2019 InterDigital, Inc. All Rights Reserved. Interactive 360° Volumetric Video With Adaptive Streaming InterDigital & Technicolor © 2019 InterDigital, Inc. All Rights Reserved.9 Challenges Solutions • Adaptive Video Fetching Based On Viewer Position • Manage Increased Server Content Without Greater Bandwidth • Setting Anchor Views • Smaller Streaming Segment Size • HW Acceleration 14 2kx2k Decoding & View Synthesis @ 30fps Interactive 360° Volumetric Video with Adaptive Streaming Channel Bandwidth & Scalability Latency Real-time Video Decoding & View Synthesis © 2019 InterDigital, Inc. All Rights Reserved.10 See how a head tracker tracks a human face, observe motion parallax, and use a joystick to track 360° video views. Interactive 360° Volumetric Video with Adaptive Streaming MPEG-DASH Server with 360° Volumetric Video Streams MPEG-DASH Client with Head Tracker / Joystick, Video Decodes/View Synthesis & Rendering Ethernet 13 Depth HEVC Streams 2048 x 2048 @ 30fps 13 Color HEVC Streams 2048 x 2048 @ 30fps Head Tracker © 2019 InterDigital, Inc. All Rights Reserved.11 up down rightleft forward backward Viewer Position/Orientation Capture Head Tracker to track head position (X,Y,Z) Observe motion parallax Joystick to change position (X, Y, Z), orientation (Yaw, Pitch, Roll) Observe motion parallax and 360° Video Head Tracker forward backward Yaw: 360° right left up down Roll (trigger) Pitch (trigger) © 2019 InterDigital, Inc. All Rights Reserved.12 Adaptive Streaming Viewer Position Adaptive Streaming © 2019 InterDigital, Inc. All Rights Reserved.13 Motion Parallax: Disocclusion from View Position Changes Position changes (x,y,z) with head tracker or joystick. Above shows ~1 feet movement away from center position. © 2019 InterDigital, Inc. All Rights Reserved.14 360° Video: Orientation Changes Orientation changes (Yaw, Pitch, Roll) with joystick © 2019 InterDigital, Inc. All Rights Reserved.15 MPEG-DASH Server MPEG-DASH Adaptive Streaming Client HEVC Video Decoder 1 HEVC Video Decoder 2 HEVC Video Decoder 3 HEVC Video Decoder 14 Real-time View Synthesis Viewport Extraction & Render Head Tracker & Joystick ... ... ... Interactive 360° Volumetric Video with Adaptive Streaming View Synthesis © 2019 InterDigital, Inc. All Rights Reserved.16 From a discrete number of camera views toward any virtual view corresponding to a user’s point of view • Visibility-based processing to compute depth information for new viewpoint • Specific processing to remove ghosting artefacts and aliasing on depth contours • Confidence-based final color definition Backups © 2019 InterDigital, Inc. All Rights Reserved.17 43” UHD TV Display Screen Shoot © 2019 InterDigital, Inc. All Rights Reserved.18 up down rightleft forward backward Head Tracker MPEG DASH Server: Video Content Captured Before-Hand © 2019 InterDigital, Inc. All Rights Reserved.19 x y z v0 v1 v2v3 v4 v9 v5 v6 v7 v8 v11v12 v10 Position (x,y,z) in meter Orientation (Yaw, Pitch, Row) in ° V0 [ 0.3000, -0.0000, 1.6500] [0, 0, 0] V1 [ 0.0927, -0.2853, 1.6500] [0, 0, 0] V2 [ -0.2427, -0.1763, 1.6500] [0, 0, 0] V3 [ -0.2427, 0.1763, 1.6500] [0, 0, 0] V4 [ 0.0927, 0.2853, 1.6500] [0, 0, 0] V5 [ -0.1500, 0.0000, 1.3902] [0, 0, 0] V6 [ 0.1500, 0.0000, 1.3902] [0, 0, 0] V7 [ -0.1500, 0.0000, 1.9098] [0, 0, 0] V8 [ 0.1500, 0.0000, 1.9098] [0, 0, 0] V9 [ 0.0000, -0.0000, 1.6500] [0, 0, 0] V10 [ 0.0000, -0.0000, 1.6500] [180, 0, 0] V11 [ -0.1500, -0.2598, 1.6500] [225, 0, 0] V12 [ -0.1500, 0.2598, 1.6500] [135, 0, 0] 13 cameras: v0~v12, color 8-bit, depth 10-bit, 2048x2048, HEVC encoded, segmented for streaming per DASH MPD Technology: • Real-time video decodes & view synthesis to produce video content with head-motion parallax support and 360° viewing experience • Real-time head tracking to detect viewing position and orientation • Adaptive multi-view streaming based on head motion tracking or joystick Targets & Objectives • Expanding user viewing experience to 360° video & head motion parallax • Bandwidth efficient adaptive streaming of 360° volumetric video over HTTP Demo Description/Visitor Experience • 360° HEVC video streams (13 color and 13 depth) stored in Video Streaming server • Client tracks viewer head or joystick movement, and streams corresponding input views through adaptive streaming. Video streams are decoded and view is synthesized at client side at 30fps. • Viewer observes real-time 360° video and motion parallax on display corresponding to his/her view position © 2019 InterDigital, Inc. All Rights Reserved. Video Streaming Server with 360° Volumetric Video Streams Streaming Client with Head Tracker / Joystick, Video Decodes/View Synthesis & Rendering Interactive 360° Volumetric Video with Adaptive Streaming