The Vault

From Geometry to Content and Content for Geometry
White Paper / Jun 2017 / ar/vr

With virtual reality (VR), the goal is to create experiences in which the user can be completely immersed; an alternative reality produced by a computer simulation and displayed to the user as a completely synthetic view generated by computer graphics.  In both VR and AR, the reality that the physical environment has such a tight relationship with the experience sets new challenges for content production and application design.  One solution to improve the relationship between the physical and virtual is to adjust virtual experiences to the physical environment, using the environment as a building block for content creation.  In this paper, Innovation Partners discusses some of these approaches and provides examples of recent solutions developed in these areas.

We found some problems with the form:
To view this content please fill out the form below to tell us about you.
Privacy Policy
Motivation / background Augmented reality (AR) is a new form of digital media, tying together digital information with the physical world surrounding the user. Often, the desired effect of AR is to insert virtual elements within the physical environment as seamlessly as possible, making the distinction between the virtual and the real difficult to acertain. To achieve this goal, the image quality of the 3D renderings must be a photorealistic reproduction of the virtual elements, as if they were part of the physical world. This requires not only high-quality computer graphics, but also a robust detection and understanding of the real world, so that the content and application behavior can function within the constraints of the physical environment. With virtual reality (VR), the goal is to create experiences in which the user can be completely immersed; an alternative reality produced by a computer simulation and displayed to the user as a completely synthetic view generated by computer graphics. The displayed graphics completely replace the visual reality for the user, but the illusion of complete immersion in the content can still break down easily if there are obvious discrepancies between the virtual world and the physical environment. A user?s inability to touch and feel virtual objects due to the lack of haptic feedback or unexpected collisions with objects in the real environment while in a virtual world can turn the experience into an unpleasant mess of disconnected and conflicting sensory stimuli. With the AR applications available today, the scenario most likely to break the illusion is the moment when a virtual element appears on top of physical objects that should have occluded the virtual objects in reality (as they are closer to the viewer than the virtual elements). With the advent of mass market adoption of both VR and AR, the need to improve the relationship between the physical and virtual is becoming more pressing. In both VR and AR, the reality that the physical environment has such a tight relationship with the experience sets new challenges for content production and application design. Solutions to minimize the negative impact of the shortcomings described above are in high demand. Until there are good solutions that allow users to touch and feel virtual objects and negate the limitations of the physical space, the only workaround is to avoid them through the use of clever content and application design. One solution to improve the relationship between the physical and virtual is to adjust virtual experiences to the physical environment, using the environment as a building block for content creation, collecting information about the environment which can then be refined to assist in the detection of user preferences and context. In this paper, we discuss some of these approaches and provide examples of recent solutions developed in these areas. From Geometry to Content and Content for Geometry Understanding the physical world and adjusting virtual experiences to it Focusing on the sensor side, the level of environmental understanding has taken a leap forward during the past few years due to the development of low cost RGB-D sensors that extend the range of camera data from purely 2D RGB image data to include depth information. At this time, these RGB-D sensors are on a path toward integration into mobile devices, enabling efficient environment scanning and analysis even on generic mobile platforms. Capabilities enabled by these RGB-D sensors include the reconstruction of Understanding the physical world and adjusting virtual experiences to it In an ideal world, AR and VR technology would enable us to create the illusion of virtual objects and worlds that would not only look real, but would also feel and behave as if physically real. However, the ideal technology that would allow users to freely inspect virtual content using all of their senses does not currently exist. In reality, today?s AR and VR applications have a limited understanding of the real physical world, and technologies allowing users to touch and feel virtual objects are almost completely lacking. The best solutions available tailor the virtual experience to the unique characteristics of the physical space that they are experienced in. This enables the seamless blending of virtual and physical. Recent advances in this area are predominantly due to the development of sensor technology and increasing processing performance. Innovation Partners | White Paper 3D models of the physical environment, as well as object recognition and tracking. One example of this is the tracking of users with RGB-D sensors who are then modeled as kinematic skeletal structures, providing information about each user?s pose and motions. For AR and VR applications, in addition to tracking, one speculated use for RGB-D sensors is capturing 3D data to be used as virtual content. However, the current quality of 3D reconstructions achieved by RGB-D scanning enable only limited use as virtual content. This is an active area that is developing fast, and in the not-so-distant future, we expect VR to enable users to visit distant places using these 3D capturing solutions. Improved 3D scanning quality generally means increased 3D data complexity, which easily creates bottlenecks for processing and distribution of the captured virtual content; thus, this issue will also require new solutions. The ability to match existing 3D assets with the reconstructed geometry of an environment is one interesting approach for improving the quality of the 3D reconstructions. One approach is to tailor virtual experiences to match the user?s physical environment by procedurally creating 3D content to match the geometric structure of that environment. By aligning 3D assets on top of physical objects with matching geometric shapes, low quality 3D reconstruction can be replaced with high quality 3D assets that are readily available in numerous online 3D model repositories. Matching the shape can be done on multiple levels. At the highest level, whole virtual scenes can be aligned with the physical space in an optimal manner to minimize discrepancies between the layouts of the real world environment and the selection of virtual scenes that best match the physical space. At a lower level, matching and replacing physical elements can be done on a per-object basis. Further, the matching can also be performed between several physical locations and virtual environments in order to provide shared virtual experiences for several users participating from different physical locations, so that all the users have minimal conflicts between physical environments and a shared virtual environment. By doing so, this solution ensures a good operational area within the virtual world for each user in a multi-user experience and, by doing so, provides the best possible experience for everyone. Besides understanding the geometry of the environment, experiences can also harness the technical capabilities of the environment to enrich the virtual experience. Particularly in future smart homes where technology is embedded and connected, much of the environment could be controlled from applications in order to improve the user experience, focused on digital content consumption. This requires AR and VR devices and applications to understand the features available in the environment, and how they can, and should, be controlled, as well as how content should be distributed depending on the potentially unique output channel combinations. One solution developed by InterDigital is capable of detecting various output devices available in the environment, harnessing them as part of the experience by distributing the content output to all available devices in a manner that best serves the virtual experience. The physical world as a building block for content creation Instead of trying to reconstruct and understand the layout and structure of the whole physical environment surrounding the user, an alternative, lighter-weight approach is to analyze camera and depth information as a sequence of individual views, i.e., single images captured by the RGB-D sensor. When an RGB-D sensor providing the data is embedded with the HMD worn by the user, these individual views correspond with the user?s view of the world. An often-described use case using this approach is the overlay of information based on the visual recognition of objects and places in the direction that the user is looking. In these examples, identified objects and locations are used as anchor points and links to where the augmented information is to be attached. The relevant information could be highlighting a mechanical component for a maintenance worker wearing an optical see-through AR HMD or linked online content, such as a restaurant review score overlayed directly into the user?s field of view when looking at a restaurant from the sidewalk. Innovation Partners | White Paper BESIDES UNDERSTANDING THE GEOMETRY OF THE ENVIRONMENT, EXPERIENCES CAN ALSO HARNESS THE TECHNICAL CAPABILITIES OF THE ENVIRONMENT TO ENRICH THE VIRTUAL EXPERIENCE Innovation Partners | White Paper This type of information layering, or AR browsing as it is sometimes called, offers a type of AR use in which relevant information is linked to the recognized visual targets and is augmented on top of them. This is the traditional approach to applying AR in a per-view basis. In addition to the recognition of full objects, the detection of just geometric features can be used as a building block for content creation. One example of how this low-level feature detection could be used to enhance the relationship between virtual and real is the segmentation and detection of isolated geometric features that can be combined with matching virtual shapes to serve as proxy objects providing haptic feedback for virtual objects. One solution developed by InterDigital details basic features detected from the environment, which are used to create artistic effects to modify the appearance of the environment, with the goal of changing the experience through these effects, rather than just using AR to relay meaningful information. Basic visual features are identified from the environment and used as an anchor point and seed for procedurally-generated fractal 3D graphics that can be synchronized to react to audio to which the user is listening. In addition to producing interesting digital trip-like experiences, similar content creation approaches can enable a simplified and efficient AR content authoring tool when the approach is combined with user input controls for operating the content creation process. Combined with features for efficient content distribution, this solution enables user generated content production and distribution, alleviating labor-intensive content production, one of the main bottlenecks hindering widespread adoption of AR. THE DETECTION OF JUST GEOMETRIC FEATURES CAN BE USED AS A BUILDING BLOCK FOR CONTENT CREATION Collecting information from the real world to use as an input In previous sections, we have discussed some approaches for using full 3D reconstruction, as well as recognition of individual elements from the sensor data to produce content during one isolated session. However, in addition to gathering and using the collected environment data in one isolated session, the collected understanding of the structure of environments can be stored and further refined to serve as an input for additional use cases. Similar to how many online services collect information about users in a progressive manner, the geometric data and refined information generated through approaches described above can be collected and further analyzed to gain even deeper insight into the environments and users. The data collected from environments in which a user spends time are extremely rich sources of information. For personal spaces, such as the home of the user, progressively collected data can be used to provide better recommendations and user profiling. For public places, such as retail shops, crowd-sourced data collection could be used to gather very detailed information about item selection in particular places as well as user demographics for those places. InterDigital has been working on a concept system for centralized and systematic collection and analysis of data generated by both AR and VR content consumption sessions that relate to environment scanning and 3D reconstructions. The described system collects information that can be used to extract further details about users, environments associated with the users, and the contexts of use. Such information enables improvement of various AR and VR experiences and services. It also allows for the development of completely new services that can identify situations where a user needs assistance, match virtual experiences with physical environments, and perform virtual product placement and advertisement. Besides AR and VR devices, the autonomous vehicles currently in development and soon expected to appear on public roads contain arrays of sensors capable of collecting very detailed environmental information from the area immediately surrounding the vehicle. While the primary use of this data is for tasks such as collision avoidance, classification of objects, and detection of lanes, centralized aggregation of environmental data captured by all such vehicles would enable the construction of extremely information-dense and constantly updating models of the physical world, which in turn would enable many new uses for emerging AR and VR technology. This is another area where InterDigital has been working on a concept for a centralized data collection system. Conclusions In this white paper, we have discussed the relationships between the physical world and virtual experiences. As we have pointed out, this relationship should be taken into consideration in order to generate the best AR and VR experiences. However, at the moment, there are areas where input and output technologies still have shortcomings that make full tailoring of experiences to the physical environment difficult. For this reason, we discussed some potential approaches to improve AR and VR applications? understanding of the physical environment and to minimize potential conflicts between virtual and real. We have also introduced related solutions that were recently developed by InterDigital. We hope that solutions such as the ones described here are developed in increasing numbers and, by doing so, populate physical spaces with virtual elements to help us enjoy and be more productive in everyday environments. Innovation Partners | White Paper WP_201705_008 Innovation Partners