With virtual reality (VR), the goal is to create experiences in which the user can be completely immersed; an alternative reality produced by a computer simulation and displayed to the user as a completely synthetic view generated by computer graphics. In both VR and AR, the reality that the physical environment has such a tight relationship with the experience sets new challenges for content production and application design. One solution to improve the relationship between the physical and virtual is to adjust virtual experiences to the physical environment, using the environment as a building block for content creation. In this paper, Innovation Partners discusses some of these approaches and provides examples of recent solutions developed in these areas.