Unstructured video based rendering, the next big thing for UAS sensors?
One of the biggest issues facing unmanned vehicle operators is how to present imaging. Cameras relay terrabytes of data but only from the current viewpoint. What if you could combine several video images and then virtually move around the environment in 3D?
The Computer Vision and Geometry Group CVG ETH, Zurich have been working on it and presented at SIGRAPH 2010
We present an algorithm designed for navigating around a performance that was filmed as a “casual” multi-view video collection: real-world footage captured on hand held cameras by a few audience members. The objective is to easily navigate in 3D, generating a video-based rendering (VBR) of a performance filmed with widely separated cameras. Casually filmed events are especially challenging because they yield footage with complicated backgrounds and camera motion. Such challenging conditions preclude the use of most algorithms that depend on correlation-based stereo or 3D shape-from-silhouettes.
Our algorithm builds on the concepts developed for the exploration of photo-collections of empty scenes. Interactive performer-specific view-interpolation is now possible through innovations in interactive rendering and offline-matting relating to i) modeling the foreground subject as video-sprites on billboards, ii) modeling the background geometry with adaptive view-dependent textures, and iii) view interpolation that follows a performer. The billboards are embedded in a simple but realistic reconstruction of the environment. The reconstructed environment provides very effective visual cues for spatial navigation as the user transitions between viewpoints.
The prototype is tested on footage from several challenging events, and demonstrates the editorial utility of the whole system and the particular value of our new billboard-to-billboard optimization.
Unstructured VBR from Jhano on Vimeo.
