Content-aware 3D reconstruction with gaze data

Abstract

3D reconstruction has been shown to be a successful method for creating accurate 3D models out of video data with moving objects. Typically, videos are captured by ordinary cameras; however, more egocentric video footage will be taken by wearable cameras. In this work, we present a 3D reconstruction pipeline that implements content awareness for combining a wearable camera (a scene camera of an eye tracker) with gaze information. The aim is to identify the object of interest (OOI) within the video sequence. The OOI is identified within each frame for boosting the results of classical Structure from Motion (SfM) approaches, using the bio-inspired approach from an earlier study. We implemented a prototype based on the concept of content-aware 3D reconstruction using gaze data. Lastly, we gave an extensive overview of possible use case scenarios in a broad range of fields, starting from spare part reconstruction in difficult-to-access areas to assistive technologies, including exoskeletons and prosthetic arms/hands. © 2017 IEEE.