Novel view synthesis with residual error feedback for FTV

The availability of multi-view images of a scene makes possible new and exciting applications, including Free-viewpoint TV (FTV). FTV allows us to change viewpoint freely in a 3D world, where the virtual viewpoint images are synthesized by Depth-Image-Based Rendering (DIBR). In this paper, we propose a new method of DIBR using multi-view images acquired in a linear camera arrangement. The proposed method improves virtual viewpoint images by predicting the residual errors. For virtual viewpoint image synthesis, it is necessary to estimate the depth maps with multi-view images. Some algorithms to estimate depth map were proposed, but it is difficult to estimate accurate depth map. As a result, rendered virtual viewpoint images have some errors due to the depth errors. Therefore, our proposed method takes into account those depth errors and improves the quality of the rendered virtual viewpoint images. In the proposed method, the virtual images of each camera position are generated using the real images from each other camera. Then, the residual errors can be calculated between the generated images and the real images acquired by the actual cameras. The residual errors are processed and fed back to predict the residual errors that can be happened to virtual viewpoint images generated by conventional method. In the experiments, PSNR could be improved for few decibels compared with the conventional method.