With the rapid progress of deep learning technology in recent years, image recognition technology is widely used in various applications such as automatic driving and face recognition. In addition, with the evolution of imaging technology, there is a growing interest in Mixed Reality (MR) technology, which is a further development of Virtual Reality (VR) and Augmented Reality (AR). We propose ReViSE (Real and Virtual Seamless Editor), a framework that combines these two technologies. ReViSE is an innovative MR framework based on a video see-through method that presents various MR representations by freely combining real and virtual objects, as shown in the left figure.

　In addition, by utilizing instance segmentation, it is possible to specify arbitrary object regions, including the background, for camera images and replace them with virtual objects. The framework structure of ReViSE is shown on the right; ReViSE consists of five core components: image acquisition, instance segmentation, instance selection, hybrid reality synthesis, and Vision Sync display.

　As an example of a system implementation based on ReViSE, a prototype of a system for a live concert was implemented. This prototype achieved 32.73 fps with moderate object extraction by selection and has the potential to achieve practical performance by further improving the extraction accuracy and processing speed. We are constantly working on our research with the aim that ReViSE will be used in the real world.