A visual system is proposed where a three-dimensional (3D) butler like avatar appears on a holographic 3D display to offer assistance for day-by-day tasks. such as answering questions about weather, showing information about stocks or enabling connection to a chat bot for further verbal interactions just to name some usage scenarios planned for UC demonstration. Of course this are only some selected scenarios only for demontration purposes. There are unlimited possibilities in the fields of assistance, but also withing other topics like education, advertising and communication.

 

One special aspect of this UC is the display technology used for visualization. It is based on true 3D Holography, a novel technology based on the interference of laser light where a detailed and deep hologram floats in the air in front or within the 3D display - the observer can see the real depth in space and can focus the details of the 3D-butler or 3D-information shown with both eyes like natural viewing. Same as with natural viewing, the observer can look around the 3D objects shown – same experience as it would be a real object, a feature which only true 3D holograms provide. The information shown beside the 3D-butler could be text data, image data (photos, maps etc.) or 3D-data (3D-models, 3D-visualizations) beside audio data played as the natural language of the butler talking with you.

 

All of those user scenarios would be supported by cloud-based services within the CHARITY project and by cloud/internet services of other companies or organizations. The system would recognize and interpret speech and would answer in natural language.

The main goal of this UC is to show that holographic 3D display technology can be driven by 3D data generated within cloud-based services. This approach needs to take the enormous delays into account but on the other hand if successful, such a solution would show that streaming of interactive content like games to holographic 3D devices is possible without any degraded user experience. The key-enabler is to use a multi view 3D point cloud generated from the animated 3D-scene (in the cloud) and streamed to the client with the holographic 3D device. This 3D point cloud enables to immediately provide the right perspectives on the client for the look around feature of true 3D holograms. It is not needed to go rounds with long delay through cloud services when the observer changes his position in front of the holographic 3D display.

CHARITY will support this UC by providing the cloud infrastructure and taking care of service deployment and resource management. The UC will adapt to the CHARITY architecture to fit into this ecosystem. CHARITY partners will support the development and/or adaption of 3D point cloud compression and decompression methods.

SRT is currently working on the cloud services to generate 3D content within a cloud-based Unity 3D engine instance (VM) and converting multiple rendered views to 3D point clouds with the target to achieve at least a framerate of 30 Hz. Next steps are to connect to service providers like text to speech and speech to text to enable speech processing and answer generation and acquisition of information from internet service providers. Interpreting spoken requests is another topic on the agenda. CHARITY partners are working in parallel on the topics CHARITY infrastructure and 3D point cloud compression / decompression.