Holographic Assistant

2023-05-11

With this blog post we want to give an update of SRTs progress on development of CHARITY use case 1-3 – the Holographic Assistant.

The main idea of this use case is to develop and show a principle of how complex 3D content could be generated in the cloud or edge, but the local presentation on a true holographic 3D display is done with minimal delay at generating the dynamic perspectives of the running 3D scene required for true 3D holography.

Focus of our work in the last 12 months was on development on the Holographic assistant 3D content module (made with Unity 3D), the Behaviour module (written in C#) to control the Assistant and information shown and implementing interfaces to access information services like weather data and stocks data on the internet (by using several REST APIs). Furthermore a preliminary (currently slow) reference system to create a 3D point cloud stream from Unity 3D render output was implemented (using C++ and Direct 3D). It is currently running with about 5 frames per second, but will reach 30 FPS with GPU based implementation of 3D Point cloud generation. We also implemented those components into a VM running on an Edge server – above picture gives an impression of the overall architecture.

An image based encoding of 3D point cloud data using several layers to encode hidden 3D points was proposed to be able to make use of image based compression. So with that we could pipe the image stream into an FFMPEG instance (see ffmpeg.org) to compress the stream using a H264 high profile compression aiming for low number of compression artifacts. With quite low delay of <1-2 seconds using some advanced mpeg settings we are now able to stream the data using RTSP to a client system where it is decompressed by FFMPEG again and piped into client software driving the holographic 3D display. More representations and methods to encode and compress / uncompress the 3D point cloud data will be implemented and tested in the next months.

The use case will make use of the CHARITY architecture to deploy and manage all the edge and cloud resources, so that the application can start and run smoothly even on changing network conditions or issues with computation nodes. This will be supported and realized by XR service specific monitoring, analytics , decision and actuation within the CHARITY-framework.

Recently SRT participated in the CHARITY webinar “Future-Forward holographic applications” hosted by CHARITY. The recording can be found here https://youtu.be/biVxDb2PVmI. The part with Enrico presenting the use case starts at about minute 47:00. You will learn a lot about the principles of true 3D holography, differences to other technologies called “holograms”, the use case and the idea of 3D point cloud streaming.

Following pictures give an impression of current assistant look (figure 2)– rendered as single views, of course this are not a holograms.

Stay tuned for more updates on the use case in the next months.

Brief bio of author

Dipl. Inf.. Enrico Zschau (♂), currently in the Role of Head of Software Development and IT, has over 20 years practical and theoretical expertise in software design and development, content and formats, component and system prototyping, product development and service. He has over a decade experience with 3D-holography. He is leading the Software Team and IT department at SeeReal for more than 7 years.

Enrico Zschau studied at Technical University Dresden and holds a Master/German diploma in computer science with specialization in computer vision and network technology.