CHARITY Use Cases and Integration Challenges

2024-01-17

The CHARITY project envisions the creation of a Cloud framework for Holography and Cross Reality framework that, utilizing edge resources and devices, and will host demanding applications for Virtual Reality, Augmented Reality, and Holography. The progress of XR services has increased significantly due to advances in available HMD hardware, the introduction of more powerful sensors, and advances in computer graphics technologies.

Today, such applications are one of the most challenging use cases. They can benefit from an intelligent orchestration framework to support their deployment and lifecycle management over multiple domains. Whilst, we envision having numerous components of the CHARITY specifically tailored for XR. Hence, the set of use cases developed by consortium partners is essential to the overall CHARITY platform experimentation.

We divide use cases (UCs) into three groups: Real-time Holographic Applications, Immersive Virtual Training, and Mixed Reality Interactive Applications, as briefly described below. Such an overview sets the initial discussion of how each use case can leverage the CHARITY framework and how we should translate that as part of the integration and experimentation activities.

  • UC1: Real-time Holographic applications
  • Holographic Concerts – A concert using a 3D holographic display to recreate musicians playing in a concert. The performers are filmed, and live broadcasts are streamed through the CHARITY platform. From an experimentation standpoint, this implies a distributed multi-setup environment to collect and reproduce different musicians and a reproduction display.
  • Holographic Meetings – Like holographic concerts, holographic meetings will require a multi-setup experimentation setup. In this case, the goal is to have the speaker be able to be placed anywhere and transmit video to multiple monitors. Again, the CHARITY framework also has the critical role of supporting the streaming of multiple involved meeting participants.
  • Holographic Assistant - In this use case, the assistant's capabilities should leverage accessed cloud-based services offered by CHARITY or third-party cloud services. This involves experimentation with a hybrid combination of edge/cloud services, high-performance GPU processing in the edge, novel 3D Point cloud generation and compression/decompression methods and the streaming of large amounts of data (e.g., 3D Point Cloud to use as part of the 3D modelling). The assistant software should run independently of the end user device's capabilities and resources.
  • UC2: Immersive Virtual Training
  • VR Medical Training – The VR Medical Training application will be used to compute, render, and encode the images transmitted to VR HMD’s utilizing a signalling server. Based on the network characteristics, CHARITY will provide XR service run-time adaptation and support dynamic optimization of the produced stream of images. Furthermore, the CHARITY platform plays an essential role in supporting the XR service device agnostic framework as heavy processes, such as rendering and physics engines, are to be deployed in edge resources, allowing the HMD to be used as a lightweight client. One of the key challenges here is to decouple such Unity3D computations into multiple Cloud services.
  • VR Tour Creator – This scenario encompasses a Cyango application to create VR-based virtual tours. Hence, CHARITY architecture should support such video editing components, including high-bandwidth and low-latency 3D and 2D video streaming.  Such a scenario can significantly benefit from multiple and different CHARITY orchestration and processing mechanisms, such as adaptive video streaming or cloud/edge optimizations which should be considered in the experimentation activities.
  • UC3: Mixed Reality Interactive applications
  • Collaborative Gaming – Like the previous one, this use case, an AR-based multi-player game depends on CHARITY processing mechanisms (e.g., Mesh Merger) and intelligent orchestration mechanisms for game server deployment. Therefore, such a challenging integration scenario combining the outcomes of multiple tasks should be carefully considered in the integration and experimentation plans. 
  • Manned-Unmanned Operations Trainer – This use case, a VR-based training simulator, will leverage the dynamic adaptation mechanisms proposed in CHARITY to exploit cloud-edge continuum capabilities efficiently. Such a scenario, which should be part of the experimentation plans, involves a challenging integration of monitoring components to enable (near) real-time monitoring of the available resources and planning mechanisms to dynamically adapt the behaviour containers and applications.

CHARITY seeks to create the base of a better adoption of more advanced media applications by a large community of users and companies, even outside the consortium.