Energy, data and computational-efficient mechanisms supporting dynamically adaptive and network-aware services
2023-07-20
One of the work package of CHARITY, the WP3, is entirely devoted to research and development activities to support the development of complex and highly demanding (in terms of computation and/or bandwidth resources) XR applications. The new technologies and algorithms developed in the ambit of this work package, that regard different aspects of advanced XR applications, will be part of the CHARITY platform and some of them will be integrated into some of the Use Cases (UCs) of the CHARITY project. Each Use Case regards one advanced XR application. These UCs are used to demonstrate concretely the benefit of the CHARITY technologies and solutions.
Monitor the available networking and computing resources and taking advantage of this information to make more efficient the deployment and the behaviour of XR applications is one of the goal of the CHARITY project. A flexible monitoring framework has been designed and is under realization. This monitoring framework is based on the open-source Prometheus technology. It will be used by the different UCs of the CHARITY project to monitor their metrics of interest (e.g. bandwidth, computing resources, etc.). The monitoring framework collect such salient information for two purposes. First, to provide to the CHARITY orchestrator, the intelligent management system that acts to satisfy the request of deployment of applications according to the available resources, all the necessary information to make its decision. Second, to make possible that a running XR application reacts when a reduction in terms of computing or networks resources happen. Every XR application can react in different ways to these circumstances, depending on its characteristics. In the R&D work conducted in the WP3, we are investigating a methodology that can be adopted by different XR applications to adapt their behaviour according to the available resources. This general run-time adaptation mechanisms is based on a variant of the MAPE-K Loop approach, an approach used in software design which consists in leaving some aspects of the application tweakable from external configuration. This adaptation solution has been implemented in the flight simulator of the Collins Aerospace, one of the CHARITY UC, related to the Manned-Unmanned Operations Trainer Application.
Another main component developed in this work package, is a data management system for the intelligent data storage and data access. This system takes into account the high degree of heterogeneity that characterises the computational resources that often are encountered in real-world applications. CHES is also designed to be a lightweight system so that it can also be used on edge devices with limited hardware capabilities, such as a Raspberry Pi. CHES is released as an open-source software under GPL 3 license. The readers interested in a detailed description of the CHES can refer to a previous post (2/14/2023) on this blog (https://www.charity-project.eu/en/blog/the-charity-edge-storage-ches).
Virtual Reality applications often require high realism in rendering and physical simulation. This is particularly important for certain applications, such as for VR systems used for medical purposes (training, simulation). The VR Medical Training Application UC of CHARITY is one of these types of virtual reality applications. This VR UC is currently being optimized by exploiting multi-threading to make the rendering and the physics part even more efficient. Preliminary results are encouraging, for what concerns the physics simulation, while multi-threaded rendering has shown some limits for applications that use the Unity framework.
The immersive applications, to reach high-quality levels of experience, require ultra-low latency and large bandwidth resources. In CHARITY, we are investigating the usage of adaptive rendering algorithm to reduce the rendering computation, and hence, the overall latency of the applications.
Some data services to satisfy the needs of XR applications are also under development. The first regards the transmission of huge 3D point clouds. The streaming of 3D points may be useful for many XR applications. In CHARITY, it is used by the UC1-3 Holo Assistant, an Use Case where a virtual assistant is shown on an innovative holographic display. A second data service under development is the Mesh Merger. The Mesh Merger makes possible to build and update an indoor virtual environment using a streaming of geometry pieces acquired by the mobile phones of users. This permit to set up quickly an environment for AR applications and to maintain it up-to-date. In CHARITY, we tested the Mesh Merger on an AR collaborative game.
Finally, we conclude this overview of the WP3 R&D activities talking of an advanced platform for the creation of virtual tours based on 360 video, called Cyango Cloud Studio (https://www.cyango.com/cloudstudio). This platform, which allows the integration of 360 video, annotations, and 3D models, allows the users to create in an easy way and effective way immersive tours. This platform is continuously updated with new features and new technological solutions to improve its performance are under test.
Keywords: XR applications, computing/networking resource monitoring, intelligent data management systems, multi-threaded rendering, adaptive rendering, compression/streaming of 3D content, 360 video.
Brief bio of author
Massimiliano Corsini is a senior researcher at the Visual Computing Laboratory of the ISTI-CNR in Pisa, Italy. His research interests are in the field of Computer Graphics, Computer Vision, Machine Learning, and Image Processing. He got a Ph.D. studies in Information and Telecommunication Engineering from the Department of Information Engineering of Florence University. His current research work regards the applications of Machine Learning, Computer Graphics and Computer Vision to Digital Humanities, Monitoring, Mobile Robotics, and other applications, and Visual Analytics. In these fields, he developed new algorithms and software tools documented in more than 70 publications in peer-review international conferences and journals. He has been involved in several, at different levels, in several International and European projects.
