cXR+ Semantic Compression: Towards Networked Immersive Environments

Extended Reality (XR) applications, which may encompass Augmented Reality (AR) and Virtual Reality (VR), commonly involve immersive virtual environments that are based on real physical environments. Transmitting the extensive color and depth image data for representing a physical environment over a communication network to create a corresponding immersive environment at a remote location is very challenging due to the enormous data volumes and the time-constraints of real-time immersion. We explore semantic compression, which conveys the semantic meaning of the data through color codes (CCs) to reduce the transmitted data volume. The creation of an immersive environment conventionally involves the five steps (with corresponding output): calibration (single-frame point cloud), registration (registered point cloud), volume reconstruction (voxel frame), marching cubes (meshes), and rendering (3D environment). We develop the novel cXR$+$ semantic compression that splits the volume reconstruction into a client-side virtual network function (VNF) that represents the registered point cloud as CCs for network transmission to a server-side VNF that decodes the CCs to obtain the voxel frames so as to complete the volume reconstruction. We consider an analogous splitting of the calibration into two VNFs with direct CC compression of the color and depth image data as a comparison benchmark. Additionally, we consider the transmission of the raw (uncompressed) image data as well as the volume reconstruction at the client-side and subsequent transmission of the voxel frames as benchmarks. We conduct measurements of the compression (computing) time, transmission time, and reconstruction accuracy for four real-world data sets so as to elucidate the trade-offs in employing semantic compression for transmitting immersive environments over communication networks.