A multi-modal smart switching based image transmission using semantic communication

Abstract

The conventional paradigm of communication primarily concentrates on the transmission of raw data, often disregarding its contextual meaning. However, to tackle the exponential growth in data demands along with the limited availability of transmission bandwidth, there is an increasing need to transition from Shannon’s classical information-theoretic communication to a more advanced framework centered on semantics. This work presents a multi-modal semantic-based communication method for the transmission of high-definition images aimed at optimizing the transmitted data volume while maintaining a high throughput and mean intersection over union score. To this end, two architectural models are explored: a denser ResNet-based and a lightweight U-Net-based. Depending on the required QoS and resource availability, the raw image is either semantically segmented to obtain a fine-grained, pixel-level classification of the image or represented as label semantics, which provides only a higher-level, object-based, or region-based classification prior to its transmission. The experimental results show that such an adaptive semantic image processing approach leads to around 63% reduction in the transmitted data volume without compromising on the quality of image reconstruction.