Segmentation and Bitstream Encoding of Foreground Objects in HEVC Encoder for Edge Computing Environment

Abstract

Segmentation of foreground object is pivotal in identifying the other finer details about objects in a scene. Our work serves to investigate ways to not only segment but encode the foreground objects in the HEVC encoder with minimal change in original HEVC encoder performance in terms of bitrate increase and encoding time. We achieve this by reutilizing the intermediate residual data at the encoder to segment the foreground activity in each frame and finally encoding the same in the final compressed bitstream using specific provisions of HEVC high level bitstream syntax. The method operates entirely in the HEVC encoder loop, taking residual data of the frame, it divides the entire frame into 8 × 8 target patches and applies the proposed algorithm Median of Discrete Variance (MoDV) to classify the target block of each frame of the video sequence as foreground or background. The foreground information in each frame is then encoded into the compressed bitstream by harnessing our proposed format of supplemental enhancement information (SEI) Network Abstraction Layer (NAL) units to tag the location of the foreground activity. Along with the segmentation accuracy, change in encoder performance is also measured to judge the various trade-offs. We conclude by testing its efficacy on a variety of videos with difference frame resolutions and background conditions.