Synergistic exploitation of localized spectral-spatial and temporal information with DNNs for multisensor-multitemporal image-based crop classification

Abstract

The challenge of performing efficient and reliable crop classification with multisensor multitemporal (MSMT) images in mixed land cover scenarios i.e. presence of small land parcels (area < 20,000-meter square) of crops and other land covers such as built-up or grasslands, is significant. Specially in countries (ex. India) where diverse crops are practiced in small land parcels. This challenge can be addressed if deep neural network (DNN) based models can exploit all three i.e. spatial, spectral, and temporal information of a crop, present in the MSMT images, efficiently and effectively. Therefore, this study presents a novel DNN based model that exploits all three information in a synergistic fashion to achieve improved crop classification. At first, the model increases the significance of local spectral information via a strategy that creates versions of spectral band set wherein neighbourhood of spectral bands is permuted. Then, the model utilizes three-dimensional convolutions, in a time-distributed fashion, to extract local spectral-spatial features. Finally, the model utilizes bidirectional long short-term memory or LSTM-RNNs to extract the temporal information embedded in the time-distributed feature-space created after the convolutions. The developed model is trained and evaluated on Sentinel-1 and Sentinel-2 MSMT data to achieve a 6-class classification including two major crops grown in the region. One of the proposed models namely Perm-3D-CRNN-v1 showed a 97.77 % overall accuracy on test samples and reflected satisfactory on quantitative analysis. The localized spectral-spatial convolutions created prominent class-specific features whereas the bidirectional information flow in the recurrent layer improved the exploitation of crop-phenology type features making the model perform efficiently.