[2107.06278] Per-Pixel Classification is Not All You Need for Semantic Segmentation

Significance

Predicting set of instance masks is better than dense prediction

Keypoints

  • Propose a mask classification based semantic segmentation framework
  • Demonstrate semantic segmentation performance of the proposed method

Review

Background

Semantic segmentation is a task that aims to partition an image into regions with different semantic categories. Application of deep learning methods for semantic segmentation has largely improved the performance of the methods, and these models are usually trained by classifying each pixels into one of the $K$ semantic classes (dense prediction). An alternative approach to the dense prediction is the mask classification, which is to generate a set of $N$ binary masks and classify each masks into one of the $K$ classes. 210714-1 Per-pixel classification (left) and mask classification While the mask classification approach is usually used for instance-level segmentation tasks, the authors show that it can also perform well on the semantic segmentation by proposing a new framework MaskFormer.

Keypoints

Propose a mask classification based semantic segmentation framework

210714-2 Schematic illustration of the proposed method

The proposed MaskFormer consists of pixel-level module, Transformer module, and segmentation module. The pixel-level module first takes an image to generate a low-resolution image feature map with a backbone model. This image feature map is upsampled with a pixel decoder model to generate per-pixel embeddings. Transformer module takes the image feature map from the pixel-level module, and computes $N$ per-segment embeddings. Finally, the segmentation module applies a linear classifier on top of the per-segment embeddings. The per-segment embeddings are used for both the mask predictions and the class predictions.

Demonstrate semantic segmentation performance of the proposed method

The performance of the proposed MaskFormer is compared with baseline models on the ADE2K, Cityscapes, and the COCO-Stuff datasets. 210714-3 Comparative study on the ADE2K dataset It can be seen that the proposed method outperforms other baseline methods with either CNN or Transformer backbones. A notable finding is that the performance gain of the mask classification based method is more apparent when the number of classes is larger. 210714-4 Comparison of mask classification and per-pixel baselines on four datasets

Panoptic segmentation on COCO panoptic dataset also demonstrates excellent performance of the MaskFormer over baselines. 210714-5 Panoptic segmentation results on COCO panoptic dataset

Further ablation study results are referred to the original paper

Related

Share

Comment

#image-generation #multi-modal #language-model #retrieval-augmentation #robotics #forecasting #psychiatry #instruction-tuning #diffusion-model #notice #graph-neural-network #responsible-ai #privacy-preserving #scaling #mixture-of-experts #generative-adversarial-network #speech-model #contrastive-learning #self-supervised #image-representation #image-processing #object-detection #pseudo-labeling #scene-text-detection #neural-architecture-search #data-sampling #long-tail #graph-representation #zero-shot #metric-learning #federated-learning #weight-matrix #low-rank #vision-transformer #computer-vision #normalizing-flow #invertible-neural-network #super-resolution #image-manipulation #thread-summarization #natural-language-processing #domain-adaptation #knowledge-distillation #scene-text #model-compression #semantic-segmentation #instance-segmentation #video-understanding #code-generation #graph-generation #image-translation #data-augmentation #model-pruning #signal-processing #text-generation #text-classification #music-representation #transfer-learning #link-prediction #counterfactual-learning #medical-imaging #acceleration #transformer #style-transfer #novel-view-synthesis #point-cloud #spiking-neural-network #optimization #multi-layer-perceptron #adversarial-training #visual-search #image-retrieval #negative-sampling #action-localization #weakly-supervised #data-compression #hypergraph #adversarial-attack #submodularity #active-learning #deblurring #object-tracking #pyramid-structure #loss-function #gradient-descent #generalization #bug-fix #orthogonality #explainability #saliency-mapping #information-theory #question-answering #knowledge-graph #robustness #limited-data #recommender-system #anomaly-detection #gaussian-discriminant-analysis #molecular-graph #video-processing