[2107.05624] End-to-end Multi-modal Video Temporal Grounding

Significance

Adding depth and flow to RGB improves video understanding

Keypoints

  • Propose a method for video temporal grounding incorporating depth and optical flow information
  • Demonstrate performance of the proposed method by experiments

Review

Background

Text-guided video temporal grounding refers to a task which aims to localize the segment of a video based on a given text query. Most previous methods consider only the RGB image frames from the video, which cannot efficiently capture action or complex scenes. This study tries to address this issue by additionally incorporating optical flow and depth information of the video.

Keypoints

Propose a method for video temporal grounding incorporating depth and optical flow information

210713-1 Schematic illustration of the proposed method

The idea is to first extract rgb, depth, and optical flow features with encoders $E_{r}$, $E_{d}$ and $E_{f}$. The local-global interaction modules (LGI) further incorporate the textual feature into each visual modality to derive multi-modal features $M_{r}$, $M_{d}$, and $M_{f}$. Interaction of these multi-modal features are modelled with co-attentional Transformer (Co-TRM) which the query and the key/value pairs are interleaved between the RGB feature and the other features. The final output $[t_{s}, t_{e}]$ are estimated by feeding each feature into a fully-connected (FC) layer, followed by a regression module (REG).

To further facilitate the multi-modal training, the authors add SimCLR based contrastive loss to the final objective where the positive and negative samples are defined as the actions from the same and different categories, respectively.

Demonstrate performance of the proposed method by experiments

The proposed method (DRFT) is compared with baseline methods including CTRL, RWM, MAN, TripNet, PfTML-GA, DRN, and LGI on the Charades-STA and ActivityNet Captions datasets. 210713-2 Comparative study of the proposed method It can be seen that the performance of the proposed method outperforms baseline methods in terms of Recall at temporal Intersection over Union thresholds (R@IoU) and the mean temporal IoU.

210713-3 Ablation study results From the ablation studies, it can be seen that the proposed method is especially degraded in performance when the transformer is not present.

210713-4 Sample results from the Charades-STA dataset Some sample results further demonstrate that fully incorporating the multi-modal video features can improve the performance when compared to single-stream RGB baseline video grounding model.

Related

Share

#language-model #responsible-ai #privacy-preserving #scaling #mixture-of-experts #image-generation #diffusion-model #generative-adversarial-network #speech-model #multi-modal #contrastive-learning #self-supervised #image-representation #image-processing #object-detection #pseudo-labeling #scene-text-detection #neural-architecture-search #notice #data-sampling #long-tail #graph-neural-network #graph-representation #zero-shot #metric-learning #federated-learning #weight-matrix #low-rank #vision-transformer #computer-vision #normalizing-flow #invertible-neural-network #super-resolution #image-manipulation #thread-summarization #natural-language-processing #domain-adaptation #knowledge-distillation #scene-text #model-compression #semantic-segmentation #instance-segmentation #video-understanding #code-generation #graph-generation #image-translation #data-augmentation #model-pruning #signal-processing #text-generation #text-classification #music-representation #transfer-learning #link-prediction #counterfactual-learning #medical-imaging #acceleration #transformer #style-transfer #novel-view-synthesis #point-cloud #spiking-neural-network #optimization #multi-layer-perceptron #adversarial-training #visual-search #image-retrieval #negative-sampling #action-localization #weakly-supervised #data-compression #hypergraph #adversarial-attack #submodularity #active-learning #deblurring #object-tracking #pyramid-structure #loss-function #gradient-descent #generalization #bug-fix #orthogonality #explainability #saliency-mapping #information-theory #question-answering #knowledge-graph #robustness #limited-data #recommender-system #anomaly-detection #gaussian-discriminant-analysis #molecular-graph #video-processing