Maximization and restoration: Action segmentation through dilation passing and temporal reconstruction

• We propose a divide-and-conquer method that first maximizes frame accuracy and then reconstructs the features to reduce over-segmentation. • Dilation passing network propagates long- and short-range features enabling better understanding of the relation between frames. • Temporal reconstruction network uses a convolutional encoder-decoder to capture local context for temporal consistency among frames. • Our model achieves meaningful results over the state-of-the-art models on three challenging datasets. Action segmentation aims to split videos into segments of different actions. Recent work focuses on dealing with long-range dependencies of long, untrimmed videos, but still suffers from over-segmentation and performance saturation due to increased model complexity. This paper addresses the aforementioned issues through a divide-and-conquer strategy that first maximizes the frame-wise classification accuracy of the model and then reduces the over-segmentation errors. This strategy is implemented with the Dilation Passing and Reconstruction Network, composed of the Dilation Passing Network, which primarily aims to increase accuracy by propagating information of different dilations, and the Temporal Reconstruction Network, which reduces over-segmentation errors by temporally encoding and decoding the output features from the Dilation Passing Network. We also propose a weighted temporal mean squared error loss that further reduces over-segmentation. Through evaluations on the 50Salads, GTEA, and Breakfast datasets, we show that our model achieves significant results compared to existing state-of-the-art models.