MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipation

Syed Talal Wasim^1,2, Hamid Suleman^1,2, Olga Zatsarynna^1,2, Muzammal Naseer³, Juergen Gall^1,2

¹University of Bonn ²Lamarr Institute for Machine Learning and Artificial Intelligence ³Khalifa University
ICCV 2025

arXiv

The current state-of-the-art method employs standard Mamba blocks for stochastic dense action anticipation. In contrast, we introduce the MixMamba block, where each block dynamically selects an appropriate A matrix based on the input sequence.

Abstract

We present MixANT, a novel architecture for stochastic long-term dense anticipation of human activities. While recent State Space Models (SSMs) like Mamba have shown promise through input-dependent selectivity on three key parameters, the critical forget-gate (A matrix) controlling temporal memory remains static. We address this limitation by introducing a mixture of experts approach that dynamically selects contextually relevant A matrices based on input features, enhancing representational capacity without sacrificing computational efficiency. Extensive experiments on the 50Salads, Breakfast, and Assembly101 datasets demonstrate that MixANT consistently outperforms state-of-the-art methods across all evaluation settings. Our results highlight the importance of input-dependent forget-gate mechanisms for reliable prediction of human behavior in diverse real-world scenarios.

Overall task of stochastic long-term dense action anticipation with its inputs and outputs. For the Breakfast and 50Salads datasets, we extract I3D features from previous works, while for Assembly101 we use TSM-features. The features are then concatenated with zero padding to represent future frames. A Gaussian noise vector is sampled and added to the input tensor, which is then processed by the MixANT model to generate per-frame labels. Importantly, each noise vector produces a single sample, and multiple outputs are generated by repeating the process with different noise vectors while maintaining the same input sample.

Our architecture and its components. In the MixANT architecture (a), the first K₀ blocks are standard Mamba blocks and the rest (K-K₀) are MixMamba blocks (b). Each MixMamba block contains a bi-directional MixMamba layer (c). In contrast to the standard Mamba unit (e) where A is fixed for all input sequences, the MixMamba unit (d) contains a router that selects the relevant A matrix depending on the input sequence.

Poster

TODO

BibTeX

@inproceedings{wasim2024mixant,
        author    = {Syed Talal Wasim and Hamid Suleman and Olga Zatsarynna and Muzammal Naseer and Juergen Gall},
        title     = {MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipation},
        booktitle = {ICCV},
        year      = {2025},
    }