Mamba: Linear-Time Sequence Modeling with Selective State Spaces

Mamba is a structured State Space Model generalizes the step size in descretization, and the ABCD matrices to be input dependent. In doing so, it loses the property that an SSSM is also convolution, but the use of modern GPUs in an effective way helps counter the loss in performance that this causes.

Better scaling law?

Initial experiments upto 2.8B parameters suggest that Mamba has a superior scaling law to transformers, and it can handle longer context widths while avoiding attention's quadratic complexity.