The dynamics of neuron populations during diverse behaviours evolve on low-dimensional manifolds. However, it remains challenging to disentangle the role of manifold geometry and dynamics in encoding task variables. Here, we introduce an unsupervised geometric deep learning framework for representing non-linear dynamical systems based on statistical distributions of local dynamical features. Our method provides geometry-aware or geometry-agnostic representations for robustly comparing dynamical systems based on sparse measurements. Our representations are generalisable to compare computations across systems, interpretable to discover a geometric correspondence between neural dynamics and kinematics in a primate reaching task, and intrinsically encode temporal information to give rise to a decoding algorithm with state-of-the-art accuracy. Our results suggest that using the manifold structure over temporal information is important to develop better decoding algorithms and assimilate data across experiments.