RNA velocity has provided a promising approach for inferring cellular state transition from single-cell RNA sequencing (scRNA-seq) data. Conventional RNA velocity models infer universal kinetics from all cells in a scRNA-seq experiment, resulting in unpredictable performance in the experiments with multi-stage and/or -lineage transition of cell states. Here, we present cellDancer, a scalable Deep Neural Network (DNN) framework, to locally infer velocity for each cell from its neighbors on gene space and then relay cell-dependent velocities of all cells. We showed that cellDancer is efficient to overcome the fundamental limitation of existing RNA velocity models in multi-stage transition during gastrulation erythroid maturation and the multi-lineage differentiation in hippocampus development. Moreover, cellDancer provides the cell-specific prediction of transcription, splicing, and degradation rates which illuminates mechanisms of transcriptome regulation.