The vision-based smart driving technologies to increase road safety are the popular research topics for modern automobile industries. The development of precise moving object detection with continuously tracking capability is the most important technology for such purpose nowadays. In this paper, we propose an improved object detection system, which combines a selected object detector with its enhancements and long short term memory (LSTM) modules, to improve the detection performance for smart driving systems. First, from a selected object detector, we combine all vehicle classes and bypassing low-level features to improve its detection performance. After the spatial association of the detected objects, the outputs of the improved objector are then fed into the proposed double-layer LSTM (dLSTM) modules to successfully improve the detection of the vehicles in various conditions, including the newly-appeared, the detected and the gradually-disappearing vehicles. With stage-by-stage evaluations, the experimental results show that the proposed vehicle detection system with the improved procedures and dLSTM modules can precisely detect the vehicles without increasing computations.