Atrial fibrillation (AF) and atrial flutter (AFL) are the most common and easily confused arrhythmias. However, the existing AF and AFL detection approaches depend on visual examination of electrocardiogram (ECG), which is taxing and time-consuming. Motivated by state refinement module for LSTM (SR-LSTM), we investigate an improved SR-LSTM (ISR-LSTM) that is capable of adaptively refining the current states of sample points in ECG leveraging a sample point-wise attention and a representation gate mechanism for automated AF and AFL signals detection. To confirm the effectiveness, we conducted ablation experiments across two public databases. The results demonstrate the effectiveness of ISR-LSTM while achieving a significant improvement than state-of-the-art algorithms with an accuracy of 99.2% and 98.5%. To our knowledge, ISR-LSTM is the first work that integrates the interaction in ECG into deep network frame providing interpretability.