Handwriting recognition has always been one of the most important and challenging issues in pattern recognition and machine learning due to its diverse applications. In this paper, we investigate the problem of Arabic offline handwritten text recognition. Existing approaches mainly use a combination of convolutional and recurrent layers for image understanding and connectionist temporal classification for text generation. Due to the sequential nature of recurrent neural networks, these methods suffer from a lack of parallelization. In addition, since these models cannot model linguistic rules, an external language model is often used in the post-processing stage to increase accuracy. To address these problems, we consider two different architectures, namely the Transformer Transducer and the standard sequence-to-sequence Transformer, and compare them in terms of accuracy and performance. Our approach can model language dependencies and relies only on the attention mechanism, making it more parallelizable and less complex. We adopt pre-trained Transformers for both image understanding and language modeling. Evaluation results on the Arabic KHATT dataset show that our proposed method outperforms the current state-of-the-art on Arabic handwritten text recognition task.