In sectors such as finance and healthcare, where data governance is subject to rigorous regulatory requirements, the exchange and utilization of data are particularly challenging. Federated Learning (FL) has risen as a pioneering distributed machine learning paradigm that enables collaborative model training across multiple institutions while maintaining data decentralization. This approach inherently heightens data privacy by sharing only model weights, rather than raw data. Despite its advantages, FL is vulnerable to adversarial threats, particularly poisoning attacks during model aggregation, a process typically managed by a central server. To counteract the vulnerabilities of traditional FL frameworks, Byzantine-robust federated learning (BRFL) systems have been introduced, which rely on robust aggregation rules to mitigate the impact of malicious attacks. However, in these systems, neural network models still possess the capacity to memorize and potentially expose individual training instances inadvertently. This presents a significant privacy risk, as attackers could reconstruct private data by leveraging the information contained in the model itself. Existing solutions fall short of providing a viable, privacy-preserving BRFL system that is both completely secure against information leakage and computationally efficient. To address these concerns, we propose Lancelot, an innovative and computationally efficient BRFL framework that employs fully homomorphic encryption (FHE) to safeguard against malicious client activities while preserving data privacy. Lancelot features a novel interactive sorting mechanism called masked-based encrypted sorting. This method successfully circumvents the multiplication depth limitations of ciphertext, ensuring zero information leakage. Furthermore, we incorporate cryptographic enhancements, such as Lazy Relinearization and Dynamic Hoisting, alongside GPU hardware acceleration, to achieve a level of computational efficiency that makes Lancelot a viable option for practical implementation. Our extensive testing, including medical imaging diagnostics and widely used public image datasets, demonstrates that Lancelot significantly outperforms existing methods, offering more than a twenty-fold increase in processing speed while maintaining data privacy. The Lancelot framework thus stands as a potent solution to the pressing issue of privacy in secure, multi-centric scientific collaborations, paving the way for safer and more efficient federated learning applications.