Adopting Convolutional Neural Networks (CNNs) in daily routine of primary diagnosis requires not only near-perfect precision, but also a sufficient degree of transparency and explainability of the decision making. With physicians being accountable for the diagnosis, it is fundamental that CNNs provide a clear interpretation of their learning paradigm, ensuring that relevant pathology features are being considered. Building on top of successfully existing techniques such as multi-task learning, domain adversarial training and concept-based interpretability, this paper addresses the challenge of introducing diagnostic factors in the training objectives. Here we show that our architecture, by learning end-to-end an uncertainty-based weighting combination of multi-task and adversarial losses, is encouraged to focus on pathology features such as density and pleomorphism of nuclei, e.g. variations in size and appearance, while discarding misleading features such as staining differences. Our results on the classification of tumor in breast lymph node tissue scans show significantly improved generalization, with best average AUC 0.89 (0.01) against the baseline AUC 0.86 (0.005). This result is a starting point towards building interpretable multi-task architectures that are robust to data heterogeneity. Our code is available at https://bit.ly/356yQ2u.