In drug discovery, rapid and accurate prediction of protein-ligand binding affinities is a pivotal task for lead optimization with acceptable on-target potency as well as pharmacological efficacy. Furthermore, researchers hope high correlation between a docking score and a pose with key interactive residues, though scoring functions as a free energy surrogate of a protein-ligand complex have failed to provide the collinearity. Recently, various machine learning or deep learning methods have been proposed to overcome the drawback of scoring functions. Despite their high accuracy, their featurization process is complex and requires high cost for its interpretation (less compatible for human recognition). Here, we propose SMPLIP-Score (Substructural Molecular and Protein-Ligand Interaction Pattern Score), a simple interpretable predictor of the absolute binding affinity. Our simple featurization embedded the interaction fingerprint pattern on the ligand-binding site environment and molecular fragments of ligands into an input vectorized matrix for learning layers (random forest or deep neural network). Despite lower complexity than state-of-the-art models, SMPLIP-Score achieved comparable performance, a Pearson’s correlation coefficient up to 0.80 and a RMSE up to 1.18 in pK units on several benchmark datasets (PDBbind v.2015, Astex Diverse Set, CSAR NRC HiQ, FEP, PDBbind NMR, and CASF-2016). For this model, generality, predictive power, ranking power, and robustness also were examined with direct interpretation of feature matrices for specific targets.