To enhance the scalability and adaptability of on-chip diffractive optical neural network (DONN), this study introduces a novel network architecture for multi-task processing with in-memory optical computing. On-chip DONNs are celebrated for their compact structures and capability to transduce a substantial volume of parameters into optical form, yet they encounter challenges in scalability and multitasking. Leveraging the principles of transfer learning, this approach involves embedding the majority of parameters into fixed in-memory optical components and a minority into adjustable electrical components. Furthermore, with deep regression algorithm in modeling, a compact optical neural network achieve to handle diverse tasks. In this work, two ultra-compact in-memory diffraction-based chips with integration of more than 60,000 parameters/mm^2 were fabricated, employing deep neural network model and the hard parameter sharing algorithm, to perform multifaceted classification and regression tasks, respectively. The experimental results demonstrate that these chips achieve accuracies comparable to those of electrical networks while significantly reducing the power-intensive digital computation by 90%. Our work heralds strong potential for advancing in-memory optical computing frameworks and next generation of artificial intelligence platforms.