The machining difficulty of TC4 titanium alloy is higher than that of other metal alloys because of its poor thermal conductivity, high chemical reactivity and so on. Under this background, a system of longitudinal-torsional ultrasonic grinding and nanofluid minimum quantity lubrication (LTUG&NMQL) was established to achieve the high-quality machining of titanium alloy, and its special machining mechanism is firstly analyzed, then the surface residual stress (SRS) model under LTUG&NMQL was built. The theoretical analysis shows that the total residual stress is related to the grinding arc length of single abrasive grain in longitudinal-torsional ultrasonic grinding (lc), the average contact area (sm), the friction coefficient (µ) and the heat flux density of workpiece (qw), especially, under LTUG&NMQL condition, lc is changed by ultrasonic vibration, and sm, µ and qw are altered by nanofluid minimum quantity lubrication. Subsequently, the SRS values are different from those of common grinding. Finally, the grinding experiments were carried out to investigate the influences of LTUG&NMQL on the SRC of titanium alloy, and the experimental results showed that the surface compressive residual stress of titanium alloy under LTUG&NMQL was higher than that under common grinding and longitudinal-torsional ultrasonic grinding when the same machining parameters were used, and it increased with the flow rates of nanofluids and the ultrasonic amplitudes, which showed that LTUG&NMQL may improve the service performance of the parts. This study provides a theoretical foundation and technical support for the high-performance machining of TC4 titanium alloy.