A smooth evolution of 16×100Gbps hybrid mode division multiplexing (MDM)/coherent-next generation passive optical network (C-NGPON) system based on wavelength complement coding is proposed. Probabilistically shaped quadrature amplitude modulation (PS-QAM) modulation established alone as the key solution to embrace advanced coherent transponders. The simulation results show that the proposed hybrid MDM/C-NGPON system can offers a high split ratio of 512 in uplink and 256 in downlink as compared to conventional passive optical networks (PONs). For legacy PON signal coexistence with PON signals as well as complement signal the minimum received power of -19dBm for uplink and − 9dBm for downlink can be obtained effectively. Also, optical to signal noise ratio (OSNR) of 20dB can be achieved forPS-512QAM as compared to uniform-shaped 128QAM at 10KHz laser linewidth and it also offers power penalty of 0.5dBunder normalized generalized mutual information (NGMI) limit of 0.857. In addition, the system offers a long-haul transmission distance of 250km considering a shaping factor of 4.7 in uplink and 4.8 in downlink, at NGMI limit. Besides this, the system throughput upto 800Gbps for uplink and 700Gbps for downlink can also be obtained successfully. Finally, it is shown that proposed work shows superiority over other existing systems.