Accurately locating and segmenting polyps from colon images is crucial for the treatment of rectal cancer. However, the environment of rectal polyps is characterized by high noise, diverse sizes, complex boundaries, and a high demand for detailed information, making the task challenging. The acquisition and processing of polyp features are central to the research on polyp segmentation methods. This paper introduces an Adaptive Fusion Of Composite Attention Convolutional Neural Network (AFCNet) for polyp image segmentation. First, this work combine depth-wise separable convolutions and convolutional attention mechanisms with a multi-branch structure to better supplement missing details and unearth potential critical features. Secondly, we employ a multi-scale cross structure and an adaptive multi-scale feature harmonization module to balance the contribution of features at different levels, thus fully integrating information across scales to maximize the utilization of previously acquired features. Lastly, we propose an upsampling feature retrospective module to filter detailed information and use the concept of gating units to filter out interfering information. Extensive experiments on five publicly available polyp segmentation datasets demonstrate the effectiveness of our AFCNet in enhancing the accuracy of polyp segmentation.The experimental results indicate that AFCNet significantly outperforms state-of-the-art models. AFCNet is an effective polyp segmentation network, and due to its excellent generalization ability, it can also be applied to other medical image segmentation tasks.