Construction and Classification Prediction of Risk Assessment Iindicators for Water Environment Treatment PPP Projects

doi:10.21203/rs.3.rs-2845690/v1

Download PDF

Research Article

Construction and Classification Prediction of Risk Assessment Iindicators for Water Environment Treatment PPP Projects

https://doi.org/10.21203/rs.3.rs-2845690/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Water pollution control is crucial for ecological environmental safety and sustainable socio-economic development. Public Private Partnership (PPP) collaboration is an important approach for water pollution control, but it faces numerous risks. Accurately assessing and predicting these risks is essential for ensuring effective water pollution management. This study aims to develop an effective risk classification prediction model for water environment treatment PPP projects, addressing the limitations of traditional methods. First, based on the relevant research on the risk assessment system for water environment treatment PPP projects, a risk data feature set of water environment treatment PPP projects consisting of four subsystems, namely, natural environment, ecological environment, socio-economic, and engineering entity, is proposed. Second, the association between different feature indicators and project risk levels is analyzed from a statistical perspective, and the contribution value of risk features is obtained. Then, an ensemble learning model based on Stack-ing is established to predict the risks of water environment treatment PPP projects. To improve the model's performance, a weighted voting mechanism is designed by introducing weight factors to adjust the relative importance of base learners during the voting process, allowing the model to better exploit the differences between base learners and improve prediction accuracy. Finally, an empirical analysis is conducted on the Phase I project of the comprehensive management of the water environment system in the central urban area of Jiujiang City, China, verifying the effectiveness and accuracy of the risk assessment system and evaluation model constructed in this study. Experimental results show that the constructed Water Environment Treatment Project Risk Support Vector Machine (WETPR-SVM) model outper-forms other traditional single machine learning classification models in terms of accuracy, macro-average precision, macro-average recall, and macro-average value, providing an effective method for risk classification prediction of water environment treatment PPP projects.

Water environment treatment

PPP projects

Government

Risk classification

Ensemble learning

Any surface or groundwater body's quality is the result of a combination of natural and human processes. The deterioration of water quality has become a global concern as population growth, industrial and agricultural activities, and the threat of significant changes to the hydrological cycle due to climate change intensify (Li et al.2021;Su et al. 2022) .Water environment treatment is essential for protecting water resources, preserving ecological equilibrium, and promoting sustainable economic growth. Water environment treatment projects are essential for advancing water environment treatment and are frequently implemented through Public-Private Partnerships (PPP), which offer benefits such as relieving government fiscal pressures and effectively enhancing management efficiency, and are therefore gaining increasing attention.

In recent years, China's investment in water environment treatment PPP projects has been increasing annually. However, from a practical standpoint, many of these projects have failed to achieve their objectives due to inadequate risk management (Su et al.2022; Wang et al.2021). On one hand, numerous risk factors are involved in the construction and maintenance of water environment treatment projects, including local climate, hydrology, geology, flora and fauna, and socioeconomic environment. These factors present high levels of uncertainty and danger, leading to significant challenges in project risk management (Lu et al.2017). On the other hand, the primary motivation for social capital to participate in ecological protection and restoration is driven by interest. That is, with the support of government policies and considering the market prospects and economic benefits of ecological restoration, investments are made to generate profits. In contrast to social capital, governments place greater emphasis on the subsequent environmental management effects of projects to maximize social and public benefits, while private investors pursue short-term profit maximization without considering long-term operations (Tang et al.2021;Wang et al.2019). This conflict of interest often leads private investors to engage in opportunistic behavior, such as providing low-quality products or services, resulting in insufficient treatment of polluted rivers. When the government's project risk management is lacking, these behaviors are more likely to occur, posing potential threats to public safety and the sustainability of society. These issues fundamentally stem from the overlapping attributes of PPP projects and water environment treatment. Decision-makers often overlook the public welfare and public attributes of water environment treatment projects during risk management, and the main creators and participants of these attributes are the government. Public-private collaboration is a process of complementing resources and weighing the pros and cons (Demirel et al.2022). Since water environment treatment PPP projects possess public welfare and public attributes, they cannot generate ideal returns. Social capital is less willing to invest, and due to insufficient financial funds, the government has become one of the main bearers of risk in water environment treatment PPP projects. However, risk management from the government's perspective must consider public risks more, which is also one of the core objectives of water environment treatment projects. In existing research, there is little discussion from the perspective of government management of water environment treatment PPP projects, and even less analysis of the interrelationships between risk indicators (Xue et al.2020;Liu et al.2018). Therefore, it is urgent to construct risk assessment and prediction models for water environment treatment PPP projects that consider public welfare and public attributes and analyze risk factors of water environment treatment PPP projects from the government management perspective, thus providing more effective guidance for risk control in water environment treatment PPP projects.

Water environment treatment is the process of transitioning between stable states in water environment systems. It is clear that merely managing environmental risks during the initial and final phases of water environment treatment projects cannot truly solve the problem. It is essential to adhere to the natural ecological characteristics of water environment systems, to consider time as a significant factor in water environment treatment projects, and to employ appropriate methods for targeted risk classification and prediction.In fact, accurate risk forecasting is a crucial factor in determining the success or failure of PPP project risk management. General methods for risk prediction in PPP projects can be categorized as qualitative, e.g., literature reviews, case studies and questionnaires (Li et al.2022); Semi-quantitative, e.g., analytical hierarchy process, game theory, and fuzzy comprehensive evaluation (Zhang et al.2021); Guantitative, e.g., artificial neural networks and value for money (Wang et al.2021). Expert judgment data is one of the most frequently employed data sources in PPP risk management. Nonetheless, the subjectivity and ambiguity inherent in this data resource pose significant challenges to the accuracy of risk analysis and assessment, and there is no unified standard for judgment in place (Owolabi et al.2020). The complexity and specificity of public-private partnership (PPP) projects for water environment treatment generate numerous and diverse data sources, necessitating extensive monitoring, sampling, and analysis to support research. This situation, which is characterized by data from multiple sources and heterogeneity, poses significant challenges to project risk management. For instance, water environment treatment data collected from cascading hydropower stations in the lower reaches of the Jinsha River include soil and water conservation monitoring data, water environment monitoring data, acoustic environment monitoring data, and terrestrial and aquatic ecological monitoring data (Mokhtar et al.2020). Traditional PPP project risk assessment and prediction methods that rely on expert survey questionnaires are characterized by their complexity, low results' credibility, and inapplicability to large-scale multi-source heterogeneous datasets. They are especially unsuitable for public-private partnership (PPP) projects involving extensive multi-source heterogeneous data for water environment treatment. Machine learning, a potential solution to these problems, has been widely applied in the PPP domain (Zheng et al.2021). Owolabi et al. used machine learning models including regression trees, support vector machines, and deep neural networks to predict potential delays in PPP projects (Owolabi et al.2020). Despite the fact that a small number of researchers have examined the classification of construction project risks using techniques such as machine learning, the vast majority have relied on single machine learning algorithms for risk classification, and the accuracy and generalizability of single algorithms on different datasets require improvement (Huang et al.2022). Consequently, a number of researchers have utilized ensemble combination models to investigate the processing of multi-source heterogeneous data, yielding new insights. Chou et al. used ensemble machine learning models to foresee disputes in PPP projects before they occurred (Chou et al.2013).

This study focuses on the issue of government risk management in water environment treatment PPP projects, aiming to improve the success rate of such projects. Based on the multi-source heterogeneous characteristics of risk data in water environment treatment projects, this study designs a risk feature set from the government's perspective and discusses the specific indicators' contributions to project risk. An empirical analysis is conducted using the Phase I comprehensive water environment treatment project in Jiujiang's central urban area. To predict risks in water environment treatment PPP projects, this research employs ensemble learning techniques incorporating classifier combinations. The objective of this study is to expand the methods of engineering risk management, enhance the rationality, comprehensiveness, and accuracy of project risk prediction, and provide a warning guide for governments to manage risks in water environment treatment PPP projects. This research holds significant implications for improving the quality of environmental public goods and services supply, as well as enhancing the capability and efficiency of water pollution prevention and control.

2.1 Ensemble Learning Method Based on Stacking

Ensemble learning is a technique for integrating multiple distinct algorithms or the same algorithm with varied parameters into a single model. Boosting, Bagging, and Stacking are methods of ensemble learning that are frequently employed (Wang et al.2023). Stacking-based ensemble learning, which generates a new model by combining the prediction information from multiple models via self-organizing sampling or cross-validation, has superior performance compared to a single algorithm. Therefore, the Stacking method is used to build an ensemble learning model in this study (Chung et al.2023).

Selecting precise and diverse machine learning classifiers is required during the construction of the base classifier. In this study, we select diverse algorithms from six machine learning algorithms with superior classification performance, namely K-nearest neighbors (KNN), classification and regression tree (CART), linear discriminant analysis (LDA), Nave Bayes classifier (NB), Support Vector Machine (SVM), and Water Environment Treatment Project Risk Support Vector Machine (WETPR-SVM) classifier for the base classifier ensemble. The logistic regression algorithm, which is commonly used in Stacking ensemble models, is used to select the meta-classifier.

As shown in Fig. 1, this study compares the combination model based on Stacking ensemble learning to traditional single models. First, the risk feature set of Water environment treatment PPP projects is determined, then risk data are introduced, labeled, and preprocessed, and the dataset is divided into training and test sets. Then, classification is carried out utilizing a support vector machine classifier, and data fitting is carried out utilizing various kernel functions. The final prediction results are obtained via an enhanced voting mechanism, and the individual prediction models are combined via the ensemble learning mechanism to form a comprehensive prediction model that serves as the final risk grading prediction model for water environment treatment PPP projects. The model is then evaluated using commonly employed model evaluation algorithms in machine learning, and the training model's performance is continuously enhanced by training effects.

2.2 Model Interpretability Method

Due to the limited interpretability of machine learning in comparison to conventional generalized linear models, even though some models can measure feature importance, the specific effect scales of features are not intuitive. This issue is resolved by the emergence of the SHAP (Shapley Additive Explanations) model explanation. The SHAP explanation model interprets features by calculating each feature's contribution to the predicted value, and the values employed (SHAP values) can quantitatively characterize each feature's contribution (Nordin et al.2023). The greater the SHAP value, the more the feature contributes to the predicted value (Li et al.2022). This study calculates the contribution of multiple risk features in water environment treatment PPP projects using the SHAP explanation model, which has good computational performance and intuitive characteristics.

As shown in Eq. (1), the SHAP value of a feature in the SHAP explanation model is the weighted sum of all possible feature value combinations.

$${\varphi }_{\text{j}}\left(\text{v}\text{a}\text{l}\right)=\sum _{S\subseteq \{1,\cdots ,p\}\setminus \left\{j\right\}} \frac{\left|S\right|!(p-|S|-1)!}{p!}(\text{v}\text{a}\text{l}(S\cup \left\{j\right\})-\text{v}\text{a}\text{l}(S\left)\right)$$

Where: $S$ is a subset of features used in the model, indicating that $j$ is not included in the set $S$; $p$ is the number of features; $val\left(S\right)$ is the prediction of feature values in set $S$; ${\varphi }_{\text{j}}$ represents the contribution of the $j$ feature to $val$.

2.3 Improved Voting Mechanism

Integrated learning, which combines multiple predictive models, often yields significantly improved accuracy and generalization performance compared to single predictive models. This combination can be achieved through voting mechanisms. However, risk prediction in water environment treatment PPP (Public-Private Partnership) projects is characterized by high complexity. Additionally, the presence of numerous missing and anomalous values in the data during certain periods may result in some base learners exhibiting higher accuracy or suitability for addressing specific issues. In such cases, traditional weighted voting strategies might not fully capitalize on the features and differences of each base learner, affecting the overall prediction accuracy.Therefore, this paper proposes a weighted voting scheme designed to better adapt to the uncertainty and complexity in risk prediction for water environment treatment PPP projects, as depicted in Fig. 2. ${W}_{1}\sim {W}_{\text{n}}$represent the predicted outcomes of each model, with values of 1 (indicating stability) or -1 (indicating instability). $W$ denotes the integrated result of the $n$ predictive models, calculated as $W={\sum }_{i=1}^{n}{\alpha }_{i}\ast {W}_{\text{i}}$. When $W>0$, the project is deemed stable; when $W<0$, the project is considered unstable; and when $W=0$, the project requires verification through alternative means or is conservatively categorized as unstable. The weighting factor ${\alpha }_{\text{i}}$ represents the relative importance of each base learner during the voting process and can be learned through the training dataset.

A feasible approach is to employ cross-validation to assess the performance of different base learners and allocate weights accordingly. More specifically, the accuracy scores of the base learners can be used as their weighting factors.

3.1 Data Collection and Analysis

The majority of research on risks in public-private partnership (PPP) projects focuses on risk identification and classification, risk analysis and evaluation, and risk allocation and management strategies (Wang et al.2018). Extensive research has been conducted over the past decade to investigate risk management issues in PPP projects, identifying various types of risks, such as financial, operational, political, and environmental risks (Xu et al.2010). Water environment systems are dynamic, complex, open systems with temporal, spatial, and volumetric variations. This complexity results in distinct techno-economic characteristics of water environment treatment PPP projects compared to purely commercial PPP projects, including strong quasi-public interest, high difficulty in integrating governance technologies, complex assessment of governance effects, and difficult project coordination and collaboration (An et al.2018). Current research difficulty is identifying risk factors in water environment treatment PPP projects.

Using keywords or subject terms in both Chinese and English, such as "PPP", "risk", and "water environment treatment", a combined search was conducted in databases such as CNKI, ISI Web of Science, and ScienceDirect to find relevant literature for risk factor analysis. This resulted in a preliminary list of risk factors for water environment treatment PPP projects, as detailed in Table 1 below. The existing literature mainly discusses risks caused by the government, risks caused by social capital, and risks generated by the external environment.Government-caused risks primarily stem from government involvement in project management, including tax adjustments (Li et al.2022; Liu et al.2018), government intervention and credit issues (Wang et al.2021; Cu et al.2019), and inadequacies in existing laws, regulations, and regulatory systems (Su et al.2022; Feng et al.2022). Social capital-caused risks mainly arise from actual project construction and operation, such as completion risks (Su et al.2022; Feng et al.2022), construction technology risks (Zhang et al.2021; Zhang et al.2021), contract change risks (Su et al.2022; Feng et al.2022; Li et al.2019), delay risks (Su et al.2022; Li et al.2022; Li et al.2019), cost overrun risks (Su et al.2022; El-Kholy et al.2021), insufficient project revenue risks (Su et al.2022; El-Kholy et al.2021), dispute and infringement risks (Wang et al.2019; Fu et al.2023; Chou et al.2013), and social capital change risks (El-Kholy et al.2021; Wang et al.2019). External environment risks refer to risks directly or indirectly caused by the external environment, including environmental damage risks (Su et al.2022; An et al.2018; Owolabi et al.2020), geological condition risks (Feng et al.2022; Cui et al.2019), social stability risks (Li et al.2021; Wang et al.2019), public satisfaction (Li et al.2020; Fu et al.2023), inflation risks (Zhang et al.2021; Wang et al.2018), and force majeure (Li et al.2018; Wang et al.2018).

Although these summarized risk indicators can provide some reference for this study, they mostly analyze the project itself and do not consider the different categories of risks that different project participants should bear. Moreover, many of them involve qualitative data, which is often difficult to obtain comprehensively in practice.

Table 1

Preliminary List of Risk Factors for Water environment treatment PPP projects.
Risk Type	Specific Risk Indicators
Government-induced	Tax adjustment risk
	Government intervention and credit issues
	Inadequate legal and regulatory frameworks
Social capital-induced	Quality completion risk
	Construction technology risk
	Contract change risk
	Schedule delay risk
	Operational cost overrun risk
	Project revenue shortfall risk
	Dispute and infringement risk
	Social capital change risk
External environment-induced	Environmental damage risk
	Geological condition risk
	Social stability risk
	Public opinion risk (public satisfaction)
	Inflation risk
	Force majeure risk (political, natural conditions)

The primary objectives of water environment treatment PPP projects, from the government's perspective, are to restore aquatic ecosystems, address water pollution issues, and maximize environmental, social, and economic benefits (Li et al.2022). From the perspective of government regulation, the public interest and public attributes of water environment treatment projects must be considered. Evidently, government departments bear the supervisory responsibility for project operation and ecological restoration, which is often overlooked or less emphasized by other stakeholders. Consequently, many risk incidents arise from this aspect (Li et al.2020). In light of these realities, this study includes continuous operation risks and natural ecological environment risks among the categories of risks that the government must prioritize in public-private partnership (PPP) projects for water environment treatment. In addition, discussing specific water environment treatment projects allows for the collection of as much subjective risk data as possible and the application of empirical analysis to validate the model's viability.

As a result, Jiujiang City, one of the first batch of demonstration cities for green development in China's Yangtze River Economic Belt, was chosen as the area of study. Jiujiang, the only city in Jiangxi Province located along the Yangtze River, has 152 kilometers of Yangtze River shoreline and two-thirds of the water surface and shoreline of Poyang Lake, China's largest freshwater lake. High-quality water resources are Jiujiang's most important ecological assets for high-quality development. In promoting the "Yangtze River Protection", Jiujiang is therefore saddled with significant responsibilities, heavy burdens, and numerous obstacles. This PPP-modeled project has a total investment of 76.99 billion yuan, prioritizes ecological and green development, and focuses on the management of the "Two Rivers" basin system. It is modeled after the Phase I project of the comprehensive water environment treatment in the central urban area of Jiujiang.

Adjustments were made to the preliminary list of risk factors for water environment treatment PPP projects using resources such as the Chinese government's public data platform, various monitoring stations in Jiujiang City, and joint bidding units. Five years' worth of project risk factor information was compiled, and the specific contents of various risk categories were categorized. This resulted in a set of twelve risk data features for water environment treatment PPP projects, covering natural environment, ecological environment, socio-economic, and project entity subsystems. Indicators of evaluation are detailed in Table 2 below. The natural environment and ecological environment subsystems primarily reflect the government's natural ecological environment regulatory risks, whereas the socio-economic subsystem takes into account the significant public impact of water environment treatment PPP projects. The project entity subsystem, on the other hand, takes into account the government's primary role in the supervision of the sustainable operation of PPP projects, excluding engineering risks that the government does not share, such as cost overrun risks and construction technology risks.

Table 2

Risk Data Feature Table for Water environment treatment PPP projects.
System Name	Risk Feature Name	Risk Feature-Related Evaluation Indicator Set
Natural Environment Subsystem	Water environment	Hydro-sediment, Water quality, Water temperature, Water level, sediment
	Acoustic environment	Noise
	Atmospheric environment	Dust, Exhaust emissions, Local climate
	Surface environment	Solid waste, Soil nutrients, Geology, Soil erosion, Soil salinization, Soil marshification, Landslides
Ecological Environment Subsystem	Terrestrial organisms	Terrestrial animal and plant growth risks
Ecological Environment Subsystem	Aquatic organisms	Safety risks of aquatic animals, Aquatic plants, Aquatic microorganisms
Socio-economic Subsystem	Livelihood security	Public satisfaction, Employment opportunities
Socio-economic Subsystem	Local economic development	Regional industry, Regional agriculture, Urban planning, Surrounding landscape, Regional economic risk
Project Entity Subsystem	Lel risksga	Dispute, Breach, Infringement risks, Planning, standards, and contract change risks
	Operational risks	Construction risks caused by social capital, Operation and maintenance management risks
	Financial risks	Interest rate change risk, Revenue shortfall risk, Social capital change risk
	Force majeure risks	Force majeure due to political and natural conditions

Using the equal interval method, the collected data were labeled with risk levels, including low, medium, higher, and high labels. There were collected a total of 927 risk data, including 12 risk features and 37 risk feature evaluation indicators. Among them, there were 789 low-level risk data, 86 medium-level risk data, 37 higher-level risk data, and 15 high-level risk data. The vast majority of the data had a low level of risk, indicating that the collected data belonged to a small sample of unbalanced data. Deep learning strategies exemplified by multilayer neural networks are unsuitable for this scenario (Tsai et al.2023; Abdoli et al.2023). Consequently, a conventional machine learning strategy was chosen for this research.

In order to analyze the relationships between various indicators and levels, the collected data risk was presented in the form of variable scatter matrix plots. In this study, a total of 29 indicator data were collected. Here, only four indicators have been selected: water quality, local industrial economy, local climate, and soil erosion. These indicators were represented by pairwise coordinates to illustrate the connection between the various risk levels and project risk characteristics. Figure 4 demonstrates the outcomes. Under the four risk levels, the distribution of the four indicators, water quality, local industrial economy, local climate, and soil erosion, is relatively concentrated. In other words, there is a correlation between various risk indicators, as well as a correlation between the indicators and the risk classification of water environment treatment PPP projects.

3.2 Risk Feature Contribution Analysis

By analyzing the contributions of different feature indicators, we can determine the level of influence of different features on the model, allowing us to better explain and adjust the model during its construction. Figure 5 illustrates the distribution of SHAP values for risk indicator feature values. It is evident that water environment risk, operation and maintenance management risk, and local economic development have a greater impact on the prediction of risk levels for water environment treatment PPP projects. In other words, the higher these three indicators are, the greater the probability that the project risk level will be high, which is consistent with the real-world scenario (Li et al.2022).From the government's perspective, the primary objective of water environment treatment PPP projects is to strengthen water environment governance to achieve ecological restoration. Clearly, water environment risk is the most influential indicator of project risk, and achieving this objective is contingent upon the project's ability to be effectively operated and maintained. Therefore, operation and maintenance management risk is a prerequisite for the project's sustainable implementation. Moreover, the ultimate objective of the government's implementation of water environment treatment PPP projects is to improve or promote local economic development, which is directly related to the well-being of the entire society. Given the ascending order of objectives, it is reasonable that the contribution value of local economic development risk is ranked last.

3.3 Dataset construction

Based on the determined risk feature set for water environment treatment PPP projects, we filled in the missing data values. The specific method begins with setting a threshold to determine whether a feature is missing or not. If the percentage of missing values for a feature exceeds this threshold, the feature is removed. In this study, the threshold for missing feature deletion is set at 80%. If the threshold is not exceeded, the KNN algorithm is used to locate the k nearest samples to the sample with the missing value, and the average value of their corresponding features.

To ensure model accuracy and eliminate the influence of dimensions, we standardized the original risk indicator data using a standardization algorithm (Wang et al.2019). Eq. (2) represents the formula, where x_i represents the ith evaluation indicator of the nth risk feature and x_std represents the data for the standardized risk evaluation indicator.

$${x}_{std}=\frac{{x}_{i}-\frac{1}{n}\sum _{i=1}^{n} {x}_{i}}{\sqrt{{\left({x}_{i}-\frac{1}{n}\sum _{i=1}^{n} {x}_{i}\right)}^{2}}}$$

After processing missing features and dimensionless treatment, the distribution of feature values is between 0 and 1. The dataset is divided into training and testing sets in a 7:3 ratio, and a support vector machine classifier is used to classify the data.

3.4 Model Construction and Training

The final performance of the Stacking ensemble learning model is largely determined by the accuracy and similarity of the base classifiers. A superior classifier based on ensemble learning should adhere to the "good but different" principle (Chung et al.2023). Initially, we conducted experiments with the scikit-learn machine learning library for Python on the Jupyter Notebook platform. Six classification models were independently developed using machine learning: KNN classifier, CART classifier, linear LDA classifier, NB classifier, SVM classifier, and WETPR-SVM classifier. Individually, we trained them on the training set using cross-validation, random search, and learning curves to determine the optimal hyperparameter combination.

The Support Vector Machine (SVM) classifier was used to perform hyperparameter tuning (Chou et al.2013). By adjusting multiple SVM parameters and comparing the use of linear kernel function (LinearSVM), Gaussian kernel function (RBFSVM), and polynomial kernel function (Sigmoid), the classification accuracy was continuously enhanced. The results are depicted in Fig. 6, and the accuracy of the test set is shown in Table 3.

Figure 6 and Table 3 reveal that the Gaussian kernel function (RBFSVM) achieved the highest accuracy on the test set in this experiment, with a value of 0.9043. Comparatively, the test set precisions of the linear kernel function (LinearSVM) and the polynomial kernel function (Sigmoid) were 0.8191 and 0.8297, respectively. These results suggest that the Gaussian kernel function (RBFSVM) provides superior classification performance for this problem. This may be due to the fact that the Gaussian kernel function can map the data to a higher-dimensional space, rendering the data linearly separable in the higher-dimensional space. Given that water environment governance PPP project risk classification issues may involve complex nonlinear relationships, the Gaussian kernel function may be more suited to addressing such issues.

Table 3

Test set accuracy of kernel functions.
Kernel function	LinearSVM	RBFSVM	Sigmoid
Test set accuracy	0.8191	0.9043	0.8297

3.5 Model Comparison and Evaluation

This study's ensemble learning model ultimately completes the four-classification task for water environment governance PPP project risk. Therefore, we use accuracy (Accuracy), macro-average precision (Macro_P), macro-average recall (Macro_R), and macro-average F1 score (Macro_F1) as four indicators to evaluate the performance of the model.

Accuracy is the ratio of the number of project risk samples correctly classified by the model to the total number of project risk samples, which reflects the overall classification accuracy of the model (Choubin et al.2023). The formula for calculation is depicted in Eq. (3):

$$\text{Accuracy =}\frac{TP+TN}{TP+PN+FP+FN}$$

The average precision across all classes is represented by the macro-average precision. Precision is the ratio of the number of risks correctly classified into a particular category to the number of risks actually classified into that category. Eq. (4) demonstrates its formula for calculating:

$${\text{ Macro}}_{-}P=\frac{1}{n}\sum _{i=1}^{n} {P}_{i}$$

The macro-average recall is the mean of all recall values across all classes. Recall is the proportion of risk samples correctly classified by the model for a particular project risk category relative to the total number of risk samples in that category. The formula for its calculation is shown in Eq. (5):

$${\text{ Macro}}_{-}R=\frac{1}{n}\sum _{i=1}^{n} {R}_{i}$$

To evaluate the performance of a classification model in practical applications, it is frequently necessary to consider the model's precision and recall in depth. As a result, the F1 score, which is the weighted harmonic average of the two, is used as an evaluation metric. The formula for calculating the macro F1 score, which represents the mean of the F1 scores for all classes, is shown in Eq. (6):

$${\text{ Macro}}_{-}F1=\frac{1}{n}\sum _{i=1}^{n} {F}_{i}$$

In Equations (3)–(6), $TP$represents the number of positive samples predicted as positive by the model; $FP$represents the number of negative samples predicted as positive by the model; $FN$ represents the number of positive samples predicted as negative by the model; $TN$ represents the number of negative samples predicted as negative by the model; $n$ represents the number of risk feature categories, and${P}_{i}$, ${R}_{i}$ and ${F}_{i}$ represent the precision, recall, and F1 scores of the model for different categories, respectively.

To compare the performance of different classification algorithms, we trained the dataset using multiple approaches and evaluated their effectiveness. LDA is the only linear algorithm among them; the others are nonlinear. The pertinent procedures are as follows: (1) divide the training set; (2) evaluate the algorithm models using 10-fold cross-validation; (3) generate six distinct models to predict new data; and (4) compare classification accuracy. The WETPR-SVM model achieves the highest Accuracy, Macro_P, Macro_R, and Macro_F1 scores, which are 0.9025, 0.9055, 0.9026, and 0.9021, respectively, as shown in Table 4. This indicates that the WETPR-SVM model constructed in this study outperforms traditional single machine learning classification models in terms of overall performance. It is able to solve the classification problem of water environment governance PPP project risk with greater accuracy and generalizability, and it has a greater capacity for classification. Therefore, WETPR-SVM is selected as the optimal model for predicting the risk classification of water environment governance public-private partnership projects.

Table 4

Performance evaluation of prediction models.
Classifier	Accuracy	Macro_P	Macro_R	Macro_F1
KNN	0.8532	0.8561	0.8537	0.853
CART	0.8251	0.8252	0.8249	0.8253
LDA	0.8469	0.8471	0.847	0.8469
NB	0.8778	0.8781	0.8779	0.8775
SVM	0.8467	0.8465	0.8469	0.8462
WETPR-SVM	0.9025	0.9055	0.9026	0.9021

Comparing box plot 7 reveals that the range of prediction accuracy for the WETPR-SVM model is the smallest at 0.07, indicating relatively high stability and consistent performance across various datasets. In contrast, the difference between the upper and lower quartiles of prediction accuracy for the NB model is 0.2636, representing the greatest variation in prediction accuracy, which may be attributable to its inability to effectively manage multi-source heterogeneous data, missing values, and class imbalance issues. The lowest lower bound of the median prediction accuracy for the CART model is 0.8251, which may be because it is a classification algorithm based on a decision tree that is sensitive to noisy data and overfitting issues. These results indicate that the WETPR-SVM model is superior for predicting the risk classification of water environment treatment PPP projects.

In conclusion, the ensemble learning-based approach proposed in this study can better utilize the benefits of various machine learning algorithms, overcome the limitations of single algorithms in dealing with multi-source heterogeneous data, missing values, and class imbalance issues, and improve prediction accuracy and generalization ability.

In this study, we proposed an effective method for risk assessment and prediction of water environment treatment PPP projects based on ensemble learning. First, we proposed a water environment PPP project risk data feature set consisting of four subsystems: natural environment, ecological environment, socio-economic environment, and engineering entity. Second, using statistical analysis, we evaluated the association between different feature indicators and project risk levels, and determined the contribution values of risk features. Then, using the Stacking concept, we developed a two-layer ensemble learning model for classifying and predicting risks in water environment treatment PPP projects. Finally, we conducted an empirical analysis on the first phase of the comprehensive water environment system governance project in the central urban area of Jiujiang City, demonstrating the high effectiveness and accuracy of the risk assessment system and evaluation model constructed in this study.

The empirical investigation yields the following conclusions:

(1) By evaluating the risks of water environment treatment PPP projects and summarizing previous research, we have compiled a preliminary list of risk factors for water environment treatment PPP projects, including risks caused by the government, social capital, and external environment. From the government's oversight perspective, we selected the final set of risk features for water environment treatment PPP projects. The natural environment and ecological environment subsystems reflect the government's regulatory role in natural ecological environments, the socio-economic subsystem reflects the significant impact of water environment treatment PPP projects on public life and work, while the engineering entity subsystem reflects the government's regulatory role in the sustainable operation of water environment governance projects.

(2) By analyzing the correlation between risk feature indicators and risk levels, and calculating the SHAP values of risk features, it can be seen that risk feature indicators have a certain correlation at different risk levels, and their contribution values to risk prediction also vary. Water environment risk, operational risk, and local economic development risk have higher contribution values. In recent years, water pollution in Jiujiang City has significantly impacted urban development, and improving the water environment is the primary objective of governance projects. Water environment risk is the most direct risk faced by the project, operational risk involves the project's sustainability, and local economic development risk reflects one of the key factors determining project success. These factors will be areas of focus for future risk prevention and response strategies for water environment treatment PPP projects in Jiujiang City, China. Relevant government departments can use this as a basis for improving the legal and policy system construction of water environment treatment PPP projects.

(3) To address the issue of missing and abnormal values in risk data for water environment treatment PPP projects during certain periods, we innovatively designed a weighted voting mechanism. By introducing weight factors to adjust the relative importance of base learners in the voting process, the model can better utilize the differences between base learners, contributing to improved risk prediction accuracy for water environment treatment PPP projects.

(4) In terms of risk classification prediction models for water environment treatment PPP projects, the Accuracy, Macro_P, Macro_R, and Macro_F1 of the WETPR-SVM ensemble learning model developed in this research are 0.9025, 0.9055, 0.9026, and 0.9021, respectively. This model avoids the subjectivity and ambiguity of traditional expert-based scoring methods and achieves higher prediction accuracy compared to conventional machine learning models.

There are still many shortcomings in this study, and the following suggestions are made for future work. On the one hand, this study only analyzes one water environment governance project. In the next step, more actual cases will be collected to improve the risk feature set and further validate and optimize the prediction model. On the other hand, The ensemble learning model designed in this study is not sufficiently diverse in combination types. In the next step, we will explore the application of other advanced machine learning and artificial intelligence technologies in risk assessment and prediction for water environment treatment PPP projects to enhance prediction.

Li H, Wang F, Zhang C, Wang L, An X, Dong G (2021) Sustainable supplier selection for water environment treatment public-private partnership projects. Journal of Cleaner Production 324 : 129218. doi:10.1016/j.jclepro.2021.129218
Su L, Cao Y, Li H, Zhang C (2022) Water environment treatment PPP projects optimal payment mechanism based on multi-stage dynamic programming model. Engineering Construction and Architectural Management. doi:10.1108/ecam-04-2022-0291
Su L, Cao Y (2022) Performance monitoring and evaluation of water environment treatment PPP projects with multi-source heterogeneous information. Frontiers in Environmental Science 10. doi:10.3389/fenvs.2022.1024701
Wang Y, Shao Z, Tiong RLK (2021) Data-Driven Prediction of Contract Failure of Public-Private Partnership Projects. Journal of Construction Engineering and Management 147 (8). doi:10.1061/(asce)co.1943-7862.0002124
Lu S, Shang Y, Li Y (2017) A research on the application of fuzzy iteration clustering in the water conservancy project. Journal of Cleaner Production 151:356-360. doi:10.1016/j.jclepro.2017.02.064
Li H, Liang M, Zhang C, Cao Y (2022) Risk Evaluation of Water Environmental Treatment PPP Projects Based on the Intuitionistic Fuzzy MULTIMOORA Improved FMEA Method. Systems 10 (5):163. doi:10.3390/systems10050163
Tang Y, Liu M, Zhang B (2021) Can public-private partnerships (PPPs) improve the environmental performance of urban sewage treatment? Journal of Environmental Management 291:112660. doi:10.1016/j.jenvman.2021.112660
Wang H, Liu Y, Xiong W, Song J (2019) The moderating role of governance environment on the relationship between risk allocation and private investment in PPP markets: Evidence from developing countries. International Journal of Project Management 37 (1):117-130. doi:10.1016/j.ijproman.2018.10.008
Demirel HC, Leendertse W, Volker L (2022) Mechanisms for protecting returns on private investments in public infrastructure projects. International Journal of Project Management 40 (3):155-166. doi:10.1016/j.ijproman.2021.11.008
Xue Y, Wang G (2020) Analyzing the evolution of cooperation among different parties in river water environment comprehensive treatment public-private partnership projects of China. Journal of Cleaner Production 270: 121118. doi:10.1016/j.jclepro.2020.121118
Liu JF, Xue XZ (2018) River Management for Local Governments in China: From Public to Private. International Journal of Environmental Research and Public Health 15 (10): 2174. doi:10.3390/ijerph15102174
Zhang Y, He N, Li Y, Chen Y, Wang L, Ran Y (2021) Risk Assessment of Water Environment Treatment PPP Projects Based on a Cloud Model. Discrete Dynamics in Nature and Society 2021. doi:10.1155/2021/7027990
Wang L, Zhang X (2018) Bayesian Analytics for Estimating Risk Probability in PPP Waste-to-Energy Projects. Journal of Management in Engineering 34 (6): 4018047(1-13). doi:10.1061/(asce)me.1943-5479.0000658
Zheng X, Liu Y, Jiang J, Thomas LM, Su N (2021) Predicting the litigation outcome of PPP project disputes between public authority and private partner using an ensemble model. Journal of Business Economics and Management 22 (2):320-345. doi:10.3846/jbem.2021.13219
Mokhtar A, He H, Zhao H, Keo S, Bai C, Zhang C, Ma Y, Ibrahim A, Li Y, Li F, He W, Abdo AI, Zhou J (2020) Risks to water resources and development of a management strategy in the river basins of the Hengduan Mountains, Southwest China. Environmental Science-Water Research & Technology 6 (3):656-678. doi:10.1039/c9ew00883g
Owolabi HA, Bilal M, Oyedele LO, Alaka HA, Ajayi SO, Akinade OO (2020) Predicting Completion Risk in PPP Projects Using Big Data Analytics. Ieee Transactions on Engineering Management 67 (2):430-453. doi:10.1109/tem.2018.2876321
Huang IH, Chang M-J, Lin G-F (2022) An optimal integration of multiple machine learning techniques to real-time reservoir inflow forecasting. Stochastic Environmental Research and Risk Assessment 36 (6):1541-1561. doi:10.1007/s00477-021-02085-y
Chou JS, Lin C (2013) Predicting Disputes in Public-Private Partnership Projects: Classification and Ensemble Models. Journal of Computing in Civil Engineering 27 (1):51-60. doi:10.1061/(asce)cp.1943-5487.0000197
Wang S, Zhu J, Yin Y, Wang D, Cheng TCE, Wang Y (2023) Interpretable Multi-Modal Stacking-Based Ensemble Learning Method for Real Estate Appraisal. Ieee Transactions on Multimedia 25:315-328. doi:10.1109/tmm.2021.3126153
Chung D, Yun J, Lee J, Jeon Y (2023) Predictive model of employee attrition based on stacking ensemble learning. Expert Systems with Applications 215: 119364. doi:10.1016/j.eswa.2022.119364
Xu YL, Yeung JFY, Chan APC, Chan DWM, Wang SQ, Ke YJ (2010) Developing a risk assessment model for PPP projects in China - A fuzzy synthetic evaluation approach. Automation in Construction 19 (7):929-943. doi:10.1016/j.autcon.2010.06.006
Li H, Wang F, Lv L, Xia Q, Wang L (2022) Evolutionary game analysis of government supervision and private sector ecological technology innovation behavior for water environment treatment PPP projects on the basis of public participation. Canadian Journal of Civil Engineering 49 (1):41-51. doi:10.1139/cjce-2020-0640
Nordin N, Zainol Z, Noor MHM, Chan LF (2023) An explainable predictive model for suicide attempt risk using an ensemble learning and Shapley Additive Explanations (SHAP) approach. Asian Journal of Psychiatry 79: 103316. doi:10.1016/j.ajp.2022.103316
Li L, Qiao J, Yu G, Wang L, Li H-Y, Liao C, Zhu Z (2022) Interpretable tree-based ensemble model for predicting beach water quality. Water Research 211. doi:10.1016/j.watres.2022.118078
An X, Li H, Wang L, Wang Z, Ding J, Cao Y (2018) Compensation mechanism for urban water environment treatment PPP project in China. Journal of Cleaner Production 201:246-253. doi:10.1016/j.jclepro.2018.08.003
Cui CY, Wang JP, Liu Y, Coffey V (2019) Relationships among Value-for-Money Drivers of Public-Private Partnership Infrastructure Projects. Journal of Infrastructure Systems 25 (2). doi:10.1061/(asce)is.1943-555x.0000479
Feng J, Tang Y, Xue S, Zhang K (2022) Study on cooperative strategies of rural water environment governance PPP project between companies and farmers from the perspective of evolutionary game. Environment Development and Sustainability 24 (1):138-155. doi:10.1007/s10668-020-01148-4
Li Y, Wang X (2019) Using fuzzy analytic network process and ISM methods for risk as-sessment of public-private partnership: a China perspective. Journal of Civil Engineering and Management 25 (2):168-183. doi:10.3846/jcem.2019.8655
El-Kholy AM, Akal AY (2021) Assessing and allocating the financial viability risk factors in public-private partnership wastewater treatment plant projects. Engineering Construction and Architectural Management 28 (10):3014-3040. doi:10.1108/ecam-05-2020-0373
Fu L, Sun H, Fang Y, Xu K (2023) A systematic review of the public-private partnership literature published between 2012 and 2021. Journal of Civil Engineering and Management 29 (3):238-252. doi:10.3846/jcem.2023.17926
Li H, Lv L, Zuo J, Su L, Wang L, Yuan C (2020) Dynamic Reputation Incentive Mechanism for Urban Water Environment Treatment PPP Projects. Journal of Construction Engineering and Management 146 (8). doi:10.1061/(asce)co.1943-7862.0001879
Li H, Lv L, Zuo J, Bartsch K, Wang L, Xia Q (2020) Determinants of public satisfaction with an Urban Water environment treatment PPP project in Xuchang, China. Sustainable Cities and Society 60. doi:10.1016/j.scs.2020.102244
Wang Y, Cui P, Liu J (2018) Analysis of the risk-sharing ratio in PPP projects based on government minimum revenue guarantees. International Journal of Project Management 36 (6):899-909. doi:10.1016/j.ijproman.2018.01.007
Tsai C-A, Chang Y-J (2023) Efficient Selection of Gaussian Kernel SVM Parameters for Imbalanced Data. Genes 14 (3): 583. doi:10.3390/genes14030583
Abdoli M, Akbari M, Shahrabi J (2023) Bagging Supervised Autoencoder Classifier for credit scoring. Expert Systems with Applications 213:118991. doi:10.1016/j.eswa.2022.118991
Fang Z, Wang Y, Duan H, Niu R, Peng L (2022) Comparison of general kernel, multiple kernel, infinite ensemble and semi-supervised support vector machines for landslide susceptibility prediction. Stochastic Environmental Research and Risk Assessment 36 (10):3535-3556. doi:10.1007/s00477-022-02208-z
Choubin B, Hosseini FS, Rahmati O, Youshanloei MM, Jalali M (2023) Mapping of salty aeolian dust-source potential areas: Ensemble model or benchmark models? The Science of the total environment 877:163419. doi:10.1016/j.scitotenv.2023.163419

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Construction and Classification Prediction of Risk Assessment Iindicators for Water Environment Treatment PPP Projects

Status:

Version 1

Abstract

Figures

1 Introduction

2 Methods

2.1 Ensemble Learning Method Based on Stacking

2.2 Model Interpretability Method

2.3 Improved Voting Mechanism

3 Experiments And Analysis

3.1 Data Collection and Analysis

3.2 Risk Feature Contribution Analysis

3.3 Dataset construction

3.4 Model Construction and Training

3.5 Model Comparison and Evaluation

4 Conclusion

References

Additional Declarations

Status:

Version 1