A Maturity Model (MM) is needed to improve and operate globally. Even though a laboratory has been accredited for more than 20 years but no evidence to prove that the system was mature for competing internationally. Many commercial laboratories were business-oriented companies; MM was supposed to be used to measure the success of their processes and management styles. The improvement was based on 4M (Manpower, Method, Machine, Material) to enhance the operations and help understand the different maturity levels and how MM can benefit the organisation. The MM was developed based on analytes accurate certain score (AACS) for the ten most routine testing and system maturity scores (SMS), which cover overall proficiency testing performance and audit/assessment scores for the recent two cycles. The models do not fix inefficiencies; they can identify areas where organisations are not operating at standard and allow them to determine strategies to improve their operations and processes. This study was based on a commercial laboratory named ABC (not a real name), which was accredited in early 1990 and went through 8 cycles of assessments. The maturity model was applied to this laboratory and fell in the “leading” level with a total score above 80%. However, improvement is needed for better development, which are 1. Manpower: maintain competent staff by changing the management’s strategy. 2. Method: validate all the in-house methods following Analytical Laboratory Accreditation Criteria Committee (ALACC) guidelines. 3. Machine: apply good laboratory practice (GLP) for equipment sharing, specifically the equipment for a specific analyte. 4. Material: Metrological traceable reference material (RM) needed, involvement Proficiency Testing (PT) in all analytes.
Case Report
Development of Maturity Model on a Commercial Accredited Chemical Testing Laboratory in Malaysia
https://doi.org/10.21203/rs.3.rs-2490007/v1
This work is licensed under a CC BY 4.0 License
Version 1
posted
You are reading this latest preprint version
maturity model
analytes accurate certain scores
proficiency testing
audit/assessment scores
The Department of Standards Malaysia (Standards Malaysia) established the Skim Akreditasi Makmal Malaysia (SAMM) on the 15th of August 1990. The major goal of this scheme – SAMM is to provide a reliable accreditation service to testing and calibration laboratories so that test reports and calibration certifications supported by SAMM are recognised globally by International Laboratory Accreditation Co-operation (ILAC) and Asia Pacific Accreditation Cooperation (APAC). ISO/IEC 17025 is the main international standard for general requirements for the competence of testing and calibration laboratories. SAMM covers MS ISO/IEC 17025, SAMM Policy (SP), Specific Criteria (SC), Specific Technical Requirements (STR), Accreditation Policy (AP) and SAMM Circular. Implementing this SAMM in the chemical testing laboratory has made significant progress in the past ten years. Based on the Department of Standard Malaysia statistics, the total number of accreditation laboratories was 493 for the year 2010 and the year 2020 achieved 769 laboratories, which is a 35.9% increase. Implementing this laboratory quality management system (LQMS) does not guarantee an error-free laboratory. Still, it does yield a high-quality laboratory that detects errors and prevents them from reoccurring, especially in food analysis. By referring to the status report on the number of Malaysian laboratories accredited by Standards Malaysia, still in high percentage with a total of 773 as of March 2021, testing laboratories cover 73%.
This study aims to determine the maturity level of accredited laboratories in Malaysia, perhaps five years, ten years or twenty years accredited. While the issue of testing laboratories' quality assurance is not new, ISO implementation has become a trend. Accreditation is a quality assurance method that has been around for over a world. Expectations are required when all analysts pursue quality standards and processes in a new realm. To maintain laboratory quality, a maturity evaluation is needed. Thus, the top management is aware of their current status and moving forward to continue improving. With this condition, they have the need and intention for globalisation.
The most popular ISO standard for Quality Management Systems (QMS) certification is ISO 9001. ISO 9001 is the general standard focus in the management part, which covers six mandatory procedures: control of documents, control of records, internal audit, corrective action, preventive action, and control of non-conforming products [1]. Whereas ISO/IEC 17025 specifies the requirements for the competence of the testing and calibration laboratories. Some of the clauses included i) Competency of personnel, ii) Calibration and maintenance of equipment, iii) Metrological traceability, iv) Validation of methods, v) sampling, vi) Evaluation of measurement uncertainty, vii) Ensuring the validity of results [2]. The process to obtain the ISO/IEC 17025 is referred to as accreditation. Accreditation is the formal recognition that the laboratory is competent to carry out specific analyses [3]. The testing laboratories can apply for accreditation through Standard Malaysia, which is the only national standard body. The National Accreditation Body plays an essential role on the international stage and has been accepted as a signatory to the ILAC and APAC.
Accreditation for the laboratory would benefit the lab because it would boost its customer reputation. The evaluation's findings in the laboratory show that a QMS based on ISO/IEC 17025:2005 can be implemented. The accreditation process is a complicated and time-consuming procedure. Normally, it takes two years for laboratories to build up their systems [4]. When a testing laboratory can manage or manage flexibility, customer, production, value-oriented, and the primary task of its personnel, it is said to be operating effectively [5]. The application of ISO in the testing laboratory will further guarantee the quality of the analysis. The proposed method was as accurate and precise as and more sensitive than the Association of Official Agricultural Chemists (AOAC) method, which the method was validated according to the ISO/IEC 17025 standard [6]. Implementation of ISO was widespread from the food industry to the medical field. The laboratory demonstrated a higher level of quality compliance over the post-accreditation years. Overall, the ISO/IEC 17025 accreditation of the National Medicines Regulatory Authority Quality Control laboratory improved the laboratory's quality system and increased the reliability of test results [7].
Many companies have tried to gain a competitive advantage by providing high-quality services and implementing general quality management standards. No matter the type of laboratories, it was necessary to adopt the quality management method in their work, especially the application of ISO/IEC 17025, improving these laboratories' administrative and technical performance. Malaysia assigns a government agency responsible for halal regulation and certification matters, thus being the first country to hold a special position in the global halal market [8]. This is the main reason for establishing a halal laboratory to measure the quality of the national food supply. Food quality assurance programmes are compulsory to ensure the competency of laboratories to provide accurate results in food control systems consistently. This is sensible since it is much easier to assess the level of system functionality in firms where the system has been accredited to a widely established worldwide standard [9]. There are three essential elements: i) the use of established analytical methods, ii) accreditation involving third-party auditing, and iii) involvement in proficiency testing systems in most food analysis and regulation fields to assure laboratory quality [10].
The current accreditation still needs to be improved to maintain the laboratory's reputation. It is more trustworthy if it has a maturity index supporting the laboratory management system. Application of the maturity model for most management systems, such as Information technology, includes software development; productivity management which covers electric, electronic, and high-tech industries; banking management, maintenance management; manufacturing or service industry for business purposes; product specification and geometrical tolerancing as well. Process maturity can be defined as the degree to which a process is defined, managed, measured, and continuously improved. Referring to the [11], either to compare the level of maturity to other processes or to find development opportunities were the target working with the maturity model. The maturity models are a powerful factor since they assist organisations in better positioning themselves and identifying better transformation solutions in information technology management [12]. A maturity model is presented to help manufacturers determine the level of sustainable manufacturing activities such as materials, energy and water usage. All the input from the stakeholders was gathered for sustainability improvement to achieve high–level targets based on the maturity model [13]. Maturity models are decision-making aids because they support the understanding of the organisation’s current condition and encourage the implementation of measures that identify, implement and measure improvement actions; the model might lead to a better perspective to support the system [14]. The most famous maturity model is the Capability Maturity Model (CMM) from the Software Engineering Institute, which was first developed to measure the level of maturity for software development in 1987 [15]. Table 1 shows the list of common maturity models from the CMM year 1987 until Innovation Maturity Model (IMM) year 2018.
Based on the abovementioned list, the author decided to develop the maturity model for an accredited testing laboratory with five maturity levels. Similar to the service offered by the sales & marketing firm, MM of the accredited laboratory, as shown in Figure 2, was modified based on the sales maturity model of the Sales Cadence powered by the must-react system, as shown in Figure 1.
The description of the methodology will be explained by two factors: Analyte Accuracy Certain Score (AACS) and System Maturity Score (SMS).
Analyte Accuracy Certain Scores. AACS
AACS is a specific score influenced by 4M criteria (Manpower, Method, Machine and Material). Manpower is related to the analyst’s qualification, competency and experience; method refers to the validated or verified method; the machine depends on measuring instruments and the analytical instruments, which are well calibrated and maintained. Material is all kinds of reference standards used in the proper procedure and proves the metrological traceability, add on for the participation in the proficiency testing or interlaboratory comparison, consumable, reagent used, and sample receiving. A combination of these 4M will develop the formulation of AACS.
Score based on pre-defined criteria:
Manpower:
Focus on the analyst who runs the testing, divided into two sections: the interview section and record tracking. The interview section covers the technique of sample running, either divided by section or run by one person. Followed by pre-treatment sampling, sample preparation – extraction process, sample analysis by instrument, resulting from the interpretation and report writing, those based on qualification, which was qualified based on Sijil Peperiksaan Malaysia, SPM, Diploma, Degree, Master and Ph.D. The experience is divided into four categories: less than one year, 1-5 years, 6-10 years and more than ten years. Years of experience directly affect the performance of the analysis. Practice helps us increase our ability to access information rapidly and automatically; thus, practice makes perfect. If the personnel do the things daily, they must start doing the right thing to obtain good performance. The method was introduced in the right way for any analysis or experiments. The training module, competency method and competency test were considered for the record tracking section. The training module was important to understand how the laboratory trained personnel, was it trained by method approach, instruments or analyst? After training, competency is needed either before case handling or during case handling. A competency test is compulsory to ensure the validity of the results (clause 7.7.2); the period of test can be done within three months, six months or annually. The total score from this section is 28 points. It covers 25% overall of 4M.
Method:
The type of method influences the validation or verification study. The method can be divided into International, regional/national standards, reputable technical organisations’ methods, relevant scientific texts or journals to treat as the laboratory’s method, the method based on the specified by the equipment manufacturer or laboratory-developed or modified method. Method validation criteria based on selectivity, robustness, linearity/calibration model, working range, the limit of detection (LOD), the limit of quantitation (LOQ), precision, trueness and uncertainty, whereas for method verification based on precision, trueness, LOD, LOQ and uncertainty only. Ten methods were selected from the laboratory, which covers food preservatives, proximate food analysis, contaminants in food, mycotoxin, drug residue and food standard. The total score from this section is 16 points and covers 15% overall of 4M.
Machine:
Referring to measuring instruments such as balance, oven and thermometer; and analytical instruments such as gas chromatography (GC), liquid chromatography (LC), ultraviolet spectrometry (UVS), atomic absorption spectroscopy (AAS) and others. The status of the instrument, type of calibration and maintenance schedule were mandatory for observation. Calibrated record and maintained records of the analytical instruments and measuring equipment. The total score from this section is 33 points and covers 30% overall of 4M.
Material:
Reference material or certified reference material based on the type and purity of the substances. Add more proficiency testing or interlaboratory comparison based on the schemes' frequency, matrices, analytes and performance. The total score from this section is 30 points and covers 30% overall of 4M, which is the same as “Machine”. Both of them are equally important; under clause 6.4, equipment including but not limited to measuring instruments, software, measurement standards, reference materials, reference data, reagents, consumables or auxiliary apparatus that are required for the correct performance of laboratory activities and that can influence the results [2].
System Maturity Score, SMS
SMS will cover scores from overall audit findings and Overall Proficiency Testing (OPT) performance. Overall audit findings include internal audits and external assessments. Internal audits are thorough reviews by an independent team that arises in findings and recommendations. An internal audit assesses performance in relation to predetermined standards. Findings are the outcomes regarding each condition; they could be a compliance or non-compliance finding verified by the evidence. A laboratory's requirement to maintain a certain task is evaluated through external assessments. These fit the strict definition of an assessment, but the assessor's expertise and experience considerably more strongly influence the findings. Assessments also result in a finding, although this conclusion is mostly based on the assessor's judgement as opposed to a set of precise requirements, based still based on the standard used. The success of the auditing program depends significantly upon selecting the right person for the task, and not everyone can be a good auditor. A proficient assessor must communicate effectively, encourage others, write and explain, be a good listener, and use a good combination of appropriate language while questioning accredited lab staff. Overall, an assessor must be a good manager, planner and situation handler. Proper knowledge, experience, and additional qualities of the assessor/auditor [16]. On the other side, the OPT covers all the proficiency testing and inter-lab comparison, which are provided by accredited or non-accredited PT providers, either at local or international levels; pre-defined criteria were based on the performance of the Z score.
Score based on pre-defined criteria:
Audit/Assessment, A score:
The finding was collected from 2 cycles of assessment covering 1st cycle (2016-2018) and 2nd cycle (2019-2021) from internal audit and external assessment; criteria are defined based on the Non – Compliance Record, NCR category 1 to 3, and category 4 is observation. The scoring system applied as follows: -
- NCR Cat 1 with 4 points
- NCR Cat 2 with 3 points
- NCR Cat 3 with 2 points
- Observation Cat 4 with 1 point
Index A was obtained by comparing the marks of the current year with the previous year, continuing for two cycles, and then averaging all the marks. The formulation of the index A (I) is based on equation (1), and the calculation of A score (A) is based on equation (2).
Overall Proficiency Testing score, OPT score:
Overall schemes for the data collection from 2016 – 2021, criteria based on the performance of Z score.
- Z<1 with 4 points
- 1≤ Z < 2 with 3 points
- 2 ≤ Z < 3 with 2 points
- Z ≥ 3 with 1 point
- Not submitted the result with no points.
OPT score is obtained by summarising all the schemes carried out in the chemical testing between the two cycles; the score is divided by the highest points and then gets the percentage average.
Maturity Model, MM
A combination of AACS and SMS with a ratio of 20:80 obtained a maturity score, which was calculated as MM. Referring to Figure 1, the scoring system started from 40% to 49%, which is declared as the Beginning level. The minimum score for an accredited laboratory is 40%. The second level is the developing level which covers 50% to 59%. The emerging level covered 60% to 69%. Followed by the fourth level, the accelerating level started from 70% to 79%. The highest level is above 80%, which is the leading level. The laboratory with this level can lead the firm without doubt or query. The laboratory achieved all the criteria from 4 M and obtained a higher score, most probably can compete at a world-class level.
The scoring system was applied in a commercial laboratory in Malaysia. It’s been a laboratory accredited for more than 25 years. For a confidential purpose, here declared the laboratory as ABC laboratory. ABC lab has been accredited since early 1990, started by five technical personnel with five testing in water samples. Until now, the scope of the accreditation expanded to environmental monitoring, food, agriculture products and materials, petroleum products and microbiology. There are more than 100 parameters accredited in different methods, which cover standard, Malaysia Standard and in-house methods. For this study, the focus area was food and food products. A total of 10 analytes were chosen to calculate the AACS. There was benzoic acid, sorbic acid, ascorbic acid, sugar profile, heavy metals, fat, protein, moisture, ash and formaldehyde. Those were the most frequent testing running by the lab.
| Analytes | Manpower (28) | Method (16) | Machine (33) | Material (33) | Total Score (110) |
|---|---|---|---|---|---|
| Benzoic Acid | 17 | 8 | 23 | 20 | 68 |
| Sorbic Acid | 17 | 8 | 23 | 19 | 67 |
| Ascorbic Acid | 17 | 8 | 23 | 11 | 59 |
| Sugar Profile | 17 | 8 | 23 | 13 | 61 |
| Heavy Metals | 17 | 8 | 22 | 24 | 71 |
| Total Fat | 25 | 8 | 25 | 24 | 82 |
| Total Protein | 17 | 8 | 25 | 22 | 72 |
| Moisture | 17 | 8 | 23 | 21 | 69 |
| Total Ash | 17 | 8 | 23 | 23 | 71 |
| Formaldehyde | 17 | 8 | 23 | 11 | 59 |
| Grand Total | 178 | 80 | 233 | 188 | 679 |
| Analytes | Manpower (%) | Method (%) | Machine (%) | Material (%) | AACS (%) |
|---|---|---|---|---|---|
| Benzoic Acid | 60.71 | 50.00 | 69.70 | 60.61 | 61.82 |
| Sorbic Acid | 60.71 | 50.00 | 69.70 | 57.58 | 60.91 |
| Ascorbic Acid | 60.71 | 50.00 | 69.70 | 33.33 | 53.64 |
| Sugar Profile | 60.71 | 50.00 | 69.70 | 39.39 | 55.45 |
| Heavy Metals | 60.71 | 50.00 | 66.67 | 72.73 | 64.55 |
| Total Fat | 89.29 | 50.00 | 75.76 | 72.73 | 74.55 |
| Total Protein | 60.71 | 50.00 | 75.76 | 66.67 | 65.45 |
| Moisture | 60.71 | 50.00 | 69.70 | 63.64 | 62.73 |
| Total Ash | 60.71 | 50.00 | 69.70 | 69.70 | 64.55 |
| Formaldehyde | 60.71 | 50.00 | 69.70 | 33.33 | 53.64 |
| Average | 63.57 | 50.00 | 70.61 | 56.97 | 61.73 |
Manpower: The grand total score for manpower based on ten analytes was 178, obtaining 63.57% overall. It is summarised that staff turnover is too frequent; within six years involved, 12 staff resigned and were replaced by 11 new staff. Among the ten analyses, only one was handled by a lab technician which more than ten years of experience; the others depended on the junior staff. Food analysis is not the main core of this laboratory. Additionally, the average of the sample receiving is lower than the water and environmental monitoring samples; thus, the top management engaged a junior staff for food analysis. Two batches of staff observed, which the staff has worked for more than ten years, then the chance for them to change jobs was slim. The main point is to maintain the team after they have worked for five years. Appreciation awards or tokens shall be a giveaway to the staff that obedient and loyal to their job. Time-based promotion might be considered from junior team to senior staff or the head of a section.
Method: The grand total score for the method is 80 and obtained 50% overall. This laboratory used manufacturer-recommended application notes and research journals as their reference method. Most of the method for food analysis was the in-house method. The in-house method needs full validation to cover selectivity, robustness, linearity, working range, detection limit, quantitation limit, precision, trueness and uncertainty. To achieve the standard method’s performance characteristic, ensure the method is fit for use. The limit of detection played an important role in the method verification to differentiate the quality of the method used. For example, Lab A and Lab B used the same method and accredited the standard method; Lab A obtained 0.05 mg/kg as the LOD, whereas Lab B obtained 0.1 mg/kg; obviously, Lab A was highly sensitive. This laboratory did validate the method, but due to the matrix being declared as a food sample, the lab did not cover all types of food, and no selectivity was done correctly. The lab focuses on low-level concentration and does not refer to the legal requirements. As an example, the concentration of BA covers 0.2 mg/L, 4.0mg/L and 8mg/L, but based on the food regulation; the highest concentration can achieve 1000mg/kg. The validation procedure did not cover the food samples' full range. Referring to the ALACC guideline [17], Table 1 shows that full validation is needed for the analyte either at low or high concentrations. Optional only for LOD/LOQ for high concentration. The lab needs to validate again based on this guideline for further improvement.
Machine: The grand total score for the machine is 233, and it obtained 70.61% overall. The commonly used measuring equipment was an analytical balance, oven and thermometer. In contrast, the analytical instrument was HPLC, spectrophotometer and ICP OES. The laboratory has a procedure for handling, storage, use, cleaning and planned maintenance of equipment, including measuring and analytical equipment. A master list of testing equipment which provided information on identity, location, calibration status, period of validity and acceptance criteria was available. The lab followed the ISO clauses strictly, so fewer issued on the noncompliance record. For the volumetric and gravimetric analysis, the equipment used was easy maintenance; the lab used all the Grade A apparatus. Due to budget constraints, some measuring equipment was in sharing conditions, which contaminants might happen, but it can be reduced if Good Laboratory Practise (GLP) is applied. Maintenance of the analytical instruments achieves a high-performance level using the updated and advanced equipment series.
Material: The grand total score for the material is 188 and obtained 56.97% overall. This material includes reference material, proficiency testing or interlaboratory comparison, software, consumable parts, chemicals or reagent used and sample receiving. Laboratory mostly uses the analytical standard treated as the reference material, which is not metrological traceable, and lab-fortified material as the quality control samples; the quality can be increased if the lab changes all the reference material to certified reference material and uses the certified QC sample to ensure the validity of the results. Based on the MS ISO 17025, clause 7.7.2, the lab shall monitor its performance by comparison with the results of other labs either involved in PT or ILC. In Malaysia, we used SAMM; thus, SP 4 applied for PT activities, the minimum requirement of PT participation included in a new field or involving different matrices. The lab encourages participating in as many PT activities as possible for better representation within one accreditation cycle [18]. Of the ten analytes, only seven analytes participate in the PT program. The lab can increase its quality by taking part in all analytes.
Based on Table 4, the average score is 61.73, and the 20% is 12.35% which include in the MM calculation. For the SMS calculation, the audit Index, I and OPT score are shown in Table 5.
| Year | 1st Cycle of assessment | 2nd cycle of assessment | Average | ||||
|---|---|---|---|---|---|---|---|
| 2016 | 2017 | 2018 | 2019 | 2020 | 2021 | ||
| OPT % | 87.50 | 87.00 | 86.03 | 90.79 | 92.31 | 82.08 | 87.62 |
| Index, I | - | 1.0000 | 0.7183 | - | 0.9705 | 0.4242 | 0.7783 |
A score based on the formula (3) is 82.54%, and 60% is 49.52%. The 40 per cent of OPT is 35.05%. Thus, the SMS is 84.57%. Total SMS contribute 80% to MM, which is 67.66%. MM is 12.35% + 67.66%, with a total of 80.01% based on the formula (4). Figure 1 shows that the level of maturity is the “Leading” level.
The main purpose of this research is to determine the maturity level of the laboratory and for further improvement based on 4M (Manpower, Method, Machine & Material) referring to the SAMM document. From this study, it’s found that this lab is the potential for globalisation which it’s achieved the leading level. However, continuous improvement can be done by recommendations below to strengthen the maturity level of the laboratory: -
1. Manpower: maintain competent staff by changing the management’s strategy.
2. Method: validate all the in-house methods following ALACC guidelines.
3. Machine: apply GLP for equipment sharing, specifically the equipment for a specific analyte.
4. Material: Metrological traceable RM needed, involvement PT in all analytes.
This study is based on questionnaires referring to records and interviews with top management to technical staff from the private laboratory. Referring to the company’s policy, only six years of records were maintained. Thus, only two cycles of the records were collected and analysed. Further research will include the expert laboratory based on AACS. The analyte with the higher AACS score will be the expert lab for that particular analyte. The contribution of the expert lab in the MS ISO/IEC 17043 will upgrade the reputation of our department’s PT provider.
- ISO 9001:2015 Quality management systems - Requirements (Second revision) (ISO 9001:2015, IDT)
- ISO/IEC 17025: 2017 General requirements for the competence of testing and calibration laboratories. Third edition.
- EURACHEM / CITAC Guide QAC 2016: Guide to Quality in Analytical Chemistry an Aid to Accreditation, Third Edition.
- F Khodabocus and K Balgobin,, 2011. Implementation and Practical Benefits of ISO/IEC 17025:2005 in a Testing Laboratory. UNIVERSITY OF MAURITIUS RESEARCH JOURNAL – Volume 17–2011.
- Irma Permata Sari, 2018. Analysis Implementation Effectiveness of ISO / IEC 17025 on Testing Laboratory. IEOM Society International, Pages 1752–1761.
- Ahmad Rois Mansur, Jungmin Oh, Hyun Sung Lee, Seung Yong Oh, 2022. Determination of ethanol in foods and beverages by magnetic stirring-assisted aqueous extraction coupled with GC-FID: A validated method for halal verification. https://doi.org/10.1016/j.foodchem.2021.130526. Food Chemistry, Vol 366, 1 January 2022.
- Mercy A. Okezue, Mojisola C. Adeyeye, Steve J. Byrn, Victor O. Abiola and Kari L. Clase, 2020. Impact of ISO/IEC 17025 laboratory accreditation in sub-Saharan Africa: a case study. Okezue et al. BMC Health Services Research 20:1065 https://doi.org/10.1186/s12913-020-05934-8
- Anis Najiha Ahmad, Ungku Fatimah Ungku Zainal Abidin, Mohhidin Othman, Russly Abdul Rahman, 2018. Overview of the halal food control system in Malaysia, Food Control 90, 352–363p. https://doi.org/10.1016/j.foodcont.2018.02.035
- Radosław Wolniak, 2019. The Level of Maturity of Quality Management Systems in Poland—Results of Empirical Research. Sustainability2019,11, 4239; doi:10.3390/su11154239
- Amanda Earnshaw, Rosemary Ann Smith, Linda Owen, 2009. How proficiency testing can improve the quality of analytical data using vitamin analysis as an example. Food chemistry 113, 781–783p
- Dahlin, G., 2020. What can we learn from process maturity models? – A literature review of models addressing process maturity, Int. J. Process Management and Benchmarking, Vol. 10, No. 4, pp.495–519.
- Jörg Becker, Ralf Knackstedt, Jens Pöppelbuß. 2009. Developing Maturity Models for IT Management – A Procedure Model and its Application. Business & Information Systems Engineering 3. DOI 10.1007/s12599-009-0044-5
- Sanjay Jain, Sudarsan Rachuri. 2014. Maturity Model Concepts for Sustainable Manufacturing. NISTIR 7989. http://dx.doi.org/10.6028/NIST.IR.7989
- Marcelo Albuquerque Oliveira and Isabel Lopes. 2019. Evaluation and improvement of maintenance management performance using a maturity model. International Journal of Productivity and Performance Management Vol. 69 No. 3, 2020 pp. 559–581. DOI 10.1108/IJPPM-07-2018-0247
- Rosenstock, C., Johnston, R. S., & Anderson, L. M., 2000. Maturity model implementation and use: a case study. Seminars & Symposium. https://www.pmi.org/learning/library/maturity-model-implementation-case-study-8882
- Aijaz Panhwar, Ateeq Rehman Memon, Azhar Naeem, M., Syed Zain ul Ibad, Aftab Kandhro, Nusrat Jalbani, Siraj ul Haque, Mehtab Ahmed, Aijaz ul Haq and Sofia Qaisar, 2020. “Responsibilities and qualities of an assessor/auditor as per ISO/IEC standards,” International Journal of Current Research, 12, (02), 10223–10226.
- Analytical Laboratory Accreditation Criteria Committee, ALACC Guide 2007: How to Meet ISO 17025 Requirements for Method Verification by AOAC international.
- SAMM Policy 4, issue 3, 1 october 2013. (Amd. 1, 9 August 2019).
Tables 1-2 are available in the supplementary files section.
Competing interests: The authors declare no competing interests.
