Outline
This prospective cross-sectional study, evaluated women reporting vaginal symptoms, seen at a single designated clinic for vulvovaginal disorders, at the Clalit Health Organization, Jerusalem, Israel. All patients were examined, diagnosed, and treated by the same provider (the first author, ALS). Inclusion criteria were (i) women with vaginal complaints: discharge, malodor, itch, burning, pain, or dryness, (ii) 18 years old and above. Exclusion criteria included (i) patients unfit to provide informed consent, (ii) an uninterpretable sample (e.g. patients who used vaginal creams or lubricants before the visit or insufficient sampling material).
During the gynecological exam, vaginal discharge samples were taken for pH levels, wet-mount microscopy, vaginal cultures, and trichomonas PCR, per standard of care in the clinic. An additional sample was taken for the investigational test diagnosis using a swab with a soft cytobrush head (Figure 1), pulled through a dedicated cartridge, and scanned in the GYNI™ investigational test table-top scanner in the clinic.
Patients were diagnosed by the physician using wet-mount microscopy and were treated according to her recommendation. The investigational test diagnosis was stored in the cloud and blinded to the physician to prevent bias. Each patient was identified using a numerical code and an investigational test number was generated by the application. Laboratory results were recorded only by the patient's code number, without any identifying details, such as name or personal identification number. The comparators used as the gold standard for each of the assessed conditions studied are shown in Table 3. Results of the three methods, the physician’s wet-mount diagnosis, the laboratory findings, and the investigational test, were summarized and compared by the chief investigator (the last author, ABC).
The sample size for this study was calculated for estimating the overall accuracy (the percent agreement on the diagonal between the investigational test and the reference diagnosis) via the level of precision required for the estimate. The level of precision is measured by the half-width of the 95% confidence interval around the proportion of interest. We calculated based on Hajian-Tilaki K27 that an accuracy level of at least 90% with a confidence interval half-width of 5% can be estimated with a minimum sample size of 139 patients. Recruitment continued to a larger sample size to obtain reasonable representation of each of the included conditions.
The results of the investigational test were compared individually to the CRS comparator of the specialist wet mount results, the candida cultures, and the STI PCR panel for detection of trichomonas vaginalis. The CRS was defined as positive if there was a positive result by either wet mount or culture/PCR. Samples were classified as negative if all comparators were negative. The comparison included the calculation of overall accuracy, sensitivity, specificity, positive predictive value, and negative predictive value according to standard equations. These measures are presented with two-sided 95% Wilson score confidence intervals. Cohen’s kappa coefficient (κ) is presented as a measure of inter-test agreement, with 95% confidence interval. All analyses were performed using Excel (Microsoft Corp. Redmond, Wash.), and SPSS (IBM Corp. IBM SPSS Statistics for Windows. Armonk, NY).
The Investigational Test
The investigational test evaluated in this study is the GYNI™ system (GynTools, Israel), a novel automated in-vitro diagnostic system, intended to aid in the diagnosis of vaginitis in symptomatic women, by qualitatively detecting the following vaginitis conditions or pathogens at the point-of-care: (1) BV, (2) candida albicans vaginitis (CA), (3) candida non-albicans vaginitis (NAC), (4) trichomoniasis, (5) vaginal atrophy (also referred to as atrophic vaginitis or Genitourinary syndrome of menopause, GSM), (6) aerobic vaginitis / desquamative inflammatory vaginitis (AV/DIV), and (7) cytolytic vaginosis (CV).
The system is aimed at providing the non-expert health-care provider with the means to obtain a wide diagnostic range with a fast, inexpensive analysis of various vulvovaginal conditions from a single swab. This is done by fully automating (a) consideration of patient’s reported symptoms (b) saline wet mount microscopy (c) KOH microscopy, and (d) pH measurement, similar to the evaluation performed in vaginitis-specializing clinics 28. The test uses machine learning computer vision in the form of a deep convolutional neural network (CNN) model that performs a multi-label classification of seven major vaginitis conditions. The model was trained on 13,500 microscopy images which were collected and classified by a specialist in a dedicated clinical trial (clinicaltrials.gov, NCT03585049). Unlike described proofs of concept for applying deep neural networks for the classification of smaller subsets of vaginitis conditions in manually gram-stained and selected microscopy images 29,30, this test does not use slide staining, therefore, minimizing the required operator labor and shortening the time period until results are available.
The investigational test system is comprised of the following components (Figure 1):
- A vaginal discharge collection tool: A disposable plastic assembly with a swab / brush-head, connected to a plastic rod (1a), that is used to collect the discharge sample from the vagina; (1b) a “floating” transparent optical tray; (1c) transparent cover and; (1d) diluent containers.
- A compact tabletop scanner: The scanner (2) includes a high-resolution color camera, illumination LEDs, magnifying lenses, and linear motion systems, both vertical and horizontal. The operator places the disposable cartridge in a cradle connected to the horizontal motion system. During insertion of the tray into the cradle, the cartridge's internal diluents are automatically sprayed on the optic tray and the tray is lifted to create optical coupling. Upon activation, the cradle begins its linear motion between the illumination sources and the camera, dozens of microscopic pictures are acquired, and the pH level is determined by a color change of pH indicator paper located in the cartridge. The collected information is then transferred to the cloud.
- Web-based user interface: A test operation website provides an interactive mechanism for data entry and test control, available via any web-connected device, such as a laptop or a smartphone.
- Cloud software platform: Test processing includes a deep learning convolutional network model for multi-label image processing classification, analysis of pH paper images, and color calibration images for pH calculation. The processing software also crossmatches the patient’s reported symptoms and pH level with the results of the computer vision classification of the microscopy images.
The results provided by the investigational test include (a) suggested diagnoses – one or more detected conditions, (b) pH level and (c) a heatmap annotation of a selected input microscopy image, obtained via guided-back propagation 31. The heatmap annotation is aimed to provide explainability 32 by visualizing the salient areas in the input image having the strongest effect on the model output. Examples of such heatmap results are shown in Figure 2. The test results are displayed online and are available to the operator for download.