Image Data
All MSK-Xs were from public datasets where animal and human ethics approvals had already been obtained and additional was not required for this study 11, 12. VetMSK-Xs were downloaded from DVMInsight built by IDEXX and Radiology Cases from the Radiological Society of North America (RSNA) 11, 12. We used the search term “limb” to identify appendicular MSK-Xs. We excluded all Xs of the skull, chest, abdomen, and spine. We used the associated report for each case to identify the lesions that were reported. There were 500 Xs from 141 animals. Each animal on average has 3–4 different views Xs of a lame limb. Furthermore, Xs of the normal limb are sometimes taken to be compared against the lame limb if the lesions are subtle. Among the 141 animals, there were 123 dogs (mean age 4.7 years), and 18 cats (mean age 5.3 years); of the dogs, 68 were male, 50 female and in 5 the gender was not specified. For the cats 13 were male and 4 were female and 1 has unspecified gender. 447 Xs are done in dogs and 53 Xs are done in cats. Out of the 500 Xs, 143 Xs were normal Xs without any lesion reported; 357 Xs were abnormal Xs with reported lesions. The distribution of the Xs was as follows: a) for dogs – 34 Xs of shoulder joint, 17 Xs of the arm, 53 Xs of elbow joint, 52 Xs of the forearm, 19 Xs of both arm and forearm, 10 Xs of carpus joint, 32 Xs of front paw, 3 Xs of phalanges, 43 Xs of the pelvis, 13 Xs of the thigh, 96 Xs of knee joint, 21 Xs of the leg, 25 Xs of both thigh and leg, 9 Xs of tarsus joint, 20 Xs of back paw; b) for cats – 1 x-ray of shoulder joint, 1 x-ray of arm, 2 Xs of elbow joint, 13 Xs of the forearm, 1 x-ray of both arm and forearm, 1 x-ray of phalanges, 1 x-ray of the pelvis, thigh and leg, 13 Xs of the pelvis, 2 Xs of the thigh, 3 Xs of knee joint, 3 Xs of the leg, 6 Xs of both thigh and leg, 3 Xs of tarsus joint, 3 Xs of the back paw. We list detailed descriptions of 500 Xs we used in our study in Table 1. We show examples of Xs without and with lesions are shown in Fig. 1.
We used the Musculoskeletal Radiograph dataset (MURA) for the human data. The MURA has 40,005 Xs from 1,863 patients. The Xs are from the upper (thoracic) limbs – shoulder, arms, elbow, forearm, wrists, and fingers 13. In the MURA, the authors defined ‘lesions’ as fractures, subluxations, degenerative joint disease, and destructive abnormalities, and examples are shown in Fig. 2.
iiVetMSK-Xs
The architecture and workflow for our approach are shown in Fig. 3. The first component is the Xs classifier. It’s built on EfficientNet. An EfficientNet in its B5 configuration is trained to classify whether an x-ray has a lesion or not. The proposed iiVetMSK-Xs is trained in two different ways: i) Direct Finetuning (DF-iiVetMSK-Xs) – finetuning the EfficientNet using the veterinary dataset; ii) MURA-Veterinary finetuning (MVF-iiVetMSK-Xs) – finetuning the EfficientNet using the MURA dataset and then the veterinary dataset sequentially. The training was done using PyTorch version 1.1.0 following the guide provided by Pytorch 14. EfficientNet was loaded from EfficientNet PyTorch developed by Melas-Kyriazi 15. All input Xs were padded to be square-shaped and resized to 1024 by 1024 pixels. We applied random rotation and contrast adjustment as data augmentation techniques. We then normalized the images using the mean and standard deviation derived from the MURA dataset. The second component was the Lesion Localizer. We used Grad-CAM 16. It can visualize areas in an image that contribute the most to the target class’ score predicted by the classifier as a heatmap17. In our iiVetMSK-Xs, the target class is the “with lesion” class. Therefore, the “hot” (red) areas in the heatmaps are areas that are the most likely to be associated with lesions identified by the classifier.
Evaluation
We evaluated the x-ray classifiers trained by the DF and MVF by comparing their diagnostic accuracy, sensitivity, and specificity for correctly classifying the Xs with lesions. For the veterinary dataset, we performed 5-fold cross-validation with the dataset divided into different training and testing set pairs five times. When preparing these, we performed a random sampling of Xs for testing without replacement. Therefore, the 5 testing subsets were mutually exclusive and did not share any common x-ray and also maintained the 7:3 ratio between Xs with and without lesions. We calculated the average diagnostic accuracy, sensitivity, and specificity over the 5 testing subsets and used these metrics to assess which training strategy performed better. The accuracy of lesion localization was determined by the findings on the heatmaps when compared to the imaging report and was carried out by the registered Vet (the first author of this manuscript) in Australia. We display an x-ray reported with osteolysis of the humeral condyle and examples of incorrect and correct lesion localization in Fig. 4.A, B, and C respectively.