  Objectives: The purpose is to apply a previously validated deep learning
algorithm to a new thyroid nodule ultrasound image dataset and compare its
performances with radiologists. Methods: Prior study presented an algorithm
which is able to detect thyroid nodules and then make malignancy
classifications with two ultrasound images. A multi-task deep convolutional
neural network was trained from 1278 nodules and originally tested with 99
separate nodules. The results were comparable with that of radiologists. The
algorithm was further tested with 378 nodules imaged with ultrasound machines
from different manufacturers and product types than the training cases. Four
experienced radiologists were requested to evaluate the nodules for comparison
with deep learning. Results: The Area Under Curve (AUC) of the deep learning
algorithm and four radiologists were calculated with parametric, binormal
estimation. For the deep learning algorithm, the AUC was 0.69 (95% CI: 0.64 -
0.75). The AUC of radiologists were 0.63 (95% CI: 0.59 - 0.67), 0.66 (95%
CI:0.61 - 0.71), 0.65 (95% CI: 0.60 - 0.70), and 0.63 (95%CI: 0.58 - 0.67).
Conclusion: In the new testing dataset, the deep learning algorithm achieved
similar performances with all four radiologists. The relative performance
difference between the algorithm and the radiologists is not significantly
affected by the difference of ultrasound scanner.
