  Automated deidentification of clinical text data is crucial due to the high
cost of manual deidentification, which has been a barrier to sharing clinical
text and the advancement of clinical natural language processing. However,
creating effective automated deidentification tools faces several challenges,
including issues in reproducibility due to differences in text processing,
evaluation methods, and a lack of consistency across clinical domains and
institutions. To address these challenges, we propose the pyclipse framework, a
unified and configurable evaluation procedure to streamline the comparison of
deidentification algorithms. Pyclipse serves as a single interface for running
open-source deidentification algorithms on local clinical data, allowing for
context-specific evaluation. To demonstrate the utility of pyclipse, we compare
six deidentification algorithms across four public and two private clinical
text datasets. We find that algorithm performance consistently falls short of
the results reported in the original papers, even when evaluated on the same
benchmark dataset. These discrepancies highlight the complexity of accurately
assessing and comparing deidentification algorithms, emphasizing the need for a
reproducible, adjustable, and extensible framework like pyclipse. Our framework
lays the foundation for a unified approach to evaluate and improve
deidentification tools, ultimately enhancing patient protection in clinical
natural language processing.
