The data files can be downloaded from the Brown Digital Repository (DOI: 10.26300/rv2a-kp40). Additionally, some limited data are contained within the code repositories, which are available on GitHub at <https://github.com/samwbell/train_unet_mng> and <https://github.com/brown-ccv/mngcount>

1. Introduction {#sec001}
===============

Phthalates (phthalic acid esters) are used to make a variety of industrial products and consumer goods, most notably to plasticize polyvinyl chloride for use in products such as vinyl sheeting and medical tubing \[[@pone.0229967.ref001]\]. Human exposure to phthalates is nearly universal, and their potential toxicity to male reproductive tract development raises concern for male reproductive health \[[@pone.0229967.ref002], [@pone.0229967.ref003], [@pone.0229967.ref004], [@pone.0229967.ref005], [@pone.0229967.ref006], recently reviewed by [@pone.0229967.ref007], [@pone.0229967.ref008]\]. In rat models with *in utero* exposure to certain phthalates, decreased testosterone production is observed \[[@pone.0229967.ref009], [@pone.0229967.ref010], [@pone.0229967.ref011], [@pone.0229967.ref012], [@pone.0229967.ref013], [@pone.0229967.ref014]\]. However, quantification of male reproductive toxicity is complicated by lack of concordance between this effect on steroidogenesis and other adverse testicular development outcomes, as histological effects on the testis can occur regardless of a reduction in measured testosterone \[[@pone.0229967.ref004], [@pone.0229967.ref003], [@pone.0229967.ref002], [@pone.0229967.ref015], [@pone.0229967.ref016]\]. One effect of phthalate exposure on fetal testis development that is consistent across species is an increase in multinucleated germ cells (MNGs), germ cells which contain two or more nuclei \[[@pone.0229967.ref009], [@pone.0229967.ref017]\]. Di-*n*-butyl phthalate and di-(2-ethylhexyl) phthalate induce MNGs in rats and mice *in utero* and in rat, mouse, and human tissue xenograft models, to levels much greater than the background rate of MNGs found in control testes \[[@pone.0229967.ref016], [@pone.0229967.ref002], [@pone.0229967.ref003], [@pone.0229967.ref018], [@pone.0229967.ref019]\], which has led researchers to conclude that, despite differences in the anti-androgenic response, rats are an appropriate model in which to study the effects of phthalates on germ cells \[[@pone.0229967.ref018]\]. The long-term impact of MNG induction on testis health is unclear. Phthalate-induced MNGs are degenerative cells that are lost through p53-dependent apoptosis in the early postnatal period \[[@pone.0229967.ref020], [@pone.0229967.ref021], [@pone.0229967.ref009], [@pone.0229967.ref022]\]. However, induction of MNGs is clearly a reproducible indicator of phthalate effect on the fetal testis. Although this is a quantitative endpoint with the potential to be used as a biomarker of seminiferous cord-mediated phthalate toxicity, thorough studies on the dose-response for induction of MNGs by most phthalates have not been conducted in any species \[[@pone.0229967.ref019]\].

The primary drivers of the low quantity of MNG count data are the time required for performing the count and the training needed for an expert to be able to confidently identify the MNGs \[[@pone.0229967.ref019]\]. Additionally, there is error inherent in recognizing an MNG in a histological section, or especially a two-dimensional image of a histological section. This is due at least in part to the high density of germ cells in seminiferous cords, the large germ cell nuclei and low ratio of cytoplasm area to nucleus area in cross-section, and the sometimes indistinct germ cell membrane. As a result of these features, different experts are able to reach different conclusions about some fraction of the cells that are identified as MNGs \[[@pone.0229967.ref019]\] began the process of addressing these issues through the creation of a semi-automated counting pipeline. Using hematoxylin-stained thin sections of fetal rat testes and a scripted process though the NIH ImageJ software \[[@pone.0229967.ref019]\], identified MNGs based on their image characteristics using computer vision principles. The primary image criteria used to identify MNGs was size. Without a formal cell segmentation routine \[[@pone.0229967.ref019]\], used image thresholding, down-sampling, blurring, and filling to identify connected image components. These connected components were filtered based on circularity to exclude non-cells, and then they were filtered by size to identify MNGs. While much of this pipeline was fully automated, several steps required human input, including the initial image processing and the identification of the thresholds for each image batch.

Here, we propose to improve on the \[[@pone.0229967.ref019]\] approach, leveraging recent innovations in convolutional neural networks to build a fully automated MNG identification code.

2. Methods {#sec002}
==========

2.1 Neural network architecture {#sec003}
-------------------------------

The base neural network architecture we chose was U-Net. Developed by \[[@pone.0229967.ref023]\], U-Net was designed for solving the cell segmentation problem. While many neural networks, when trained, take an image as an input and produce a classification probability, U-Net produces a mapped image showing the probability of a certain classification for each part of the image. For cell segmentation, U-Net produces a map of where the cell boundaries are located.

U-Net takes a 512 by 512 pixel image tile as its input. As the image tile proceeds through the U-Net, it passes through multiple convolution, max pooling, and ReLU activation function layers, resulting in an image cube with reduced x and y dimensions and enhanced z dimensions. At this layer, the deepest learning occurs. The image is then up-sampled using up-convolution, finally producing an output prediction map with values scaled from 0 to 255. The ReLU activation function, which sets all negative values to zero, helps the neural network focus in on specific areas of the image.

There are several implementations of the U-Net, and we chose a Python and Keras implementation published by GitHub user zhixuhao (<https://github.com/zhixuhao/unet>). While we modified the training, augmentation, prediction, and data management code extensively, we left the original network architecture unmodified.

Because it was originally designed for solving the cell segmentation problem, the original implementations of U-Net used a map of cell boundaries as the training data. The cells were outlined, not filled in. To apply the U-Net to the MNG problem, we structured the training data somewhat differently. Instead of the outline of each MNG, we used a map of the filled in area of each MNG. As a result, the trained network produced a prediction heat map with a likelihood of each pixel belonging to an MNG ([Fig 1](#pone.0229967.g001){ref-type="fig"}).

![A schematic showing the different stages of the automated MNG prediction process, as performed on a single panel measuring 512x512 pixels (here 1 pixel = 1 micron): a) The original panel. b) The predictions after the trained model has been applied. (In this case, the fold configuration with Fold 0 as the holdout set is being used.) The likelihood of a pixel being represented by an MNG has been mapped to the pixel brightness values on an arbitrary 0 to 255 scale, creating an MNG location heatmap. These brightness values should be seen as relative likelihood, and the exact likelihood depends on the model. Note the clearly visible MNGs and some brighter pixels around two cells that the model sees as having a possibility of being an MNG. c) The MNG location prediction image after a brightness cutoff threshold has been applied. For this fold configuration, the brightness cutoff is 240. After applying the brightness cutoff, the two MNGs close together clearly separate, and the not clearly predicted MNGs become a few specks. d) The final prediction image after a cutoff area threshold has been applied. All shapes with an area below the cutoff, 150 pixels in this case, have been removed. (We used the Green's theorem area approximation, not literal pixel areas.) This image shows the final predictions of MNG locations. e) The "true" MNG locations as identified by the human reference scorer. f) The successful and unsuccessful predictions shown superimposed on the original image. Correctly predicted MNGs are outlined in green, and the MNG that was missed is outlined in red.](pone.0229967.g001){#pone.0229967.g001}

2.2 Data acquisition {#sec004}
--------------------

Experiments involving animals were performed under a protocol approved by the Institutional Animal Care and Use Committee of the USEPA National Health and Environmental Effects Research Laboratory (Laboratory Animal Project Review \#19-03-001). Animals were housed in a facility accredited by the Association for Assessment and Accreditation of Laboratory Animal Care and maintained at 20--22°C, 45--55% humidity, and a 12:12 h photoperiod (lights off at 1800 hrs). Histological section images were obtained from the samples from animals exposed to diethyl phthalate (DEP), dipentyl phthalate (DPeP), dimethyl phthalate (DMP), di-(2-ethylhexyl) phthalate (DEHP), di-n-butyl phthalate (DBP), benzyl butyl phthalate (BBP), di-(2-ethylhexyl) tetrabromophthalate (TBPH), dioctyl terephthalate (DOTP), or corn oil vehicle reported previously by \[[@pone.0229967.ref019]\]. An additional 15 samples were obtained from rats treated with DPeP (1, 11, 33, or 100 mg/kg/d) or corn oil vehicle as previously described. Briefly, timed pregnant Sprague Dawley rats were obtained from Charles River Laboratories (Raleigh, NC). Treatment compounds were administered by daily oral gavage in 2.5 mL/kg body weight corn oil vehicle from gestation day (GD 17--21). Dams were euthanized by decapitation approximately two hours after administration of the final dose. Testes were isolated from male fetuses and treated with a modified Davidson's fixative for 15 minutes, before being transferred to 70% ethanol.

Fixed testes were dehydrated through a series of graded ethanols, embedded in paraffin wax, and sectioned at a thickness of 5 μm. Paraffin sections were mounted on glass histological slides, deparaffinized, rehydrated, and stained with hematoxylin. Hematoxylin stained sections were scanned at 40x magnification on an ImageScope CS digital slide scanner (Leica Biosystems, Buffalo Grove, IL) and saved in the ScanScope default.svs file format.

Images were down-sampled to 1 micron per pixel and compressed using the NIH's ImageJ software, then converted to 24-bit RGB PNG files. We determined that the color ratios did not contain additional useful information beyond the grayscale values, so we converted the color images to single-channel 8-bit greyscale images. We then cropped sub-images around each individual testis slice on the slide. In most cases, each slide contained four testis slices.

2.3 Manual identification of MNGs {#sec005}
---------------------------------

For each image, we hand-identified the centers of each MNG by using Leica ImageScope software to view digital slide images. In order to ensure consistency, hand-identification was performed by the human reference scorer, who is experienced in identification of MNGs. In order to quantify the accuracy of human identification, two additional researchers performed our own hand MNG identification on previously unseen slides. Human test scorer 1 examined 12 testes that he had not previously seen, identifying 44 MNGs. Using the reference human scorer's annotations as true MNG identifications, these 44 MNGs consisted of 10 false positives, and 34 true positives. There were an additional 11 false negatives. Human test scorer 2 examined fourteen images, identifying 105 MNGs, including 10 false positives, with an additional 41 false negatives in this dataset.

To quantify the match between human scorers or the model, we utilized the F1 score metric, which is the harmonic mean of the precision and the recall: $$precision = \frac{true\ positives}{true\ positives + false\ positives}$$ $$recall = \frac{true\ positives}{true\ positives + false\ negatives}$$ $$F1 = \left( \frac{precision^{- 1} + recall^{- 1}}{2} \right)^{- 1},$$ where the true positives are MNGs that were successfully identified; the false negatives are MNGs that were identified by the reference human scorer, but not by the human test scorers or the model; and the false positives are MNGs identified by the human test scorers or the model but not the reference human scorer. For the purposes of these statistics, we treated the reference human scorer's identifications as the true identifications, but we understand that likely they are not perfectly accurate. The F1 scores for human test scorers one and two were 0.764 and 0.788 respectively.

To construct the true data maps, we used the reference human scorer's identifications. After the identifications had been made, we manually identified the outlines of the MNG cells, drawing filled polygons.

2.4 Augmentation and tile selection procedure {#sec006}
---------------------------------------------

In order to increase the number of images on which the network is trained, the training set for a neural network is typically augmented by passing the true maps and raw images through a series of transformations. For each MNG, we selected 500 sample panels with the MNG center located randomly within the panel. As a result, we were able to sample the MNG with a large number of randomized horizontal and vertical displacements. For each sample panel, we iterated through all eight unique combinations of vertical and horizontal flips and 90°C and 180°C rotations. By randomizing the orientation and location of the MNGs, we ensured that the network would not falsely train on features related to the orientation and location of the MNG. Randomizing the location also increased the area of negatives to train the model on.

2.5 Training process {#sec007}
--------------------

We separated the data into five separate "folds" or batches of images. For the training set, we used three of the folds, with one fold each for the test and holdout sets. The training set was used for training the model, the test set was used for optimizing parameters, and the holdout set was used for assessing the accuracy of the model. We used five different combinations of these folds, with each fold being used once as the holdout set. In order to ensure that the folds were both representative and fully independent from each other, we used batch stratification. There were 28 slides, with four testis slices on most slides. To ensure representative batches, we sorted the slides by the number of MNGs on each slide and separated them into five groups of five and one group of three, ordered by number of MNGs. We assigned each slide from every group to a fold, making sure than no fold received more than one slide from each group and selecting the configuration that minimized the variation in the total number of MNGs in each fold. Unfortunately, we could not fully equalize the number of MNGs in each fold. One slide in Fold 0 had 152 MNGs, more than the average number of MNGs per fold, 119.2. In order to accomodate this outlier slide, Fold 0 had to have 175 MNGs, substantially more than the other folds, which ranged from 103 to 108 MNGs.

Numbering the five folds 0--4, we ran the model on five configurations. In each case, we selected the test set to be the fold numerically after the holdout set fold number (looping back to fold 0 for the test set when the holdout set was fold 4), with the other three folds going into the training set. We used the data from the training set to train the model, running it for fifteen epochs and saving the results from each epoch. For each epoch, we pass the model through the full set of augmented training data, taking the output of the previous epoch as the starting model for the next one.

The result of the trained neural network is a map of MNG probability for each pixel scaled as a brightness with integer values ranging from 0 to 255. To convert it into a binary map of MNG locations, we used brightness and area cutoffs ([Fig 1](#pone.0229967.g001){ref-type="fig"}). All pixels below the brightness cutoff were removed, and only remaining connected regions of pixels with areas of above the area cutoff were retained. For reasons of computational efficiency, the areas we used here were Green's theorem areas, which are typically slightly lower than literal pixel areas. To calculate the optimal brightness and area cutoffs, as well as the optimal epoch, we used a grid search, selecting the combination of values these three parameters that maximized the F1 score (Figs [2](#pone.0229967.g002){ref-type="fig"} and [3](#pone.0229967.g003){ref-type="fig"}). With the epoch, brightness cutoff, and area cutoff optimized and selected, we arrived at a final model that could take an image and predict the locations of the MNGs. To measure the final accuracy, we tested this model on the holdout set and measured the F1 score. By using these separate training, test, and holdout sets, we avoided overfitting, which would distort our accuracy scores.

![An example of the grid search over the test set to find the optimal cutoff area, cutoff brightness, and F1 score.\
This example shows the fold configuration with Fold 4 as the holdout set and Fold 0 as the test set. Here, F1 score is plotted for every cutoff brightness and cutoff area we simulated. We selected the parameter combination that maximized the F1 score, a cutoff area of 20 pixels and a cutoff brightness of 210, and we have indicated this particular grid point on the figure it with a black box around it. The f1 score color bar was scaled between a minimum of 0.65 and the maximum brightness, with f1 scores below 0.65 set to black. The cutoff area and cutoff brightness parameters convert the output MNG location probability map, scaled as a brightness from 0 to 255, to specific MNG location maps. First the brightness cutoff is applied, and then any contiguous regions of pixels above the cutoff smaller than the cutoff area are removed.](pone.0229967.g002){#pone.0229967.g002}

![The grid search shown in [Fig 2](#pone.0229967.g002){ref-type="fig"}, but with epoch instead of F1 score plotted.\
The cutoff area and cutoff brightness parameters convert the output MNG location probability map, scaled as a brightness from 0 to 255, to specific MNG location maps. First the brightness cutoff is applied, and then any contiguous regions of pixels above the cutoff smaller than the cutoff area are removed.](pone.0229967.g003){#pone.0229967.g003}

3. Results {#sec008}
==========

We successfully ran our model on all five of the fold configurations. There was little consistency in the optimal parameter values among the different fold configurations. There was especially great variety in the optimal epoch, with values ranging from 1 to 13 ([Table 1](#pone.0229967.t001){ref-type="table"}). The fold configuration with Fold 4 as the holdout set led to the lowest holdout set F1 score, 0.622, and its optimal cutoff area was a rather extreme outlier of 20 pixels. This fold combination used Fold 0, which had a much larger number of MNGs than the other folds, as its test set. The fold combination with the second lowest holdout set F1 score was when Fold 0 was the holdout set, and in that case the optimal cutoff area was 150 pixels, above the outlier of 20 pixels but still clearly lower than for the other folds. When Fold 0 was not part of the training set, the model had fewer MNGs to train on, so it makes sense that the performance would be worse.

10.1371/journal.pone.0229967.t001

###### Results of the grid search to optimize the epoch, cutoff brightness, and cutoff area.

![](pone.0229967.t001){#pone.0229967.t001g}

  Holdout set fold   Test set fold   Optimal Epoch   Optimal Cutoff Brightness   Optimal Cutoff Area   Test Set Matches   Test Set False Positives   Test Set False Negatives   Test Set F1 Score   Holdout Set F1 Score
  ------------------ --------------- --------------- --------------------------- --------------------- ------------------ -------------------------- -------------------------- ------------------- ----------------------
  0                  1               13              240                         150                   78                 22                         29                         0.754               .710
  1                  2               2               120                         700                   78                 4                          25                         0.843               .714
  2                  3               1               246                         400                   94                 17                         14                         0.858               .770
  3                  4               5               220                         300                   73                 17                         30                         0.756               .805
  4                  0               5               210                         20                    126                29                         49                         0.764               .622

Our best holdout set F1 score was 0.805, coming in the fold configuration with Fold 3 as the holdout set ([Table 2](#pone.0229967.t002){ref-type="table"}). This exceeded the accuracy of either human scorer, whose F1 scores were 0.764 and 0.788. However, the mean holdout set F1 score across all five fold configurations was 0.724, with a standard deviation of 0.07, falling slightly below the scores for the human scorers. On average, the model reaches near human accuracy, and in the best model, it exceeds it.

10.1371/journal.pone.0229967.t002

###### Holdout set results for the five folds as well as the two human scorers.

![](pone.0229967.t002){#pone.0229967.t002g}

                  Matches   False Positives   False Negatives   Holdout Set F1
  --------------- --------- ----------------- ----------------- ----------------
  Best (Fold 3)   91        27                17                .805
  Fold 0          119       41                56                .710
  Fold 1          71        21                36                .714
  Fold 2          72        12                31                .770
  Fold 4          61        32                42                .622
  Mean                                                          0.724
  Std. Dev.                                                     0.070
  Human 1         34        10                11                0.764
  Human 2         95        10                41                0.788

In all except the best performing fold configuration, false negatives exceeded false positives, with an average false positive to false negative ratio of 0.62. However, in the best performing model false positives exceeded false negatives 27 to 17, which corresponds to a false positive to false negative ratio of 1.59. For all the folds, the average false positive to false negative ratio was 0.81.

4. Discussion {#sec009}
=============

Ultimately, the strongest underlying limitation we faced was the training data. The accuracy of a neural network is only as good as the training data, and it can always be improved with more training data. Our model is no exception. We can see this clearly in our data because the fold configurations without Fold 0 and its especially high MNG count in the training set underperformed compared to the three fold configurations that had Fold 0 in the training set. More training data would certainly improve the model.

However, more data may not be enough to reach perfect accuracy. In addition to the limited quantity of training data, we were also limited by error inherent in the scoring procedure used to obtain training data. There are several issues that make manual MNG identification from a single slide difficult. The thin sections on each slide merely contain 2-D cross-sections of 3-D cells. While some MNGs can be clearly identified on a slide because multiple nuclei are clearly visible, in many cases the cross section does not slice through all the nuclei. In some cases, the second nucleus is only faintly visible, and in others it may not be visible at all, even though part of the cell is visible on that cross-section. In other cases, two nuclei may be very close together, and it may not be clear whether they are separated by cell membranes or not. Many cells contain localized hematoxylin-stained cytoplasmic regions that cannot be easily distinguished from the edge of a nucleus, even by a highly trained eye.

To develop the most robust possible set of training data would require serial sectioning, a pseudo-3D approach. Under this approach, three serial thin sections would be taken from each sample, with the top and bottom layers informing the identification of MNGs on the middle layer, as in \[[@pone.0229967.ref004]\]. In this approach, scoring is performed in the central section. Ambiguous MNG identifications in the central section can often be confirmed on an adjacent section, because MNGs are significantly larger than the 5 µm diameter of the sections. When an MNG can be clearly identified in the top or the bottom layer but is unclear in the middle layer, then we should be able to more confidently identify it in the middle layer. This way, we should be able to build up a much more accurate set of training data. With those more accurate training data, we should be able to train a model that may well exceed human accuracy.

The error inherent in human modeling is such that our "true" labels produced by the human reference scorer are almost certainly at least partially inaccurate. When the model differs from the "true" labels, it is always possible that the model is correct, and the "true" labels are not. Additionally, human scorers may be more likely to err in similar ways, and the F1 scores between the scores of human scorers may be somewhat elevated for this reason. For instance, human scorers may be unusually prone to miss an MNG, compared to a trained model that will always examine all of the image.

We also note that our model used the same laboratory to produce and process all the slides. A different group reproducing these results might well wind up with images with slightly different treatment, lighting, and end image conditions. We cannot ensure that our model will achieve the same level of accuracy on images produced by other groups in other laboratories. We have, in a sense, overfit the model to the particular researchers and set-up used to produce the images we have analyzed. By training the model with a much wider array of researchers and laboratory set-ups, this issue could be ameliorated through future work.

We were careful to do everything we could to minimize this sort of overfitting. One crucial step we took was to ensure that each slide would fall in its own fold. We did not split up multiple testes on a single slide and place them in different folds. As a result, we controlled against overfitting to the particular conditions of an individual slide, such as lighting. Had we split up the slides, we could have achieved much more equal folds, without large variability in the total number of MNGs and the types of MNGs in each fold.

The resulting variability among the folds is a major driver of error in accurately assessing the F1 scores. Even among folds with similar MNG counts, some folds will be easier for the model to count than others. For instance, in the model with the best holdout set score, where Fold 3 is the holdout set, the holdout set score, 0.805, is higher than the test set score, 0.756. In all the other cases, the holdout set score is lower than the test set score, which makes sense because the model in the test set has been optimized to produce the best possible F1 score for the test set. So the high F1 score when Fold 3 is the holdout set may be partially driven by Fold 3 being easier to count. When Fold 3 is used as the test set, it produces a slightly higher test set F1 score than any other fold configuration, which is additional evidence that Fold 3 may simply be easier for the model to count.

5. Conclusions {#sec010}
==============

The MNG identification problem is a difficult task for humans to perform consistently. We have shown that a convolutional neural network using the U-Net architecture can approach near human accuracy and, in the case of the best model, exceed it. This new automated approach is significantly faster and involves much less human input, which will facilitate the generation of dose-response data for induction of MNGs by phthalates. The code for applying the trained model can be downloaded from github.com/brown-ccv/mngcount, and the code for training the model can be found at github.com/samwbell/train_unet_mng.

We would like to thank Christy Lambright and Earl Gray from the Environmental Protection Agency for conducting the rat phthalate exposures and providing the tissue samples used in this study. We would like to thank Sklyar Loeb and Rebka Ephrem for their assistance with cell boundary identification during the creation of the training data. The manuscript has been subjected to review by the U.S. EPA Center for Public Health and Environmental Assessment and approved for publication, but the views expressed do not necessarily reflect the views or the policy of the U.S. EPA. Part of this research was conducted using computational resources and services at the Center for Computation and Visualization, Brown University.

10.1371/journal.pone.0229967.r001

Decision Letter 0

Schlatt

Stefan

Academic Editor

© 2020 Stefan Schlatt

2020

Stefan Schlatt

This is an open access article distributed under the terms of the

Creative Commons Attribution License

, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

11 Mar 2020

PONE-D-20-04638

Automated identification of multinucleated germ cells with U-Net

PLOS ONE

Dear Dr. Bell,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE's publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

This paper describes an automated process to determine the number of multinucleated cells in testicular tissue sections. I agree with the reviewers that the study is of importance but also support their critical views on some details. The authors need to deal with the very thoughtful comments of the reviewers, provide adequate responses and revise the paper accordingly.

We would appreciate receiving your revised manuscript by Apr 25 2020 11:59PM. When you are ready to submit your revision, log on to <https://www.editorialmanager.com/pone/> and select the \'Submissions Needing Revision\' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: <http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols>

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled \'Response to Reviewers\'.A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled \'Revised Manuscript with Track Changes\'.An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled \'Manuscript\'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Stefan Schlatt

Academic Editor

PLOS ONE

Journal Requirements:

1\. When submitting your revision, we need you to address these additional requirements.

 

Please ensure that your manuscript meets PLOS ONE\'s style requirements, including those for file naming. The PLOS ONE style templates can be found at

<http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf> and <http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf>

2.  Thank you for stating the following in the Competing Interests section:

\"Justin Conley is employed by the US Environmental Protection Agency (USEPA).  This manuscript has been internally reviewed by the USEPA.\".

\* Please confirm that this does not alter your adherence to all PLOS ONE policies on sharing data and materials, by including the following statement: \"This does not alter our adherence to  PLOS ONE policies on sharing data and materials." (as detailed online in our guide for authors <http://journals.plos.org/plosone/s/competing-interests>).  If there are restrictions on sharing of data and/or materials, please state these. Please note that we cannot proceed with consideration of your article until this information has been declared.

\* Please include your updated Competing Interests statement in your cover letter; we will change the online submission form on your behalf.

Please know it is PLOS ONE policy for corresponding authors to declare, on behalf of all authors, all potential competing interests for the purposes of transparency. PLOS defines a competing interest as anything that interferes with, or could reasonably be perceived as interfering with, the full and objective presentation, peer review, editorial decision-making, or publication of research or non-research articles submitted to one of the journals. Competing interests can be financial or non-financial, professional, or personal. Competing interests can arise in relationship to an organization or another person. Please follow this link to our website for more details on competing interests: <http://journals.plos.org/plosone/s/competing-interests>

\[Note: HTML markup is below. Please do not edit.\]

Reviewers\' comments:

Reviewer\'s Responses to Questions

**Comments to the Author**

1\. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer \#1: Yes

Reviewer \#2: Yes

\*\*\*\*\*\*\*\*\*\*

2\. Has the statistical analysis been performed appropriately and rigorously?

Reviewer \#1: N/A

Reviewer \#2: I Don\'t Know

\*\*\*\*\*\*\*\*\*\*

3\. Have the authors made all data underlying the findings in their manuscript fully available?

The [PLOS Data policy](http://www.plosone.org/static/policies.action#sharing) requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data---e.g. participant privacy or use of data from a third party---those must be specified.

Reviewer \#1: Yes

Reviewer \#2: Yes

\*\*\*\*\*\*\*\*\*\*

4\. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer \#1: Yes

Reviewer \#2: Yes

\*\*\*\*\*\*\*\*\*\*

5\. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer \#1: The manuscript \"Automated identification of multinucleated germ cells with U-Net\" describes the establishment and validation of a neural net with a U-Net architecture to identify multinucleated germ cells (MGN). MGN are formed due to exposure of phthalates. Phthalates are reproductive toxicants that derive from plastic softeners and are most likely responsible for a diminished fertility, also given from mother to fetus during pregnancy. As MGN are not easy to identify, an automated system is needed. The authors describe in a detailed way the establishement, training and validation of the U-Net neural net. There, they also describe the limitations of the system and how to overcome these. As a result, the U-Net is found to be as good as the human reviewer in most cases.

Minor points of criticism:

Two references (e.g. Kavlock et al. 2006 and Ronneberger et al. 2015) are not cited in the reference list.

Isn\'t citing in Plos ONE performed using numbers in the text?

Reviewer \#2: The manuscript aims to design and test an automated system to quantify multinucleated gonocytes (MNG) in testicular tissue sections. The study concludes that the automated program performed close to human accuracy.

General Comments

Overall the study appears technically sound. Methods to improve accuracy and efficiency of quantification of cell types are important to establish. The main areas for improvement relate to the need for a clear definition of what constitutes an MNG and their biological importance. In addition, the interpretation of the results and the utility of the system for obtaining accurate numbers of MNGs in practice should be developed. The 'gold standard' used to determine the accuracy of human and automated counting is the opinion of one experienced observer rather than an objective measure. Whilst the authors report that the models perform close to human accuracy, the human quantification performed relatively poorly compared to the 'gold standard' in terms of a high number of false negatives and false positives.

Specific Points

Page 9 Line 40 -- 'potential toxicity' or cite direct evidence for effects of phthalates on human reproductive tract

Page 9 Line 42 -- mentioned critical window in abstract. No mention of the MPW here

Page 9 Line 49 -- there is no definition of an MNG. How many nuclei? How are they distinguished from a mitotic cell

Page 9 Line 50 -- in which species?

Page 9 Line 52 -- There is no mention of the biological importance of MNG. Given that MNGs are also identified in normal testis tissue the relevance of these should be described. Worth commenting on differences in MNG/GC aggregation between rodent and human (e.g. van den Driesche, Environmental Health Perspectives, 2015)

Page 11 Line 98 -- provide a link or citation to this implementation

Page 12 Line 116 -- explain how the pixel brightness indicates an MNG

Page 13 Line 134 -- provide ethics reference number for the animal work

Page 16 Line 214 -- how were the 3 folds for the training set chosen

Page 18 Line 258 -- it is not clear how this relates to the values on the figures?

Page 19 Line 280 -- what are the units of measurement for the '20'

Page 19 Line 282 -- explain the significance of this finding

Page 20 Table 1 -- final column '0.710' etc

Page 20 Line 294 -- how much more accurate is the model for this holdout set?

Page 21 Line 297 -- the wording here is confusing and difficult to interpret

Page 21 Line 299 -- why is this important? Why is it better to have more false negatives than positives? Is it not more relevant to say how many incorrect classifications there were?

Page 21 Table 2 -- the number of incorrect classifications is still very high. There are 32-46% incorrect classifications in the five folds compared with \~35% incorrect classifications in the human data. For discussion on whether any of the methods are accurate enough

Page 22 Line 309 -- how are the authors planning to improve the training data?

Page 23 Line 333 -- why has the additional training data not been performed?

Page 23 Line 336 -- is there a possibility of having more than one 'expert' to determine consistency and agreement for 'gold standard'

Page 24 Line 351 -- discuss whether there are other ways of identifying MNGs e.g. other techniques or cellular products etc

Page 25 Line 378 -- there is no definitive conclusion on whether human assessment or the new system are accurate methods for quantification of MNGs. This makes it difficult for the reader to know how to apply the findings to their own research

\*\*\*\*\*\*\*\*\*\*

6\. PLOS authors have the option to publish the peer review history of their article ([what does this mean?](https://journals.plos.org/plosone/s/editorial-and-peer-review-process#loc-peer-review-history)). If published, this will include your full peer review and any attached files.

If you choose "no", your identity will remain anonymous but your review may still be made public.

**Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our [Privacy Policy](https://www.plos.org/privacy-policy).

Reviewer \#1: No

Reviewer \#2: No

\[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link \"View Attachments\". If this link does not appear, there are no attachment files to be viewed.\]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, <https://pacev2.apexcovantage.com/>. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at <figures@plos.org>. Please note that Supporting Information files do not need this step.

10.1371/journal.pone.0229967.r002

Author response to Decision Letter 0

18 May 2020

We have uploaded a specific response to reviewers document, where our responses are color-coded to differentiate them from the reviewers\' comments. We believe that document is more readable.

However, we are pasting its text here, with our responses indicated by \"\>\>\>\" instead of color-coding:

Reviewer \#1: The manuscript \"Automated identification of multinucleated germ cells with U-Net\" describes the establishment and validation of a neural net with a U-Net architecture to identify multinucleated germ cells (MGN). MGN are formed due to exposure of phthalates. Phthalates are reproductive toxicants that derive from plastic softeners and are most likely responsible for a diminished fertility, also given from mother to fetus during pregnancy. As MGN are not easy to identify, an automated system is needed. The authors describe in a detailed way the establishement, training and validation of the U-Net neural net. There, they also describe the limitations of the system and how to overcome these. As a result, the U-Net is found to be as good as the human reviewer in most cases.

Minor points of criticism:

Two references (e.g. Kavlock et al. 2006 and Ronneberger et al. 2015) are not cited in the reference list.

\>\>\>Added them. We do apologize for this omission.

Isn\'t citing in Plos ONE performed using numbers in the text?

\>\>\>We do apologize for not using numbers in the text, and we have gone back and added them in.

Reviewer \#2: The manuscript aims to design and test an automated system to quantify multinucleated gonocytes (MNG) in testicular tissue sections. The study concludes that the automated program performed close to human accuracy.

General Comments

Overall the study appears technically sound. Methods to improve accuracy and efficiency of quantification of cell types are important to establish. The main areas for improvement relate to the need for a clear definition of what constitutes an MNG and their biological importance. In addition, the interpretation of the results and the utility of the system for obtaining accurate numbers of MNGs in practice should be developed. The 'gold standard' used to determine the accuracy of human and automated counting is the opinion of one experienced observer rather than an objective measure. Whilst the authors report that the models perform close to human accuracy, the human quantification performed relatively poorly compared to the 'gold standard' in terms of a high number of false negatives and false positives.

\>\>\>Both in this paper and in general, an MNG is defined as any germ cell with more than one nucleus in shared cytoplasmic space. We assume that the reviewer means that the performance of non-expert scorers against the expert scorer ("gold standard") was relatively poor. We agree and would argue that the poor performance of humans on this task is part of the motivation for this research. Because MNG identification is a problem that humans struggle at, automation is both more difficult and more necessary. Further, as with any assessment of pathology, there will be some error inherent in identification of histological features, even when performed by an expert.

Specific Points

Page 9 Line 40 -- 'potential toxicity' or cite direct evidence for effects of phthalates on human reproductive tract

\>\>\>We have added the word "potential" to reflect the uncertainty about effect levels in humans. However, given that there is considerable evidence of toxicity in experimental models using human tissue and in the epidemiological literature, we have added additional citations to support this claim.

Page 9 Line 42 -- mentioned critical window in abstract. No mention of the MPW here

\>\>\>The critical window is not really essential to this study, so we have removed the mention of it in the abstract.

Page 9 Line 49 -- there is no definition of an MNG. How many nuclei? How are they distinguished from a mitotic cell

\>\>\>Added a clarification that MNGs are defined as containing two or more nuclei. MNGs do not display the chromatin condensation that is observed during mitosis. In this study, samples were obtained on gestation day 21, which falls within the quiescent period when germ cells are mitotically inactive (described in Culty et al. 2013. Biol Reprod 89:46). We have also previously shown that the sensitive window for MNG induction begins on GD 18, coincident with the initiation of the quiescent period, and that MNGs form through a non-proliferative mechanism, presumably the collapse of intercellular bridges (Spade et al. 2015. Biol Reprod 93:110).

Page 9 Line 50 -- in which species?

\>\>\>This was altered to clarify that few studies have quantified the dose-response for MNG induction by most phthalates in any species. The next paragraph expands on the reasons for this, which include the technical difficulty and time required to count MNGs.

Page 9 Line 52 -- There is no mention of the biological importance of MNG. Given that MNGs are also identified in normal testis tissue the relevance of these should be described. Worth commenting on differences in MNG/GC aggregation between rodent and human (e.g. van den Driesche, Environmental Health Perspectives, 2015)

\>\>\>We have added a discussion on the biological significance of MNGs and their induction by phthalates in rats, mice and humans. Thank you for the suggested reference.

Page 11 Line 98 -- provide a link or citation to this implementation

\>\>\>Link added.

Page 12 Line 116 -- explain how the pixel brightness indicates an MNG

\>\>\>Added some clarifications. This is a bit difficult to explain because there is not explicit formula. The model is set up to try to reproduce the binary training images, where the pixels covered by MNGs have a pixel value of 255, and the pixels not covered by MNGs have a pixel value of 0. In the output map after an image has been run through the trained model, the closer the pixel value is to 255 the likelier it is that it is covered by an MNG, but there is no exact formula. It is specific to the model, and because neural networks are black boxes, we don't know the exact way the model is producing it or what exactly the model means by pixel brightness.

Page 13 Line 134 -- provide ethics reference number for the animal work

\>\>\>This has been updated to indicate that the animal work was performed under Laboratory Animal Project Review \#19-03-001 at the USEPA National Health and Environmental Effects Research Laboratory.

Page 16 Line 214 -- how were the 3 folds for the training set chosen

\>\>\>This is explained below in some detail. Essentially, because there are five folds, the selection of a holdout and test set leaves only three folds left over for the training set. We chose to identify the fold configurations by which fold was used for the holdout set.

Page 18 Line 258 -- it is not clear how this relates to the values on the figures?

\>\>\>We have indicated these values on the figure with a black box.

Page 19 Line 280 -- what are the units of measurement for the '20'

\>\>\>The unit of measurement is pixels. We have added a clarification.

Page 19 Line 282 -- explain the significance of this finding

\>\>\>Added a brief sentence explaining this.

Page 20 Table 1 -- final column '0.710' etc

\>\>\>We're not quite sure how to interpret this comment.

Page 20 Line 294 -- how much more accurate is the model for this holdout set?

\>\>\>If this is asking how much more accurate the best model is than the human scorers, it had a holdout score of 0.805 compared to human scorer values of 0.764 and 0.788. We may not fully understand the question being asked here. We do apologize.

Page 21 Line 297 -- the wording here is confusing and difficult to interpret

\>\>\>We have restructured this as false positive to false negative ratios instead of percentages. We hope this will be less confusing and easier to interpret.

Page 21 Line 299 -- why is this important? Why is it better to have more false negatives than positives? Is it not more relevant to say how many incorrect classifications there were?

\>\>\>It's not inherently better to have more false negatives than false positives, but it is important because it describes how the model fails. For instance, if you have an application where you are more worried about false negatives, then you probably want a model where false negatives are less common than false positives. Maybe the goal is to make sure you don't miss anything. Plenty of machine learning algorithms are tuned in order to make many false positives but minimize false negatives because the goal is as a screening before a final pass human check. Alternatively, there are models where the goal is to make sure you don't make a false identification. Maybe you want to make sure that you don't falsely classify a testis as having an MNG when it doesn't. The F1 score incorporates both into one statistic.

Page 21 Table 2 -- the number of incorrect classifications is still very high. There are 32-46% incorrect classifications in the five folds compared with \~35% incorrect classifications in the human data.

\>\>\>This is true. The mean model performs slightly worse than the average human scorer, but the best model outperforms the average human scorer. However, to even approach human accuracy we consider a good result. The reason why there is a high rate of incorrect classifications is that this is a difficult task even for humans.

For discussion on whether any of the methods are accurate enough

Page 22 Line 309 -- how are the authors planning to improve the training data?

\>\>\>We discuss this in the next two paragraphs. We believe that more accurate training data could be obtained by using three serial thin sections. By using the central section for scoring and the sections above and below the central section to confirm MNG identifications, we believe we could have much more accurate human identification of MNGs. This approach is described in the methods of Spade et al. (2014. Toxicol Sci 138:148). It would be time consuming and laborious to use for the quantity of samples included in the current paper. However, it would provide more accurate training data, which we believe we could build a much better model. In this case, the main limitation on the training data is not so much the quantity as the accuracy.

Page 23 Line 333 -- why has the additional training data not been performed?

\>\>\>This work would require additional funding. We hope to perform this in the future.

Page 23 Line 336 -- is there a possibility of having more than one 'expert' to determine consistency and agreement for 'gold standard'

\>\>\>We did not have enough collaborators on this specific project to do this, but it should be possible in theory through future work. There is a limited community of experts, but perhaps one approach could be to have a broad panel of experts reach consensus on the identifications through discussion and a voting procedure. Such an approach would require extensive collaboration and logistics, but it would be a sensible direction for future research.

Page 24 Line 351 -- discuss whether there are other ways of identifying MNGs e.g. other techniques or cellular products etc

\>\>\>There is no published method of identifying MNGs aside from identification in histological sections. The only technique we are aware of for increasing the accuracy of the count is incorporating serial thin sections in the scoring procedure.

Page 25 Line 378 -- there is no definitive conclusion on whether human assessment or the new system are accurate methods for quantification of MNGs. This makes it difficult for the reader to know how to apply the findings to their own research

\>\>\>We do not attempt to downplay the accuracy issues with human quantification of MNGs. Indeed, they are a primary motivating factor of this research. The ultimate goal is to develop a methodology that can outperform humans, and we believe we have made major progress in that direction. Despite the issues with accurately quantifying the MNG count, MNG counts are still valuable tools in assessing phthalate toxicity. We also believe it is likely that the current model could produce data of sufficient quality for dose-response assessment.

###### 

Submitted filename: Response_to_Reviewers_final.docx

###### 

Click here for additional data file.

10.1371/journal.pone.0229967.r003

Decision Letter 1

Schlatt

Stefan

Academic Editor

© 2020 Stefan Schlatt

2020

Stefan Schlatt

This is an open access article distributed under the terms of the

Creative Commons Attribution License

, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

1 Jun 2020

Automated identification of multinucleated germ cells with U-Net

PONE-D-20-04638R1

Dear Dr. Bell,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at <https://www.editorialmanager.com/pone/>, click the \"Update My Information\" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at <authorbilling@plos.org>.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact <onepress@plos.org>.

With kind regards,

Stefan Schlatt

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers\' comments:

Reviewer\'s Responses to Questions

**Comments to the Author**

1\. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the "Comments to the Author" section, enter your conflict of interest statement in the "Confidential to Editor" section, and submit your \"Accept\" recommendation.

Reviewer \#1: All comments have been addressed

Reviewer \#2: (No Response)

\*\*\*\*\*\*\*\*\*\*

2\. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer \#1: (No Response)

Reviewer \#2: Yes

\*\*\*\*\*\*\*\*\*\*

3\. Has the statistical analysis been performed appropriately and rigorously?

Reviewer \#1: (No Response)

Reviewer \#2: I Don\'t Know

\*\*\*\*\*\*\*\*\*\*

4\. Have the authors made all data underlying the findings in their manuscript fully available?

The [PLOS Data policy](http://www.plosone.org/static/policies.action#sharing) requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data---e.g. participant privacy or use of data from a third party---those must be specified.

Reviewer \#1: (No Response)

Reviewer \#2: Yes

\*\*\*\*\*\*\*\*\*\*

5\. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer \#1: (No Response)

Reviewer \#2: Yes

\*\*\*\*\*\*\*\*\*\*

6\. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer \#1: (No Response)

Reviewer \#2: The authors have addressed the reviewer comments. There are a few instances of spelling and grammar to address and the numbers in the final column of Table 2 should be corrected e.g. \'0.710\' instead of \'.710\'

\*\*\*\*\*\*\*\*\*\*

7\. PLOS authors have the option to publish the peer review history of their article ([what does this mean?](https://journals.plos.org/plosone/s/editorial-and-peer-review-process#loc-peer-review-history)). If published, this will include your full peer review and any attached files.

If you choose "no", your identity will remain anonymous but your review may still be made public.

**Do you want your identity to be public for this peer review?** For information about this choice, including consent withdrawal, please see our [Privacy Policy](https://www.plos.org/privacy-policy).

Reviewer \#1: No

Reviewer \#2: No

10.1371/journal.pone.0229967.r004

Acceptance letter

Schlatt

Stefan

Academic Editor

© 2020 Stefan Schlatt

2020

Stefan Schlatt

This is an open access article distributed under the terms of the

Creative Commons Attribution License

, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

26 Jun 2020

PONE-D-20-04638R1

Automated identification of multinucleated germ cells with U-Net

Dear Dr. Bell:

I\'m pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they\'ll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact <onepress@plos.org>.

If we can help with anything else, please email us at <plosone@plos.org>.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Stefan Schlatt

Academic Editor

PLOS ONE

[^1]: **Competing Interests:**Justin Conley is employed by the US Environmental Protection Agency (USEPA). This manuscript has been internally reviewed by the USEPA. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
