  Regular maintenance of all the assets is pivotal for proper functioning of
railway. Manual maintenance can be very cumbersome and leave room for errors.
Track anomalies like vegetation overgrowth, sun kinks affect the track
construct and result in unequal load transfer, imbalanced lateral forces on
tracks which causes further deterioration of tracks and can ultimately result
in derailment of locomotive. Hence there is a need to continuously monitor rail
track health. Track anomalies are rare with the skew as high as one anomaly in
millions of good images. We propose a method to build training data that will
make our algorithms more robust and help us detect real world track issues. The
data augmentation will have a direct effect in making us detect better
anomalies and hence improve time for railroads that is spent in manual
inspection. This paper talks about a real world use case of detecting railway
track defects from a camera mounted on a moving locomotive and tracking their
locations. The camera is engineered to withstand the environment factors on a
moving train and provide a consistent steady image at around 30 frames per
second. An image simulation pipeline of track detection, region of interest
selection, augmenting image for anomalies is implemented. Training images are
simulated for sun kink and vegetation overgrowth. Inception V3 model pretrained
on Imagenet dataset is finetuned for a 2 class classification. For the case of
vegetation overgrowth, the model generalizes well on actual vegetation images,
though it was trained and validated solely on simulated images which might have
different distribution than the actual vegetation. Sun kink classifier can
classify professionally simulated sun kink videos with a precision of 97.5%.
