  This paper considers the automatic classification of herding behavior in the
cluttered low-visibility environment that typically surrounds towed fishing
gear. The paper compares three convolutional and attention-based deep action
recognition network architectures trained end-to-end on a small set of video
sequences captured by a remotely controlled camera and classified by an expert
in fishing technology. The sequences depict a scene in front of a fishing trawl
where the conventional herding mechanism has been replaced by directed laser
light. The goal is to detect the presence of a fish in the sequence and
classify whether or not the fish reacts to the lasers. A two-stream CNN model,
a CNN-transformer hybrid, and a pure transformer model were trained end-to-end
to achieve 63%, 54%, and 60% 10-fold classification accuracy on the three-class
task when compared to the human expert. Inspection of the activation maps
learned by the three networks raises questions about the attributes of the
sequences the models may be learning, specifically whether changes in viewpoint
introduced by human camera operators that affect the position of laser lines in
the video frames may interfere with the classification. This underlines the
importance of careful experimental design when capturing scientific data for
automatic end-to-end evaluation and the usefulness of inspecting the trained
models.
