  Defining test oracles is crucial and central to test development, but manual
construction of oracles is expensive. While recent neural-based automated test
oracle generation techniques have shown promise, their real-world effectiveness
remains a compelling question requiring further exploration and understanding.
This paper investigates the effectiveness of TOGA, a recently developed
neural-based method for automatic test oracle generation by Dinella et al. TOGA
utilizes EvoSuite-generated test inputs and generates both exception and
assertion oracles. In a Defects4j study, TOGA outperformed specification,
search, and neural-based techniques, detecting 57 bugs, including 30 unique
bugs not detected by other methods. To gain a deeper understanding of its
applicability in real-world settings, we conducted a series of external,
extended, and conceptual replication studies of TOGA.
  In a large-scale study involving 25 real-world Java systems, 223.5K test
cases, and 51K injected faults, we evaluate TOGA's ability to improve
fault-detection effectiveness relative to the state-of-the-practice and the
state-of-the-art. We find that TOGA misclassifies the type of oracle needed 24%
of the time and that when it classifies correctly around 62% of the time it is
not confident enough to generate any assertion oracle. When it does generate an
assertion oracle, more than 47% of them are false positives, and the true
positive assertions only increase fault detection by 0.3% relative to prior
work. These findings expose limitations of the state-of-the-art neural-based
oracle generation technique, provide valuable insights for improvement, and
offer lessons for evaluating future automated oracle generation methods.
