  Concerns about reproducibility in artificial intelligence (AI) have emerged,
as researchers have reported unsuccessful attempts to directly reproduce
published findings in the field. Replicability, the ability to affirm a finding
using the same procedures on new data, has not been well studied. In this
paper, we examine both reproducibility and replicability of a corpus of 16
papers on table structure recognition (TSR), an AI task aimed at identifying
cell locations of tables in digital documents. We attempt to reproduce
published results using codes and datasets provided by the original authors. We
then examine replicability using a dataset similar to the original as well as a
new dataset, GenTSR, consisting of 386 annotated tables extracted from
scientific papers. Out of 16 papers studied, we reproduce results consistent
with the original in only four. Two of the four papers are identified as
replicable using the similar dataset under certain IoU values. No paper is
identified as replicable using the new dataset. We offer observations on the
causes of irreproducibility and irreplicability. All code and data are
available on Codeocean at https://codeocean.com/capsule/6680116/tree.
