  Since February of this year, KOA began to prepare, transfer, and ingest data
as they were acquired in near-real time; in most cases data are available to
observers through KOA within one minute of acquisition. Real-time ingestion
will be complete for all active instruments by the end of Summer 2022. The
observatory is supporting the development of modern Python data reduction
pipelines, which when delivered, will automatically create science-ready data
sets at the end of each night for ingestion into the archive. This presentation
will describe the infrastructure developed to support real-time data ingestion,
itself part of a larger initiative at the Observatory to modernize end-to-end
operations.
  During telescope operations, the software at WMKO is executed automatically
when a newly acquired file is recognized through monitoring a keyword-based
observatory control system; this system is used at Keck to execute virtually
all observatory functions. The monitor uses callbacks built into the control
system to begin data preparation of files for transmission to the archive on an
individual basis: scheduling scripts or file system related triggers are
unnecessary. An HTTP-based system called from the Flask micro-framework enables
file transfers between WMKO and NExScI and triggers data ingestion at NExScI.
The ingestion system at NEXScI is a compact (4 KLOC), highly fault-tolerant,
Python-based system. It uses a shared file system to transfer data from WMKO to
NExScI. The ingestion code is instrument agnostic, with instrument parameters
read from configuration files. It replaces an unwieldy (50 KLOC) C-based system
that had been in use since 2004.
