The watchdog timers, instituted soon after the restart of the first test point, protected the IWRS from network failures through-out the remainder of the test.
However, with the loss of power, as well as the numerous times the IWRS had to be halted because of hardware failures--about 25 times during the course of the test--it became important to be able to restart the system quickly, without searching the logs manually to determine the state the system was in before the malfunction.
Early in the IWRS work, however, as the hardware groups were setting their test goals, the controls group put forth a control goal of 95-percent unattended operations as a way of getting the water team to start thinking about autonomous operations.
With the IWRS, we, as monitors of the software controls system, shared many of the same concerns as our intended users, the engineers monitoring the ALS hardware.
Reviewing Performance History Although only monitoring data intermittently, the IWRS engineers needed to be able to review performance history to detect system anomalies or indicators that an anomaly was developing.
Accessing Reference Information From interactive parts of the GUIs, the user could display the skills specifications (figure 5), allowing operators to refresh their understanding of how the controls work and providing access to device nomenclature from standard IWRS drawings used in the hardware specifications.
Advanced Techniques Because the software development resources were limited, we were unable to try a number of advanced automation techniques to support intermittent monitoring of IWRS and software controls systems.
When IWRS engineers wanted to know when automated sloughing took place so they could monitor it during initial deployment, we had RAPs broadcast a sloughing message and then integrated device-level data to accompany this communication.
Nonetheless, our experience showed that just as the procedure tracking log was used regularly during the course of the IWRS tests in the water laboratory to determine the cause of anomalies, the web-based tracking log information made it possible to conduct similar activities remotely.
Resources were not available to provide automated fault diagnosis, so when major problems occurred, the team could take the IWRS "out of test" to spend several days analyzing and solving the problem.
The most rewarding experience in the two-year IWRS test came when one of the biological engineers asked the control team to use the top tier of 3T to help them with a particularly troublesome aspect of the BWP.
Toward the end of the last IWRS test-point run, 3T was sloughing one tube an hour each day and night.