I work as the primary developer and IT administrator for a small business. I want to ensure that business can continue even if I suddenly become unavailable for some reason. Much of what I do requires access to a number of servers, (through key-based ssh), cloud services, and other secure infrastructure of applications. Some of these services use MFA, either using dedicated MFA apps (like Amazon) or SMS.

How do I ensure that my "hit by a bus" plan and documentation is complete and comprehensive, but that this documentation is not itself a security risk?

The documentation will be hosted on a shared file server behind our VPN, but that can also be accessed using a third party web frontend that puts a "DropBox"-like interface on top of the base file server (i.e. authentication, desktop syncing, file sharing, etc). The files are in a location where only I, and other file server administrators can see them.

How should I manage the "secrets" (passwords, private keys, MFA access) in this documentation to ensure it remains comprehensive without compromising security?

My advice would be to remove the secrets from the drop-box and store them elsewhere.  Your instructions have to be easily human readable by anyone, but they can include instructions on how to get access to the properly secured part of the data.  That lets you separate the accessibility side of things from the security side.

Once you can think about security on its own, you can start to ask the real question of how much do you need to protect these keys?  This is a business logic question, so consult your management.  You might:


Have a password to a file that everyone knows
Have a file set up with multiple passwords so that each individual maintains their own copy.
Have a file locked by a M-out-of-N algorithm (the digital equivalent of requiring two keys to unlock a safe).
Have a M-out-of-N algorithm with one 'master' password required regardless of which group of individuals unlocks the file, and that one master is physically kept in a tamper-evident safe that you check every now and then.


Use creativity here.  Whatever you do, the decoupling of "instructions" with "sensitive information" frees you to properly safeguard the information, and then provide instructions on how to get that data later.

Your business logic decisions will also include uptime questions.  If something goes wrong in your life, how long does another admin have to take over your work before business is affected?  Consider how well replicated you want these instructions and sensitive information to be in case of server glitches.  When I was administering a server and needed to store instructions on how to restore it from backup, I used the server's own wiki to store that information for easy viewing, but obviously that wouldn't be so useful in a glitch scenario, so I also had a copy on the dev VM of that machine, saved off copies of it on 3 separate PCs and a printout.  I couldn't guarantee the printout would stay up to date, but I made sure that I could do my best.

This also points out something which is not always part of a hit by the bus plan: graceful degradation.  Not all hit-by-the-bus scenarios involve getting hit by a bus.  Some involve you just being inaccessible at an unfortunate time.  Some involve you leaving the company, but being available for a question or two.  Others... do not.  Consider layering the plan.  Small mishaps may be very well protected, while greater mishaps may still result in business loss while everyone gets things together, but nothing permanent.  To use my backup restoration plan as an example, the printed version was almost guaranteed not to be fully up to date.  But if lightning wiped out every computer for a city block, it was still more helpful than nothing.  On the other hand, if the server just thew a harddrive, and I had to restore from backup, the version I kept sync'd on the dev box was almost certainly up to date.

Example of this failing: I was a user on a network managed by KERBEROS by an admin that was distrustful of others and did not have a hit-by-a-bus plan.  When he... left, we had a hacking party to try to break his server.  In the end, our best impromptu hit-by-a-bus plan was to wipe the machines (every one of them) and start from scratch.  Note that, while this wasn't the best plan (in fact, I think its the worst plan?), the business kept moving.  We just got stagnated for about two days and had a bunch of grumpy customers.  In the words of Frank Herbert's Dune, "The spice must flow."  Even in the worst case (which may involve a curious incident involving your server's harddrive flung out of the bus and hitting you on the head, destroying all record of the hit-by-a-bus plan), business does have a way to keep moving... but I approve of trying to raise the bar a wee bit more than that!
