Starfleet should have tested M-5 by linking it to another computer-- a virtual simulator that the M-5 thinks is the real deal.
James P. Hogan addressed this in THE TWO FACES OF TOMORROW, a novel about the creation of the first "true" artificial intelligence. The story begins with an accident caused by a semi-intelligent computer performing an action that seemed like a good idea. The act was highly creative, but demonstrated a lack of "common sense" and judgment.
The dilemma is that computers of the same sort run the rest of human civilization. Going back is out of the question—that would sacrifice the many advances, economy and very lives of far too many people. And continuing with the current generation of computers is untenable following the revealing accident.
While trying to work out a solution, the reader is shown researchers working in the lab with a new generation AI that learns how to deal with the real world by working in a simulation driven by another computer. The problem is the same as the real world problem with the existing computers—reality is just too complicated to plot out in every facet. And putting a human in the loop to provide "judgment" defeats the purpose of using the computers to manage the volume of civilization's daily interactions.
So the next generation AI is placed in charge of a new O'Neill-style space colony, a smaller yet suitably detailed proxy of the world. This should
protect Earth in case the experiment gets out of hand, but the AI evolves far faster than anyone had imagined possible...