As organizations brace themselves for a possible recession, many are seeking ways to reduce operational costs. When we seek efficiency, automation is not only our friend, but often the only practical approach. In the world of IT, automation is achieved through software, and it allows us to operate in ways that are simply impossible for individual humans or even teams of humans to achieve. Automation enables us to take simple tasks and repeat them at speed and massive scale.
A few years ago, I was at a conference, and I heard a speaker say that they had eliminated human error from their IT operations because they had automated everything. It sounds reasonable, doesn’t it? Computers are designed to yield the same outcome time and time again without error. Ideally, they preclude human limitations, such as fatigue, attention loss, and lack of consistency, that could otherwise tarnish processes.
SOMETHING IS MISSING
However, I believe this logic misses one important point: We rely on software to provide our automation, and even though software programs are often generated by other software programs, humans will ultimately be involved in the process at some point.
When we build layers upon layers of software, bugs and other errors at the bottom of the stack can have far-reaching consequences. Consider the Facebook outage that occurred in 2021. Human operator error during a routine maintenance job resulted in their entire network backbone being disabled. Ultimately, this rendered Facebook, Instagram, WhatsApp, and several other Meta products unavailable for over seven hours. While the source of this outage was human error, the automation replicated it across Meta’s entire network.
An interesting aspect of the Facebook example is that an automation script was in place to provide a safety net to prevent this very scenario from playing out. The problem? By Facebook’s own admission, there was a bug in the safety-net script. This example shows how a single bug buried in a very large complex system can have far-reaching consequences.
THE CONSEQUENCES OF RELYING ON AI
At this point, some may wonder if generative AI algorithms like ChatGPT are the answer for organizations seeking more robust automation. Perhaps they can generate software without bugs—if not now, then in the future. While this might be possible, just because a piece of software doesn’t contain any coding mistakes doesn’t mean it will do exactly as we expect. There are many examples of AI and machine learning algorithms taking their creators by surprise, like recruitment algorithms and chatbots prone to making biased recommendations and statements.
Another consideration includes the consequences of connecting multiple automated systems. The need to interconnect different IT systems is almost as necessary as automation. However, that can also lead to unintended consequences.
For example, let’s say that someone decides it’s a good idea to allow System A and System B to talk to each other, while someone else independently decides to allow System B to talk to System C. Interestingly, these two parties have unwittingly created a new supersystem comprising all three systems. System A can now indirectly influence System C via System B—whether this interaction was desired or not. The emergence of such supersystems in the modern world is inevitable, but they can—and will—yield unexpected results at the speed and scale provided by automation.
WHAT LEADERS CAN DO
So, what can leaders do? Well, two things: The first action is to deploy systems that provide independent visibility of your IT systems—and not just the visibility of the behavior of individual IT systems so automation errors can be quickly detected and remedied. This approach should provide holistic visibility, too, so that unexpected behaviors emerging from unexpected interactions can also be identified.
The second action is to ensure that the results are presented to human beings for review and possibly even authorization if an action affects mission-critical systems. This is not a trivial task because of the vast amounts of information involved, and the monitoring systems themselves will inevitably rely on automation, and possibly AI, to produce their output. Do IT leaders risk repeating the Facebook example where the safety system also failed? Possibly. But I believe providing no oversight at all is arguably reckless.
Paul Barrett is CTO for NETSCOUT, overseeing development of network assurance and cybersecurity tech for the world’s largest organizations.