Data center operations individuals and teams are expected to be the facility experts on everything from backup generators, chillers, or various electrical equipment. While many systems are automated, operators are always expected to respond to emergencies. Some automated systems might fail, and the operators should be ready to respond. This is where the title comes into play… OPERATIONAL PARANOIA.
Anticipating failures is a result of experience and knowledge of engineers and operators. Operators work daily in the data centers, being the hands, feet, eyes, and ears to understand normal/nominal system and equipment operations. Being around equipment in normal operations or also in maintenance conditions gives us a really good idea of what to expect when something DOESN'T give the nominal sounds, indications, readings, etc. Excessive vibrations could be a soon-to-fail fan, high or low voltages could indicate issues before the alarm or breaker trip. Basically, having that questioning attitude and “spidey sense” about equipment and automations not operating normally is having that operational paranoia to anticipate possible or probable failures that can affect redundancy, resiliency, customer uptime, safety, and many other things.
Operational paranoia also comes into play during maintenance. Asking yourself when preparing for the maintenance and walking through the operating procedure, “If this fails to operate as expected when we do this preventative maintenance, what is the effect and how do we normalize and restore the system?” I recently read about an issue at a site where the loadbank they were using for generator maintenance was placed near the air intakes for cooling equipment, causing high supply air temps in the data center. This was a good example of lack of that operational paranoia in operations…
Can one be too paranoid? Yes absolutely, and this is where a healthy risk assessment and collaboration with other operations peers and leaders to evaluate suspicions. Usually a good risk assessment of the likelihood of a suspected event and the area of impact of the event can help determine which approach to take to operations, maintenance, or an emergency recovery. Being too operationally paranoid can cause delay of maintenance, or even worse, a lack of action on an important recovery needed to restore normal operations. Best to address concerns with peers and leaders during planning phases, and bounce ideas off of other support teams to establish those credible ideas that might cause impact, and also an opportunity to learn more about a system or procedure.
Having a HEALTHY level of operational paranoia can help you and your team operate as robustly and reliably as possible, and achieve maximum uptime and less human caused failures.
All my articles are handwritten/typed by me. By reading and sharing these, you support this work and you support real human authorship.