Preparing for the Next Worldwide Tech Outage

As tech leaders race to bring Windows systems back online after Friday’s software update by cybersecurity company CrowdStrike crashed around 8.5 million machines worldwide, experts share with CIO Journal their takeaways for preparing for the next major information technology outage.

Be familiar with how vendors develop, test and release their software

IT leaders should hold vendors deeply integrated within IT systems, such as CrowdStrike , to a “very high standard” of development, release quality and assurance, said Neil MacDonald , a Gartner vice president.

“Any security vendor has a responsibility to do extensive regression testing on all versions of Windows before an update is rolled out,” he said.

That involves asking existing vendors to explain how they write software, what testing they do and whether customers may choose how quickly to roll out an update.

“Incidents like this remind all of us in the CIO community of the importance of ensuring availability, reliability and security by prioritizing guardrails such as deployment and testing procedures and practices,” said Amy Farrow, chief information officer of IT automation and security company Infoblox.

Re-evaluate how your firm accepts software updates from ‘trusted’ vendors

While automatically accepting software updates has become the norm—and a recommended security practice—the CrowdStrike outage is a reminder to take a pause, some CIOs said.

“We still should be doing the full testing of packages and upgrades and new features,” said Paul Davis, a field chief information security officer at software development platform maker JFrog . undefined undefined Though it’s not feasible to test every update, especially for as many as hundreds of software vendors, Davis said he makes it a priority to test software patches according to their potential severity and size.

Automation, and maybe even artificial intelligence-based IT tools, can help.

“Humans are not very good at catching errors in thousands of lines of code,” said Jack Hidary, chief executive of AI and quantum company SandboxAQ. “We need AI trained to look for the interdependence of new software updates with the existing stack of software.”

Develop a disaster recovery plan

An incident rendering Windows computers unusable is similar to a natural disaster with systems knocked offline, said Gartner’s MacDonald. That’s why businesses should consider natural disaster recovery plans for maintaining the resiliency of their operations.

One way to do that is to set up a “clean room,” or an environment isolated from other systems, to use to bring critical systems back online, according to Chirag Mehta, a cybersecurity analyst at Constellation Research.

Businesses should also hold tabletop exercises to simulate risk scenarios, including IT outages and potential cyber threats, Mehta said.

Companies that back up data regularly were likely less impacted by the CrowdStrike outage, according to Victor Zyamzin, chief business officer of security company Qrator Labs. “Another suggestion for companies, and we’ve been saying that again and again for decades, is that you should have some backup procedure applied, running and regularly tested,” he said.

Review vendor and insurance contracts

For any vendor with a significant impact on company operations , MacDonald said companies can review their contracts and look for clauses indicating the vendors must provide reliable and stable software.

“That’s where you may have an advantage to say, if an update causes an outage, is there a clause in the contract that would cover that?” he said.

If it doesn’t, tech leaders can aim to negotiate a discount serving as a form of compensation at renewal time, MacDonald added.

The outage also highlights the importance of insurance in providing companies with bottom-line protection against cyber risks, said Peter Halprin, a partner with law firm Haynes Boone focused on cyber insurance.

This coverage can include protection against business income losses, such as those associated with an outage, whether caused by the insured company or a service provider, Halprin said.

Weigh the advantages and disadvantages of the various platforms

The CrowdStrike update affected only devices running Microsoft Windows-based systems , prompting fresh questions over whether enterprises should rely on Windows computers.

CrowdStrike runs on Windows devices through access to the kernel, the part of an operating system containing a computer’s core functions. That’s not the same for Apple ’s Mac operating system and Linux, which don’t allow the same level of access, said Mehta.

Some businesses have converted to Chromebooks , simple laptops developed by Alphabet -owned Google that run on the Chrome operating system . “Not all of them require deeper access to things,” Mehta said. “What are you doing on your laptop that actually requires Windows?”

There has rarely, if ever, been so much tech talent available in the job market. Yet many tech companies say good help is hard to find.

What gives?

U.S. colleges more than doubled the number of computer-science degrees awarded from 2013 to 2022, according to federal data. Then came round after round of layoffs at Google, Meta, Amazon, and others.

The Bureau of Labor Statistics predicts businesses will employ 6% fewer computer programmers in 2034 than they did last year.

All of this should, in theory, mean there is an ample supply of eager, capable engineers ready for hire.

But in their feverish pursuit of artificial-intelligence supremacy, employers say there aren’t enough people with the most in-demand skills. The few perceived as AI savants can command multimillion-dollar pay packages. On a second tier of AI savvy, workers can rake in close to $1 million a year .

Landing a job is tough for most everyone else.

Frustrated job seekers contend businesses could expand the AI talent pipeline with a little imagination. The argument is companies should accept that relatively few people have AI-specific experience because the technology is so new. They ought to focus on identifying candidates with transferable skills and let those people learn on the job.

Often, though, companies seem to hold out for dream candidates with deep backgrounds in machine learning. Many AI-related roles go unfilled for weeks or months—or get taken off job boards only to be reposted soon after.

Playing a different game

It is difficult to define what makes an AI all-star, but I’m sorry to report that it’s probably not whatever you’re doing.

Maybe you’re learning how to work more efficiently with the aid of ChatGPT and its robotic brethren. Perhaps you’re taking one of those innumerable AI certificate courses.

You might as well be playing pickup basketball at your local YMCA in hopes of being signed by the Los Angeles Lakers. The AI minds that companies truly covet are almost as rare as professional athletes.

“We’re talking about hundreds of people in the world, at the most,” says Cristóbal Valenzuela, chief executive of Runway, which makes AI image and video tools.

He describes it like this: Picture an AI model as a machine with 1,000 dials. The goal is to train the machine to detect patterns and predict outcomes. To do this, you have to feed it reams of data and know which dials to adjust—and by how much.

The universe of people with the right touch is confined to those with uncanny intuition, genius-level smarts or the foresight (possibly luck) to go into AI many years ago, before it was all the rage.

As a venture-backed startup with about 120 employees, Runway doesn’t necessarily vie with Silicon Valley giants for the AI job market’s version of LeBron James. But when I spoke with Valenzuela recently, his company was advertising base salaries of up to $440,000 for an engineering manager and $490,000 for a director of machine learning.

A job listing like one of these might attract 2,000 applicants in a week, Valenzuela says, and there is a decent chance he won’t pick any of them. A lot of people who claim to be AI literate merely produce “workslop”—generic, low-quality material. He spends a lot of time reading academic journals and browsing GitHub portfolios, and recruiting people whose work impresses him.

In addition to an uncommon skill set, companies trying to win in the hypercompetitive AI arena are scouting for commitment bordering on fanaticism .

Daniel Park is seeking three new members for his nine-person startup. He says he will wait a year or longer if that’s what it takes to fill roles with advertised base salaries of up to $500,000.

He’s looking for “prodigies” willing to work seven days a week. Much of the team lives together in a six-bedroom house in San Francisco.

If this sounds like a lonely existence, Park’s team members may be able to solve their own problem. His company, Pickle, aims to develop personalised AI companions akin to Tony Stark’s Jarvis in “Iron Man.”

Overlooked

James Strawn wasn’t an AI early adopter, and the father of two teenagers doesn’t want to sacrifice his personal life for a job. He is beginning to wonder whether there is still a place for people like him in the tech sector.

He was laid off over the summer after 25 years at Adobe , where he was a senior software quality-assurance engineer. Strawn, 55, started as a contractor and recalls his hiring as a leap of faith by the company.

He had been an artist and graphic designer. The managers who interviewed him figured he could use that background to help make Illustrator and other Adobe software more user-friendly.

Looking for work now, he doesn’t see the same willingness by companies to take a chance on someone whose résumé isn’t a perfect match to the job description. He’s had one interview since his layoff.

“I always thought my years of experience at a high-profile company would at least be enough to get me interviews where I could explain how I could contribute,” says Strawn, who is taking foundational AI courses. “It’s just not like that.”

The trouble for people starting out in AI—whether recent grads or job switchers like Strawn—is that companies see them as a dime a dozen.

“There’s this AI arms race, and the fact of the matter is entry-level people aren’t going to help you win it,” says Matt Massucci, CEO of the tech recruiting firm Hirewell. “There’s this concept of the 10x engineer—the one engineer who can do the work of 10. That’s what companies are really leaning into and paying for.”

He adds that companies can automate some low-level engineering tasks, which frees up more money to throw at high-end talent.

It’s a dynamic that creates a few handsomely paid haves and a lot more have-nots.