Preparing for the Next Worldwide Tech Outage

As tech leaders race to bring Windows systems back online after Friday’s software update by cybersecurity company CrowdStrike crashed around 8.5 million machines worldwide, experts share with CIO Journal their takeaways for preparing for the next major information technology outage.

Be familiar with how vendors develop, test and release their software

IT leaders should hold vendors deeply integrated within IT systems, such as CrowdStrike , to a “very high standard” of development, release quality and assurance, said Neil MacDonald , a Gartner vice president.

“Any security vendor has a responsibility to do extensive regression testing on all versions of Windows before an update is rolled out,” he said.

That involves asking existing vendors to explain how they write software, what testing they do and whether customers may choose how quickly to roll out an update.

“Incidents like this remind all of us in the CIO community of the importance of ensuring availability, reliability and security by prioritizing guardrails such as deployment and testing procedures and practices,” said Amy Farrow, chief information officer of IT automation and security company Infoblox.

Re-evaluate how your firm accepts software updates from ‘trusted’ vendors

While automatically accepting software updates has become the norm—and a recommended security practice—the CrowdStrike outage is a reminder to take a pause, some CIOs said.

“We still should be doing the full testing of packages and upgrades and new features,” said Paul Davis, a field chief information security officer at software development platform maker JFrog . undefined undefined Though it’s not feasible to test every update, especially for as many as hundreds of software vendors, Davis said he makes it a priority to test software patches according to their potential severity and size.

Automation, and maybe even artificial intelligence-based IT tools, can help.

“Humans are not very good at catching errors in thousands of lines of code,” said Jack Hidary, chief executive of AI and quantum company SandboxAQ. “We need AI trained to look for the interdependence of new software updates with the existing stack of software.”

Develop a disaster recovery plan

An incident rendering Windows computers unusable is similar to a natural disaster with systems knocked offline, said Gartner’s MacDonald. That’s why businesses should consider natural disaster recovery plans for maintaining the resiliency of their operations.

One way to do that is to set up a “clean room,” or an environment isolated from other systems, to use to bring critical systems back online, according to Chirag Mehta, a cybersecurity analyst at Constellation Research.

Businesses should also hold tabletop exercises to simulate risk scenarios, including IT outages and potential cyber threats, Mehta said.

Companies that back up data regularly were likely less impacted by the CrowdStrike outage, according to Victor Zyamzin, chief business officer of security company Qrator Labs. “Another suggestion for companies, and we’ve been saying that again and again for decades, is that you should have some backup procedure applied, running and regularly tested,” he said.

Review vendor and insurance contracts

For any vendor with a significant impact on company operations , MacDonald said companies can review their contracts and look for clauses indicating the vendors must provide reliable and stable software.

“That’s where you may have an advantage to say, if an update causes an outage, is there a clause in the contract that would cover that?” he said.

If it doesn’t, tech leaders can aim to negotiate a discount serving as a form of compensation at renewal time, MacDonald added.

The outage also highlights the importance of insurance in providing companies with bottom-line protection against cyber risks, said Peter Halprin, a partner with law firm Haynes Boone focused on cyber insurance.

This coverage can include protection against business income losses, such as those associated with an outage, whether caused by the insured company or a service provider, Halprin said.

Weigh the advantages and disadvantages of the various platforms

The CrowdStrike update affected only devices running Microsoft Windows-based systems , prompting fresh questions over whether enterprises should rely on Windows computers.

CrowdStrike runs on Windows devices through access to the kernel, the part of an operating system containing a computer’s core functions. That’s not the same for Apple ’s Mac operating system and Linux, which don’t allow the same level of access, said Mehta.

Some businesses have converted to Chromebooks , simple laptops developed by Alphabet -owned Google that run on the Chrome operating system . “Not all of them require deeper access to things,” Mehta said. “What are you doing on your laptop that actually requires Windows?”

Berkshire began liquidating its big stake in the banking company in mid-July—and has already unloaded about 15% of its interest. The selling has been fairly aggressive and has totaled about $6 billion. (Berkshire still holds 883 million shares, an 11.3% interest worth $35 billion based on its most recent filing on Aug. 30.)

The selling has prompted speculation about when CEO Warren Buffett, who oversees Berkshire’s $300 billion equity portfolio, will stop. The sales have depressed Bank of America stock, which has underperformed peers since Berkshire began its sell program. The stock closed down 0.9% Thursday at $40.14.

It’s possible that Berkshire will stop selling when the stake drops to 700 million shares. Taxes and history would be the reasons why.

Berkshire accumulated its Bank of America stake in two stages—and at vastly different prices. Berkshire’s initial stake came in 2017 , when it swapped $5 billion of Bank of America preferred stock for 700 million shares of common stock via warrants it received as part of the original preferred investment in 2011.

Berkshire got a sweet deal in that 2011 transaction. At the time, Bank of America was looking for a Buffett imprimatur—and the bank’s stock price was weak and under $10 a share.

Berkshire paid about $7 a share for that initial stake of 700 million common shares. The rest of the Berkshire stake, more than 300 million shares, was mostly purchased in 2018 at around $30 a share.

With Bank of America stock currently trading around $40, Berkshire faces a high tax burden from selling shares from the original stake of 700 million shares, given the low cost basis, and a much lighter tax hit from unloading the rest. Berkshire is subject to corporate taxes—an estimated 25% including local taxes—on gains on any sales of stock. The tax bite is stark.

Berkshire might own $2 to $3 a share in taxes on sales of high-cost stock and $8 a share on low-cost stock purchased for $7 a share.

New York tax expert Robert Willens says corporations, like individuals, can specify the particular lots when they sell stock with multiple cost levels.

“If stock is held in the custody of a broker, an adequate identification is made if the taxpayer specifies to the broker having custody of the stock the particular stock to be sold and, within a reasonable time thereafter, confirmation of such specification is set forth in a written document from the broker,” Willens told Barron’s in an email.

He assumes that Berkshire will identify the high-cost Bank of America stock for the recent sales to minimize its tax liability.

If sellers don’t specify, they generally are subject to “first in, first out,” or FIFO, accounting, meaning that the stock bought first would be subject to any tax on gains.

Buffett tends to be tax-averse—and that may prompt him to keep the original stake of 700 million shares. He could also mull any loyalty he may feel toward Bank of America CEO Brian Moynihan , whom Buffett has praised in the past.

Another reason for Berkshire to hold Bank of America is that it’s the company’s only big equity holding among traditional banks after selling shares of U.S. Bancorp , Bank of New York Mellon , JPMorgan Chase , and Wells Fargo in recent years.

Buffett, however, often eliminates stock holdings after he begins selling them down, as he did with the other bank stocks. Berkshire does retain a smaller stake of about $3 billion in Citigroup.

There could be a new filing on sales of Bank of America stock by Berkshire on Thursday evening. It has been three business days since the last one.

Berkshire must file within two business days of any sales of Bank of America stock since it owns more than 10%. The conglomerate will need to get its stake under about 777 million shares, about 100 million below the current level, before it can avoid the two-day filing rule.

It should be said that taxes haven’t deterred Buffett from selling over half of Berkshire’s stake in Apple this year—an estimated $85 billion or more of stock. Barron’s has estimated that Berkshire may owe $15 billion on the bulk of the sales that occurred in the second quarter.

Berkshire now holds 400 million shares of Apple and Barron’s has argued that Buffett may be finished reducing the Apple stake at that round number, which is the same number of shares that Berkshire has held in Coca-Cola for more than two decades.

Buffett may like round numbers—and 700 million could be just the right figure for Bank of America.