Part of the delay was not merely that DNS needs time to propagate, but that McDonald’s would have needed to send the change via different DNS resolvers. “This was likely a DNSSEC (Domain Name System Security Extensions) change intended to improve their security.”
Wilkes also suspected that a TTL (time to live) setting played a role. “No one likely had time to lower the TTL to have a recovery time of five minutes,” he said, which would further explain the lengthy delays.
Terry Dunlap, co-founder and managing partner of Gray Hat Academy, also believed the McDonald’s outage appeared to be an attempt to quickly block a potentially imminent attack. “They were saying ‘Give me a life vest. I don’t want to be drowned by the wave that is coming.’”
More strategically, Dunlap was not a fan of the statements McDonald’s issued.
“It’s much better to be proactive and as detailed as possible upfront,” he said. “I don’t think that the statements conveyed the level of warm and fuzzies needed. I would recommend going into more details. How did you respond to it? Why did it happen? What impacts have occurred that you are not telling me? (The McDonald’s statements) create more questions than answers.”
This appropriately raises yet again the enterprise risk coming from third-parties — especially those who, as might be the case with McDonald’s, act on their own and cause problems for the enterprise IT team.
“Every company is being flyspecked for their third-party risk management right now,” said Brian Levine, a managing director with Ernst & Young (EY). “Third-party risk management is increasingly being put under the microscope today by courts, regulators and companies.”
McDonald’s did not initially file an SEC report on the incident. Given that Wall Street did not react in any serious way to the McDonald’s outage, it’s unlikely McDonald’s would consider the outage material. As for the third-party POS provider, it’s unclear whether it filed a report as its identity has yet to be confirmed.
Among the important lessons here for all enterprise IT, is to give careful thought to outage statements. Anything beyond, “Something happened. We are investigating and will report more once facts are known and verified” is going to leave clues.
Vague implications are not your friend. If you are ready to say something, say it. If you are not, say nothing. Splitting the middle as McDonald’s did won’t likely serve your long-term interests (not unlike eating McDonald’s food). But at least a quarter-pounder tastes good and is filling.
The McDonald’s outage statement was neither.
