Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Cloud

7/14/2015
10:30 AM
Anthony Lye
Anthony Lye
Commentary
Connect Directly
Twitter
LinkedIn
RSS
E-Mail vvv
50%
50%

Inside A Vicious DDoS Attack

What it's really like to fend off a relentless distributed denial-of-service attack.

On Sunday, May 17, cybercriminals launched a vicious distributed denial of service (DDoS) attack against my company, HotSchedules, a cloud-based service supporting more than 2 million workers in restaurants, hospitality, and retail industries.  For 45 hours, from Sunday night to Tuesday afternoon, the assailants prevented employees from checking and managing their schedules  -- and never stated a motive.

However, what began as every CEO’s nightmare ultimately became a source of pride and valuable lessons for me. HotSchedules had fended off many such attacks over the past 16 years without any interruption to service. This one was exceptionally ferocious, and I believe that other companies can benefit from what we learned.

Thanks to Hollywood, we tend to imagine cyberattacks as technical battles, fought in the bunkers of the Internet by good and bad hackers sitting in front of black screens with green code. In reality, cyber-battles are fought by everyone in an organization. If you run a company, an attack will test your culture, teamwork, and collective dedication to its mission. Either your team will find a way to fulfill its commitment to customers – or it will become paralyzed. There is a technical fix to every cyber challenge, but the human response determines whether you win or lose in the eyes of your customers.

The human element
Sunday evening, upon receiving news of the DDoS attack, the entire HotSchedules team rushed to our offices. No one was called or specifically asked to do anything. Throughout the conflict, people took initiative and never asked for permission to act.

The highest priority for any business is to continue serving customers. We still had access to our database, so our engineers pulled rosters and passed them to our customer service team, which had swelled with implementation team members, customer success managers, marketers, HR, and almost everyone in non-technical roles. They downloaded the schedules and emailed copies to each customer. For the next 48 hours, our team worked the phones, email, and social media with almost no sleep. When customers called in, we provided rosters, schedules, shift swaps, and other core services.

The second highest priority is transparency. Once we knew what had happened, we emailed customers immediately. Our policy was to “leave no conversation behind,” an idea we originally learned from BTC Revolutions, a digital and social marketing and strategy agency that trained our team. We posted announcements and updates on our website, Facebook, Twitter, and made direct calls to customers the next morning and answered every post and tweet we saw.

Our team worked without sleeping for two days yet kept up a level of intensity that was unbelievable. The data center floor became a makeshift sleeping area for engineers who refused to leave their post for more than a power nap. Companies either have or don’t have this human element. It comes down to who you hire, how you train people, and the sense of purpose employees find in their work. As CEO, I could not have felt more proud of our team.

The cyber battle
While all customer-facing team members maintained service and communicated with customers, our IT groups addressed the cyber threat. None of our data was compromised, but we were unprepared for such a vicious attack.

The assailants hit with 10- to 15 gigabytes per second (Gbps) of traffic – 250 times our standard rate. When I called my peers in banking and telecom for advice, they were stunned. This was more than enough traffic to bring down most commercial businesses. It required a whole team of assailants, careful orchestration, and an expensive investment in network resources.  

It’s difficult to know if and when the attack is over. Wherever we moved, the DDoS followed. At one point, we attempted to publish our service on another IP address, but the criminals took us down again. We made the difficult decision to publish “black holes” that discard incoming traffic. It’s a way of saying, “You got us – we’re not letting you do any further damage.”

Ultimately, our sleepless security engineers re-engineered the whole service on a subnet protected by Akamai’s cloud security solution, which can withstand over 321 Gbps of traffic. At 2:37 pm CT on Tuesday, May 19, we were up and running again.

The take away
The FBI has not been able to determine who committed this crime and why; there were no ransom demands, claims, or motives declared. Although thousands of servers in multiple countries hit us, we know that 60 percent of the attack originated overseas. I doubt we’ll ever identify the attackers.

Considering that 90 percent of companies suffer DDoS attacks, protection is worth it, no matter how much it may cost. Retreating from the cloud would be as absurd as downgrading from cars to horses in order to avoid crashes. Given the costs of on-premises technology, restaurant software is prohibitively expensive without cloud and mobile computing. We’re not about to abandon an entire industry to fear.  

This DDoS attack was the most stressful event in my business career, but it had a few silver linings. We never lost any data, and we maintained some level of service to our customers, who were very supportive. I learned that I have a team of co-workers who will go through 45 hours of hell to do what’s right for our customers. That is more valuable than any piece of technology. 

Anthony provides strategic direction and business guidance for the company, including the integration of multiple technology solutions into one integrated platform. Prior to his role as President and CEO, Anthony was Chief Product Officer for Red Book Connect. managing ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Mobile Banking Malware Up 50% in First Half of 2019
Kelly Sheridan, Staff Editor, Dark Reading,  1/17/2020
Exploits Released for As-Yet Unpatched Critical Citrix Flaw
Jai Vijayan, Contributing Writer,  1/13/2020
Microsoft to Officially End Support for Windows 7, Server 2008
Kelly Sheridan, Staff Editor, Dark Reading,  1/13/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Write a Caption, Win a Starbucks Card! Click Here
Latest Comment: This comment is waiting for review by our moderators.
Current Issue
The Year in Security: 2019
This Tech Digest provides a wrap up and overview of the year's top cybersecurity news stories. It was a year of new twists on old threats, with fears of another WannaCry-type worm and of a possible botnet army of Wi-Fi routers. But 2019 also underscored the risk of firmware and trusted security tools harboring dangerous holes that cybercriminals and nation-state hackers could readily abuse. Read more.
Flash Poll
[Just Released] How Enterprises are Attacking the Cybersecurity Problem
[Just Released] How Enterprises are Attacking the Cybersecurity Problem
Organizations have invested in a sweeping array of security technologies to address challenges associated with the growing number of cybersecurity attacks. However, the complexity involved in managing these technologies is emerging as a major problem. Read this report to find out what your peers biggest security challenges are and the technologies they are using to address them.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-7227
PUBLISHED: 2020-01-18
Westermo MRD-315 1.7.3 and 1.7.4 devices have an information disclosure vulnerability that allows an authenticated remote attacker to retrieve the source code of different functions of the web application via requests that lack certain mandatory parameters. This affects ifaces-diag.asp, system.asp, ...
CVE-2019-15625
PUBLISHED: 2020-01-18
A memory usage vulnerability exists in Trend Micro Password Manager 3.8 that could allow an attacker with access and permissions to the victim's memory processes to extract sensitive information.
CVE-2019-19696
PUBLISHED: 2020-01-18
A RootCA vulnerability found in Trend Micro Password Manager for Windows and macOS exists where the localhost.key of RootCA.crt might be improperly accessed by an unauthorized party and could be used to create malicious self-signed SSL certificates, allowing an attacker to misdirect a user to phishi...
CVE-2019-19697
PUBLISHED: 2020-01-18
An arbitrary code execution vulnerability exists in the Trend Micro Security 2019 (v15) consumer family of products which could allow an attacker to gain elevated privileges and tamper with protected services by disabling or otherwise preventing them to start. An attacker must already have administr...
CVE-2019-20357
PUBLISHED: 2020-01-18
A Persistent Arbitrary Code Execution vulnerability exists in the Trend Micro Security 2020 (v160 and 2019 (v15) consumer familiy of products which could potentially allow an attacker the ability to create a malicious program to escalate privileges and attain persistence on a vulnerable system.