Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Perimeter

9/26/2014
10:20 AM
Paul Vixie
Paul Vixie
Commentary
Connect Directly
Facebook
Twitter
LinkedIn
Google+
RSS
E-Mail vvv
50%
50%

Shellshocked: A Future Of ‘Hair On Fire’ Bugs

Most computers affected by Bash will be updated within 10 years. The rest will be vulnerable for the lifespans of all humans now living. This should concern us. But then, global warming should also concern us.

It seems that the more we integrate our lives and our national economies into the Internet, the less safe we, and our privacy, and our money, become. The human population will eventually grow weary of the endless parade of worldwide hair-on-fire technology problems like Y2K, Conficker, Heartbleed, and, as of this week, Shellshock.

By “weary” I mean thoughts like “Another day, another bug, I have no time for this, I’ll upgrade everything when I get the time.” We may already be there, given that six full years after the Conficker worm was announced, we still see about a million unique IP addresses per day in the Conficker sinkhole. Those can’t all be student researchers checking to see if the botnet still exists. Sadly, every hour of delay dramatically increases the likelihood that the device will become more deeply infected before it is patched -- after which time, patches won’t actually make anything better.

Aggregate attack surface is a progressive concept. When the total number of infected or vulnerable computers, and the total number of infections and vulnerabilities themselves, were small, the criminal miscreants of the world did not have as much free infrastructure to draw from when attacking new targets. Make no mistake, the bad guys don’t pay for hardware or connectivity -- they’re using ours instead, since ours is free, and this kind of indirection helps them hide their tracks and misdirect our investigations. Today the computational resources available to the bad guys have grown to the point where no victim is out of reach, and no attacker has to be all that frugal or even quiet about its attacks.

Shellshock is the name of a bug in “Bash,” an acronym for Bourne Again Shell, a command line interpreter present on most computers in the world except for Windows. Bash is in Linux, and Linux is in just about every embedded computer including smart TVs, smartphones and tablets, home gateways, wireless access points, Internet servers, and many industrial control systems.

There is not a quality problem with Linux per se. In fact, Linux may be the most reviewed, most tested, highest quality (per line of code) system in history. However, there’s an awful lot of code and an awful lot of interfaces -- connections between different parts of code -- and with complexity comes error just as surely as night follows day. More importantly, many devices containing Bash are not field upgradeable, either for cost reasons or because their makers died out. Even among devices that are still upgradeable, most are silent, unknown trolls in dark closets with no monitoring or auditing or management at all.

Fix they did, but…
Within a few hours of the announcement of Shellshock and the first software update to fix the bug (which fix did, by the way, turn out to be incomplete), those of us who watch the Internet for malicious activity saw hundreds of researchers scanning billions of Internet addresses, all trying to measure and catalogue attack surface.

It’s safe to assume that some of those researchers have evil intent, such as adding the vulnerable computers to a botnet, to be used later for launching DDoS attacks or similar malfeasance. And as always during times of emergency, misinformation and misunderstanding was everywhere, even in some official announcements from people and companies and agencies whose purpose in life is to keep us informed and safe against online threats.

It’s conservative to estimate that for every modern computing device we actually know is a computing device, and whose manufacturer still has the ability to update its software, there are at least 10 other computing devices that we don’t know much about, which will not have their software updated ever, and which have years or even decades of life left.

Experience tells us to expect that the computers that will be updated will be mostly updated within about 10 years. Most of the rest will be vulnerable until they die, which will take some decades, and a few million of them will still be running and still be vulnerable to Shellshock for the lifespans of all humans now living. This should concern us, but then, global warming should also concern us.

As the saying goes, “When you’re stuck in a hole, the first thing to do is stop digging.” Given the inevitability of software bugs and the growing dependence on technology for banking, communications, infrastructure, agriculture, and food supply, can we afford to continue driving innovation guided only by near-term profit, where technology’s winners are always those who pursue time-to-market over quality?

By all means, let us patch Bash wherever we can find it. But that’s busy work. The vulnerable Bash instances that we won’t find vastly outnumber those we will, and our future is going to be dominated by leftovers from an endless parade of hair-on-fire bugs that we eventually learn to live with when the next one comes along and steals our attention.

Dr. Paul Vixie is an Internet pioneer and thought leader who designed, implemented, and deployed several Domain Name System (DNS) protocol extensions and applications that are used throughout the Internet today. He is CEO of Farsight Security Inc. Previously, he served as ... View Full Bio
Comment  | 
Print  | 
More Insights
Comments
Oldest First  |  Newest First  |  Threaded View
Page 1 / 3   >   >>
Kelly Jackson Higgins
50%
50%
Kelly Jackson Higgins,
User Rank: Strategist
9/26/2014 | 4:09:35 PM
hair-on-fire bug fatigue
Your point about this being another in a series of major flaws we'll face is so true, @Paul. What is worrisome is how this could spawn bug fatigue, almost like we're seeing in the wave of retail data breaches. Another day, another major Internet flaw is exposed. How does the industry avoid that?
Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
9/26/2014 | 4:14:13 PM
Great analysis but is it really so hopeless?
It patching is simply busy work, what real work needs to be done? I for one don't look forward to a future with my hair on fire watching the bug parade go on and on.... 
paulvixie
50%
50%
paulvixie,
User Rank: Author
9/26/2014 | 4:57:21 PM
Re: Great analysis but is it really so hopeless?
<< It patching is simply busy work, what real work needs to be done? I for one don't look forward to a future with my hair on fire watching the bug parade go on and on....  >>

the situation at hand is the result of mass numbers of people acting in their short-term self interest, to the detriment of the whole, sort of in a "tragedy of the commons" remake. my comparison of this to global warming is not an accident. historically, the way we incentivize mass numbers of people to act in their longer-term self interest as opposed to their short-term self interest, has involved some combination of three things:

1. give them better choices (so, innovation)

2. increase the cost of some of their current choices (so, regulation)

3. make them better aware of their long-term self interest, and their choices (so, education)

one innovation i've been pondering for the last few years in the face of a similarly difficult problem which is the wide spread lack of source address validation at the edge of the internet, is a certification programme for devices which have been well tested under difficult conditions, along the lines of the "underwriters laboratory" sticker i looked for before buying a toaster oven for my kitchen.

but to your question ("what real work needs to be done?"), i think the first thing we've got to do is stop panicking every time one of these bugs comes along. we still have millions of devices vulnerable to attacks that had our hair on fire one, two, three, four, and five years ago -- clearly setting our hair on fire didn't have a big impact. so, do patch, but, also consider what i said on a private security forum yesterday:

correct reaction would be strategic: get an inventory of the contents of every smart device your agency or your company owns or operates or depends upon, and enact a phase-out plan that replaces every non-upgradeable or un-auditable device with something you can actually control. let normal apple/redhat/$vendor upgrade/patch take care of their products on your network in due course. note for example that the first patch released by redhat yesterday did not go far enough to fix this bug.

thank you for your question, it's an important one. --vix

kstaron
50%
50%
kstaron,
User Rank: Apprentice
9/26/2014 | 6:36:25 PM
Ok, so now what?
Okay ,so it's out there, other than patching what can we do about it? It's not feasible to throw all the devices out and start fresh, so where does that leave us? As consumers what can we do to protect ourselves?
paulvixie
50%
50%
paulvixie,
User Rank: Author
9/26/2014 | 6:44:22 PM
Re: Ok, so now what?
<< Okay ,so it's out there, other than patching what can we do about it? It's not feasible to throw all the devices out and start fresh, so where does that leave us? As consumers what can we do to protect ourselves? >>

my recommendation is as follows:

correct reaction would be strategic: get an inventory of the contents of every smart device your agency or your company owns or operates or depends upon, and enact a phase-out plan that replaces every non-upgradeable or un-auditable device with something you can actually control. let normal apple/redhat/$vendor upgrade/patch take care of their products on your network in due course. note for example that the first patch released by redhat yesterday did not go far enough to fix this bug.

i hope this helps. --vix

Marilyn Cohodas
50%
50%
Marilyn Cohodas,
User Rank: Strategist
9/29/2014 | 8:08:13 AM
Re: Ok, so now what?
Thanks for these links, Paul, not to mention your spot-on advice for dealing with Shellshock, both long- and short-term. Hope you'll keep us apprised of new insights/developoments as this story continues to unfold. 
aws0513
50%
50%
aws0513,
User Rank: Ninja
9/29/2014 | 8:34:19 AM
Re: Great analysis but is it really so hopeless?
Great article Paul.

Since the announcement of the bug finding, I have not considered this a full on "sky is falling" situation.  My reaction was more in line with my military background. It wasn't that we were not aware of the gravity of the situation, we just reacted as we always react to high risk vulnerability announcements.  Business as usual for this IT security program.

First we went into full assessment mode to identify as many systems that could be affected as possible.  This meant taking our inventory and removing the obvious non-players, actioning the known affected systems, and validation of the unknowns.  We also ramped up monitoring practices for the time being until we feel most of the immediate risk has been mitigated.
We were fortunate in that we just finished implementing an inventory management practice that gives us a very comprehensive data set to start with.  BTW...  Inventory data on enterprise systems is probably the most powerful tool an IT security team can have.  The classic "know thyself" factor.  Having such a data set allows for efficient triage and planning.  It is some overhead to keep an inventory current, but worth every penny when the sky does seem to be falling.

As it stands today, we are looking good on the patching front thanks to some dedicated work over the weekend.  Now we are still sorting through and validating each unknown system/device to determine impact and remediation options.  The fire fight is ongoing, but we have reinforced the weak points and are assessing the rest of the infrastructure to see if there are more.
paulvixie
50%
50%
paulvixie,
User Rank: Author
9/29/2014 | 9:06:35 AM
if you patched over the weekend, you're out of date, and vulnerable, again.
<< As it stands today, we are looking good on the patching front thanks to some dedicated work over the weekend. >>

if you patched over the weekend, you're out of date, and vulnerable, again. here's the latest as of this moment:

Shellshock (CVE-2014-6271CVE-2014-7169CVE-2014-7186CVE-2014-7187CVE-2014-6277) is a vulnerability in GNU's bash shell that gives attackers access to run remote commands on a vulnerable system. If your system has not updated bash in since Sun Sep 28 2014: 1:11AM EST (See patch history), you're most definitely vulnerable and have been since first boot. This security vulnerability affects versions 1.14 (released in 1994) to the most recent version 4.3 according to NVD.

(that text is from https://shellshocker.net/)

Kelly Jackson Higgins
50%
50%
Kelly Jackson Higgins,
User Rank: Strategist
9/29/2014 | 9:21:44 AM
Re: if you patched over the weekend, you're out of date, and vulnerable, again.
So @Paul, how would orgs who patched over the weekend actually know that their patch has to be redone? Is US-CERT or ICS-CERT planning to issue an alert to let them know? There's so much information flying around, it must be a nightmare to stay on top of this. 
Page 1 / 3   >   >>
5 Ways to Up Your Threat Management Game
Wayne Reynolds, Advisory CISO, Kudelski Security,  2/26/2020
Google Adds More Security Features Via Chronicle Division
Robert Lemos, Contributing Writer,  2/25/2020
Cybersecurity Industry: It's Time to Stop the Victim Blame Game
Jessica Smith, Senior Vice President, The Crypsis Group,  2/25/2020
Register for Dark Reading Newsletters
White Papers
Video
Cartoon
Current Issue
6 Emerging Cyber Threats That Enterprises Face in 2020
This Tech Digest gives an in-depth look at six emerging cyber threats that enterprises could face in 2020. Download your copy today!
Flash Poll
State of Cybersecurity Incident Response
State of Cybersecurity Incident Response
Data breaches and regulations have forced organizations to pay closer attention to the security incident response function. However, security leaders may be overestimating their ability to detect and respond to security incidents. Read this report to find out more.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-9463
PUBLISHED: 2020-02-28
Centreon 19.10 allows remote authenticated users to execute arbitrary OS commands via shell metacharacters in the server_ip field in JSON data in an api/internal.php?object=centreon_configuration_remote request.
CVE-2020-5247
PUBLISHED: 2020-02-28
In Puma (RubyGem) before 4.3.2 and 3.12.2, if an application using Puma allows untrusted input in a response header, an attacker can use newline characters (i.e. `CR`, `LF` or`/r`, `/n`) to end the header and inject malicious content, such as additional headers or an entirely new response body. This...
CVE-2020-9447
PUBLISHED: 2020-02-28
The file-upload feature in GwtUpload 1.0.3 allows XSS via a crafted filename.
CVE-2019-10064
PUBLISHED: 2020-02-28
hostapd before 2.6, in EAP mode, makes calls to the rand() and random() standard library functions without any preceding srand() or srandom() call, which results in inappropriate use of deterministic values. This was fixed in conjunction with CVE-2016-10743.
CVE-2019-8741
PUBLISHED: 2020-02-28
A denial of service issue was addressed with improved input validation.