Comic URL: http://www.lefthandedtoons.com/1835/

Remember Heartbleed? If you believe the hype today, Shellshock is in that league and with an equally awesome name albeit bereft of a cool logo (someone in the marketing department of these vulns needs to get on that). But in all seriousness, it does have the potential to be a biggie and as I did with Heartbleed, I wanted to put together something definitive both for me to get to grips with the situation and for others to dissect the hype from the true underlying risk.

To set the scene, let me share some content from Robert Graham’s blog post who has been doing some excellent analysis on this. Imagine an HTTP request like this:

target = 0.0.0.0/0
port = 80
banners = true
http-user-agent = shellshock-scan (http://blog.erratasec.com/2014/09/bash-shellshock-scan-of-internet.html)
http-header = Cookie:() { :; }; ping -c 3 209.126.230.74
http-header = Host:() { :; }; ping -c 3 209.126.230.74
http-header = Referer:() { :; }; ping -c 3 209.126.230.74

Which, when issued against a range of vulnerable IP addresses, results in this:

Put succinctly, Robert has just orchestrated a bunch of external machines to ping him simply by issuing a carefully crafted request over the web. What’s really worrying is that he has effectively caused these machines to issue an arbitrary command (albeit a rather benign ping) and that opens up a whole world of very serious possibilities. Let me explain.

What is Bash and why do we need it?

Skip this if it’s old news, but context is important for those unfamiliar with Bash so let’s establish a baseline understanding. Bash is a *nix shell or in other words, an interpreter that allows you to orchestrate commands on Unix and Linux systems, typically by connecting over SSH or Telnet. It can also operate as a parser for CGI scripts on a web server such as we’d typically see running on Apache. It’s been around since the late 80s where it evolved from earlier shell implementations (the name is derived from the Bourne shell) and is enormously popular. There are other shells out there for Unix variants, the thing about Bash though is that it’s the default shell for Linux and Mac OS X which are obviously extremely prevalent operating systems. That’s a major factor in why this risk is so significant – the ubiquity of Bash – and it’s being described as “one of the most installed utilities on any Linux system”.

You can get a sense of the Bash footprint when you look at the latest Netcraft web server stats:

When half the net is running Apache (which is typically found on Linux), that’s a significant size of a very, very large pie. That same Netcraft article is reporting that we’ve just passed the one billion websites mark too and whilst a heap of those are sharing the same hosts, that’s still a whole lot of so think along the lines of 500M+ Bash installations. Oh – that’s just web servers too, don’t forget there are a heap of other servers running Linux and we’ll come back to other devices with Bash a bit later too.

Bash can be used for a whole range of typical administrative functions, everything from configuring websites through to controlling embedded software on a device like a webcam. Naturally this is not functionality that’s intended to be open to the world and in theory, we’re talking about authenticated users executing commands they’ve been authorised to run. In theory.

What’s the bug?

Let me start with the CVE from NIST vulnerability database because it gives a good sense of the severity (highlight mine):

GNU Bash through 4.3 processes trailing strings after function definitions in the values of environment variables, which allows remote attackers to execute arbitrary code via a crafted environment, as demonstrated by vectors involving the ForceCommand feature in OpenSSH sshd, the mod_cgi and mod_cgid modules in the Apache HTTP Server, scripts executed by unspecified DHCP clients, and other situations in which setting the environment occurs across a privilege boundary from Bash execution.

They go on to rate it a “10 out of 10” for severity or in other words, as bad as it gets. This is compounded by the fact that it’s easy to execute the attack (access complexity is low) and perhaps most significantly, there is no authentication required when exploiting Bash via CGI scripts. The summary above is a little convoluted though so let’s boil it down to the mechanics of the bug.

The risk centres around the ability to arbitrarily define environment variables within a Bash shell which specify a function definition. The trouble begins when Bash continues to process shell commands after the function definition resulting in what we’d classify as a “code injection attack”. Let’s look at Robert’s example again and we’ll just take this line:

http-header = Cookie:() { :; }; ping -c 3 209.126.230.74

The function definition is () { :; }; and the shell command is the ping statement and subsequent parameters. When this is processed within the context of a Bash shell, the arbitrary command is executed. In a web context, this would mean via a mechanism such as a CGI script and not necessarily as a request header either. It’s worth having a read through the seclists.org advisory where they go into more detail, including stating that the path and query string could be potential vectors for the attack.

Of course one means of mitigating this particular attack vector is simply to disable any CGI functionality that makes calls to a shell and indeed some are recommending this. In many cases though, that’s going to be a seriously breaking change and at the very least, one that going to require some extensive testing to ensure it doesn’t cause immediate problems in the website which in many cases, it will.

The HTTP proof above is a simple but effective one, albeit just one implementation over a common protocol. Once you start throwing in Telnet and SSH and apparently even DHCP, the scope increases dramatically so by no means are we just talking about exploiting web app servers here. (Apparently the risk is only present in SSH post-auth, but at such an early stage of the public disclosure we’ll inevitably see other attack vectors emerge yet.)

What you also need to remember is that the scope of potential damage stretches well beyond pinging an arbitrary address as in Robert’s example, that’s simply a neat little proof that he could orchestrate a machine to issue a shell command. The question becomes this: What damage could an attacker do when they can execute a shell command of their choosing on any vulnerable machine?

What are the potential ramifications?

The potential is enormous – “getting shell” on a box has always been a major win for an attacker because of the control it offers them over the target environment. Access to internal data, reconfiguration of environments, publication of their own malicious code etc. It’s almost limitless and it’s also readily automatable. There are many, many examples of exploits out there already that could easily be fired off against a large volume of machines.

Unfortunately when it comes to arbitrary code execution in a shell on up to half the websites on the internet, the potential is pretty broad. One of the obvious (and particularly nasty) ones is dumping internal files for public retrieval. Password files and configuration files with credentials are the obvious ones, but could conceivably extend to any other files on the system.

Likewise, the same approach could be applied to write files to the system. This is potentially the easiest website defacement vector we’ve ever seen, not to mention a very easy way of distributing malware

Or how about this: one word I keep seeing a lot is “worm”:

When we talk about worm in a malicious computing context, we’re talking about a self-replicating attack where a malicious actor creates code that is able to propagate across targets. For example, we saw a very effective implementation of this with Samy’s MySpace XSS Worm where some carefully crafted JavaScript managed to “infect” a million victims’ pages in less than a day.

The worry with Shellshock is that an attack of this nature could replicate at an alarming rate, particularly early on while the majority of machines remain at risk. In theory, this could take the form of an infected machine scanning for other targets and propagating the attack to them. This would be by no means limited to public facing machines either; get this behind the corporate firewall and the sky’s the limit.

People are working on exploiting this right now. This is what makes these early days so interesting as the arms race between those scrambling to patch and those scrambling to attack heats up.

Which versions of Bash are affected?

The headlines state everything through 4.3 or in other words, about 25 years’ worth of Bash versions. Given everyone keeps comparing this to Heartbleed, consider that the impacted versions of OpenSSL spanned a mere two years which is a drop in the ocean compared to Shellshock. Yes people upgrade their versions, but no they don’t do it consistently and whichever way you cut it, the breadth of at-risk machines is going to be significantly higher with Shellshock than what it was with Heartbleed.

But the risk may well extend beyond 4.3 as well. Already we’re seeing reports of patches not being entirely effective and given the speed with which they’re being rolled out, that’s not all that surprising. This is the sort of thing those impacted by it want to keep a very close eye on, not just “patch and forget”.

When did we first learn of it and how long have we been at risk?

The first mention I’ve found on the public airwaves was this very brief summary on seclists.org which works out at about 18:00 GMT on Wednesday (about 2am today for those of us on the eastern end of Australia). The detail came in the advisory I mentioned earlier an hour later so getting towards late afternoon Wednesday in Europe or the middle of the day in the US. It’s still very fresh news with all the usual press speculation and Chicken Little predications; it’s too early to observe any widespread exploitation in the wild, but that could also come very soon if the risk lives up to its potential.

Scroll back beyond just what has been disclosed publicly and the bug was apparently discovered last week by Stéphane Chazelas, a “Unix/Linux, network and telecom specialist” bloke in the UK. Having said that, in Akamai’s post on the bug, they talk about it having been present for “an extended period of time” and of course vulnerable versions of Bash go back two and a half decades now. The question, as with Heartbleed, will be whether or not malicious actors were aware of this before now and indeed whether they were actively exploiting it.

Are our “things” affected?

This is where it gets interesting – we have a lot of “things” potentially running Bash. Of course when I use this term I’m referring to the “Internet of Things” (IoT) which is the increasing prevalence of whacking an IP address and a wireless adaptor into everything from our cutlery to our door locks to our light globes.

Many IoT devices run embedded Linux distributions with Bash. These very same devices have already been shown to demonstrate serious security vulnerabilities in other areas, for example LIFX light globes just a couple of months ago were found to be leaking wifi credentials. Whilst not a Bash vulnerability like Shellshock, it shows us that by connecting our things we’re entering a whole new world of vulnerabilities in places that were never at risk before.

This brings with it many new challenges; for example, who is actively thinking they should regularly patch their light bulbs? Also consider the longevity of the devices this software is appearing in and whether they’re actually actively maintained. In a case like the vulnerable Trendnet cameras from a couple of years ago, there are undoubtedly a huge number of them still sitting on the web because in terms of patching, they’re pretty much a “set and forget” proposition. In fact in that case there’s an entire Twitter account dedicated to broadcasting the images it has captured of unsuspecting owners of vulnerable versions. It’s a big problem with no easy fixes and its going to stick with us for a very long time.

But Bash shells are also present in many more common devices, for example our home routers which are generally internet-facing. Remember when you last patched the firmware on your router? Ok, if you’re reading this then maybe you’re the type of technical person who actually does patch their router, but put yourself in the shoes of Average Joe Consumer and ask yourself that again. Exactly.

All our things are on the Microsoft stack, are we at risk?

Short answer “no”, long answer “yes”. I’ll tackle the easy one first – Bash is not found natively on Windows and whilst there are Bash implementations for Windows, it’s certainly not common and it’s not going to be found on consumer PCs. It’s also not clear if products like win-bash are actually vulnerable to Shellshock in the first place.

The longer answer is that just because you operate in a predominantly Microsoft-centric environment doesn’t mean that you don’t have Bash running on machines servicing other discrete purposes within that environment. When I wrote about Heartbleed, I referenced Nick Craver’s post on moving Stack Overflow towards SSL and referred to this diagram of their infrastructure:

There are non-Microsoft components sitting in front of their Microsoft application stack, components that the traffic needs to pass through before it hits the web servers. These are also components that may have elevated privileges behind the firewall – what’s the impact if Shellshock is exploited on those? It could be significant and that’s the point I’m making here; Shellshock has the potential to impact assets beyond just at-risk Bash implementations when it exists in a broader ecosystem of other machines.

I’m a system admin – what can I do?

Firstly, discovering if you’re at risk is trivial as it’s such an easily reproducible risk. There’s a very simple test The Register suggests which is just running this command within your shell:

env X="() { :;} ; echo busted" /bin/sh -c "echo stuff"

You get “busted” echo’d back out and you’ve successfully exploited the bug.

Of course the priority here is going to be patching at risk systems and the patch essentially boils down to ensuring no code can be executed after the end of a Bash function. Linux distros such as Red Hat are releasing guidance on patching the risk so jump on that as a matter of priority.

We’ll inevitably also see definitions for intrusion detection systems too and certainly there will be common patterns to look for here. That may well prove a good immediate term implementation for many organisations, particularly where there may be onerous testing requirements before rolling out patches to at-risk systems. Qualys’ are aiming to have a definition to detect the attack pretty quickly and inevitably other IDS providers are working on this around the clock as well.

Other more drastic options include replacing Bash with an alternate shell implementation or cordoning off at-risk systems, both of which could have far-reaching ramifications and are unlikely to be decisions taken lightly. But that’s probably going to be the nature of this bug for many people – hard decisions that could have tangible business impact in order to avoid potentially much more significant ramifications.

The other issue which will now start to come up a lot is the question of whether Shellshock has already been exploited in an environment. This can be hard to determine if there’s no logging of the attack vectors (there often won’t be if it’s passed by HTTP request header or POST body), but it’s more likely to be caught than with Heartbleed when short of full on pcaps, the heartbeat payloads would not normally have been logged anywhere. But still, the most common response to “were we attacked via Shellshock” is going to be this:

unfortunately, this isn't "No, we have evidence that there were no compromises;" rather, "we don't have evidence that spans the lifetime of this vulnerability."  We doubt many people do - and this leaves system owners in the uncomfortable position of not knowing what, if any, compromises might have happened.

Let the speculation about whether the NSA was in on this begin…

I’m a consumer – what can I do?

It depends. Shellshock affects Macs so if you’re running OS X, at this stage that appears to be at risk which on the one hand is bad due to the prevalence of OS X but on the other hand will be easily (and hopefully quickly) remediated due to a pretty well-proven update mechanism (i.e. Apple can remotely push updates to the machine).

If you’re on a Mac, the risk is easily tested for as described in this Stack Exchange answer:

It’s an easy test, although I doubt the average Mac user is going to feel comfortable stepping through the suggested fix which involves recompiling Bash.

The bigger worry is the devices with no easy patching path, for example your router. Short of checking in with the manufacturer’s website for updated firmware, this is going to be a really hard nut to crack. Often routers provided by ISPs are locked down so that consumers aren’t randomly changing either config or firmware and there’s not always a remote upgrade path they can trigger either. Combine that with the massive array of devices and ages that are out there and this could be particularly tricky. Of course it’s also not the sort of thing your average consumer is going to be comfortable doing themselves either.

In short, the advice to consumers is this: watch for security updates, particularly on OS X. Also keep an eye on any advice you may get from your ISP or other providers of devices you have that run embedded software. Do be cautious of emails requesting information or instructing you to run software – events like this are often followed by phishing attacks that capitalise on consumers’ fears. Hoaxes presently have people putting their iPhones in the microwave so don’t for a moment think that they won’t run a random piece of software sent to them via email as a “fix” for Shellshock!

Summary

In all likelihood, we haven’t even begun the fathom the breadth of this vulnerability. Of course there are a lot of comparisons being made to Heartbleed and there are a number of things we learned from that exercise. One is that it took a bit of time to sink in as we realised the extent to which we were dependent on OpenSSL. The other is that it had a very long tail – months after it hit there were still hundreds of thousands of known hosts left vulnerable.

But in one way, the Heartbleed comparison isn’t fair – this is potentially far worse. Heartbleed allowed remote access to small amount of data in the memory of affected machines. Shellshock is enabling remote code injection of arbitrary commands pre-auth which is potentially far more dire. In that regard, I have to agree with Robert:

It’s very, very early days yet – only half a day since it first hit the airwaves at the time of writing – and I suspect that so far we’re only scratching the surface of what is yet to come.