It has been a while since I wrote a blog post. This past Tuesday, April 22nd, I celebrated my first year with Codenomicon. Yes, that same company that named the Heartbleed bug, created the Heartbleed.com website, and created the snazzy logo that we all recognize as the first logo ever created for a computer bug. It has certainly been an interesting year, to say the least.
I was brought into Codenomicon because of a few reasons, and carefully considered their offer before joining. The biggest draw for me was their new focus on health care and control systems testing. One thing I immediately took note of was the vast applicability of their testing tools in the emerging medical device security space. After spending some time getting to know some of the team (mostly headquartered in Finland, but with a very significant US presence) I quickly came to the conclusion that these were some very brilliant people doing some very interesting work. I had no idea to what extent they would continue to hold my interest at the time, but I have to say that one year later they still continue to amaze me.
Not long after I joined, the FDA decided to purchase our Defensics fuzz testing tools as the first tool in their planned cybersecurity testing lab. This was great news for us, as it immediately caught the attention of the medical device and hospital communities. At the time we were a relatively unknown niche player (and in fact, still are to most) that was suddenly on their radar screen. This created a lot of opportunity for us to show off what our testing tools can do, and got us into the doors of several major health care organizations, including both hospitals and device manufacturers. They all wanted to see what we could do, and we happily demonstrated how our tools could render their devices non-functional. Some like to say we broke the devices with our tools, but we really broke nothing. The devices (or more specifically the code running on the devices) was already broken. Our tools simply found where it was broken. We literally discovered zero-day vulnerabilties right in front of the prospective customer, and have to say it was often quite quick and easy to do. We saw quite a number of concerned looks.
It has been a busy year...as I said. First one medical device customer bought our tools, then another, then another, then another...and it kept going like that...and is still growing almost as fast as we can keep up with it. I spent a lot of time on the road speaking at medical device events on the topic of security, contributed to article, book chapters, and the list goes on. I did a lot of traveling in the last year, and I must say that despite being very interesting, it was quite exhausting.
This past March (and going into April) ended being a crazy month of travel for me. Between March 1st and April 9th I was on the road around 30 days. It was really exhausting most of the time, including several trans-contintal flights, and one trip to the Czech Republic (which, by the way, is really beautiful). All of it was business travel, except for one trip back home to Cleveland towards the end of March to deal with a family matter.
Ahhh yes, Cleveland Ohio. The semi-frozen tundra I call "home". I spent 21 years there growing up, then lived in Florida for 13 years, then went back to Cleveland for 1 winter, before deciding to move to Northern California. I felt I made a wise choice...at least the moving out of Cleveland part.
Don't get me wrong! I enjoyed my years in Cleveland, despite hating winter after about the age of 7. My family lives there, as well as my friends, and the wonderful Case Western Reserve University, where I attended college. Despite living in the hugely Asian populated San Francisco Bay Area my favorite Chinese restaurant is still in Cleveland. My all time favorite Kosher Half-Sour deli pickles can only be found at the famous Corky and Lenny's deli in Woodmere, and I make sure to stock up on the rare occasions I go home for a visit. I like to hang out at my favorite cigar store in Mayfield Ohio, where the 80+ year old owner (who I have known for over 25 years) still sits around with a bunch of old Italian-American curmudgeons, puffing on cigars and yelling at whatever talking head shows up on the widescreen TV, while he asks us if we would like an espresso to go with our favorite smoke. Heck! It IS home.
Yet after all that business travel, I found myself going back for personal reasons. Not fun personal reasons either. It was a family death, and I was tired, a bit stressed, and it happened to be snowing when I landed. Yes, it was snowing in Cleveland at the end of March. That happens a lot. In fact, I can remember many 80 degree days in early spring that were followed a few days later with blizzards. The last one I remember was in the year 2000, about 3 weeks before I decided to move to California. It was a sort of confirmation that I had made a wise decision...or so that is how I took it.
The snow was only a day long, and I was in Cleveland for only 5 days, but managed to catch a nasty chest cold while I was there. It was on the last day I was there, so I flew back home and spent the next several days downing some shots of Nyquil and sleeping. A few days later I felt much better, and was thankful because I had an upcoming trip to Boston (yet another trans-continental flight) and did not want to travel sick. All seemed well until the weekend came, and my nasty cough came back. It continued to build up over the next few days, and I found myself flying to Boston feeling very under the weather. It was while I was on the plane that I got the first message from our R&D team about Heartbleed. We were not calling it Heartbleed then, mind you. It was just an email saying our team had discovered this bug in OpenSSL while testing a new feature in our Defensics testing tools (the feature is called SafeGuard), and we were going to change all of our certificates, and we all would have to change our passwords, and to NOT change anything until they said to do so, because it would do no good until the bug was fixed. They also told us that the bug had been reported to the Finland's national CERT, which is not something I can recall having seen before in the year I had been there.
Now please understand I work for a security company that literally spends all of their time finding bugs (well, almost all of their time, they also like to sit around and sweat in saunas in Finland, as I understand). We find literally THOUSANDS of bugs every year as a matter of course. So when I get an email from our company telling us of a bug that affects us, it means something more than normal daily news from the trenches. I figured this must be serious, but had no idea how serious it really was.
A few hours later I got another message telling me of a website we had launched for the bug we had now dubbed Heartbleed, and was also shown the bleeding heart logo. The message announced that we were hosting this site to inform the public, since OpenSSL had gone public due to a report they had received from someone at Google. I was semi-lightheaded at the time, since my own personal bug was taking hold, so I was more than a bit confused by all of this. I dozed off with visions of bleeding hearts dancing in my head.
Once I landed in Boston I had lots to do in preparation for a big meeting the next morning. I got some dinner and went to bed, at this point feeling quite nasty from the cough. I got up the next morning and took some daytime cold medicine (Dayquil as I recall) and had some breakfast. I managed to get through a day of meetings without passing out, but by the time it was all over I was feeling some chills, and knew I had a fever. This was not good. Thankfully, one of my work colleagues had some Nyquil. I took some of it back to my hotel room and also got some other cold medicine. I had a nasty cough and it was not getting any better. I got a bowl of clam chowder for dinner (after all, I was in Boston), and then decided to do a little reading and go to bed.
Well, as I perused the news I started to notice all this talk of Heartbleed. It was right there front and center on every computer website, news site...it was everywhere. Again, I was partially delirious from the chest cold, but I started reading the news, our internal messages, and our newly minted website. I quickly realized that this was a lot more serious than I first imagined. Moreover, I noticed our company name showing up everywhere. Things were definitely NOT like they had been before. As I continued to read I began realizing that Heartbleed affected everything that had the affected version of OpenSSL/TLS on it, and that was a LOT of systems and devices. I read about patches being available, and other ways to mitigate, but also realized that it would be a long time before every device that is affected was fixed.
I took the Nyquil and other cold medicine, and was still dealing with the nasty cough, which was making it impossible to get to sleep. Back home I remember I had some great codeine-based cough syrup which, despite making me really sleepy, was great at stopping coughs. I had nothing of the sort with me in Boston, but did have a leftover Vicodin from a prescription given to me when I had a car accident back in October (not my fault, by the way). I knew that contained codeine, and took that. It worked. The cough subsided, and I drifted away into lala land. My doctor late told me that was a wise thing to do.
I was awakened soon after drifting by a phone call from a reporter with a medical journal, who wanted to know how Heartbleed could possibly affect medical devices and healthcare systems in general. I managed to deliver what turned out to be a fairly cogent interview, which he published immediately. I drifted off to sleep.
The next day I had another meeting, and managed to get through it with some Dayquil. I went to the airport feeling quite ill at this point, and while waiting for my long flight home, I got the first call from one of my contacts at DHS, asking if we could do a webinar, as everyone at DHS and upward was very concerned. I said I would make it happen for them, and took my long flight home. I went to the doctor the following day and was informed that I had Pneumonia...and put on heavy doses of antibiotics, which, I am happy to say, seems to have worked.
During the downtime at home I was asked to put together some informative webinars, which I and my team did very quickly. It occurred to me as I helped deliver the message that the community of those affected did not seem to all get just how serious an issue Heartbleed really is. The attack is ridiculously simple to mount, is completely undetectable, and affects EVERYTHING that is running the affected versions of OpenSSL. That means small handheld devices, phones, VPNs, routers, mesh network equipment, general networking equipment...just about everything. Again, I want to emphasize that an attack is UNDETECTABLE. Most users are likely completely unaware if they have an affected version of OpenSSL. Some users that may be aware cannot simply patch devices right away. An example of this is health care. Although the FDA allows patching of devices for security, the device manufacturer must still test the patch for any regressive behavior, and that is no small task. Once the patch is deployed all certificates must be revoked, public and private keys must be re-generated, new certificates must be deployed, and then (and only then) can users change usernames and passwords. While websites can potentially do this all quickly, any one of those steps can take a very long time in healthcare.
Well, I continue to deliver webcasts, as well as field lots of inquiries, and review multiple requests to sit on panels and in meetings to discuss Heartbleed. I never expected to be part of the team that discovered what some have called the biggest bug to ever hit the Internet, but here I am, and I have to say I expect things to get more and more interesting as time goes on.
Perhaps I might also consider building a sauna. When in Rome...as the saying goes.
