Fuzzing Medical Devices...The FDA Certainly Will Be

In August of 2012 the US Government Accountability Office (GAO) released a report titled "Medical Device - FDA Should Expand Its Consideration of Information Security for Certain Types of Devices", which essentially stated that the FDA should do something to address the growing cybersecurity issues researchers had uncovered in medical devices.  While the Food and Drug Administration (FDA) was essentially told what they were expected to do, they were not told how they had to do it.

While it may seem, to some, that the FDA was falling short in their need to address cybersecurity, it is important to note that the FDA does indeed provide guidance (and has for many years) related to cybersecurity from a functional perspective.  In fact, with all issues related to safety (and we are talking about cybersecurity as it relates to safety here), the FDA is quite thorough in addressing safety as it relates to unintentional misuse of functionality.  What is different today is that the FDA is now tasked with addressing intentional misuse...and that is where things become complicated.

The reason this is so complicated is because intentional misuse (or all the ways something can be used incorrectly...malicious or otherwise) is infinite.  That is why hackers, researchers, or malicious actors have so much to work with.  Moreover, hiring a hacker to constantly try to constantly hack away at your medical devices during the 18 month to 2 year development phase can become quite cost prohibitive.

In conversations I had with the FDA, who happen to be a very busy and underfunded agency, it was clear to me that they wanted to figure out a way to shrink this infinite space into something reasonably manageable, and they began seeking the advice of the security community...and the security community was happy to help.  What is particularly great about having discussions with the FDA is that they are, by and large, scientists.  Security researchers...despite the rather underground nature they have worked in for so long...are also scientists.  While some may argue against that assertion...others will agree.

Scientists like empirical evidence, and are driven more by curiosity than by dollars.  I am not saying they are not budget conscious...because they must be in order to conduct research.  What I am saying is that they are more concerned with "what if" than "how much does it cost".  This, as many of us our painfully aware, differs from the corporate world.  If you talk to a hacker for any length of time, you will see the similarities to scientists pretty quickly.

In late April of this year I joined Codenomicon, which is a company that is arguably the world leader in fuzzing technology.  For those who do not know what fuzzing is, I strongly suggest you do some "Googling" and read about it.  In short, it is the practice of exercising (some may say bombarding) a target with malformed data until it produces an error...or simply dies.  The malformed traffic that causes the error is thereby deemed a vulnerability.  A good fuzzing tool keeps track of what causes the error, and allows it to be replayed as needed to help developers remediate the error.  

So let's get back to the FDA.  Codenomicon demonstrated their fuzzing tools (known as Defensics) to the FDA, and they were more than a little impressed.  As we were informed, they had decided to build a cybersecurity testing lab, and wanted to bring in our fuzzing tools as the first of many tools to come.  It took a while to get from the initial conversation to the final award, but on July 12, 2013 the FDA posted a solicitation for Codenomicon Defensics, and we were awarded the contract on August 13th, 2013. 

Needless to say, this has certainly generated a lot of buzz in the medical device industry.  The FDA released draft guidance in June, 2013 stating they are expecting vulnerability assessments as part of the documentation submitted to the FDA, then states they are building a test lab, and will incorporate fuzzing into their lab.  This, of course, is part of the answer to how they are going to address cybersecurity.

I look forward to working with the FDA in making sure our medical devices are secured.  This is a great first step toward that goal.

The Archimedes Medical Device Security Group

The illustrious and ever so articulate Kevin Fu, who has emerged as the premier academic in the world of medical device security in the past several years, held his first Archimedes workshop at the University of Michigan this past May 9th and 10th.

This invitation-only event (which I was proudly invited to participate in) brought together 65 top security professionals, medical device manufacturers, health care system representatives, academics, doctors...and just about everyone else who has a stake in medical device security (except regulators and patients).  I do not recall ever having been around so many PhD's in my life.

The purpose of this event was to have an open discussion of the challenges associated with securing medical devices, and what we might all do to help resolve the issues.

The key points that came out of the event are as follows:

• Health care organizations and medical device manufacturers are making assumptions about the issues without looking at the whole picture.
• We simply do not have enough data about what the real issues are and what everyone is doing to address the issues to determine how serious the problem may be...or how far along we are...or are not.
• Trying to come up with new ways to address security may not be as prudent as re-purposing what others have already done in other industries (particularly the Industrial Control System space).
• It is difficult to get anyone to take responsibility for the issues.  Everyone hands it off to someone else (some more than others...some not at all...to be fair).
• Viewing security in terms of return on investment is pure folly...and will get nowhere.
• Vendors are not ready to provide what customers (health care providers) are not demanding, and health care providers are not ready to demand anything.

There were certainly others that came out, but most importantly the people at this event REALLY cared about talking about the issues...and were fully engaged.  This is what I found most important, because I have been working on medical device security for nearly 6 years, and for at least 4 of those 6 years I was often the only person in the room who had anything to say about the subject, and had to deal with a lot of blank stares, or comments like "Oh yes, privacy is very important in health care."  It is finally dawning on the health care community at large that we are NOT talking about privacy any more.  We are talking about safety.

Did we solve any problems?  Probably not...except for the problem of open and honest communication, which seems to have been resolved for at least this small event.

I'll take my baby steps and be quite content with them, and thank Dr. Kevin Fu (and company) for making something like this happen.  Getting smart people who really care together in a room with a common goal is often not a bad thing, and can move things forward in untold ways.

Why Huawei and ZTE Are Potential Red Herrings

A Red Herring can be described as a distraction or scapegoat to divert attention from bigger issues by focusing on a smaller issue.  Sometimes those who try to divert the attention to the Red Herring issue do so intentionally, sometimes it is done out of ignorance.  In any case, it generally has the same net effect of prolonging the solution.

In a recent 60 Minutes episode, Huawei was portrayed as a massive Chinese networking equipment manufacturer that was making great strides in the marketplace globally, initially in Asia and Europe, and working their way towards our shores in the USA, with some early market wins in the American Breadbasket.  The 60 Minutes story talked about how Huawei was very secretive about what they do, and because they build the communication equipment that will ostensibly be the backbone of global communications, that gives them free range to potentially put in back doors, or otherwise take control of global communications.

This was followed by a US House Intelligence Committee Report that articulates that Huawei and ZTE  (another Chinese networking equipment maker) are bonafide threats to our national security.  It pulls no punches as it lays out the gory details.

I have to say that all of this is true, in my opinion, but by no means addresses the much bigger issue at hand.  Consider, if you will, that nearly ALL communication equipment used globally today (certainly in the US) is made in China, and ALL of it can be provisioned with the same back doors.  That popular smartphone you and all your friends carry around and carry on conversations with, send emails with, submit documents through is likely made in China.  That wireless router that your laptop, tablet, desktop, phone are all communicating with, attached to that switch in your office or home network are all likely made in China.  We have literally MILLIONS and MILLIONS of communications devices where we have little to no visibility of the supply chain.  Even the "US Makers" of networking equipment have significant (and often ALL) components made and provisioned in China.

I bring this up because I have worked on projects to address some of these security issues with companies that provided components to communication equipment manufacturers.  While some manufacturers have taken some steps to address these issues (mostly ones who have been breeched and shamed), others have done nothing at all.  In at least one case I am aware of a major manufacturer only addressed a very small subset of their equipment, which was essentially their high end networking equipment, and all but ignored their lower end (and far more popular and prevalent) devices.

The American public, not knowing any better, may indeed believe that the US Government is doing us a great justice by performing this study, issuing this report, and taking some steps to address this issue.  I would have to say that this may be a good first step, and an eye opener, but we are FAR from addressing the real issue.

Let's hope we all wake up and take notice.

Our Government, The TSA, and Medical Device Security

Let's face it...we deal with security in a very reactive way, and we often end up with some real progress on the security front, along with some very real screwups.

I want to step back to 9/11 for a moment.  It is vivid and clear in most (if not all of our minds).  After the 9/11 attacks happened, we witnessed a time when our government had a lot of power (and most of it remains today) to do almost anything they wanted to do in the name of securing our nation.  For the first time in my life I witnessed a Congress the passed sweeping laws that created a somewhat muted police state, cloaked as "The Patriot Act".  Those that chose to defy our leaders were labeled as miscreants (in some cases), or told that they were being unpatriotic.

Some things that came out of 9/11 were pretty good.  Locking of cockpit doors is one of them.  Another is a heightened sense of awareness by airline passengers, who are not likely to sit idly as terrorists attempt to mess around on an airplane.

Other things...not so much.

Anyone who travels today must now be subject to the circus we know as the security line at any major airport, which is a barricade often manned by everything from well experienced and concientious agents, to agents that find their absolute power quite satisfying, and not in a good way.  We are forced to relinquish event the tiniest pocket knife, but are permitted to carry our laptops on the plane, which has a glass screen that could easily be broken, leaving us with a razor sharp length of glass that could do far more damage in the hands of a would be terrorist.  There have been instances of agents who have become so attentive of water bottles, that they miss handguns in carryon items.  Elderly people in wheelchairs are often forced to go through lengthy searches, and in at least one case I have personally witnessed several agents taking someone out of a wheelchair who could not stand up, and forcing him to go through a search while he totters on the brink of falling.

One would hope, after more than a decade of dealing with security issues post 9/11, that we would be better at this than we are, and I certainly do not feel more secure.  I am simply more annoyed.  I don't fear terrorists at the airport.  I fear TSA agents who routinely search checked bags, and sometimes steal the contents.  I saw a story last week about a TSA agent who routinely stole iPads out of checked bags.  The TSA response was something like "We are looking into it."  Try as you will, but you are not likely to win an argument or case against the TSA.  They are, after all, supremely powerful in this day and age.

This brings me to the issue of medical device security.  I am seeing a lot of news stories lately where our government is being pressured to do something about medical device security.  Without question, it is an issue that has to be addressed.  The FDA is currently being pressed upon to act on cybersecurity issues, and it remains challenging, to say the least.

We face a situation today where Congress is likely to require the FDA to take a more active role in cyber security, and I am concerned that if this is done in a reactive and hurried manner, we face the possibility of overreaction.  Cyber security is not complicated, but it is a constant learning process that requires immersion to fully understand.  The FDA is currently one very busy agency, and in my conversations with the FDA, I have discovered that they are very challenged in keeping up with the workload they are presently facing, which would lead me to conclude that they are going to be very challenged in  properly consider all the nuances of any cyber security decisions they may be forced to make.  As someone who has worked with (and continues to work with ) medical device manufacturers, I have learned that patient safety, reliability of therapies, and ease of use are of utmost importance to manufacturers and patients alike, and decisions made about how to implement security must be tested to the nth degree before proceeding with implementation.  Congress must understand that if anything is mandated, there will indeed be manufacturers who will focus more on compliance over creating a security culture, and what the FDA (and Congress) should focus on is first understanding what a security culture should look like, and how the risk profile  changes with any security related decision made by a device manufacturer.

It is important to understand that, as a traveller, I often have alternatives to dealing with air travel, and the repercussions of the inconvenience are generally limited to a specific instance (a trip).  The repercussions of bad decisions made on the medical device front are far more serious in nature.

I am off my soapbox.

Ethical Hacking, Health Care, and Irrationality

As a security professional, I am often intrigued and frequently fascinated by some of the clever things that security researchers come up with.  I remember hearing of one presentation where a researcher was able to tunnel into a laser printer that was exposed to the internet, and stop a print job, causing the paper to catch fire in the fuser (at least I believe that is how he did it).

Another interesting discussion I was made aware of was someone who could remotely access cars parked in a parking lot and set off the car alarms.  Very clever indeed.

Another researcher showed how he could attack an ATM and get it to spit out cash.  Fascinating!

The fact is that people who build these devices and add features that allow creative ways to access them build them for functional purposes.  The functionality is complex enough to build, so it is not very likely that they consider non-functional uses of such devices when building them.  Even if they did, someone is likely to give them some blank stares and furrowed brows if they spend too much engineering time considering how not to use the device.

This, of course, is completely at odds with the way security researchers (hackers, if you will) look at things.  They do not spend much time looking at what makes something work, but instead on what makes it not work...or work in a way that was not intended by the designer.  As luck would have it (for the hackers), there are literally infinite possibilities in the non-functional world.

This, of course, leads to a lot of ways to potentially entertain and certainly alarm more than a few people.  In many cases, there are ways to mitigate some of the risk associated with these findings.  In the case of the laser printer, simply  segregating the network is fairly straightforward.  With ATMs, we can always go back to tellers (until we find a fix).  We can always disable the network features of automobiles as well.  These "fixes" may inconvenience us, but at least they do not diminish our quality of life in any major way.  As humans, we cope well.

From an ethical persecutive, one can argue that the work of security researchers is completely necessary to move secure design and development forward.  Let's face it...good security happens in a reactive manner close to 100% of the time.  Going back to my earlier point, engineers are primarily tasked with building functionality into products.  There are ways to do it with security built in from the ground up, but we are still a long way from getting the engineering world to embrace that.  We will eventually get there, but not likely until the public consciousness is raised.

Nonetheless, I had the opportunity to review an excellent document recently titled "The Menlo Report: Ethical Principles Guiding Information and Communication Technology Research" (http://www.cyber.st.dhs.gov/wp-content/uploads/2011/12/MenloPrinciplesCORE-20110915-r560.pdf) which "...proposes a framework for ethical guidelines for computer and information security research, based on the principles set forth in the 1979 Belmont Report, a seminal guide for ethical research in the biomedical and behavioral sciences."  This report really hit home with me, and I want to explain why, but first let me tell you a couple of stories.

I want to start with a story about flu shots.  Back in the 1980s I worked at a resort in Florida and one day I heard that a nice old chap who was one of the dock masters had gone to his doctor for a flu shot (his first ever) and had an allergic reaction to the vaccine, went into anaphylactic shock, and died.

Wow!!!

That sealed the deal for me.  Despite all the prodding I had gotten from those around me as I grew older, I decided that there was no way I was going to get a flu shot.  I mean...c'mon...I DON'T WANT TO DIE OF AN ALLERGIC REACTION!

This sat well with me for many years...until I got the mother of all flu attacks in 2004.  I remember laying in bed in sheer misery for two weeks, first fearing that I was going to die of all the pain and inability to breathe...and then, towards the end, almost wishing I would just die.  Let me tell you, a cold is NOT the flu.  I have had bad colds, and this was the flu, and it was utter HELL.

It occurred to me, once I was feeling better again, that the one instance in my entire life of anyone having a severe reaction to the flu vaccine was not a rational justification for that two weeks of misery, which was likely to happen (and perhaps even kill me as I got older) again.  I certainly like to think of myself as being intelligent, but I have a way of rationalizing things to suit my purpose outside of global empirical information...and sometimes it bites me in the you know what.

What is perhaps even more alarming is that, prior to my awakening to the benefits of vaccination, I almost prevented my first child from getting vaccinated because of all the hysteria surrounding alleged incidences of autism from vaccinations.  Were it not for the calm and patient persuasion of the vaccination nurse at the local medical center, who explained to me that the likelihood of devastating childhood maladies was indeed quite high for my baby if he did not get vaccinated, I may have exposed him to several diseases that came back in the last decade (namely polio and whooping cough), no doubt at least in part from the irrational fears brought about by vaccination naysayers.

Fear and uncertainty has a way of getting us to do things outside of the realm of reason at times.  It is the essence of propaganda, marketing hype, and political circuses.  After watching "Jaws" in the 1970s it was not until I had spent over a decade living in Florida, where the waters are literally thick with sharks, that I realized that the likelihood of getting attacked by a shark was FAR smaller than the likelihood of getting skin cancer...which several of my Florida friends and associates did contract.  Media hysteria and Hollywood stunts have a way of tugging at our hearts and warping reality...indeed they do.

So this brings me back to the point I am trying to make (and thank you for being patient).  Lately, there have been more than a few media-rich and Hollywood-like stunts portraying some of the dangers of security flaws found in medical devices, and this is simply not sitting well with me.  Unlike printers, ATMs, and automobiles, medical devices are currently causing patients that need them to experience a much better quality of life than they would have without them...and in many cases they are keeping them alive.  If one looks at sections C 3, 3.1, 3.2, and 3.3 of the aforementioned report, some very salient points emerge:

C 3 Beneficience
"...the Beneficence principle reflects the concept of appropriately balancing probable harm and likelihood of enhanced welfare resulting from the research. Translating this principle to ICTR demands a framework for systematic identification of risks and benefits for a range of stakeholders, diligent analysis of how harms are minimized and benefits are maximized, preemptive planning to mitigate any realized harms..."


C 3.1 Identification of Potential Benefits and Harms
"...researchers should identify benefits and potential harms from the research for all relevant stakeholders, including society as a whole, based on objective, generally accepted facts or studies..."


"...One helpful approach to identifying harms is to review the laws and regulations that apply to an ICTR activity, and analyze the underlying individual and public interests that the research might negatively impact..."


"Because laws may be unclear or open to interpretation, a narrow focus that only considers acts impacting the integrity or availability of information and information systems might overlook a broader range of harms that may not be explicitly protected by law."


C 3.2 Balancing Risks and Benefits
"...the researcher should systematically assess risks and benefits across all stakeholders. Researchers should be mindful that risks to subjects are being weighed against the benefit to society, not to to either the research subjects or the researchers themselves. Researcher actions should be measured using a standard of a reasonable researcher, who exercises the knowledge, skills, attention, and judgment that the community requires of its members to protect their interests and the interests of others.
When ICT is involved, burdens and risks can extend beyond “the human subject,” making the quantification of potential harm more difficult than with direct intervention. It can be difficult to balance risks and benefits with novel research whose value may be speculative or delayed, or whose realized harm may be perceived differently across stakeholders. If there are plausible risks, researchers bear the burden of showing specific, evidence-based consideration that they can manage those risks."


C 3.3 Mitigation of Realized Harms
"Despite appropriate precautions and attempts to balance risks and benefits in ICTR, research may cause unintended side effects that harm stakeholders."

Please understand that I have taken excerpts out of the report, and I expect the reader of this blog posting to look at the entire article.

The point should be relatively clear by now.  The work of security researchers is invaluable, but prior to the hacking of medical devices, the societal risks have not hit home quite at the level they are now.  The fact is that any patient who chooses to minimize their risk of having their device hacked as a result of the media hype is likely to make a decision that is likely to cause them far greater harm than the likelihood of the device being criminally hacked.

It is my hope that security researchers delving into medical device hacking take this under serious consideration, because (as Peter Parker - The Amazing Spiderman's uncle once said) with great knowledge comes great responsibility.

First Smart Grid Documentary Ever !

It's an interesting ride...so hang on!

I am fascinated by reality much more than I am fascinated by anything in the world of fantasy.  I mean, think about it for a moment.  In my life I have watched us go from phones with rotary dials and coiled cords that always got bizarrely tangled, tethered to walls via mysterious outlets, to handheld computers that allow you to place video calls, and allow you to have conversations with them as they reply to your commands with a sexy voice.

It seems like humans are capable of building anything if they see a need for it to be built, and that is the most fascinating story ever told...and it has been told many times in all of our lives, and as far as time goes back.

We all build things...create things if you will...for our own reasons, or for those who employ us.  It seems to me that the most interesting things built are built by those who driven by a desire to make something great, or make something better, and not necessarily for a paycheck.

Don't get me wrong...a paycheck is nice, and definitely a necessity in life, but it is rarely the driver to those who want to build great things.  What drives people to build great things is the human need to prove to themselves (and others) that they can create great things, or make other things better.

It's a fascinating story to watch unfold, if you will take the time to discover it.  If you like your iPhone, you really should get yourself a copy of Steve Job's biography, and understand what drove him to create the device that literally changed the way we consume information, navigate, and communicate.

Besides reading books, my favorite way of consuming these fascinating tales of how things came to be as they are today is through watching documentaries...and lot's of them.  I remember the first time I discovered that through Netflix streaming services I could watch literally hundreds of documentaries on nearly as many topics...and so I did (much to the behest of my small children).  It seems like I cannot get enough of them.  There are so many interesting stories to be told, and the documentarians seem to do a fine job of getting past the hype, marketing spin, and myths surrounding so many subjects worth exploring.

So let's fast forward to my security conference.  Back in 2010 my company was hired by a company (a silicon vendor) to produce a whitepaper that outlined the Smart Grid security landscape.  I dove right in, as I normally do, and attempted to capture the essence of the Smart Grid as quickly as I could.  What I soon discovered is that, although the Smart Grid was rapidly evolving, our understanding of the Smart Grid was changing with every passing moment.  The issue of Smart Grid security was particularly challenging to grasp, since the topic is very sensitive in nature to most, and those involved in the Smart Grid security ecosystem still had a lot to learn.  While I discovered pockets of knowledge here and there (e.g. NIST, OpenSG, DHS ICSJWG), there was no place I could go to truly immerse myself in the dialogue that I felt needed to happen.  There were lots of Smart Grid conferences out there, but they covered the topic of Smart Grid security at a very minimal level at best.  There were also lots of security conferences out there, but Smart Grid was only a tiny portion of the event.  I felt that we needed a Smart Grid security event, and created the first Smart Grid security conference that I knew of in the United States.  I was shocked to have around 100 people show up for the first event, and it led to two more after that (and my 4th event is coming up at www.GridSec.com, which is focused on not only Smart Grid, but also energy infrastructure security).

As someone who was working within the ecosystem,  I was able to bring in some great speakers, and gain the trust and support of some very key players.  I always sought to evolve the conference as the industry evolved, and decided that the next event (upcoming March 27-29 in Irving, Texas) should involve people at the CxO level, and went on a quest to find at least one utility CxO who would speak on the sensitive topic of security.  I have to say, it was a lofty goal and was not easy, but persistence pays off, and I was put in touch with Dave Hallquist, the CEO of the Vermont Electric Cooperative, who agreed to speak.  That, in and of itself, was absolutely fantastic.

What happened next (a few days later) became even more interesting.  Dave's son, Derek Hallquist, is a documentary film maker, and contacted me asking if he could film his father at my upcoming conference, since he had partnered with documentary film producer Aaron Woolf (of "King Corn" fame), and they were going to follow Dave Hallquist around the country as he went from conference to conference interacting with people in the Smart Grid world.  It was to be the first Smart Grid documentary ever created, and they planned to submit it to the Sundance Film Festival.

Needless to say, I was flabbergasted.  Not only was the story of the Smart Grid going to be told in a documentary (and we are still in the very early stages of the Smart Grid), but it was going to break ground at my conference.

This has, of course, unleashed a storm of interest and support from everyone I know in the industry.  Utilities are all thrilled, vendors are all thrilled, and all those who have helped me make this happen are all thrilled.  I was wondering when a Smart Grid documentary was going to come to fruition.

...now I know.

Please Attend This Seminal Event!

Sign Up At www.GridSec.com

Humbled By The Outpouring Of Support

This past week I discovered that an article I submitted to the Information Systems Security Association (ISSA) Journal had been selected for publication, and made the cover.  Although I am part of the editorial board (the shark tank, as we call it), I submitted it anonymously, and made the cut.  Needless to say, I was thrilled.

The article was about medical device security, and I have a Google Alert set up for medical device security.  The day after my article was published, I saw a Google Alert that pointed to an article with a similar title.  My article is called "Oh, Hackable You!" and the similarly titled article was "The Hackable You."  Interesting.

When I went to the website, I realized that the author of the article had, quite literally, completely plagiarized my article.  He changed the introduction a bit, copied and pasted the entire rest of the article WORD FOR WORD, and then changed the conclusion a bit.  It was obvious and willful fraud, and I was livid.

I immediately posted this on my Twitter feed, and what happened next truly reminded me why I absolutely love working with the information security community.  My dear friend Travis Goodspeed (who has over 2700 followers) re-tweeted it and then embarked on a quest to find out more about this person, who, as it turns out, is a serial plagiarist.  He quickly discovered that dozens of members of the infosec world had been plagiarized by this person, and let them all know that this had happened, which unleashed a Twitter storm like nothing I had ever witnessed.  Within hours the organization he works for had pulled the blog, issued a public apology, and called me (and at least one of the other writers) and personally apologized for the incident(s).

What amazes me about the information security community is that it has evolved into a very tight brotherhood, independent of any "official" regulatory body.  Every member of the community is charged with the duty of policing even other member, and NOBODY gets a pass go.  Anyone who tries to enter the infosec world and attempt to sell snake oil is immediately smacked down by the community.  It took me years of hard work to get to the point in my career where the community accepted me as one of their own, and I have to say that I am completely overwhelmed by the support, and knowledge that by brothers (and sisters) in the information security world are there for me...and I for them.

Thank you!