GraniteKey

Humbled By The Outpouring Of Support

2011-12-05T03:39:00.000-08:00

This past week I discovered that an article I submitted to the Information Systems Security Association (ISSA) Journal had been selected for publication, and made the cover. Although I am part of the editorial board (the shark tank, as we call it), I submitted it anonymously, and made the cut. Needless to say, I was thrilled.

The article was about medical device security, and I have a Google Alert set up for medical device security. The day after my article was published, I saw a Google Alert that pointed to an article with a similar title. My article is called "Oh, Hackable You!" and the similarly titled article was "The Hackable You." Interesting.

When I went to the website, I realized that the author of the article had, quite literally, completely plagiarized my article. He changed the introduction a bit, copied and pasted the entire rest of the article WORD FOR WORD, and then changed the conclusion a bit. It was obvious and willful fraud, and I was livid.

I immediately posted this on my Twitter feed, and what happened next truly reminded me why I absolutely love working with the information security community. My dear friend Travis Goodspeed (who has over 2700 followers) re-tweeted it and then embarked on a quest to find out more about this person, who, as it turns out, is a serial plagiarist. He quickly discovered that dozens of members of the infosec world had been plagiarized by this person, and let them all know that this had happened, which unleashed a Twitter storm like nothing I had ever witnessed. Within hours the organization he works for had pulled the blog, issued a public apology, and called me (and at least one of the other writers) and personally apologized for the incident(s).

What amazes me about the information security community is that it has evolved into a very tight brotherhood, independent of any "official" regulatory body. Every member of the community is charged with the duty of policing even other member, and NOBODY gets a pass go. Anyone who tries to enter the infosec world and attempt to sell snake oil is immediately smacked down by the community. It took me years of hard work to get to the point in my career where the community accepted me as one of their own, and I have to say that I am completely overwhelmed by the support, and knowledge that by brothers (and sisters) in the information security world are there for me...and I for them.

Thank you!

The SCADA Within Us

2011-11-05T09:06:00.000-07:00

I have been saying this for quite some time now, and I was absolutely thrilled when someone from the health care industry came up to me and said "We are running SCADA systems in health care." For those who do not know what the acronym stands for, it is "Supervisory Control and Data Acquisition".

Let's examine this for a moment.

Supervisory - Medical systems are indeed used to supervise patients. That is exactly what they do.

Control - Medical systems are indeed used to control patient procedures at many levels. That is exactly what they do.

Data Acquisition - Medical systems record patient data constantly, and use this information to make decisions. That is exactly what they do.

Yup! They are SCADA systems.

I just returned from the Amphion Medical Forum in Minneapolis, home of Medtronic (the largest medical technology company in the world). Medtronic is very concerned with medical device security, and they are now beginning to understand the potential impact of mounting interest among the attack sector in hacking SCADA systems. Rest assured they are taking this VERY seriously, and this is an absolutely fantastic bit of news for the health care community, because they are the most likely organization to make an impact on health care security. I applaud Medtronic executives for their decision to aggressively address these issues.

One of the most interesting discussions I had with a member of the Medtronic engineering staff, who seemed very familiar with SCADA systems, was the very unique challenges the medical device industry is facing. One challenge is that they cannot easily address physical security of many medical devices, since they are frequently found in patients (e.g. insulin pumps, pacemakers) or in their homes (e.g. monitors). While it is possible to educate patients about this, it is nearly impossible to control physical security. Another issue is that, even if devices are designed with firmware that can be updated, there is no easy way to update the firmware in devices implanted in the human body, and for several reasons. One obvious reason is...well...because it is implanted in a human body. Another reason is because many of these devices operate on coin sized batteries, and many of you know that firmware updated dramatically decrease battery life. Let's not forget, by the way, that a failed firmware update on an implanted device that puts it in a DOS state is also very serious.

On the subject of power, if you think that the "traditional" SCADA systems have resource constraints, you are not even close to the resource constraints of some of these medical devices. Let's not forget the need for reliability as well.

The health care industry is taking this very seriously, but there are some major challenges to address...and this is very high priority.

Health care touches each and every life on Earth. I look forward to working with the health care industry to get this under control.

What We Really Want Is A Hot Meal, Good Health, And Electricity

2011-10-14T09:54:00.000-07:00

Those of you that know me are perhaps aware that I have have spent large portions of my life working in 3 somewhat distinct areas: Food Service, Health Care Security, and Smart Grid Security. All 3 disciplines have taught me a few things that I carry with me every day.

I am no longer in the Food Service industry...thank God! If any of you have ever watched Hell's Kitchen on TV, trust me...it is not far from reality. Working in high technology means better pay, less heavy lifting, and weekends and holidays off (more or less).

Still, I learned some things in the Food Service industry that serve as valuable lessons to this very day. One thing I learned is that regardless of how hard you work, you are inevitably judged for the last good (or bad) deed you accomplished, often irrespective of your history. Memories are short, and you always have an opportunity to either redeem yourself, or fall flat on your face. The choice is yours.

Another thing I learned about the Food Service industry is that they have 2 objectives:

Make Food
Get Paid For The Food

Hey! What can I say? I am nothing if I am not perceptive.

As it turns out, this carries over into both the Health Care and Energy industries. The Health Care industry wants to deliver health and get paid for it. The Energy industry wants to deliver energy and get paid for it.

We can apply this logic to just about any industry we choose, as it turns out :-)

Okay, so I am here to talk about security. What does all of this have to do with security?

As it turns out, security is essentially about safety (or perhaps safety is really about security). The two go hand in hand...and perhaps can be conflated in some (if not all) cases.

So let's go back to my life in foodservice for a moment. Having spent many years working as a chef in restaurants, I noticed a few things about safety that were recurring themes. One was that every single restaurant I worked in had a fire safety system installed by a competent installer, and (most importantly), the fires safety system itself was built by a competent manufacturer. After this was done, the fire inspector would perform an inspection and make sure it satisfied the requirements for fire safety, and the fire inspector would periodically return to make sure all was in order. Eventually, we saw the arrival of the National Fire Protection Association's Certified Fire Protection Specialist Certification Program, which is ANSI accredited. Additionally, UL has a program in place for approval of fire safety systems (e.g. sprinklers) in use today.

Having worked in a restaurant where the fire safety system has triggered, I have to admit that it is very effective. However, in retrospect, the fact that I find most interesting is that not one restaurant, hotel, or resort (and I worked for some big resorts) had any staff on board who was responsible for the design, implementation, and maintenance of the fire safety system.

They simply hired someone to put on in, got it inspected, and then went on with the business of making and serving food. I have to say, it works splendidly.

Imagine that!

So let's take this back to the Health Care and Energy industries for a moment. We need to understand that what we have to do in the security world is get to that point where health care and utility staffs can focus as much of their time as possible on delivering what they are in the business of delivering. We are currently living in an environment where we have place nearly all the burden for securing health care and energy systems on those who are ill suited for the job. Sure, they are getting better...by hiring staff to help get them up to speed, and reaching out to professionals, but is this necessarily the desired end state.

I fully realize that the food service industry is not saddled with the enormous burden of protecting their network stack from intrusion, and that no level of cyber attack is likely to mess with the integrity of their signature dish covered with delicious Béarnaise sauce. Yet the threat of fire is very real, generally quite devastating, and ever present. Nonetheless, we have managed to create a management system that is both extremely effective and extraordinarily simple to live with.

...and let's look at the health care industry for a moment.

We are all familiar with the FDA (the Food and Drug Administration). Hospitals use health care equipment and use drugs that are FDA approved...and absolutely do not use any health care equipment or drugs that are not FDA approved. Okay...at least they better not...or face stiff fines and immediate shutdown (believe me, the FDA is hardcore about their rules). Although it is a US organization, FDA approval is so highly regarded globally that most nations accept FDA approval as a "green light" for use in their own countries. Health care providers do not have to manage staff to ascertain the safety in using FDA approved products. They simply stick with the FDA approved products and (ostensibly) use them to deliver good health care.

I fully believe that we will eventually come to terms with cyber security issues, as we have come to terms with fire, and as we have come to terms with "snake oil" health care solutions of the past. As Paul Kocher of Cryptography Research indicate during his excellent keynote at my Smart Grid Security Summit this past month, security today is still struggling with the same "snake oil" issues that health care had to deal with in the past. As we continue to move forward with addressing cyber security issues, we all need to keep in mind that a lot of what we hear is going to be "snake oil", and we should look towards how other safety issues have been addressed in the past, and perhaps learn some valuable lessons.

Okay...now I'm hungry.

Upcoming Event: Amphion Medical Forum

2011-10-10T09:45:00.000-07:00

I have been invited to moderate a panel at the Amphion Medical Forum on November 3rd, 2011 in Minneapolis, Minnesota. This fantastic event features security experts who specialize in studying, understanding, testing, and addressing security issues related to connected medical devices.

What you may or may not know is that nearly every piece of medical equipment that collects and records data today (heart monitors, X-Ray machines, MRIs, IV Monitors...and the list goes on and on) has a communications stack of some built in, or will have one soon. Recent demonstrations at Blackhat, for example, have re-awakened our consciousness to the seriousness of security issues surrounding medical devices (if this attack in 2008 was not enough).

If this is of interest to you, join me at the Amphion Medical Forum on November 3rd, where you will have an opportunity to listen to some of the most brilliant minds in the world of medical device security, as well as meet them face to face.

Oh...and by the way...IT'S FREE !

See you there!

Mike Ahmadi

P.S. To guarantee yourself an invitation, use priority code "GraniteKey"

My Sally Field Moment

2011-10-10T07:49:00.000-07:00

My third Smart Grid Security Summit has drawn to a close. This past week in San Diego was a seminal event in my life as a conference chairman. For the last 3 weeks I have been working out a hundred plus details that no amount of advance preparation ever prepares you for. Anyone who has ever put on a conference is keenly aware of that. For those who have not, I would describe it as something akin to the excitement of the descent from the peak of a roller coaster coupled with the fact that you decided to finish you children's corn dogs.

When I stated the Smart Grid Security Summit my intention was to build my network and get some like-minded people together to chat about what was, and continues to be, an important topic. We had around 100 people show up, and 1 sponsor (SAIC). We were so proud of that event, and I still harbor fierce loyalty for those who helped make that event what it was. We knew we had something, and built on it. The second event was held in Knoxville in early 2011, and we had around 10 times the sponsorship, and double the attendance. Most importantly, we had asset owners coming to the event to both participate as speakers and join the crowd of attendees. We were sure we had something of value at this point. Let's face it, Knoxville is a really nice place, but it is certainly not a "conference boondoggle" location. People showed up because they had a thirst for knowledge and because they wanted to communicate with people who understand what they need, and we delivered that.

The third event say us partner with the Energy Sector Security Consortium (EnergySec), and we were blessed with lots of great sponsorship, and perhaps the finest selection of speakers and attendees to date (although that is a tough call, since both of our other events had fantastic speakers and attendees). It just seems to keep getting better and better as time goes by. I tried to take the time to speak to everyone I ran into at this event, with around 15 sponsors and around 250 attendees, but found myself nearly overwhelmed by the outpouring of interest in the event, the massive amount of networking going on, the fantastic sessions, and the constant outpouring of love from all who took the time to come up to me and tell me what a fantastic event our little conference has grown into.

I cannot help thinking about that famous Sally Field moment, when she accepted the Oscar for her starring role in the 1984 drama "Places In The Heart". She took the stage after receiving the Oscar and gushed "I haven't had an orthodox career, and I've wanted more than anything to have your respect. The first time I didn't feel it, but this time I feel it, and I can't deny the fact that you like me, right now, you like me!"

Peer acceptance is what we all crave in our careers, regardless of what we may think or say about the subject. I am humbled by everyone's acceptance and love, and will continue to deliver the quality you have all come to expect.

Kindest Regards,

Mike Ahmadi
Conference Chairman

Smart Grid Security East 2011: AMI Vendor Roundtable

2011-09-15T20:22:00.000-07:00

This is the video taken of the AMI Vendor Roundtable panel at the Smart Grid Security East conference in March 2011.

The presenters were:

Edward Beroset, Director of Technology & Standards, Elster Solutions Inc.
Stephen Chasko, Principal Security Engineer, Landis+Gyr
Walter Sikora, VP of Security Solutions |Industrial Defender
Ido Dubrawsky, Principal Software Engineer/Security, Itron

We hope you will join us at the EnergySec Smart Grid Security Summit West 2011 conference from October 3-5 in San Diego, California.

See you at the next event - www.smartgridsecuritysummit.com

Smart Grid Security East 2011: Panel - How Utilities Are Managing Security

2011-09-11T13:59:00.000-07:00

This is the video taken of the "How Utilities Are Managing Security" panel at the Smart Grid Security East Conference in March 2011.

The presenters were:

David Batz, Manager, Cyber & Infrastructure Security, Edison Electric Institute (EEI)

Ward Pyles, Senior Security Analyst, Southern Company

James Sample, Director of Enterprise Information Security, Tennessee Valley Authority (who has recently been promoted to CISO of Pacific Gas & Electric)

Robert Humphrey, Senior IT Security Analyst, Duke Energy

Moderator: Bob Lockhart, Senior Analyst, Pike Research

I am pleased to report that all of these panelists (and more) will be returning to the EnergySec Smart Grid Security Summit West conference from October 3-5 in Sand Diego, California. We hope to see you there!

See you at the next event - www.smartgridsecuritysummit.com

Smart Grid Security East 2011: NISTIR 7628 - Progress Report

2011-09-09T20:34:00.000-07:00

This is the video taken of the NISTIR 7628 Progress Report session at the Smart Grid Security East Conference in March 2011.

The presenters were:
Annabelle Lee, Technical Executive - Cyber Security, EPRI
William Hunteman, Senior Advisor For Cyber Security, US Department of Energy (DOE)
Daniel Thanos, Chief Cyber Security Architect, GE Digital Energy
Sandy Bacik, Principal Consultant, EnerNex
Mike Coop, ThinkSmartGrid
Moderator: Mike Ahmadi

Please join us for the EnergySec Smart Grid Security Summit from October 3-5, 2011 in San Diego, California.

See you at the next event - www.smartgridsecuritysummit.com

Smart Grid Security East 2011: Keynote Address - Annabelle Lee

2011-09-08T16:45:00.000-07:00

This is the video taken of Annabelle Lee's fantastic keynote at the Smart Grid Security East Conference in March 2011. Please join us for the EnergySec Smart Grid Security Summit from October 3-5, 2011 in San Diego, California.

See you at the next event - www.smartgridsecuritysummit.com

The Importance of Context When Discussing Smart Grid Security

2011-09-08T16:38:00.000-07:00

This letter was originally posted on the excellent Smart Grid Security Blog. It is a letter from former NERC CSO Michael Assante to the global community of stakeholders who are working diligently to keep our critical infrastructure safe from attackers.

I recently had an opportunity to learn about the importance of context. I tried to help someone understand the challenges of regulation and cyber security in the context of smart grid technology deployments and electric infrastructure, and learned once again how polarized this topic can become. Certainly many can appreciate the challenge of communicating with clarity on this topic, as it can be nuanced, highly-technical, process-laden, and mired in the details of a little-followed piece of history and U.S. federal and state law.

Let me begin by providing some of the context, or background, that explains why I work hard to help develop a better understanding of how cyber security impacts operational technology in critical infrastructures. As a boy I was fascinated with the engineering required to generate and deliver electricity. To me, the power system represented a grand achievement that demonstrated what dedicated men and women could accomplish.

My father worked for a utility and was rightfully proud of the public service his company delivered to homes, schools, manufacturing plants, and hospitals. He worked with impressive machines that excavated coal, and cutting edge control centers with analog light displays. But the thing that made the biggest impact on me was the dedication with which my father and his colleagues performed jobs, and their uniform sense of mission, as they clearly understood that what they did made people’s lives better. I was quick to appreciate the vision, investment, and effort that enabled vast natural resources like coal and hydro-power to be turned into electricity, which was then transported and delivered over vast distances to every household and business.

The success of the electricity industry in designing, building and maintaining an incredible system of systems, continues to inspire children and adults alike. It has grown to become a critical infrastructure that underpins modern society. The delivery of highly-affordable and reliable electricity has paved the way for the industrial and technological revolutions that have transformed global economies. It is ironic that over the last forty years of progress, we have also created a significant set of challenges that need to be addressed as a consequence of our continued innovation.

The rapid advancement and application of digital technology has improved electric system operations, reliability, and process efficiency. But it carries with it a heavy responsibility. We must now safeguard this increasingly ubiquitous element of the grid from those who would seek to disrupt technology and cause harm.

This dilemma of digital technology is that, like electricity, it enables great things but can cause great damage if not managed properly. There is one very important difference, though. The nature of electricity is understood sufficiently to prudently manage the risks it can present, whereas cyber threats are constantly evolving and are co-adaptive (the threat will consider the protections you have employed and find ways to circumvent or compromise them). This has led me to conclude that many of the difficulties we experience addressing cyber security come less from how the electricity industry behaves, and originate more from the complex nature of digital technology and the unique risks it engenders.

Many of you know that I have often shared my thoughts on the difficulties of managing cyber risk in the complex and vast systems that comprise power grids. There are a number of necessary constraints, such as the golden rule of “first, do no harm” (do not negatively impact system reliability and safety). Other challenges have more to do with state of industrial control system technology and the tough job of keeping up with the rapid changes in technology and the evolving capabilities of would-be cyber attackers.

NERC and the industry have pioneered the use of mandatory reliability standards as one tool to manage risks to reliability across the complex weave of entities that comprise the bulk power system in North America. I am confident that progress will continue to be made by NERC and the industry, but it takes time to learn what works well when dealing with the scale of the bulk power system and specifically, when trying to address the difficult-to-bound risk that comes from cyber threats. I, like many others, understand that we must continually evaluate the processes we use to develop and manage the CIP standards. We must consider the effectiveness of the standards requirements when compared to how digital systems are being compromised by current cyber attackers. Cognizant of the risks of unintended consequences, we need to fully understand the behaviors we are promoting by using standards that require strict compliance. Finally, we need to be mindful of the spirit and goal of the standards and the importance of providing enough flexibility so that utility security programs can adapt to best confront the threats they face.

I have had the pleasure of working alongside of some of the most gifted experts in power engineering and industrial control system security over the years. The power industry has a rich collection of experts often passionately inclined to work together as a community to solve complex problems. Their expertise is essential in determining how to best apply cyber defenses in the highly-specialized environments of power generation, transmission, and distribution. We would also, however, benefit from the experience and learnings of other industries’ cyber professionals who themselves labor to defend highly-targeted networks. I have grown to appreciate the adaptive nature of cyber threats and importance of maintaining a current understanding of how systems are compromised. NERC has engaged with the U.S. government to benefit from its understanding and should continue to look for opportunities to learn from government and cyber security experts from other industries bent on tackling this common problem.

Context matters in how we think about these problems, in how we frame our concerns, and in how we formulate new approaches so that we may attain the many benefits of new technologies while managing the risk. I am confident that we will begin to engineer away the worst consequences, continually find more effective practices and develop the necessary skills to better address sophisticated and ever changing cyber threats. This is a difficult task that will continue to require our best efforts, to include regulation. It is a task that demands a prudent approach as the effectiveness of our investments needs to be measurable and demonstrable. We must continue to innovate if we're to fully enjoy the many benefits of affordable and reliable electricity.

Michael can be reached at michael.assante@nbise.org

Michael will also be speaking at the EnergySec Smart Grid Security Summit West in San Diego, California, October 3-5, 2011.

Customer Data: Authorization, Privacy and Security - Smart Grid Security Summit East 2011

2011-09-07T21:44:00.000-07:00

This is the video taken of the Customer Data: Authorization, Privacy and Security session at the Smart Grid Security East Conference in March 2011. The presenters were:

Sandy Bacik, Principal Consultant, Enernex
Megan Hertzler, Director of Data Privacy, Xcel Energy Services
Boris Segalis, Partner, Information Law Group
Moderator: Chris Kotting, ThinkSmartGrid

Please join us for the EnergySec Smart Grid Security Summit from October 3-5, 2011 in San Diego, California.

Due to the extreme popularity of the Privacy in the Smart Grid, we will be hosting a pre-conference workshop. Please make sure you sign quickly as space is limited. You can sign up at http://www.smartgridsecuritysummit.com/Info/RegistrationInfo.aspx.

See you at the next event - www.smartgridsecuritysummit.com

Michael Assante Keynote - Smart Grid Security Summit East 2011

2011-08-29T22:30:00.000-07:00

This is the video taken of Michael Assante's fantastic keynote at the Smart Grid Security East Conference in March 2011. Please join us for the EnergySec Smart Grid Security Summit from October 3-5, 2011 in San Diego, California.

Michael will be joining us at again from October 3-5 in San Diego.

See you at the next event - www.smartgridsecuritysummit.com

DOE Smart Grid Security Grant Recipients - Smart Grid Security East 2011

2011-08-23T14:53:00.000-07:00

This is the video taken from the DOE Smart Grid Security Grant Recipients session at the Smart Grid Security East Conference in March 2011. Please join us for the EnergySec Smart Grid Security Summit from October 3-5, 2011 in San Diego, California.

Panel: The US Department Of Energy (DOE) Smart Grid Security Grant Recipients

The DOE has allocated substantial funds to both private enterprises and non-profit agencies in their quest to improve the security of our power grid. This public/private partnership is intended to capture the best and brightest and drive Smart Grid security to level it needs to be in order to ensure a smooth transition to a smart and stable infrastructure. Join this panel of DOE recipients and DOE sponsor representatives in a discussion of what we can expect as a result of this partnership.

Moderator: Mike Ahmadi

Panelists:

William J. Hunteman, Senior Advisor For Cyber Security, US Department of Energy

Seth Bromberger, Executive Vice President, Energy Security Consortium

Craig Miller, Project Manager, National Rural Electric Cooperative Association (NRECA)

Dr. Hal Aldridge, Director of Engineering, Sypris Electronics

Annabelle Lee, Technical Executive – Cyber Security, Electric Power Research Institute (EPRI)

See you at the next event - www.smartgridsecuritysummit.com

Panel: DOE, FERC, NERC - Smart Grid Security East 2011

2011-08-04T09:18:00.000-07:00

This is the video taken from the DOE, FERC, NERC session at the Smart Grid Security East Conference in March 2011. Please join us for the EnergySec Smart Grid Security Summit from October 3-5, 2011 in San Diego, California.

The Department of Energy, Federal Energy Regulatory Commission, and North American Electric Reliability Corporation are unquestionably three of the most watched Federal agencies in the Smart Grid deployment world today. Join this in an interactive discussion about how they are working together to secure our grid.

Moderator: Andy Bochman, Security Lead, IBM

Panelists"
William J. Hunteman, Senior Advisor For Cyber Security, US Department of Energy

Jason Christopher, Technical Project Lead for Smart Grid Security, Federal Energy Regulatory Commission (FERC)

Mark G. Lauby, Vice President, Reliability Assessments and Performance Analysis North American Electric Reliability Corporation (NERC)

See you at the next event - www.smartgridsecuritysummit.com

Smart Grid Security East 2011 - Harmonizing Federal and State PUC Guidelines

2011-07-29T20:52:00.000-07:00

This is the video taken from the Harmonizing Federal and State PUC Guidelines session at the Smart Grid Security East Conference in March 2011. Please join us for the EnergySec Smart Grid Security Summit from October 3-5, 2011 in San Diego, California.

While Federal agencies may indeed have jurisdiction of some parts of the Smart Grid, a large part of the Smart Grid falls directly under State jurisdiction, and certainly most of AMI. This session will present the perspectives of State Public Utility Commissions in various stages of deployment.

Panelists:
Alan Rivaldo, Cyber Security Analyst, Public Utility Commission Of Texas

Christopher Villarreal, Regulatory Analyst, California Public Utility Commission (CPUC)

Craig Miller, Project Manager at National Rural Electric Cooperative Association (NRECA)

Moderator: Chris Kotting - ThinkSmartGrid

See you at the next event - www.smartgridsecuritysummit.com

The NRECA Cooperative Research Network Security Strategy

2011-07-14T07:28:00.000-07:00

It was through a conversation I was having with Christopher Villarreal, Regulatory Analyst with the California Public Utility Commission (CPUC), that I was first made aware of Craig Miller, who is the Project Manager for the National Rural Electric Cooperative Association (NRECA). Chris is generally a soft spoken guy, and that makes me pay a bit more attention to him when he talks. He told me that I really needed to reach out to Craig Miller and include him in my Smart Grid Security Summit as a speaker, since Craig seemed to know what he was talking about with respect to cybersecurity.

It took me a while, but I finally got through to Craig (he is a busy guy). I have to admit, being someone who has had a lot of conversations with "the big boys" in the world of Smart Grid security, I was not expecting the level of knowledge and professionalism that the NRECA exhibited. Suffice it to say, the members of the NRECA are well served by the organization.

Let me explain.

Craig was a panelist at my Smart Grid Security East conference in Knoxville, TN this past March, 2011, and he was easily one of the most popular panelists at the event. He does not mince words when he speaks. He is a consummate straight shooter in every sense of the word, and gets down to business right away. When asked about what the NRECA is doing to help their COOP network address security, he will tell you that they are defining a "process of continuous improvement", and goes on to explain that rather than telling their members what to do, they offer detailed and ACTIONABLE guidance, as well as continual educational programs. It reminds me of the saying "Give a man a fish and he can feed himself for a day. Teach a man to fish, and he can feed himself forever.".

Back in March, it was all great talk, and I (and many others) left the event wondering how this program worked. It did not take long to find out. In May of 2011 (2 months after my conference) the NRECA released A Guide to Developing a Cyber Security and Risk Mitigation Plan, and made it publicly available for all to see. It is a fantastic collection of materials, put together with the assistance of Cigital, and besides providing an fantastic collection of well referenced cybersecurity guidance (much of it based on the NISTIR 7628 guidance document), it provides templates and plenty of "getting started" materials and templates.

Why is this so important? I'm glad you asked...

It may come as a surprise to many of you, but the fact is that most facilities that generate power in our great nation are not staffed with massive IT departments, and much less security experts. This is true in general, and certainly true in the COOP world. Providing guidance is important, but providing ACTIONABLE guidance is far more important. This is important because cybersecurity is quite daunting to the uninitiated. Showing someone how to do it (rather than telling them what to do) is what the NRECA CRN program focuses on. They do not dictate to the COOP network (remember, the NRECA works for the COOP network, and not the other way around). They offer well researched guidance and continual support.

Craig Miller will be returning to my conference in October, 2011 (www.smartgridsecuritysummit.com), and if you get a chance to read the NRECA documents prior to that event, please do so, and make sure you make it to my conference, where you can meet the man himself, and I am sure he will be happy to answer your questions.

Just be prepared for straight answers...he does not mince words.

Videos: NERC CIP Compliance Workshop - Smart Grid Security East 2011

2011-07-12T07:53:00.000-07:00

The North American Electric Reliability Corporation (NERC) enforces electric reliability standards under the authority of the Federal Energy Regulatory Commission (FERC). A large part of these enforcement efforts include Critical Infrastructure Protection (CIP), which is currently a key area of cyber security enforcement for NERC, and the set of guidelines are referred to as the NERC CIP guidelines. Organizations who are subject to enforcement under NERC CIP face fines of up to 1 million dollars per day for failing to comply with set requirements. This workshop will focus on the following:

Understanding NERC CIP Requirements
How to prepare for a NERC CIP Audit
Tips and Findings from organizations that have experienced a NERC CIP Audit
Overview of the direction NERC CIP is heading

Gib Sorebo, AVP CyberSecurity, SAIC
Mike Echols, Critical Infrastructure Protection Manager, Salt River Project
Jim Brenton, Regional Security Coordinator, Electric Reliability Council of Texas (ERCOT)
Steve Applegate, Cyber Security Threat and Vulnerability Program Manager, North American Electric Reliability
Joshua Axelrod, Director of Professional Services, AlertEnterprise
Lior Frenkel, Co-founder and CEO, Waterfall Security Solutions Ltd.

Everyone Wants Their Pound Of Flesh

2011-07-11T15:37:00.000-07:00

I was directed to an article this morning titled "CRS: Smart grid cybersecurity standards potentially subject to conflict of interest", which points to a paper from the Federation of American Scientists (FAS) titled "The Smart Grid and Cybersecurity— Regulatory Policy and Issues". If you scroll down to the section called "Policy Concerns" (beginning on page 13) you will find the following:

"...While reliability standards are mandatory, the ERO process for developing regulations is somewhat unusual in that the regulations are essentially being established by the entities who are being regulated. This can potentially be an issue when cost of compliance is a concern, and acceptable standards may conceivably result from the option with the lowest costs. While FERC ultimately has approval authority over the regulations NERC submits and can remand such regulations it judges as not satisfying requirements, any such revisions are ultimately subject to NERC stakeholder approval..."

We need to first clearly understand that everything is ultimately a conflict of interest in the regulatory world. There are few people who take an altruistic approach to making rules. Our entire US government system is driven by lobbyists who all come to Congress looking for their "pound of flesh", and they are generally very successful at it. It is not different in the world of cybersecurity. Organizations are being tasked with addressing cybersecurity for the smart grid. What we have in terms of participation is a few large utilities who have a vested interest in avoiding regulations that would make their lives more difficult, consultants who stand to gain if rules should lead stakeholders to hire consultants to help address the requirements, and vendors who either want to avoid regulations that would harm their business models, or who want to fight for regulations that would bring them more business. All of these "volunteers" to the effort are there for strategic reasons, and I am not exempt from that.

Will this potentially lead to better cybersecurity? I would say yes.

One of the best ways to get people to implement better security is to get people interested in and talking about and learning more about security. This stimulates heated discussions and lots of geek talk. It is how I learned a lot of what I know. It is also reasonable to conclude that having lots of cybersecurity experts involved in the process will probably lead to some solid technical reviews. I can tell you from direct experience that there are a lot of security vendors involved in the Smart Grid security process, and they all try to convince everyone else that what they do is the way to go, but there are also a lot of people who are willing to (and enjoy) scrutinize every word they say.

Still, a conflict of interest will always exist, and unless Congress or any other regulators want to take some time to understand security, it is perhaps best if they allow the process to continue as it is going, conflicts of interest and all.

Just my opinon.

The Revolutionary War, The Civil War, and Cyber War

2011-06-23T09:48:00.000-07:00

I watched a fascinating series on Netflix Streaming a few weeks ago (I love my Apple TV and Netflix Streaming). The series is called "America: The Story of Us". There were 2 episodes that I found most fascinating. One was about the US Revolutionary War, and the other was about the US Civil War. Both of these episodes caused me to draw parallels to our modern society, and the current state of affairs with respect to cyber security (or perhaps insecurity is more appropriate).

Let me explain.

The US entered the Revolutionary War facing off against the British. If Vegas oddsmakers had been around back then, it is quite likely that the odds that "The Colonists" (that's the US) would have emerged victorious were something like 1 in 1000. The British war machine was honed to a razor's edge, and they kicked butt everywhere they went. They were organized, sharp dressers, and knew how to march like nobody's business.

Back then wars were fought in a somewhat organized matter. Two enemies faced off in a field somewhere, and fighting ensued. You saw your enemy before you got a chance to knock him off, or he knocked you off. The side that had the most troops had a major advantage. One advantage was the intimidation factor. The other was sheer numbers. Everyone kept shooting at each other until one side was badly beaten. The side with more troops generally ended up with more men standing when the battle was over, and emerged victorious.

The Colonists decided to change things up a bit. Rather than wait until the British got to where they were going an setting up a battle front, the Colonists decided to arm themselves with German style hunting rifles (with rifled barrels), and pick off troops sniper style. The Colonists hid behind trees, rocks, or wherever they could take cover and simply waited for the nicely organized march of British soldiers to cross their path. Additionally, The Colonists started targeting the leadership (generals) rather than the lower ranking troops, and began picking them off first.

This OUTRAGED the British. How dare The Colonists fight in such an "uncivilized" manner?

Civility is an interesting concept. It is civility that prevents many of us from doing what we would really like to do when someone really ticks us off. The concept of civility is what the "upper crust" of society counts on to keep things in order. Once the concept of civility is abandoned, all bets are off, and power tends to shift very quickly. Generally, those at the top try to remain civil, and those who have abandoned civility end up kicking some serious butt.

What is perhaps most interesting is that those who choose to abandon civility often feel that they are not the ones who have become uncivil. I am sure that The Colonists did not feel they were being uncivil. They were fighting an enemy that was oppressing them, as they saw it. As far as The Colonists were concerned, the oppression was an uncivil act.

So this brings me to the hacktivism we are witnessing lately. Anonymous and LulzSec have decided that the time has come to take down corporatism and oppressive regimes. What concerns me the most about this is the fact that we are currently living in a world where economic conditions have led to millions of unemployed people with intimate knowledge of the internal workings of corporate and government organizations being out on the street with an axe to grind. Loyalty has become a foreign concept to many of these people, and "civility" is in the process of being re-defined. When Joe Plumber decides that the actions of Anonymous (or Julian Assange, or LulzSec) are better aligned with his interests, things quickly get ugly.

Are we there yet? I don't believe so. However, I am very concerned about the current state of "the system", and that brings me to the US Civil War.

Abe Lincoln was a clever man. He knew how to harness resources and technology. The North had a vast network of railroads. It was much bigger than what the South had. Abe Lincoln decided that it would be a good idea to move troops using railcars rather than making them march, or ride horses. Moreover, the owners of the railroads struck a deal with The Union, allowing The Union to take over the railways for the war effort. This, coupled with a massive network of telegraph lines (built along the railways) allowed Lincoln to move resources (people and information) much faster than The Rebels. It was technological warfare at its finest, and the Union became unbeatable as a result.

Circle back to the "cyber war" we are potentially facing. The underground hactivist community is manned by quite a few "geeks" with very good knowledge of the cyber "railways". Communication also seems to be fairly good. I am not sure how this compares to the knowledge and communication on the other side, but I am going to assume that "the good guys" are perhaps a bit less motivated to freely share information.

I could be way off on these comparisons, and my assessment of the situation, but maybe I am not. I think I am at least partially on point here, and that may very well be a cause for great concern moving forward.

AMI Security 101 Workshop At Smart Grid Security East 2011

2011-06-19T12:27:00.000-07:00

The AMI Security 101 Workshop was the most popular workshop at the Smart Grid Security East 2011 Conference held in Knoxville Tennessee on February 28th. We videotaped it for the world to enjoy. We were blessed with some of the best leaders in the AMI security space, and we sincerely hope you enjoy the presentations.

We will hold the AMI Security 201 Workshop at the EnergySec Smart Grid Security Summit West 2011 on October 3rd, 2011. The AMI Security 201 Workshop will take a more in depth and more technical approach in discussing AMI Security.

Travis Goodspeed at Smart Grid Security East 2011

2011-06-18T14:05:00.000-07:00

I am finally getting around to posting videos of the Smart Grid Security East conference presentations. Among the most interesting (and undoubtedly the most popular) were the stylistic exploits of Travis Goodspeed.

Travis is a very interesting person. He is quite young, quite brilliant, and is one of the most polite individuals I have ever met. Please take a look at these videos, and be thankful that he is well intentioned in his research.

A Utility CEO Who Is Talking About Security

2011-06-04T03:58:00.000-07:00

Wow! It truly amazes me to hear that the CEO of a large utility is speaking up about the importance of cyber security in the Smart Grid. Tom Fanning of Southern Company has made the news in the past few days with his declaration that "cyber security issues must be resolved before a so-called smart electricity grid can be fully built" and "Southern Co. hires hackers to identify vulnerabilities" and "the power company gets attacked frequently."

Okay, now let's be fair here. Southern Company is not the only utility that cares about security, and they are not the only utility that hires hackers to identify vulnerabilities. They are, however, the ONLY utility I am aware of where the CEO has decided to come forward and speak out about their security posture publicly.

Why is this so significant? Simply put, this is a declaration at the highest level in an organization that security has not been relegated to a lower position. It is a declaration that "the buck stops here" with respect to security. You got my respect, Mr. Tom Fanning!

So what else has Southern Company done to back this security stance? Well, let me tell you.

Southern Company is the FIRST utility I know of in the entire USA to force a vendor to CERTIFY the security of their product through a third party. Yes, you heard that correctly, they essentially told their AMI vendor (SENSUS) that if they wanted to do business with Southern Company, they had to submit to the Wurldtech Achilles Practices Certification (APC) process. SENSUS went through this process, and achieved Bronze Level Certification. The Wurldtech Achilles Practices Certification is a certification program originally designed to certify vendors for the Gas and Oil industry, and the requirements are outlined in a document known as the WIB, which is a set of security evaluation requirements originally initiated in The Netherlands by Wurldtech and Royal Dutch Shell. Wurltdtech worked with Southern Company to scope a set of certification requirements that could be applied to the electric industry, and SENSUS immediately went to work. The rest is history.

Wurldtech did not stop there, however. Nate Kube of Wurldtech and Ted Angevaare of Shell Oil worked with standards veterans Dennis Holstein and Tom Phinney, who have submitted the WIB requirements to the International Electrotechnical Commission (IEC) as a proposal known as IEC 62443-2-4.

Upon learning about this, I decided to get involved. I have been working closely with NIST, OpenSG, and the DHS ICSJWG for the better part of two years in trying to get some baseline security standards in place for the Smart Grid. When I learned about what Southern Company had done, and that it had led to a proposed international standard, I knew this was significant. I immediately communicated this information to Marianne Swanson of NIST (who is currently chairing the NIST Smart Grid Interoperability Panel Cyber Security Working Group), and she asked me if I would be willing to take the lead in aligning the IEC requirements with the NIST IR 7628 security requirements. I agreed to do so, and was immediately joined by several of the most active members of the NIST CSWG in building this task force. I was also selected as a member of the US Technical Advisory Group for TC65 (the working group for the IEC 62443-2-4 proposed standard). Since then, I have managed to engage several large AMI vendors, silicon producers, security product vendors, and consultants in the process. According to Ward Pyles, Security Analyst at Southern Company, and Nate Kube of Wurldtech, several other utilities in the USA and overseas have now become involved in the process.

What is so significant about this is the fact that it took the EXTRAORDINARY leadership of Southern Company to plant a stake in the ground, and demand that their vendors go the extra step towards assuring that a security baseline had been met. What we all must understand is that the utility is the customer to the Smart Grid product vendor, and the vendor WILL build security into their products if the utility demands it, and forcing a vendor to certify to a third party audit is the only true assurance that a baseline is being met during procurement. It is still critically important for a utility to perform their own security validation (which Southern Company does), but knowing what the baseline is up front saves the vendor, the utility, and the rate payer (you and I) a lot of time and money.

I hope other utilities follow the lead that Southern Company has established.

James Lewis, Peter Schiff, Nassim Nicholas Taleb

2011-05-01T11:26:00.000-07:00

I remember buying my first home (the one I still live in today) back in 2003. I remember the mortgage broker cajoling me to take advantage of the no money down loans that were being handed out like candy, and watching a world where everyone I knew was jumping on the gravy train. I took on a primary and secondary loan. One at a crazy high interest rate, and the other was interest only. Everyone was doing it, and there seemed to be no risk involved. Homes were skyrocketing in value (mine rose a WHOPPING 60% in value before the crash of 2008), and everyone was making money.

It made no fianancial sense whatsoever to me, and it certainly did not make financial sense to Peter Schiff. He was the guy that people laughed at on FOX News interviews for claiming that the market was going to collapse. Nassim Nicholas Taleb also felt it was all going to end in a bad way, and said so in his book "The Black Swan". I would recommend everyone alive today read this book.

So, this morning I read an article on Infosecurity.com titled "Cybersecurity threat rhetoric not supported by evidence, researchers argue", which references a paper written by Jerry Britto and Tate Watkins of George Mason University. It is a well written paper. It is well cited and referenced, and I am sure Britto and Watkins are a couple of smart guys. The paper refers to another paper "Securing Cyberspace for the 44th Presidency: A Report of the CSIS Commission on Cybersecurity for the 44th Presidency", which was a product of a project directed by James A. Lewis, who is the senior fellow and director of the Technology and Public Policy Program at the Center for Strategic and International Studies (CSIS). Britto and Watkins are critical of some of the assertions in the CSIS paper, and I can certainly follow their logic. For example, here is a quote from the Britto and Watkins paper:

"Nevertheless, the Commission report and the cybersecurity bills it inspired prescribe regulation of the Internet. The report asserts plainly: “It is undeniable that an appropriate level of cybersecurity cannot be achieved without regulation, as market forces alone will never provide the level of security necessary to achieve national security objectives.”52 But without any verifiable evidence of a threat, how is one to know what exactly is the “appropriate level of cybersecurity” and whether market forces are providing it? How is one to judge whether the recommendations that make up the bulk of the Commission’s report are necessary or appropriate?"

I cannot argue against the point that without empirical evidence, one cannot determine what is an appropriate level of security, IF an appropriate level of security is predicated by empirical evidence bound by a narrowly defined context.

So what the heck am I trying to say here? Let's go back to the financial collapse of 2008. Despite the assertions made by Schiff and Taleb that the financial situation was headed for disaster (and they were not the only ones predicting this), the "geniuses" in the high towers and in the hallowed halls of our government sponsored institutions decided that there simply was not enough evidence to get them to change their ways. This was despite the fact that we had lived through a Great Depression, and despite the fact that financial analysts interviewed after the collapse claimed that they had indeed told those in charge the that things were not going to end well (and in some cases were terminated for saying so). This was a new way of doing business, as some claimed, and the old rules did not apply. Everyone is making money, so shut up and don't rock the boat.

So the market did collapse after all. We now have LOTS of empirical evidence that our privately run financial system gave us the shaft, and yet some of the very instruments that led to the financial demise of millions of people worldwide remain unregulated.

So this leads me to the conclusion of the Britto and Watkins paper:

"Cybersecurity is an important policy issue, but the alarmist rhetoric coming out of Washington that focuses on worst-case scenarios is unhelpful and dangerous. Aspects of current cyber policy discourse parallel the run-up to the Iraq War and pose the same dangers. Pre-war threat inflation and conflation of threats led us into war on shaky evidence. By focusing on doomsday scenarios and conflating cyber threats, government officials threaten to legislate, regulate, or spend in the name of cybersecurity based largely on fear, misplaced rhetoric, conflated threats, and credulous reporting. The public should have access to classified evidence of cyber threats, and further examination of the risks posed by those threats, before sound policies can be proposed, let alone enacted.
Furthermore, we cannot ignore parallels between the military-industrial complex and the burgeoning cybersecurity industry. As President Eisenhower noted, we must have checks and balances on the close relationships between parties in government, defense, and industry. Relationships between these parties and their potential conflicts of interest must be considered when weighing cybersecurity policy recommendations and proposals.
Before enacting policy in response to cyber threats, policymakers should consider a few things. First, they should end the cyber rhetoric. The alarmist rhetoric currently dominating the policy discourse is unhelpful and potentially dangerous. Next, they should declassify evidence relating to cyber threats. Overclassification is a widely acknowledged problem, and declassification would allow the public to verify before trusting blindly. They must also disentangle the disparate cyber threats so that they can determine who is best suited to address which threats. In cases of cyber crime and cyber espionage, for instance, private network owners may be best suited and may have the best incentive to protect their own valuable data, information, and reputations. After disentangling threats, policymakers can then assess whether a market failure or systemic problem exists when it comes to addressing each threat. Finally, they can estimate the costs and benefits of regulation and its alternatives and determine the most effective and efficient way to address disparate cyber threats.
No one wants a “cyber Katrina” or a “digital Pearl Harbor.” But honestly assessing cyber threats and appropriate responses does not mean that we have to learn to stop worrying and love the cyber bomb."

While I have to agree that being alarmist tends to turn some people off, I cannot help but think of the tongue-in-cheek saying "Just because I'm paranoid doesn't mean they're not watching me." I wholeheartedly agree that our government is not good at spending our money in a reasonable manner, and I feel they do not spend our money wisely even when given correct and verifiable information. In fact, I currently have very little evidence to support that our government does anything based on a well thought out assessment of facts. While Mason and Britto point out a plausible and seemingly correct way to go about this, perhaps they should consider that Mr. Lewis does indeed live in a world where he faces policy makers regularly, and there seems to be no hurry to address the issues at hand. Furthermore, in January of 2011 (this year) CSIS published a followup to the previous paper titled "Cybersecurity Two Years Later" where some specific significant attacks are cited:

"2010 should have been the year of cybersecurity. It began with a major penetration of Google and other Fortune 500 companies, saw the Department of Defense describe how its classified networks had been compromised, watched the Stuxnet worm cut through industrial control systems, and ended with annoying denial of service attacks over Wikileaks. These public incidents were accompanied by many other exploits against government agencies, companies, and consumers. They show how the United States is reliant on, but cannot secure, the networks of digital devices that make up cyberspace. As a nation, we must do more to reduce risk, and we must do it soon."

The report also points to a document titled "Significant Cyber Incidents Since 2006", which (as of the date this blog was posted) list 70 significant incidents (not only in the US).

So how do we solve this problem? Making claims that "the sky is falling" tends to lead to public ridicule (Schiff and Taleb know that well). Theoretical attacks tend to become marginalized ignored due to a critical flaw in risk based formulas that tends to zero out risk for attacks that have never occurred (which is what I am presenting on at the ICSJWG conference on May 3rd, 2011 in Dallas). Finally, empirical evidence only seems to lead to some policy changes as long as it is accompanied by an alarmist outcry (e.g. Congress made some minor changes to our financial system while the world was screaming about the collapse of our economy, but seemed to wind down when the crying stopped, and the risks still exist despite plenty of empirical evidence).

The reality is this: We are not going to do much about cybersecurity until we all feel it in a major way, and then maybe we will get better at dealing with the issues, but maybe not.

One thing is certain, however, Mr. Lewis and the CSIS team will suddenly seemed a lot smarter.

By the way, I got into a less risky loan a year after buying my house. The alarmist rhetoric hit home with me.

Trivial Key Extraction From Electromagnetic Emissions

2011-04-27T11:04:00.000-07:00

I am at day 2 of a 2 day workshop at Cryptography Research in San Francisco, California. The focus of this workshop is on side channel attacks using both Power Analysis (SPA and DPA) and EM analysis to extract secrets (such as keys) from various systems and devices. By far, the most interesting demonstration I have seen is one where both RSA and ECC keys are extracted from a mobile device using a hobbyist antenna and some basic equipment (and software tools developed by Cryptography Research). The total cost for the equipment is less than $2000 (even less if you scrounge around on Ebay, to be sure).

In this attack, a mobile device performing a cryptographic operation is held 10 feet away from an antenna, and with a few seconds of signal sampling, they are able to extract a key by analyzing peaks on a spectrograph. One of the questions I asked is if a more powerful antenna could potentially read the EM from a longer distance, and was told that by simply focusing a parabolic dish at a target (similar to what has been done for reading long distance WiFi signals), the traces can be gathered from very long distances.

I found this quite fascinating, since I am not sure what (if any) protection currently exists for products being used in Smart Grid deployments. There are indeed ways to protect devices against both EM and SPA/DPA attacks, but I am currently unaware of what protections exist. Moreover, as we learned in the workshop, typically most devices "leak" this information in more ways than one, and what they typically discover in many implementations they test, known simple EM,SPA,DPA attack vectors are not considered during engineering (not always, but often enough to furrow ones brow).

I am not sure how serious an issue this may be, but it does raise some concern, when you consider that an attacker does not necessarily have to set foot on someone's property to gather enough information to extract a key from something like a meter (or a cellphone, or anything else where secrets may be stored). One of the basic tenants of protecting against attacks is to prevent scalable attacks. In other words, design the system so that if I get one secret I can only do one bad thing, and make it hard enough to get multiple secrets that an attacker simply gets exhausted with the "workload", and moves on to something else. If an attacker has to get his hands on each and every device to perform an attack, one can see how this becomes non-trivial. However, if an attacker can focus an antenna setup at (for example) a bank of meters on a wall for an apartment complex, now you may have something to write home about.

For those of you not familiar with Cryptography Research Inc., you can find out more about them at www.cryptography.com . Paul Kocher, who is one of the founders, is co-creator of SSL 3.0. This is a well established, and well respected research organization, with an impressive pedigree.

I have a video of Gilbert Goodwill from Cryptography Research (one of the workshop instructors) demoing an EM attack at RSA 2011 on my YouTube channel:

Forgive the background noise (the RSA Expo Hall is quite noisy). They will be demoing this at my EnergySec Smart Grid Security Summit in October, 2011.

This is indeed something to think about.

Executive Level Apathy For Security...Maybe Not So Much

2011-04-07T10:24:00.000-07:00

I read an article in Information Week this morning titled "76% Of Energy Utilities Breached In Past Year", and while I found most of it rather sensationalistic and perhaps a bit boorish (I mean, c'mon, 76% of all businesses AND government agencies have probably been breached in the past year...at least according to the boundaries defined in this article), one part stood out:

"71% of people surveyed said that "the management team in their organization does not understand or appreciate the value of IT security."...Executive-level apathy or misunderstanding over information security is surprising..."

I have been thinking about this notion of "Executive Level Apathy" for a while now, and I have come to the conclusion that executives are doing exactly what anyone would do if put in their positions and under the circumstances. Hear me out for a moment before throwing daggers.

One of the primary (if not THE primary) responsibilities of a CEO of an investor owned utility is to make sure the investors get what they paid for. This can include anyone that invests in mutual funds and ETFs that include utility stocks as part of their portfolios. When those stock prices (and dividends) go up, everyone is happy. When they drop, everyone gets grouchy. Anyone who invests knows the scene quite well.

There are a lot of factors that cause stock values to fluctuate, but suffice it to say that the more money a company spends on things that do not generate a return on investment, the lower the bottom line becomes. In some cases utilities have to deal with MASSIVE expenditures fixing problems that, while they are fully responsible for them, generate no ROI (e.g. explosions, environmental messes). I am talking about very real issues that are vivid in nature, and absolutely have to make it to the top of the list of "things we gotta take care of like yesterday".

So lets circle back around to security. In the article the author points out that the average cost of fixing one of these breaches at an energy utility was $156,000. If we take a look at my local utility (PG&E) revenues for 2010, a quick search on the Internet reveals that they took in $13.8 billion dollars. That comes out to 0.00113%. So, let's assume that PG&E maybe gets hit a bit more than the "average" reported by writer of this article. Lets assume they get hit 100 times more, for a grand total of $15,600,000. That brings us to a "whopping" 0.113%.

Okay, I am not saying they should not be concerned with security, but when one considers the costs of doing business and managing budgets on a great scale, it is easy to see that a $156,000 (or even $15,600,000) problem can work it's way down the list of "things I gotta deal with right away as the CEO".

Don't misunderstand me, I dislike filthy rich CEOs like any other red-blooded American worrying about paying his mortgage in our tough economy (although I am perhaps more jealous than anything else), but my very inquisitive nature forces me to peel back the layers of the onion enough to at least try to get some perspective on this, and the truth is that a 0.00113% to 0.113% problem is not something to get worked up about. We, as a society, have created a specific role for such top level executives which FORCES them to focus on what really matters, and today that is measured in the short term (1 budget quarter at a time).

It is, however, VITALLY important to pay attention to security (and CEOs know this) because there is a potential for a MASSIVE loss in revenues given the right circumstances, but how is anyone to know what the right amount of money is to spend on managing the issue? If a company spends $1 million, $5 million, or $20 million to protect themselves against such breaches (and potentially larger ones), how do we determine if it is enough? As stockholders we end up paying for it, and that does not usually make us happy. As customers we also pay for it, because utilities are guaranteed recovery (from us) for such expenditures.

So how much are we all willing to pay for security?

We, as a society, generally get what we demand...eventually. While it may sometimes seem like executive apathy abounds, the truth is that WE are just as apathetic (hopefully not me, but as a society in general) about security. Consumers are simply not demanding security...and what would they demand anyway? With SUBSTANTIALLY less than 1% loss to cybersecurity breaches today in the utility space, what kind of empirical information is likely to motivate a consumer?

Utilities can always do more, and executives can always be more concerned, but exactly how much more should they do, and how much more should they be concerned? Frankly, until something really bad happens, I am not sure anyone will be able to answer that question.

Sorry if this seems like a downer to the security minded (and believe me, I am one of them), but I can't really demonize the guys in the high towers on this one. I would like to see them speak publicly about cybersecurity issues, and that is something they could do as a form of outreach to the community, but in terms of being more pro-active, I certainly don't see how I would (or could) do anything different.

Just my opinion. Take it for what it's worth.

OpenSG And How Utilities Are Missing Out In Smart Grid Security Opportunities

2011-04-03T09:51:00.000-07:00

Industry groups are often a good thing for the industries they exist for. Industry groups allow member organizations to converge and discuss ideas, and hopefully come up with a unified way to improve stakeholder positions.

In order for an industry group to create value for the industry it serves, it is important to have a lot of participation by stakeholders. It is also important to make sure that stakeholder participation is not skewed to serve one (or more) stakeholder categories over others.

This brings me to OpenSG. For those not familiar with OpenSG (Open Smart Grid), it is a subcommittee created under the UCAIug (UCA International Users Group) to facilitate the creation and adoption of standards, methods, and guidelines for Smart Grid deployment. This is a utility industry group, and the intended outcome of the work being done under OpenSG is to come up with a consensus based set of standards and guidelines for all utilities deploying Smart Grid.

Under the OpenSG umbrella, there exists an SG Security Working group, and I have been involved with this group for approximately 1 1/2 years. In this time (and before I threw my hat in), there has been a lot of progress, and one can view some of the output by going to http://www.smartgridipedia.org/. One of the more notable pieces of work output is AMI Security Profile 2.0. This document was used by NIST (among other documents) to assist in the creation of the vaunted NISTIR 7628 document, and is currently being utilized by some larger utilities (such as PG&E) as a guideline for AMI security deployment.

I could go on and on about the task forces under the SG Security Working Group (such as the newly formed Embedded Security Task Force), but this is a blog posting, and keeping it short is important.

Okay, so let me tell you what concerns me. The OpenSG SG Security Working Group is quite vendor heavy, and utility light. Sure, we have great participation by some of the "big boys" (e.g. PG&E, SCE, Southern Company, Virginia Dominion, FPL), but we have nowhere near an adequate quorum of utilities participating in OpenSG. When you consider there are over 2300 investor owned utilities in the USA, as an industry group I would suggest that at least 50% (or 1150) of those utilities should be ACTIVELY involved.

Why is this important? Well, if it is not already obvious, because utilities probably have a better idea of how their industry works than the vendors do. Every stakeholder that makes up any industry group is there for one reason, and that is to discover and create opportunities for commerce, or protect their current business model and bottom line (and hopefully make it fatter). This is normal human nature in the business world, so spare the daggers.

In a vendor heavy environment, the industry group is obviously skewed in favor of the stakeholders who do not have the most at stake. Security decisions made by utilities are decisions that are going to have far reaching impact for a long time. While some may find comfort in the fact that large utilities are participating with vendors in making decisions for the industry (and regardless of participation in OpenSG the decisions made by the big boys will affect every utility), it is not wise to assume that what is good for the big utilities is good for the little guys as well. There is a SIGNIFICANT difference between the security management and deployment capabilities of a utility with 5000 employees and one with 20 employees (or even less). If vendors and large utilities decide on something like (as an example) a utility managed PKI system for key management, and products are designed to work with such a system, then a small utility may be a bit hampered in their ability to comfortably deploy.

I am not saying that the SG Security Working Group is not cognizant of this. We certainly are and discuss it regularly. However, without DIRECT input from the ACTUAL stakeholder organizations, we are forced to make educated guesses, and rely on extrapolation and conjecture in our decision making process.

Utilities who are preparing to deploy Smart Grid technologies have lots of questions and more than a few concerns about security. There are a lot of very brilliant people working in OpenSG that are happy to freely share a wealth of information about Smart Grid security, and we can all learn from more participation. We are still at an embryonic state with respect to Smart Grid security, so a little participation is sure to gain anyone who participates a lot of expertise over a short period of time.

Come on in. The water's fine.

The Limitations Of Voluntary Efforts

2011-03-27T06:57:00.000-07:00

James Lewis of the Center for Strategic and International Studies (CSIS) is one of my favorite figures in the world of cybersecurity. I would venture that both Mr. Lewis and the very wise Michael Assante truly get at the heart of the issues we face in the world of cybersecurity like no others. They do not cover everything, but the issues they do discuss are profound in nature.

Both Mr. Lewis and Mr. Assante like to use analogies as part of their discussions, and I like that as well. Michael Assante co-wrote a great piece on Roman Aqueducts with current NERC CSO Mark Weatherford (Assante is the former CSO), where they use analogies to compare Roman Aqueducts to the Smart Grid. I do not want to go into detail about the article, so make sure you read it. I can assure you it is quite well done.

James Lewis likes to keep things a bit closer to home, and uses a somewhat Socratic method to stimulate critical thinking. This is extremely evident in a recent testimony to the House Committee on Homeland Security. In his testimony, Mr. Lewis makes a number of absolutely wonderful points, but the part that resounded with me was the following:

"There is no other area of national security were we rely on voluntary action reinforced by incentives. A policy of voluntary efforts for better cybersecurity reinforced by incentives is not a serious effort to protect national security against real damage and a growing threat. These proposals are best seen as intended to block reform rather than to promote cybersecurity."

In order to understand this, we need a little background. Nearly all progress being made on cybersecurity in the USA is due to voluntary efforts. I personally volunteer my time to participate in NIST, OpenSG, and DHS working groups to address cybersecurity for the Smart Grid, as well as additional work I do for California for health care security. This is not an altruistic endeavor. My incentive to do so is either because I am paid to do so by a client, or because I want to develop a skill and become a subject matter expert for the purpose of exploring opportunities for commerce. This works out well in my case, because my efforts have helped me pay my bills. I am not getting rich doing this, but I am also not eating ramen noodles for dinner every night.

As it turns out, I am not alone. There are literally hundreds (if not thousands) of people volunteering their efforts to the cause for the same exact reasons. Of those, about 5% to 10% regularly contribute something besides attending a meeting or a conference call. There is some progress being made, but it is slow, and most of it has no teeth whatsoever. Why is that? Because nobody is in charge, and since there is no consideration for efforts, there is nothing anyone in charge could do anyway.

Don't misinterpret what I am saying here. I do not believe voluntary efforts are a bad thing. Heck! Our US Voluntary Militia of the 1700's did a fine job whooping some butt back in the day. Yet one has to understand that once the threat turned into a battle, it was no longer a good idea to sit around and hope that the local blacksmith was going to show up for an attack against the Redcoats.

What our Congress seems to not "get" is that we are currently fighting a daily battle against the bad guys, and the bad guys are winning. We are not fighting a war in the "classic" sense, but we are definitely getting our butts kicked more often than we would like to admit it. It leads one to question just what does Congress and our President use as a determining factor for pouring money into a national security effort. In fact, I wonder if national security really has anything to do with it at all. When we light up skies overseas with bombs, and take down villages on the ground, are we doing this to protect anything, or are we simply trying to show the world that we still have plenty of firepower to go around? Are we protecting our interests, or are we trying to get the President re-elected, or is a member of Congress trying to get re-elected, or more campaign contributions, or whatever?

Of course it could simply be that Congress and Mr. President simply do not feel the battle, because most of what goes on in the world of cyber attacks is not broadcast on the nightly news, and even if it was it is not likely to have the impact of bombs dropping and villagers screaming with blood running from their temples. It could also be tradition. Our Congress is all about tradition in many ways, and one of them is the long tradition of spending a lot of money overseas fighting battles, and NOT fighting domestic cybercrime. Sure, there have been some token payments, but nothing approaching the billion dollars per day we are now spending fighting wars overseas.

At my last Smart Grid Security Conference I had two people from state public utility commissions as speakers. Bill Hunteman from the US Department of Energy (a great and intelligent person, by the way) told me he was pleased to see that the PUC's were willing to send them to events such as mine where they can interact with others and discuss cybersecurity issues. When I told Mr. Hunteman that the only reason they came was because I paid for their travel expenses, he was surprised to hear that state PUC's had no budget for such events. This is particularly alarming when you consider that state PUC's are tasked with making decisions about cybersecurity for all of the distribution and "user" portion of the Smart Grid (i.e. Smart Meters, Advanced Metering Infrastructure, Home Area Networking), and nobody at the Federal level has any authority over this.

Think about this for a moment. We have 50 states with public utility commissions who have little or no budget allocated for cybersecurity expertise being tasked to come up with rules for Smart Grid cybersecurity. That is perhaps as effective as gong to your state Department of Motor vehicles and asking them to come up with some rules for earthquake proofing bridges...in their spare time, with no budget for hiring bridge experts.

So hopefully, once we are done starting new wars with countries, and have wrapped up some of the other wars we are fighting, our government will consider funding some of the efforts to deal with our coming cyberwar, because once the daily battles turn into a full blown war, we are not going to be ready for it.

Not at this rate.

RSA SecurID and The Smart Grid

2011-03-23T07:40:00.000-07:00

The compromise of RSA's SecurID system is one of several security-related hot topics this week. I am still not sure how significant the compromise is (some say it is not overly significant, others claim it is a massive problem), but one thing is quite clear to me and many others I know in the security world - It is not a major surprise that their security has been compromised.

Security gets compromised. That is what happens with security. Some organizations have great track records. As I understand it RSA SecurID has a 20 year track record. Cryptography Research claims an 8 year perfect record for their CryptoFirewall product (used to protect, among other technologies, pay tv). DES and SHA1 had their days in the sun, but all good things come to an end, and the newer and (hopefully) better technologies take their place. Heck! There was a time when the Mac loving world believed that Macs were immune to security compromises. Think again!

RSA will fix whatever is broken. They are a good company with a long history of knowing what they are doing. Sure, they make mistakes along the way, but they are a good provider of security products. The fact is that a compromise helps build better security. As users of security technologies, we should EXPECT compromise at some point, and be prepared for it. I am a careful driver, who does not text as I drive, and stays within the speed limit (more or less), and I wear my seatbelt and back up out of parking spots nice and slow. Nonetheless, I am fully aware that operating my motor vehicle puts me at high risk for an automobile accident. I can avoid automobile accidents entirely only by never getting anywhere near an automotive vehicle. In the world of technology it is the same story. If I play in the cyber world I am going to face cybersecurity incidents.

That brings me to the Smart Grid, and perhaps more specifically Smart Meters (or AMI in general). Utility Commissions throughout the US (and perhaps the world) are hoping that rate cases cover AMI products that are going to be "good to go" for somewhere around 15 or 20 years. This strikes me (and others I have spoken to) as somewhat of a pipe dream. RSA SecurID is built to function as a sort of "Fort Knox" of security systems, and it lasted about 20 years. AMI products are simply not designed that way today, and it may be a while before they are. Simply put, utilities do not require that level of security and do not want to pay for it...and neither do consumers today (who will ultimately foot the bill due to recovery rules). It is more likely that AMI systems rolling out TODAY (and not 3 years ago) may remain "secure" for a maximum of 10 years.

Think about this for a minute. They are sitting on the outside of homes. There are no set requirements for security. The protocols, designs, and general security knowledge of vendors vary. This is new territory and we are in the earliest stages of deployment. We have to expect that we are not likely to get this right on the first few tries. We have to also expect that we are going to learn (and have learned) some valuable security lessons as we proceed.

This creates a bit of an issue for utility commissions and consumers, because we have to pay for replacing devices that fail to remain secure over time. As consumers, we are used to having to upgrade technology as time progresses. Who keeps a computer or cell phone for 10 years today? Okay, there are some who do, but not many. Technology simply moves too fast and a few years down the line anyone who uses technology to get things done simply accepts that in order to continue reaping the benefits of technology, upgrades are a given.

Ahhh...and that is the key! The consumer needs to experience the benefits.

I have a Smart Meter on my house and since I switched to off-peak pricing I have seen a drop in my power bill of approximately 30%. I simply do not use power very much in the middle of the day. Sure, it goes up a bit in the summer when I use my AC more, but it plummets in the winter. This is significant for me because my winter bills used to eclipse my summer bills. As a consumer, I am happy to have a Smart Meter on my house because these savings would not be possible without a Smart Meter (or so I am told). As a consumer, if I see a benefit, I will pay to play.

As a consumer I am also aware that I have not experienced any security-related issues...yet. It could be a long time before I do, but I am aware that the high tech nature of the Smart Grid opens up a gaping hole from a security perspective. That is the nature of technology. As a user of pocket computers (that is what we should be calling mobile phones and devices), I am aware of the major security holes that tag along with the technology, and also pray that bad things will not happen on the mobile device front. Nonetheless, I know that it is indeed quite likely that somewhere within the next decade an enterprising attacker will figure out a way to exploit our favorite new technologies, and we are not likely to go back to doing things the old fashioned way as a result of the compromise.

Let us accept the fact that this is the world we live in, continue working to build better security, avoid freaking out when things go wrong, and reap the benefits that inevitably come with new technologies.

We live and we learn.

Congress, Mr. President, You Are About To Get Served

2011-03-20T07:17:00.000-07:00

I was enjoying lunch yesterday afternoon with my family when my iPhone sent me a push notification from CNN that the US had mounted an attack on Libya. My brow immediately became furrowed at this news, since we are all keenly aware of the billions we are spending on wars today, and Libya now represents more opportunities to spend untold billions fighting yet another war (spare me the correction for using the term "war").

At my recent Smart Grid Security East Conference, I had a great panel with representatives from the US Department of Energy (DOE), The Federal Energy Regulatory Commission (FERC), and The North American Electric Reliability Corporation (NERC). I called this the "Super Panel", since we had all the major Federal decision making organizations on stage at once discussing Smart Grid security. I asked a simple question "From a distribution perspective, meaning the part that deals with Smart Meters and the consumer, who is in charge of security?" The answer was the same across the board. The Federal government is not in charge. It is up to the States, meaning the Public Utility Commissions.

I then brought up a point that I continuously keep bringing up whenever anyone will listen. State PUCs do not have any resources to address security. The California Public Utility Commission, which is one of the largest in the country, has no staff dedicated to Smart Grid security, and very little in the way of knowledge of security. I know this because I have spent quite a bit of time working with the CPUC (voluntarily) in the past year. Some of them are eager to learn, to be sure, but they are a long way from being able to make decisions that will adequately address security issues.

I brought this up to Bill Hunteman, who is the Senior Advisor for Cyber Security for the US Department of Energy (and a very wise man), and he told me he is well aware of this issue, and the DOE is well aware of this issue, but (for now) there is nothing they can do about it, because they simply do not have the funding to address this issue. I asked about where the funding needs to come from and he (and the rest of the panel) told me that it has to come from Congress.

This seems simple enough. Congress likes to spend money to fight wars, and we are being attacked on a constant and consistent basis on the cyber front, so what's the problem?

I am not sure what is going on here. Mr. Hunteman also mentioned (on another panel) that the Federal government is considering pulling some of the DOE funds they had allocated for addressing the Smart Grid, which was in response to a question I had asked regarding the likelihood that Congress would release additional funding to address cyber security for the Smart Grid. This made me feel more than a bit concerned, since we are currently addressing the security of our critical infrastructure through the voluntary efforts of a bunch of people who are essentially only doing so in hopes of future opportunities for commerce. Make NO MISTAKE ABOUT IT!

Congress and our President have authorized somewhere close to a billion dollars a day for our ongoing war efforts. I have heard estimates of between $700 million and a billion a day. If we took 1 billion dollars out of the budget for fighting wars, that would mean that we could give each state $20 million in financial resources to address cyber security. I know that the California Public Utility Commission could certainly use the money.

This past week the big security news was about a compromise of RSA's SecureID system. This is used extensively in both enterprise and government, and has caused a great deal of concern in the security industry. We are still not sure how severe the damage was, but we nobody I know in the security world is particularly surprised that the exploit occurred. We EXPECT exploits of this nature to happen, and we KNOW they are only going to get bigger and more sophisticated.

It is really only a matter of time before our critical infrastructure is hit with the mother of all attacks, and when that happens I am not sure Congress and Mr. President are going to be able to offer anything to us.

Again, just my opinion.

The Smart Grid Brass Ring For Consumers

2011-03-15T10:03:00.000-07:00

Since I live and breathe Smart Grid these days (even though I am focused solely on Smart Grid security), I am constantly dumbfounded at the lack of awareness of the Smart Grid by most people I speak to "on the street". I go to parties, social engagements, or wherever I may travel, and am asked what I do for a living, and when I tell whoever is inquiring that I work in Smart Grid Cyber Security, the inevitable question is nearly always "What is a smart grid?"

It is particularly interesting when one of my neighbors asks me and I take him (or her) to the meter on the side of their house and point to the recently installed Smart Meter and explain what it is and how it works. Sometimes I hear comments like "Oh yeah, I heard some people have had their bills go up." or "I hear these cause cancer."

So we are essentially with either no perception of the Smart Grid, or a collection of sound bites that fail to tell the story.

...but what is the story?

What I mean by this is "What is the story from a consumer perspective?" Why do I want a Smart Meter, or a Smart Grid anyway?

Years ago, before I had a computer, I heard of this "Internet". It was all over the place, this Internet discussion. Internet this, Internet that, email, surfing, World Wide Web, @this, .com that, blah, blah, blah. It was a big joke to those of us who did not partake in the festivities...at least for a little while.

Then the internet became cool. No, I don't mean cool in a hipster with gel in his hair sense. I mean that the Internet became an environment where everyone could get something out of it beyond the interesting technology that makes it all work. It became a fixture. It started to take off.

The Internet did not really take off until it became something that delivered value to those who partook in it. Once we realized that we could save money shopping online, save stamps with online banking, save fuel with online shopping, download music, movies, news, entertainment, etc. we all took to it like ants to picnics. We are hooked.

Then came the whole smart phone revolution. We discovered that we could not only carry this cool Internet with us, but we could also download interesting "apps" that we could do interesting things with, and suddenly discovered that these apps were something we could no longer live without.

Okay...maybe that is a stretch, but those of us who live happily in the app using world will probably agree that apps are a great thing, and they endear us to our devices.

So that brings me around to the Smart Grid, and consumer adoption. This morning I saw an article about a new iPhone application called JouleBug, which makes a game out of saving energy. The application is rather low tech, but it is sort of interesting with good graphics. It illustrates a conversation I have had with some people in the Smart Grid space, where I insist that what it will take for consumers to "buy into" the Smart Grid is a combination of some sort of savings on power consumption AND an interesting way to interact with the ecosystem. Sound bites on news programs and lower bills alone will not win us over. An cool iPhone application (as an example) with cool graphics, push notifications, easy to use, and a general fun feeling may indeed be a winner. If it gets us to change our energy usage in a positive way...even better. After all, changing consumer behavior is really what the Smart Grid is about.

We, as humans, are not far removed from the creatures that flock towards bright and shiny things. Perhaps we are a bit shallow in that regard, but if it gets us to save energy, then so be it.

Just my opinion.

Knowing What To Ask For

2011-03-10T11:02:00.000-08:00

One of my favorite conference speakers has to be Robert Former of Itron. Robert is the in-house penetration tester at Itron. He is paid to break things, and it is a job he thoroughly enjoys. Like many vulnerability testers, he is essentially a no nonsense guy that shoots from the hip. If you ask Robert a question about security, you are likely to get a very practical (and honest) answer.

One of the comments Robert has made (on more than one occasion) is that AMI vendors deliver what their customers ask for. Early deployments of Smart Meters lacked many of the security features of today's Smart Meters for two reasons. One reason was that many of the security concerns we face today simply did not exist in our conscience back then. Sure, some will argue that we should have known better, and we should have learned our lesson from blah blah blah, but the reality of how we deal with security is only partially pro-active. Think about this for a moment. You are not going to walk about town in a bulletproof vest until you realize that there are bullets flying. You may take a few precautions if you hear news of some people getting shot, but barricading yourself in body armor is not likely unless you are living in a war zone.

So with AMI, despite all the glorious hype we sometimes see, rest assured we are nowhere near a war zone. Yes, there have been a few shots fired over the bow, but I have yet to hear of any casualties (or, for that matter, any injuries whatsoever).

The other reason why early meters lacked security found in today's meters is because utilities simply did not demand it. Utilities wanted (and still want) inexpensive, reliable, and easy to manage meters. Adding security to a meter can directly impact all three of these criteria. Early deployments were focused on just getting everything to work, and many still are.

Yet we live and we learn...or so we hope. The fact is that utilities are now keenly aware of the need for security, and they are now beginning to demand it from vendors.

However, this is not necessarily working out as well as it should.

I have had conversations with several vendors who have told me that some potential customers have essentially copied and pasted the entire NIST IR 7628 final report (which is 3 volumes) and said something akin to "do this" to their vendors. As someone who is currently working on developing testing and certification guidelines for Smart Grid security as part of the NIST Testing and Certification CSWG working sub-group, I can assure you that this is not a good idea. This is like handing a copy of "Larousse Gastronomique" to a caterer and saying "cook this".

Knowing what to ask for is crucially important for a utility. Without some type of guidance, utilities are not going to be very effective at making demands. In fact, without knowledge of what they ask for, utilities are likely to accept anything they are given as a response to their demands. I mean, how are they going to verify anything anyway?

The work being done in OpenSG is seeking to rectify this. There are a number of prominent utilities working in OpenSG, but the majority of utilities in the USA are not active members of OpenSG. There is a wealth of information available to anyone who wants it, and anyone (utility or not) can participate in the work. By educating themselves about security, utilities can create RFP's in an informed manner, and they can also take advantage of the tools and people available to help them verify that they are getting what they demand. Getting involved is easy. Send an email to darren@utilisec.org (who is the current chair), or Bobby@enernex.com (the current co-chair) and you will be on your way.

The answers are out there.

Travis Goodspeed Outside The Box

2011-03-06T07:23:00.000-08:00

You are not likely to forget your first encounter with the very neighborly Travis Goodspeed. He is a rather lanky young man (age 24 as of this posting) hailing from the Knoxville, TN area, who speaks with the slightest Southern drawl, and sports a rather impressive crop of dreadlocks. Travis is an extraordinarily polite, easygoing, and friendly person who is very sociable and is quite fond of West Coast Style IPA beer, which he longs for when he visits Germany (where malty lager is the brew of choice).

Travis likes to challenge security assertions. He likes to shave, etch, probe, and otherwise infiltrate computer chips in his quest to discover what secrets lie within. He insists he does this for fun, and after sitting with him for a bit and listening to his exploits, I am convinced he must be having the time of his life. I am also glad he is not one of "the bad guys".

I had the pleasure of having Travis join us at my Smart Grid Security East conference, as a panelist and as a fixture in my expo area, where he set up shop with some of his tools and toys (some homegrown, some off the shelf) and proceeded to show the crowd how he managed to hack the "security" of a Microsoft Wireless Keyboard. Mind you, this was not an old keyboard, but one he had recently purchased. Apparently Microsoft decided to use the MAC address as the key for this keyboard communication scheme. Travis showed how, using a rather interesting badge he had created, he was able to monitor every keystroke typed into the keyboard, and display it on a monitor.

I find this particularly intriguing, because for the last year or so I have been working on security guidelines for the State of California Office of Health Information Integrity as part of the Security Steering Committee for the Privacy and Security Advisory Board. We have been creating guidelines addressing security for health information exchanges in order to help ensure that health care organizations in California align themselves with requirement under the HIPAA HITECH privacy and security regulations. While we have done all we can to deal with issues such as how people should interface with systems, and how data should be handled in the system (mind you, it is not perfect, but we are working hard on the issues), something like a wireless keyboard communication protocol is so far out of scope it may as well be a discussion on the topic of corn pads.

We still live in a world where, from a security perspective, device manufacturers are essentially exempt from any liability for making silly choices. Microsoft has enough money and brain trust to address this issue properly. They could easily implement a design that transcends this level of silliness, but they choose not to do so. Yet a health care organization that decides to replace their keyboards with the cool wireless ones available from their hardware supplier is one Travis Goodspeed Hope Badge away from having everything they type into the electronic health record becoming publicly available information.

People like Travis (and there are a lot of people like him, both good and not so good) think way outside of the box. Organizations that spend millions (and even billions) of dollars trying to secure their systems who fail to understand this should prepare for lots of sleepless nights, and many sour looks as they face their boards of directors.

The Greatest Outcome

2011-03-04T09:23:00.000-08:00

Well, 6 months of hard work, planning, endless phone calls, emails, accolades, assaults, cancellations, headaches, strong cups of coffee, sponsorship groveling, blogging, writing, and finally making it all happen are over.

My Smart Grid Security East conference was a success. With the help of all the wonderful speakers, sponsors, and my own team, we made it happen. We have a bunch of videos to edit and post on the site, and some papers, presentations, and slide shows to upload, but the lion's share of the work is done.

It is quite overwhelming when so many brilliant people tell you how wonderful something you built is. These are all people I respect TREMENDOUSLY, and I cannot help feeling elated by their approval. It is both exhilarating and humbling. I would love to list them all in this blog posting, but you would probably stop reading halfway through the list, and I want to bring up a far more important point...so hang on.

As my conference drew to a close, I reflected on all the moments I felt defined my sense of accomplishment for this event. There was the opportunity to sit down to lunch with Bill Hunteman, Senior Advisor for Cyber Security for the US Department of Energy, who chatted openly about the DOE's roll and challenges. Then there was the opportunity to enjoy breakfast with Matt Carpenter and Michael Assante. I had an opportunity to converse extensively with the young, brilliant, and very "neighborly" Travis Goodspeed, who exposes security flaws in between pints of his favorite IPA's. I enjoyed countless meals and moments with Daniel Thanos of GE Energy, and Bobby Brown of EnerNex, and Erich Gunther (who wears so many hats...well, lets just say he is everywhere).

Then there were all the wonderful AMI security minds. There was Ed Beroset of Elster, and Stephen Chasko of Landis+Gyr, and Ido Dubrawsky and Robert Former of Itron. All brilliant people working hard to build the products that we will rely on to securely manage our energy infrastructure going forward.

I literally could go on for several pages, but suffice it to say I had many "moments" with some great people.

Yet, life has a way of showing you what really matters right at the moment when you think you have it all figured out. As I was leaving the conference hall at the final moments of the event, I was approached by a lovely young lady, who goes by the name of Summer. She had won a free pass to my conference through a contest Andy Bochman of the Smart Grid Security Blog held.

She had contacted me by email after she won, and was thrilled because she is focusing her studies on Smart Grid cyber security, and she was attending school at the nearby Tennessee Tech University. I welcomed her and told her she could bring along someone else from the school as a guest of the conference.

As Summer approached me, her young face broke out into a huge smile, and she profusely thanked me for the event and the opportunity to hear from the brightest minds in the world of Smart Grid security. She then told me that the person she had brought with her had decided to change his thesis topic to Smart Grid security.

It was at that moment that I felt truly humbled. As I get older (and hopefully wiser), and raise my children to be the best that they can be, the things that matter the most to me are maturing. There was a time when I felt it would be great to be remarkably wealthy (okay, I still think that would be great), or achieve great fame (granted, that wouldn't be so bad either), but what matters the most is when you find a way to be the change you want to see. I look at so much of what our youth has to deal with today, and often wonder how they can possibly cope with the mess they have been given. I wonder where they can look for any guidance that will in any way affect them in a positive manner, and at that moment realized that something I had created served to positively influence at least 2 young minds.

Wouldn't we all like to do that?

Be the change you want to see...

Keeping Out Of The Trough

2011-02-11T06:22:00.000-08:00

I am absolutely thrilled and truly blessed beyond all words to be hosting the Smart Grid Security East conference (co-hosted by EnerNex) in Knoxville, TN from February 28th to March 2nd. I have had the great pleasure of working with some of the finest security minds in the Smart Energy industry for the last year, and most of them are speaking at my event. If you are not aware of this by now, you should take a closer look. I think you will be duly impressed.

What struck me as an absolutely fantastic illustration of what the Smart Grid security landscape looks like today was a slide that Ido Dubrawsky of Itron sent me (and gave Bobby Brown of EnerNex credit for):

This one slide really says it all for me, and completely illustrates my goal for creating this conference. The people I have been working with in the NIST, OpenSG, and DHS groups for the last year are indeed (by and large) luminaries in the world of Smart Grid security. Everyone acknowledges that there are issues to deal with, and we are well above "The Trough" and are rapidly moving along the path of true productivity in addressing security. Have we "solved" the security problems of the Smart Grid...no.

Will we ever "solve" the problem?

Well, this is like asking the question "Will we ever solve the problem of obesity?".

There is an easy way to solve the obesity problem. Simply stop eating. Of course the problem with that solution is that it tends to lead to bigger problems. However, we have discovered that putting a little thought into what we eat, and combining it with some other good practices, tends to lead to a pretty darn good quality of life and avoids obesity (I am not a good example of this, but I am working on it). If you find yourself not quite understanding how to tackle this problem, there are plenty of people out there to help you.

The same holds true for Smart Grid security. It is an ongoing issue that comes about as the result of an idea to implement a system which will ultimately serve to address some major energy issues throughout the world, and ultimately improve our quality of life. It is not being ignored, and if want to know more about what is being done to address the issues, the answers are out there.

I hope some of you can make it to my conference.

The Smart Grid Security Misinformation Network

2010-12-15T07:36:00.000-08:00

It seems that a lot of the news we hear about Smart Grid security seems to focus on how we are all potentially doomed due to the lack of attention being given to Smart Grid security. Bad news does seem to get a lot of attention, so I can certainly see how this may be a great way to attract readers. I have a Google Alert set to "Smart Grid Security", and every evening I get an email with the latest headlines. It seems to come in waves, but I get a lot of links to random postings where the author proclaims that not enough attention is being given to Smart Grid security.

I am not sure what "enough" really means in the eyes of many of these authors, but I will say that there are a lot of people paying very close attention to Smart Grid security. I personally belong to 2 of 12 NIST Smart Grid Cyber Security Working Groups (http://collaborate.nist.gov/twiki-sggrid/bin/view/SmartGrid/WorkingGroupInfo) and these groups generally meet for 1 hour per week. Members from every corner of the energy and security industries regularly attend these meetings, and the discussions and associated tasks are certainly focused on securing our Smart Grid. The NIST CSWG is also where the NISTIR 7628 Security Guidelines came from, which was a collaborative effort of over 400 people from the energy, security, legal, regulatory, government, educational, and general technology industries. Many of these same people are still quite active in the efforts of the NIST Smart Grid Interoperability Panel (SGIP) and NIST CSWG. Besides the NIST effort, several standards development organizations have become involved in working towards developing standards for securing the smart grid.

The US Department of Homeland Security has put together a comprehensive Industrial Control Systems Joint Working Group (ICSJWG), which is open to anyone who wants to help (http://www.us-cert.gov/control_systems/icsjwg/index.html). I am also a member of this group, and recently attended a conference (also open to anyone who wishes to attend) in Seattle Washington. The speakers were all excellent (okay, I was one of them), and the presentations are all freely available at http://www.us-cert.gov/control_systems/icsjwg/presentations.html .

Under the UCA International Users Group (UCAIUG) there exists the vaunted and very active Open Smart Grid (OpenSG) users group, with several active Smart Grid security groups operating under their umbrella. Literally hundreds of people (many of the same working with the NIST groups) meet regularly to discuss security, take on tasks, and publish documentation which has been utilized by NIST to help develop their special publications (including NISTIR 7628), and by both utilities and public utility commissions to guide their security efforts and regulatory efforts.

The Federal Energy Regulatory Commission (FERC) has worked with the North American Electric Reliability Corporation (NERC), who have developed critical infrastructure protection requirements (NERC CIPS), which are used by utilities for auditing the security in bulk generation and transmission. The US Department of Energy (DOE) has granted millions of dollars to organizations who are charged with researching and developing security methods to protect our energy infrastructure.

There are several active Smart Grid and Industrial Control Systems active mailing lists, and several LinkedIN groups focused on Smart Grid security discussions and collaboration. Several research organizations (most notably Pike Research) have invested enormous efforts on researching and reporting on the topic of Smart Grid security, and the security product and vendor community has come out in force to address the challenges that are constantly being discovered and discussed among Smart Grid security professionals.

Of course, I must take the opportunity to also give myself a shameless plug, since I created the Smart Grid Security Summit (www.smartgridsecuritysummit.com), which took place this past Summer, and this has led to the upcoming Smart Grid Security East conference (www.smartgridsecurityeast.com), where representatives from all the above mentioned organizations (and a lot more) will be presenting on nearly every Smart Grid security topic there is to talk about. I certainly hope some of you can make it to the event. It will be worth your time if Smart Grid security information is what you seek. You can also freely join and attend meetings of the NIST, DHS, and OpenSG groups. Anyone interested in helping is welcome.

Otherwise, please continue to peruse the fear, uncertainty, and doubt (FUD) driven news headlines. If nothing else, they are quite entertaining.

WikiLeaks and Why "Plan B" is now more important than "Plan A"

2010-12-05T08:07:00.000-08:00

We all understand the idea of "Plan A" and "Plan B". Plan A is the plan we put in place that is meant to work as planned. In security, it is the plan we hope will ensure the CIA (Confidentiality, Integrity, and Availability) triad is in place. We put a lot of effort into Plan A, and then the more intelligent among us will put some effort into a Plan B. This is the plan we switch over to in the event Plan A fails.

This is generally the "damage control" mode plan. This is the plan we all hope we never have to go to, since by this point something very bad has happened. This could be something like...say perhaps...all of our national secret and top secret information getting leaked on a website.

That can be a really bad thing...

By now we are all probably keenly aware that the masterminds behind the WikiLeaks website have decided that information must all be publicly shared no matter what. Someone asked me my opinion about this several days ago....if I thought it was a good thing of bad thing...and my response was simple. I do not have an opinion about it being a good thing or bad thing. What I do know is that it is something that exists, and we must now figure out a way to deal with it, because it is NOT going to go away...EVER! It is like winter in Cleveland...deal with it.

I know this may sound harsh, but that is what we are facing. We live in an age where information ebbs and flows (and overflows) like water in an ocean. It comes to us as a gentle and calm breeze, or as a hurricane. It drifts down like snowflakes, or comes crashing down like an avalanche.

Okay...enough analogies...you get the picture. The truth is, we simply no longer have the control of information we once thought we had. The very nature in which we communicate today has created an environment were massively scalable information storms can occur. In the "good old days" we communicated by sending letters and talking. Today we communicate by generating data that gets pumped into "The Cloud", and then BLINDLY trust that it will only get to the intended recipients and nobody else.

Isn't that cute...

The Plan A way of dealing with information has been to protect the confidentiality, integrity, and availability of the information for as long as information has been important to us (essentially forever, but perhaps more so today in the information age). While we have created some absolutely fantastic systems for insuring both the integrity and availability of information over the last several decades, it seems that the very systems we have built have made it increasingly more difficult to insure confidentiality. Through the application of Moore's Law we have created systems with insane amounts of processing power, and have driven down the cost of these systems to almost nothing (I say almost nothing because you can find computers for free these days...at least in the San Francisco Bay Area), meaning that anyone can get their hands on the tools needed to both obtain and distribute information. There was a time when getting confidential information meant breaking encryption or applying brute force or dictionary attacks on systems. While this is still true today, we now live in a world where there are so many people accessing systems throughout the world, we no longer need to break into systems to get a hold of sensitive information. Today somebody who has authorized access to information either copies it or sends it into the cloud for all to consume. What makes this so difficult to control is that there are so many who have access to information, and either through direct access or aggregation the information can be assembled into nice little information bombs.

In other words, confidentiality has become nearly impossible to both achieve and manage.

This makes Plan A an incredibly difficult plan to manage, and certainly makes our reliance on Plan A more and more difficult to justify from a due diligence/due care perspective. We simply live in an age where we MUST assume compromise. We must accept the fact that, at some point, confidentiality goes out the window. Time to look at Plan B.

I am not sure what the US Government is doing with respect to Plan B. I saw an article where the US Government is warning college students to not talk about WikiLeaks...or else. I see some efforts to shut down the WikiLeaks site, and cut off funding sources. I imagine these are all some valid steps to take...in an act of desperation. Okay, maybe it is not desperation, but it certainly seems desperate. I mean...c'mon...do we really believe this is going to do anything more that irritate a bunch of college students who already do not like our government to begin with, and who are perhaps infinitely more savvy about the information age?

I am fairly certain that Plan B has not been given the level of attention it should have been given. It is very difficult for people who are intelligent AND arrogant (a bad but common combination) to consider the possibility that their best laid plans may have a fatal flaw. Consequently, anything more than a cursory level of attention to Plan B is considered an admission that maybe they are not as smart as they think they are. Perish the thought!

The truth is, Plan B has ALWAYS been more important than Plan A. By the time you get to point where you need to use Plan B, things have generally gotten very bad. This is now the time were you must not only figure out how to keep things operational, but also undo the damage that caused Plan A to fail. This is the "do or die" moment.

We certainly need to continually focus on protecting information. We do indeed have systems and methods available to us today that can buy us some time in the race between those who need to protect information and those who want to uncover it. We simply need to understand that at some point the information we so dearly protected is likely to be become publicly available, and use that mentality to weather the information age. It may take some time, but I am sure we will eventually get to a point where we can deal with this...much like I dealt with 21 years of Cleveland winters.

Can I Steal A Vowel ? Never Mind...I Don't Need To.

2010-11-11T07:24:00.000-08:00

The human mind is capable of some amazing feats. The conceptual capabilities of a young child, for example, astound me. I am a fairly good chess player, and recently my 6 year old son decided he wanted to learn chess (having seen "Wizard Chess" played in a Harry Potter movie), and I decided to indulge him.

Being a firm believer that children are far more capable of what some tend to give then credit for, I play him just as hard as I would play anyone, which meant the first few games resulted in quick checkmates. It did not take long for him to figure out how to think ahead and use inference as a strategic method, and he has since managed to achieve stalemate. Games now frequently last for an hour or more. Not bad for a 6 year old. Pappa is so proud!

What amazes me is watching him "think". I swear I can almost feel his brain thinking, and I swear I can sense his "brain muscles" getting stronger. I also believe that ANYONE can build those "brain muscles" if the drive exists to do so.

Let's consider an interesting story I read this morning. It was an article on esquire.com about a Wheel of Fortune contestant who solved the puzzle with 1 letter (and a freebie apostrophe). I read the story and realized that she was using a highly tuned level of inference in order to arrive at her conclusion. It reminded me of a conversation I recently had with Dr. Fred Cohen (we occasionally meet at the local Peet's for coffee and conversation). He stated that he believes that inference is impossible to prevent, and I have to say that I tend to agree with him.

A hacker (researcher, penetration tester, whatever term you like) is presented with overwhelming amounts of information surrounding a system all the time. In fact, the challenge is not where to find the information, but how to filter out what does not matter. With a little mental exercise, this can be accomplished very quickly...mainly because most organizations charged with protecting information are inherently lazy, and fail to understand the power of aggregation and inference. I have discovered countless pieces of company "confidential" information from piecing together bits of information available in various "sanitized" versions of documents. Bear in mind, I am not a "hacker" (at least not in the modern sense of the word), but I get how hackers think...at least to some degree.

I think about this a lot when I consider Smart Grid technologies, as well as health care information technologies. As these technologies grow we are going to see new sources of information emerge, and in our inherent somewhat lackadaisical manner of dealing with security at the decision making helm of our corporate culture, we will create plenty of early opportunities for aggregation and inference.

Things are going to get interesting....

Smart Grid Hackenomics

2010-11-08T06:56:00.000-08:00

I recently attended (and presented at) the Department of Homeland Security Industrial Control Systems Joint Working Group (DHS ICSJWG) meeting in Seattle Washington. It was a interesting event, and STUXNET seemed to be the hot topic everyone was discussing. Most of the sessions were quite good, and many were informative.

When I attend these types of events, I often find the side conversations I have with attendees more interesting than the conference itself. I had the opportunity to chat with people who work at DHS, FBI, NRC...and just about any other 3 letter agency seeking to get a handle on cyber security issues. It does my heart good to know that our government is indeed serious about cyber security, and truly seeking knowledge.

The most interesting discussion I had, however, was on the last day. It was during a lunch break with one of the attendees, and we started a discussion on the economics of attacking the Smart Grid. Essentially, we agreed that "hobbyist" attackers and "nation-state" attacks are perhaps not the types of threats that should (or do) cause great levels of concern at the C-level's of stakeholder companies. At the highest decision making level of any organization directly affected by security threats, the only issue that consistently keeps them awake at night is money...or rather the loss of money. In fact, when we talk about security, we must constantly understand that an enterprise's chief (and arguably exclusive) security concern is in securing their ability to keep making money (and not lose money).

In other words, if security does not lead to more $$$, expect some rolling eyes. Likewise, if a lack of security leads to a loss of $$$, expect some wide eyes. This is the beginning of my Theory of Hackenomics.

In our discussion, we used the financial industry as an example of an economic model that makes a lot of sense to organized criminal enterprises. In the former Soviet Union, there are criminal enterprise organizations that provide tools and support services (for a fee) to criminals who want to make a career out of exploiting security holes in the financial industry. This is a very popular target for criminals because it is both large in size, and the direct result of a successful attack is immediate access to cash. So as part of my theory I want to state the following: The quicker an attack leads to cash for the attacker, the greater the likelihood that the attack moves from theory to reality.

This is, however, only part of the theory. The other part has to do with volume. For organized crime to get involved, the volume needs to be big enough to take the risk. Remember, organized crime is just as concerned with risk as corporations are (some will argue that corporations are the "new" organized crime anyhow). Therefore a quick path to cash that does not include a large enough volume is not necessarily a win for organized crime.

Another important issue to consider is keeping the attack as "clean" as possible, in order to make collecting and retaining the cash as easy as possible. A good example of this is how financial firms created Credit Default Swaps as a way to hedge high risk investments. This instrument allowed the potential for a large return on the chance that those who took out those crazy loans on overpriced homes (and such) would default. Well, as it turns out, those who purchased Credit Default Swaps seem to have done quite well. It was essentially a low risk method of shorting the entire financial system, and it is perfectly legal under today's laws.

So now this brings me to what became an interesting part of the lunch discussion. I postulated that if a large stakeholder in the Smart Grid ecosystem (in other words, a large publicly traded utility or AMI product vendor) was vulnerable to a major Smart Grid related attack, and an attacker held onto a 0-Day vulnerability, he could potentially sell the 0-Day vulnerability for a lot of money to a large criminal enterprise, who could then short the stock of the utility or product vendor, and then publicly announce the vulnerability. Granted, this would require some coordinated effort, but if done correctly, one could make a killing when the stock plummeted on the bad news. The news alone would probably drop the price enough to make a lot of money with a high enough volume. The news immediately followed by an actual attack would probably lead to a very big win for the criminal enterprise.

As we continue to have lunch, we discussed a few more ideas, and I thought of a few more over the last several weeks (I am not going to go into them here), and I came to the conclusion that Smart Grid Hackenomics may indeed be an interesting discipline for criminal organizations to investigate...and they probably already are.

Hopefully, the C-Level people at stakeholder organizations have thought of this as well.

Mobile Application Insecurity

2010-11-07T13:59:00.000-08:00

Being someone who has develops secure mobile applications, I am consistently dumbfounded at large enterprises (who should know better) that fail to secure their mobile applications. A recent article in The Wall Street Journal highlighted some findings by viaForensics which pointed out several banking applications for mobile devices that store passwords unencrypted on devices.

The banking industry is no stranger to security concerns. They are indeed one of the largest purchasers of security products and services globally. The rush to bring mobile applications to the marketplace by enterprises has not overlooked financial firms, however, and they are simply not applying basic principals of secure application development - such as build security in from the very beginning, and test the security before deploying the applications. I am absolutely floored by the number of financial applications available on the iPhone (for example) that do not require something as simple as a PIN to enter the application after storing the password (let alone encrypting the password).

It is carelessness at best, and completely irresponsible at worst. Banks, Large Enterprises, and Health Care organizations should make maximizing security a priority with any and every application that deals with ANY potentially sensitive information...and they consistently fail to do so often enough to convince me that there will be a lot more breaches before things get better.

What I also find remarkable is how a company like Apple, who scrutinizes application submissions and regularly rejects applications that use foul language, show nudity, or (God forbid) replicates Apple functionality. Yet Apple does not bother to reject applications submitted by banking and health care organizations (the latter being something I am personally well aware of) that fail to encrypt information. Is this their responsibility?

Yes it is!

Security is everyone's responsibility, and until we understand that, we will continue down the same path with every new technology, platform, and latest and greatest thing that comes down the pike.

You can bank on that.

"Smart Grids Don’t Present Any New Security Threats" (According To At Least One Man's Opinion)

2010-09-10T16:14:00.000-07:00

I read an interesting interview on TMCnet this morning. It was an interview with Chris King, who is the Chief Strategy and Regulatory Officer at eMeter. One of the questions he was asked...well, let me just quote it directly:

Q: Don’t smart grids potentially present a major security threat?

A: Smart grids don’t present any new security threats. Utilities have controlled millions of customer-owned air conditioners, water heaters, and other devices for decades with no security breaches. In fact, the technologies being deployed today are more secure than ever.

Hmmm....

I would suggest that Chris King might consider taking a look at NISTIR 7628, Volume 3, Chapter 7.

Let me quote from that specific section:

7.1 Scope

...First, we have identified a number of evident and specific security problems in the Smart Grid that are amenable to and should have open and interoperable solutions but which are not obviously solved by existing standards, de facto standards, or best practices. This list includes only cyber security problems that have some specific relevance to or uniqueness in the Smart Grid. Thus we do not list general cyber security problems such as poor software engineering practices, key management, etc., unless these problems have some unique twist when considered in the context of the Smart Grid. We have continued to add to this list of problems as we came across problems not yet documented...

This chapter then continues on for a bit over 30 more pages (including references) to articulate the specific security issues identified in the Smart Grid (so far). You know, the ones that Chris King essentially says are not there.

Perhaps Chris King has not read NISTIR 7628, or he simply does not agree with more than 400 people who contributed to NISTIR 7628, let alone the plethora of "unofficial" discoveries made by security consultants worldwide. I would strongly suggest that he takes a good hard look at NISTIR 7628 (at least at Volume 3, Chapter 7) and then revisits his last statement.

I am sure he is a very smart person, and perhaps he was misquoted (that can happen). I would love it if he would comment on this.

NISTIR 7628 Is Final...So Now What?

2010-09-05T09:38:00.000-07:00

The entire Smart Grid deployment and cyber security world has been waiting for NISTIR 7628 to move from "Draft" status to "Final" status for nearly one and a half years. This magnificent effort, which included over 400 participants from many industries, government agencies, public and private groups, and just plain interested individuals, has culminated in 3 volumes that essentially read like an encyclopedia of cyber security best practices and technical jargon, complete with tables, drawings, and lots of arrows pointing all over the place. It is an impressive compendium of knowledge, and you can get your very own copy by going here.

So what does this all mean to the world of Smart Grid security? Does this make us more secure?

Well, as things stand right now, not exactly.

First of all, let's understand something about NIST and NISTIR 7628. The title is both prescient and potentially misleading. Here is the title for Volume 1:

Guidelines for Smart Grid Cyber Security: Vol. 1, Smart Grid Cyber Security Strategy, Architecture, and High-Level Requirements

Look carefully at the first word and the title and bear in mind that, for all legal intents and purposes that is all that matters. It is a "Guideline". I know it says "Requirements" at the end of the sentence, but understand that NIST does not dictate requirements to anyone who has the authority to enforce anything. The only requirements NIST has any authority over is the requirements NIST sets forth to comply with NIST standards (i.e. there are certain specific requirements that an entity must meet in order to become FIPS certified).

Why do I say this is potentially misleading? Well, because unless an authoritative body passes a rule, law, or mandate of some sort that requires the adoption of all or part of the recommendations in NISTIR 7628, it is nothing more than a magnificent exercise.

The simple existence of Smart Grid security guidelines does not make the Smart Grid more secure. The correct implementation of Smart Grid security standards, however, can.

Yet simply pointing at the NISTIR 7628 and saying "do this" will not suffice. This is because NISTIR 7628 is a collection of NIST standards and recommendations. While this may seem sufficient for some, it is still too open ended to serve as anything close to prescriptive. In fact, NISTIR 7628 is not intended to be prescriptive, and it says so in section 2.2 of Volume 1:

"This list of technologies and services is not intended to be prescriptive; rather, it is to be used as guidance."

This leads to the obvious conclusion that NISTIR 7628 is not intended to serve as "the rulebook", but to assist the rulemakers in writing "the rulebook".

So who are the rulemakers?

Well, that is a good question, and one that is not so easy to answer without first understanding that it all depends on what part of the Smart Grid we are talking about.

To try to simplify this as much as possible, and forgive me if this is oversimplified (or overly complicated as the case may be).

We can break the power Smart Grid into three categories:

1. Generation - Where the power is generated (i.e. the power plant)

2. Transmission - How the power gets from the power plant to the substations that send it to those who use it.

3. Distribution - The part of the organization that the user directly interfaces with (the ones who read your meter and send you a bill and shut off your power if you do not pay your bill).

So Generation and Transmission are generally not considered part of AMI (Advanced Metering Infrastructure). AMI is the part of the Smart Grid where smart meters live. Generation and Transmission currently fall under the jurisdiction of the Federal Government, and are therefore subject to the whims of the Federal Energy Regulatory Commission (FERC) and the North American Electric Reliability Corporation (NERC). NERC is not a Federal agency, but is given authority by FERC to conduct audits, levy fines, and all sorts of interesting stuff that tends to keep utilities in various stages of insomnia and cold sweats.

Distribution, on the other hand, falls under the jurisdiction of the individual States, and consequently the Public Utility Commission (PUC) of a given state.

So what this means is that FERC, NERC, and the State PUC's must now take a long and hard look at NISTIR 7628 (not to say they have not already been doing so) and try to synthesize some specific regulations based upon what is contained in this very verbose 3 volume set. This is no easy task, as one can imagine. Let's examine one particular section, taken from Volume 1:

4.2.1.8 Physical Security Environment

...In determining the appropriate level of physical protections required for a device, it is important to consider both the operating environment and the value and sensitivity of the data protected by the device. Therefore, the specification of cryptographic module physical protections is a management task in which both environmental hazard and data value are taken into consideration. For example, management may conclude that a module protecting low value information and deployed in an environment with physical protections and controls, such as equipment cages, locks, cameras, and security guards, etc., requires no additional physical protections and may be implemented in software executing on a general purpose computer system. However, in the same environment, cryptographic modules protecting high value or sensitive information, such as root keys, may require strong physical security...

If, for example, you are the CPUC (California Public Utility Commission) and are attempting to create a requirement based upon this section for physical protection of cryptographic modules (and the data contained within them), one must first define what "high value or sensitive information is". The root key mentioned is a good example, but what about other information stored on the device? What is the information? Is it also sensitive? Who determines if it is sensitive or not?

If the CPUC then determines that the information stored is not overly sensitive (i.e. not a root key), then it is important to ensure that the scope of the information stored on such modules does not "creep" to a point where it may indeed become sensitive. This is no easy task, because sometimes what is deemed safe today does not always remain safe going forward. A good example of this is a Social Security Number. There was a time when nobody had a problem sharing their Social Security Number with anyone. Heck! In many cases it was your ID number for school, work, military, etc. What happened, however, is that the scope of the Social Security Number expanded, and it was soon discovered that if you knew someone's number you could do all sorts of bad things with it.

If the CPUC determines that the information is indeed sensitive, then they are tasked with determining what standard for protection of such information must serve as a baseline (i.e. FIPS 140-2).

Providing they can accomplish these tasks, they must then determine if and how they are going to audit (and potentially certify) such requirements.

...but first they have to determine what is in scope and what is not in scope, and why. This in and of itself requires the PUC's (and FERC and NERC) to have an intimate understand of what parts of NISTIR 7628 (and potentially other guidelines, such as the excellent work done by the UCAIUG AMI-SEC Task Force, which is specifically credited for their contributions to NISTIR 7628 within Volume 1) apply to their purview. Looking at this at the Federal level, one might conclude that they have enough resources to tackle this task, but having listened to FERC Commissioner Philip Moeller's keynote address at my Smart Grid Cyber Security Summit last month, in which he stated "We don't have all the answers, we need all of you to help.", I am led to believe that we still have a long way to go.

...and it is even more challenging for State PUC's. The CPUC is a fairly well staffed organization, being that California is indeed a very large State. Nonetheless, the CPUC does not currently have anything close to a comprehensive understanding of cyber security. To be fair, why would they? In its many years of existence they have never had to deal with cyber security issues with respect to regulation of utilities, and up until the passage of California SB 17 it has never been their responsibility. However, being staffed with some very intelligent (and diligent) people, and now being responsible for making decisions relating to cyber security and the Smart Grid, the CPUC has indeed taken it upon themselves to rise to the occasion. I have personally attended two public hearings at the CPUC where Smart Grid security was discussed, contributed to requests for comments from the CPUC regarding cyber security, and the CPUC is planning a public hearing to specifically discuss NISTIR 7628 with the NISTIR 7628 team at the CPUC at the end of September, 2010 (currently planned for September 28th and 28th), as well as additional workshops to hash out the details of Smart Grid security.

This is all good stuff!

...but what about other PUC's? Some States (from what I have been told by members of the CPUC) have PUC's that could fit into a small room with plenty of space to spare for filing cabinets, chairs, and tables. In other words, they are woefully understaffed and underfunded. How are they going to manage cyber security?

Well, one answer is contained in one of my favorite sayings "As goes California, so goes The Nation." Their eyes are on California, and what California decides is quite likely to serve as a template for the rest of the nation. Some have also argued that Texas is also serving as a template. While this may be true, I have a sneaking suspicion that California will likely prevail as a trendsetter. Only time will tell, I imagine.

The great news is that there seems to be no shortage of people who are willing to volunteer their time in working through these challenges. It may not be entirely altruistic in nature (hey, everyone wants a piece of the Smart Grid security market pie, including yours truly), but the fact remains that we are indeed well served by some of the great minds working on the effort. PG&E has a cyber security team currently led by CISO Dave Tyson (who came from the security team of eBay) and PG&E has been dealing with Smart Grid security for longer than just about any utility in the world. The UCAIUG AMI-SEC Task Force is still working hard and growing stronger with every meeting (I try to attend and contribute as often as possible). Many AMI vendors are currently specifically dedicating resources to cyber security efforts, and are working together in a spirit of "coopetition", where they cooperatively share information with each other despite being competitors. Anyone who attended my conference is well aware of just how many organizations are involved in this effort, and the list keeps growing.

We still have a lot of work to do, but we have come a long way, and I am not even close to tired yet! NISTIR 7628 is worthy of being celebrated for finally being completed, but now the real work begins.

The Top Level Disconnect

2010-08-15T13:28:00.000-07:00

The first Smart Grid Cyber Security Summit has come and gone, and was much more successful than I ever imagined it would be (especially for being the first of its kind). Being the conference chairman, I was thrilled at all the attendees who took the time to come up to me during networking breaks and thank me for holding the conference, and consistently asking me "When are you holding the next one." I was amazed at the attendees who traveled from Europe and Asia, and all the government defense contractors, AMI vendor security specialists, utility representatives, consultants, members of the media, and FERC Commissioner Moeller...all who came to take part in what many felt was a fantastic opportunity for stakeholders at many levels of the Smart Grid security ecosystem.

What struck me as interesting, however, was a point that was raised by Dr. Fred Cohen, who delivered the keynote address on the 2nd day of the conference. His presentation focused on the lack of (and need for) security expertise in the ever expanding cyber world, and some proposed solutions (which included a bit of self promotion, since Dr. Cohen is currently operating a school specializing in security related academics). Something he said struck a chord in me, and while I have been aware of the situation for quite some time, I have been pondering it quite a bit the last several days. What Dr. Cohen did was posed a question to the audience, asking how many attendees were CEO's (or top level executives) of either AMI vendor organizations or utilities. Nobody raised their hand (and, in fact, nobody at that level signed up to attend the conference), and Dr. Cohen proceeded to point out that until management at that level takes an active role in dealing with security, we will continue to witness a shortfall in security.

Okay, in all fairness this was the first Smart Grid Cyber Security Summit, and CEO's are busy people, and who the heck am I anyhow? Yet his point still rings true in the security world.

The fact is that CEO's simply do not participate in the cyber security ecosystem at any appreciable level, and that leads to the obvious question "Why should they?"

In a word...MONEY!

A CEO's job, after all, is to make sure the organization's income level goes up and the amount of money leaving the company does not go up faster than what is coming in. That is the essence of what being a successful CEO is all about. If the company is publicly traded, then it is all about keeping the stock price from falling. No matter what anyone tells you, that is the name of the game, and always has been, and always will be.

So that brings us back to security. One of the most difficult expenditures to justify to a CEO is the cost of security. Trying to demonstrate a return on investment for security is next to impossible. Security is simply not considered a feature a customer is willing to pay for. Rather, it is something that customers expect to be part of "the package". Customers of AMI vendors, for example, want a decent meter at a low price that also happens to be secure because there are enough people "out there" making enough noise about Smart Grid security to get their attention. This noise includes the government, bloggers, the media, security "hobbyists", security professionals, and privacy proponents...to name just a few. While this may be enough to get the attention of decision makers, it is generally not enough to get decision makers to dedicate any more resources than necessary to divert the attention from their organizations to someone else's. If a top level decision maker believes the attention is (or may) negatively impact the bottom line, more resources are generally expended. Now I have to say that I may be painting the corporate world with a very broad brush, and I am sure that there are some high level executives that want to do the right thing because it is the right thing to do, but their ultimate survival depends on keeping the company cash flow positive and profitable. Stockholders simply do not reward any other behavior.

...and security can be very expensive. It is especially expensive if it is poorly done, and really amounts to a waste of time and money in such cases. If it is not part of the design, it can mean lost revenues due to customers going to a competitor, or it can amount to outright devastating losses in the event of a serious malicious attack. Imagine an AMI vendor that installs 20 million meters and it is later determined that the meters are vulnerable to a very serious security related threat that requires an outright replacement of meters. I am not talking about something theoretical, but rather a vulnerability that turns into a real world exploit. An attacker does not need to, for example, shut down power to millions of people in order for the exploit to prove effective. A few thousand is plenty (maybe even less). It does not take a massive failure of all systems to negatively impact the reputation (and market cap) of a company.

Just look at what happened to both Toyota and BP. It only took a few failures for Toyota to lose billions in market cap, and 1 major failure for BP to lose so much market cap that it dramatically impacted the retirement accounts of millions of British citizens. If a major publicly traded utility should become the victim of such unfortunate circumstances, what potential economic impact could this translate to? The answer is really a big unknown.

It seems prudent, at this point, for CEO's (and other top level executives) of organizations involved in the Smart Grid to become a bit more involved in actively participating in the world of Smart Grid security. One utility representative at the conference mentioned that the high level executives literally pour money into security when they discover that they are about to be audited by NERC, but other times are not so willing to open the coffers. This really does not make sense, and is not indicative of due diligence. If high level executives had a better understanding of the ecosystem, and the concerns of stakeholders, and the dynamic environment surrounding Smart Grid security, then they could make better and more informed decisions on where and how to dedicate resources. It comes down to being proactive rather than reactive.

I am planning to hold another Smart Grid Cyber Security Summit in the near future. I will make sure to reach out to the CEO's of utilities, AMI vendors, and other stakeholder organizations involved in building the Smart Grid. I am hoping they will view this as an opportunity to become part of the solution.

...because otherwise they may indeed be part of the problem.

The Importance of Trusted Relationships

2010-07-04T06:37:00.000-07:00

As a security professional, I have had the opportunity to work with many different companies in the ever expanding world of security. Some of these companies have been very large (multi-billion dollar companies), and other have been quite small. The larger companies have the dubious distinction of being able to pour enormous amounts of marketing dollars into convincing the world that they are at the leading edge with respect to security. Unfortunately, this is absolutely no indicator whatsoever of the security posture of a company. In fact, as I have discovered on more than one occasion, some companies will opt to spend enormous amounts of money on everything EXCEPT building better security, hoping to convince their potential customers that they are the right choice to go with.

I think it is no big secret that many corporations seem to have no problem "embellishing" when it comes to the information they choose to share with the world. We have all seen enough of this at this point in our lives to know it is "just the way it is" in the world of business. We simply accept the fact that some companies choose to create their own versions of reality, and make choices to do business with them despite what we may believe about them. For example, we may not believe that an oil company is as committed to safety or environmental soundness as their public relations department may say they are, but we still choose to buy their petroleum products.

The fact is, most of us are not overly concerned about an oil company's safety record or what they are doing to make our environment better when we purchase fuel. If an company does not have a good safety record or destroys our environment we simply do not make the connection between that and our lives when we are at the fuel pump. We have other things on our minds.

With other products, it is perhaps a bit different. If a company that produces food or drugs is found to be acting in a scandalous manner, we tend to become a bit more nervous (perhaps more with drugs than food). Finding out that a drug company is being run by a bunch of corrupt and non-trustworthy people may indeed be cause for concern (at least it would be with me). At a more granular level, finding out that my personal physician is seedy lowlife would certainly make me ask my HMO to provide me with a new doctor. The fact is that when we are forced to trust our lives to a company or person, we want to make sure we are dealing with PEOPLE who can be counted on.

You see, dear reader, an organization is portrayed as being an entity (i.e. a corporation), but we all know that the organization is ultimately a collection of people. Despite the attempt by such organizations to make it about the entity, it is always the people who make or break it.

This certainly holds true in the world of security. When it comes to security products (i.e. security hardware), there are many companies to choose from. In fact, most of the security hardware available today (such as security chips) have become a commodity. When I speak to vendors of AMI (Smart Grid) products, or to organizations interested in implementing security products in health care organizations, one of the first questions that comes up is "How stable and reliable is the company making the security products?".

Organization who are making decisions about security products are transitioning from those who simply wanted to look like they were doing something to ones who are expending resources on products and services that do what they are supposed to do. This is largely driven by the nearly insatiable appetite the hacking community seems to have for breaking down security systems. When I present a security product line to a company, they ask a lot of questions. This is a welcome departure from several years back, when a company simply asked us what they could buy that fit within a given budget. Today, they want to be sure they are making the right decision for the long haul.

What I have found is that is seems to be very important that the organizations making security decisions trust the organizations they do business with at a much deeper level than ever before. I have intimate one-on-one discussions with security professionals and decision makers in companies who literally want my opinion of the companies I represent. They ask questions like "Do you think these guys are going to be around a while?" and "Are they trustworthy?" and "How do you find them compared to Company B?".

While remaining as tactful as I can, I always tell the truth, because these days most people I speak to in the security world VERIFY what I tell them. I know this because on more than one occasion I have had them return to me and say "I checked out what you said, and found out it was true." At first, I was taken aback by this (at least momentarily), but now I find it absolutely refreshing. In fact, sometimes I take the time to send citations for the claims I make, in order to make it easier for them to verify what I tell them.

You see, ultimately security is built on trust. The character of the people who make up an organization is as important (if not more important) as the products they build. Once I begin questioning the integrity of the people who make up the team of a security organization, I question the stability of the company, and ultimately the products they build. Anyone can build a security product line, given the right resources. However, it is only companies with integrity can build a security product line they can stand behind, and no matter how big or small the company may be, that is what I believe everyone should look for, and it always starts with the people.

Sorry Health Care...Game Over!

2010-05-22T07:39:00.000-07:00

Back in the day when pinball machines were all the rage at game arcades (and the only video game was pong), one could bear witness to a room full of pre-pubescent boys and girls (mostly boys as I recall) pumping quarters into machines and batting steel balls with rubber coated "flippers" in order to prevent the ball from falling into the shoot. The longer you could keep batting that ball the more points you would score. If you were particularly well versed in the art of pinball machines you could shake and tilt the machine to a degree (avoiding the inevitable "tilt" caused by being over-zealous) and perhaps score more points by making the ball move where you wanted it to move.

Nonetheless, this mastery of batting the ball and shaking and gyrating the machine came to an end for even the most skilled of pinball wizards. Many players would continue to shake and gyrate the machine after their last steel ball (back than you got at least 3 balls) had fallen, but there was simply no denying the reality of what the scoreboard prominently displayed in bold letters...GAME OVER !

Sure, those with paper routes or other sources of quarters could keep pumping legal tender into the system to give it another go around, but the result was inevitably the same. Eventually you have to give into reality. You can't bat the ball around forever.

The idea of health care organizations having to take responsibility for security and privacy in an ever expanding digital age is certainly not new. The first HIPAA regulations passed in 1996. I am no math genius, but that is about 14 years by my calculations. In 14 years, however, health care organizations and providers have been lax in dealing with security. I currently serve on the CalPSAB security steering committee, and that seems to be something we all agree on (actually, we seem to agree on a lot more than that). Having just returned from the Safeguarding Health Information: Building Assurance through HIPAA Security conference in Washington DC, it seems quite clear that the Office of Civil Rights (OCR) and Federal Trade Commission (FTC) are also aware that a lack of due diligence on the part of health care practitioners (and business associates) with respect to security and privacy has gone on long enough.

Nonetheless, we still see organizations (such as The American Medical Association, American Osteopathic Association and Medical Society of the District of Columbia) fishing for more quarters to pump into the machine.

Hey! Why not? They have plenty.

In an article published on the excellent Health Data Management Blog, the author references a lawsuit filed by the aforementioned entities. The essence of the lawsuit is that health care organizations do not want to fall under the authority of the FTC with respect to the "Red Flags" rule the FTC currently requires creditors to abide by. The (ridiculous) claim being made by the filers of the lawsuit is that (from the article):

Among other factors, the medical associations argue that physicians are not commonly referred to as "creditors," nor are patients ordinarily thought of as "account holders" or "customers."

Wow! Am I understanding this correctly? This is coming down to a definition of what a "creditor" or "customer" is?

At the Washington DC meeting one of my takeaways was that OCR is really putting the hammer down, and perhaps they should consider less "stick" and more "carrot" in dealing with organizations that have to comply with the rules. However, when I witness the equivalent of a bunch of pinball wizards banging on a machine as they fish for more chances to avoid the inevitability of owning up to the fact that batting balls around eventually loses its charm, I shed some of my sympathy.

The Health Care Industry simply cannot keep playing this game forever. It is time to focus their energy on ways to address security and privacy concerns in a meaningful way, and stop fighting what is inevitable.

GAME OVER!

Privacy: A Prescription For Disaster

2010-04-21T06:17:00.000-07:00

I have been watching the world of cyber security unfold for the last several years in a manner I would best describe as divergently focused. As a security professional who frequently engages in deep (almost philosophical) discussions with other security professionals (or more appropriately, Security Warriors) I am constantly amused at the frustrations we seem to share about privacy being the biggest driver of security in emerging technological initiatives.

I recently wrote about this on the topic of Smart Grid security, and Gib Sorebo of SAIC followed up on his blog with his opinion. Gib and I have had some long and great discussions about the issue of privacy in a world of security vulnerabilities, and the one area that seems to get us both preaching is the issue of privacy as it relates to health care. Simply put, this is a good time to shift the security discussion to something that really matters, and I am sorry to say that privacy needs to leave the room for a while.

Okay, I am sure the entire world of privacy evangelists are probably going to want to send me that nasty fruitcake (or worse) they have been holding onto for the last 20 years after reading that last statement, but please hear me out before you head over to the post office. Privacy IS important and DOES MATTER to me and probably every security professional in this world. I am a strong supporter of privacy, and consistently do all I can to protect my privacy. I refuse to give my address and phone number at stores that ask for it when I pay with cash or ask me for that information for any reason whatsoever. I refuse to share ANY information with ANY entity that requests it that I deem is not on a need-to-know list, and have held up lines in stores, banks, and other places (sorry to all of you who stood behind me) defending my rights to my own information. Privacy is indeed very important in the digital world we are now completely enveloped in.

...but it has got to stop being a part of health care security discussions, or we are probably going to end up with a lot of dead people as a result.

In fact, we already are ending up with seriously damaged patients in the age of digital health care. I read an article on The Huffington Post this morning titled "Electronic Medical Record Shift: Signs Of Harm Emerge As Doctors Move From Paper" which pointed out how either bad information or a failure in software has led to patient trauma (heart attacks, seizures). The article did not speak of security issues that led to failures in these systems, yet the failures found in these systems serve to illustrate what I have been talking about for years. If a system is vulnerable to penetration and compromise by an attacker, the attacker can cause a lot more harm than a patient would suffer as a result of a privacy breach.

Let me specifically paint a scenario based upon the Huffington Post article. The first sentence of the article speaks of hospital workers misreading medical dosage information and dispensing 10 times the normal dose of a medication, leading to a patient heart attack. Under HIPAA HITECH, if an attacker should enter a system and change a single patient record (perhaps a patient who is a political figure) for a medical dosage to purposely cause a heart attack or death, the health care organization would be in violation of a privacy law, but could not be held liable for the death of the patient due to a failure in data integrity. In my opinion (and the opinion of others I have spoken to about this issue), there is something very wrong with this.

The problem becomes even more complicated when you add medical devices to the system. Medical devices have become increasingly "smart" and are now trusted devices on health care networks. Devices perform many functions in health care, and the information some devices are trusted with gathering is often used to make life or death decisions. A device which automates the process of typing blood and then sends the information to a patient record database is indeed one type of device that would fall into this category description. If an attacker could spoof such a device he could then populate the database with incorrect information that could kill a patient. Moreover, some medical devices have firmware that can be updated (and in some cases over a network connection), which opens up the possibility of rogue firmware that could be purposely introduced to cause havoc.

I bring this up because the cost of failure due to a privacy breach simply pales in comparison to the potential cost of failure due to a failure to deliver correct information to the system. One leads to embarrassment and potential financial headaches, the other leads to death. Why is this distinction important? Well, except for obvious reasons, it is important because in a world where risks are mitigated based on costs of failure from a LEGAL perspective (i.e. a finable offense), the actual cost of failure due to a privacy breach is infinitesimally small compared to to somebody dying. A good lawyer can potentially turn a $1.5 million dollar fine (the maximum fine for a single instance under HITECH) down considerably if he or she could convince a judge/jury that the punishment does not really fit the crime.

It happens all the time, in fact, in other industries. At one time I worked for a company that dealt in motor oil who faced millions of dollars in fines from the EPA for statutory violations, but the fine was reduced to a few thousand dollars because the violation simply did not lead to anyone being harmed. The potential for harm was very high (as is true with medical record breaches), but if nobody is actually harmed then a slap on the wrist is a common punishment (especially if you have a good lawyer). Sure, it may cost you in legal fees, but if you already have a staff of lawyers anyway it is not so hard to stomach.

HITECH is a good step in the right direction for better security, but it still completely fails to address the bigger issues. As we continue to build out our "internet of health care" and interconnect data sources at a national (and eventually global) level, the security risks grow at a nearly exponential rate. This is because attackers like to attack systems more as they get bigger simply because it has a bigger impact. We should not wait for theoretical dangers to manifest themselves before we address these issues. Security vulnerabilities of large infrastructures are well known enough today that a failure to pro-actively address them is simply nothing more than negligence, and the health care industry should act more responsibly.

They know better.

The Grid Reliability and Infrastructure Defense Act- Better Late Than Never

2010-04-18T09:42:00.000-07:00

As I have discussed many times in the past, security is primarily driven by compliance.

Wait...let me back up for a moment.

While many organizations (and particularly those who are involved in The Smart Grid) are indeed elevating security on the priority scale of "things we gotta do", we can be certain that any organization that has felt the pain of an attack will do more to secure their deployments than one who has not had the displeasure of being "owned" by an attacker. While some may argue that this is not the best way to get security into a system, I will argue that it is indeed the most effective driver of security. It is human nature to react to known dangers rather than proactively defend themselves against them. Moreover, we tend to proactively secure ourselves only if the known threats are directly experienced. Simply knowing someone who has been mugged in "the city" is not enough to get most people to become exceedingly aware of their surroundings, after all.

So with the Smart Grid we are in a situation where vulnerabilities have been discovered, and many more have been theorized. While nearly everyone who is involved in Smart Grid is indeed paying attention to security, turning that into "action items" remains a bit nebulous. Utilities who are actively deploying AMI (such as PG&E and SCE) are indeed focusing what I believe are tremendous (and competent) resources on Smart Grid security, and others are paying close attention (as I have gathered from various Smart Grid groups I am involved in). Vendors have created cyber security specific positions and departments. Security consultants are now specializing in smart grid security consulting. The US Government has several groups addressing the issues (FERC, NERC, NIST, DHS, DoD) in various capacities, and the list goes on and on.

The reason I say this is all a bit nebulous is because so far we have been lacking an authoritative mandate for Smart Grid security. Sure, NERC has been working on compliance and auditing standards (NERC-CIP 002-009), but neither NERC nor any other entity has the CLEAR authority to "lay down the law" as far as Smart Grid security is concerned. Each individual state has the power to halt Smart Grid deployments (I would surmise) for any reason whatsoever, but at a national level it is still very laissez-faire. The unfortunate negative consequence of this is that states (such as California) have adopted a bit of a "hurry up and wait" mentality about security (despite the fact that California doing this with voting machines was an epic disaster). This is never a good thing, because if (and when) security issues manifest themselves, the typical response is to halt progress until a resolution is reached (again, such as happened with voting machines). This is, to say the least, very irresponsible, because as far as the Smart Grid is concerned we NEED to have it deployed NOW in order to deal with the ever increasing demand for electricity. Consider electric cars, for example. Exactly how do we expect to manage load if California has millions of electric cars plugged in and charging on a hot summer day? Our current system can barely manage the load with no electric cars on the road, with high peak air conditioning usage days leading to power outages. We NEED the Smart Grid.

I was happy to see an article on TheHill.com that spoke of the House passing the Grid Reliability and Infrastructure Defense Act (GRID) which seeks to up the ante on FERC to take control of security issues affecting the Smart Grid. I am not generally fond of Congress passing laws that serve to penalize those who do not comply, as this generally leads to more consternation and less solution (in my opinion). So I was happy to see a section of this bill which seem to instead focus on providing resources to entities that are deploying the Smart Grid. From the bill:

COST RECOVERY.—If the Commission determines that owners, operators, or users of the bulk-power system or of defense critical electric infrastructure have incurred substantial costs to comply with an order under this subsection and that such costs were prudently incurred and cannot reasonably be recovered through regulated rates or market prices for the electric energy or services sold by such owners, operators, or users, the Commission shall, after notice and an opportunity for comment, establish a mechanism that permits such owners, operators, or users to recover such costs.

Now I know this is not very specific, but it does seem to address perhaps the biggest concern businesses involved in Smart Grid deployment may have in addressing security - COST $$$$.

It is not a law yet, and it may indeed go through some changes (perhaps not for the better) as it makes its way towards becoming a law, but I have high hopes.

...and hope springs eternal.

The Need For A Security Paradigm Shift

2010-04-10T04:40:00.000-07:00

I remember years ago, when Stephen Covey's bestseller The Seven Habits Of Highly Effective People was making its rounds throughout the business world, the introduction of the word "paradigm" in my vocabulary. I was working in a resort way back then and our director of operations used to love walking around and tossing the term out like Rockefeller gave away dimes to the poor. He was a great operations director, and certainly was not deserving of the gentle ribbing he took for the liberal use of a term that nobody in my world seemed to want to care about. Frankly, most of us cared more about changes in our scheduled shifts more than we cared about "paradigm shifts".

Still, I did indeed listen intently to what he had to say. I liked him a lot, and he liked me. He convinced me to read Covey's book, and I gained a better understanding of several concepts, most importantly the concept of the paradigm shift.

To summarize my understanding of it in as few words as possible is perhaps something I am incapable of, so I defer to a definition I found while perusing the venerable Wikipedia. Here is the section I found best describes it:

the historian of science Thomas Kuhn gave paradigm its contemporary meaning when he adopted the word to refer to the set of practices that define a scientific discipline at any particular period of time. Kuhn himself came to prefer the terms exemplar and normal science, which have more precise philosophical meanings. However in his book The Structure of Scientific Revolutions Kuhn defines a scientific paradigm as:

what is to be observed and scrutinized
the kind of questions that are supposed to be asked and probed for answers in relation to this subject
how these questions are to be structured
how the results of scientific investigations should be interpreted

The bullet points capture the essence of what I believe is absolutely critical as we continue to discuss the topic of securing the smart grid.

Anyone who knows me knows that I am generally a very positive person, and generally give most people the benefit of a doubt. However, you also know that I tend to not suffer foolishness lightly. I call things like I see them, and although I am sometimes way off base, I am on target often enough to cause those I target (using my Socratic methods) to feel a bit uncomfortable. To those of you who I have made uncomfortable, my apologies for making you feel uncomfortable. My intention is not to get you to dislike me. My intention is to get you to see things differently, or to get you to shift your paradigm.

What led me to this blog posting was an article I read titled Securing The Smart Grid by Elinor Mills. This article is a combination of what I believe is sound information layered with generous doses of conjecture. I am not going to get into what I believe is conjecture at this point, since that will indeed take more time than I have this morning. What I did find worthy of calling out, however, was a quote made by Jesse Berst of Smart Grid News:

Jesse Berst, managing director of the Global Smart Energy consultancy and founder of Smart Grid News, said he didn't see any reason why the energy industry wouldn't be able to secure the infrastructure as it modernizes.

"The physical security concerns me more than the cyber security because we've solved the cyber (security issues) for other big consequential infrastructures (like financial and Internet) and I think we can solve it to that same degree of safety for this one," Berst said.

Now let me preface this by saying that I believe Jesse's contributions to the entire world of Smart Grid are indeed beyond admirable. I read Smart Grid News on a daily basis, and find it to be a wealth of information. I will also be the first to admit that he is light years ahead of me (and perhaps a lot of people) in his understanding of The Smart Grid.

...however, his statement "...we've solved the cyber (security issues) for other big consequential infrastructures (like financial and Internet) and I think we can solve it to that same degree of safety for this one," truly left my mouth hanging open.

Are you kidding me?

Okay, maybe CNet took that out of context. God knows the media seems to do that with more frequency than we would like to see. So I will indeed work with the assumption that this may be the case, and dissect this statement as one that may indeed be put forth by someone who may not be aware of how a security professional (such as myself) might view it.

Let's start with the first part of the statement "..we've solved the cyber (security issues) for other big consequential infrastructures (like financial and Internet)".

Really?

I am not entirely sure where to start with this one. Let's just take financial to begin with, and describe how we have "solved" those issues. Despite having "solved" the cyber security issues with respect to the financial world, the financial industry still loses billions per year due to cyber attacks, and then passes these losses on to the consumer. One "solution" the financial industry put forth several years ago was PCI Compliance, which simply shifts losses to merchants, who then are forced to raise prices to cover the losses. Another "solution" is to jack up credit card fees and interest rates ("risk management" as they like to call it) to cover the losses that the financial industry cannot pass on to the merchants. Sadly, there is no way any consumer can avoid falling into this abyss. If I do not want to use credit cards I am hampered by having to write checks or use cash for anything and everything. I also must live with the cost of failure that cyber crimes impose on my merchants through higher prices.

Such is life. Do I get by despite this mess? Certainly! Is the problem "solved"? Nope! In fact, I am not sure it can ever be solved. What I am sure of is that we seem to be able to live with this particular cost of failure in cyber security, and that may indeed be good enough (for now).

It is the second part of the statement "I think we can solve it to that same degree of safety for this one", however, that got me to bolt out of bed and start writing. Here is where the entire world of Smart Grid security "apologists" need to go through the mother of all paradigm shifts. Solving the security issues to "the same degree of safety" where The Smart Grid is concerned does not quite seem to cut it, now does it? Let me explain.

Let's consider the cost of failure.

While we live in a world of hyperbole in the world of cyber insecurity, we now also live in a world where some of the inherent weaknesses in the Smart Grid security arena have made the transition from theoretical to proof of concept. Perhaps the most famous of these is the infamous Aurora Attack seen on 60 Minutes (that was when my phone started ringing). What that showed us was that the cost of failure in security could lead to power being shut down in some areas for months. Now I know that some of you are going to want to attack this by telling me that we can simply redirect power from elsewhere, and that is indeed true, but what you would probably leave out of that statement is the fact that redirecting power during the middle of a blistering summer heat wave is next to impossible, and I am quite sure that a malicious attacker (not just a script kiddie) is keenly aware of that.

...and there is more. A lot more in fact. Utilities are keenly aware of the issues, and so is our government, and they do indeed care A LOT about security. Way more than the media is willing to give them credit for. In fact, I have never seen an industry embrace the importance of security with such fervor as the power industry has in the last year. In California both PG&E and SCE have invested considerable resources in dealing with these issues. One member of the cyber security team at PG&E sent me a message at 10:50 PM several nights ago in response to a question I had (regarding a conference I am planning). I was surprised that he replied so late and he informed me he was still at work! When I asked him why, he told me that he (and others on his team) often work late. When I see this level of dedication from security professionals I do indeed feel quite comfortable about the work being done towards securing our grid, and so should others (in my opinion).

Nonetheless, there is a dire need for a paradigm shift in the discussions surrounding Smart Grid security. We cannot use examples where the cost of failure truly pales in comparison to the cost of failure when we have no electricity. Imagine, if you will, a scenario where someone hacks your bank account and takes all of your money. You contact that bank and it can take several weeks to get your money back. I know this to be true because I know someone who went through such a nightmare. Nonetheless, she did not go hungry or die. She had food in her house, and credit cards, and family and friends. It really amounted to a nasty inconvenience, and she got all of her money back eventually. The "degree of safety" built into the system was indeed more than adequate to deal with this situation, but nobody who has their nose to the grindstone in the world of Smart Grid security would consider this to be a valid presumption of having the situation under control. We are fortunate enough to have only a small (yet significant) fraction of our power infrastructure on The Smart Grid, and everyone involved is working hard to deal with the issues at hand.

Let us avoid out of context statements making their way into the public consciousness, which will inevitably lead to a loss in credibility for those who are working hard to resolve these issues. The public loves to dwell on the negative even more than the media loves to talk about it.

In other words, let's not fuel the naysayers.

Bravo PG&E! The Proactive Approach Always Wins

2010-04-08T07:47:00.000-07:00

I am once again making my way through comments to the CPUC, and was extremely pleased to find a comment made by PG&E:

PG&E AGREES WITH THE CUSTOMER PRIVACY GOALS AND POLICIES RECOMMENDED BY CONSUMER PRIVACY ADVOCATES

A number of consumer groups have provided specific recommendations regarding customer privacy goals and standards in this proceeding, including the Center for Democracy and Technology, Electronic Frontier Foundation, Consumer Federation of California, TURN and the Division of Ratepayer Advocates.5 These recommendations generally urge the Commission to move cautiously and very carefully in updating or revising its existing rules on customer privacy, particularly third-party access to customer data. In addition, the consumer privacy advocates point out that there are certain sources of national “best practices” for protecting consumer privacy in all industries that the Commission should consider and endorse, such as the “Fair Information Practices Principles” developed over the years and cited by various federal agencies, such as the Department of Homeland Security.6

As PG&E pointed out in its opening comments, we adhere to existing, strict and precise Commission and statutory rules and standards providing for protection of customer privacy, and do not see the need to dilute or reduce those protections. However, after reviewing the comments and presentations by consumer privacy advocates, PG&E agrees that it is timely in

this proceeding for the Commission and all parties to “benchmark” the existing customer privacy protections in the Public Utilities Code and utility tariffs against the national consumer privacy standards and goals applicable to other industries and consumer services. This is particularly important to the extent that the Commission will need to establish and be able to enforce these privacy protections and cyber-security protocols against third-parties who may be granted access by customers to sensitive or confidential personal information. For this purpose, we agree that the Commission and interested parties should start with review of the “Fair Information Practices Principles” and other national consumer privacy laws and guidelines, and evaluate whether enhancements or improvements in current Commission and utility practices should be considered in light of those national guidelines. PG&E recommends that this “benchmarking” effort be an integral element of the utilities’ SB 17 deployment plans.

My take on this is that PG&E is indeed committed to not only dealing with security issues head on, but is also sensitive to the concerns of the various agencies who are voicing their concerns. Moreover, they are willing to voice their commitment directly with the CPUC and essentially tell the CPUC that the ball is in their court.

I look forward to other key players in Smart Grid coming forward with this level of commitment! PG&E has been a leader in Smart Grid since day one, and leadership is what drives excellence.

I anxiously await the CPUC's decision on this.

Smart Grid Security and the CPUC

2010-03-19T08:43:00.000-07:00

I am sure everyone can cite an example in life where various entities (including oneself) are forced into accepting responsibilities they may or may not be prepared to accept. This generally comes about in a somewhat organic manner, and sometimes hits a point where the duty of care rises in a nearly exponential manner, which creates a situation where all activity is then reactive in nature. This is not necessarily the best position to be in, but it happens quite frequently.

An example of this would be the extremely rapid growth of the automotive age in the USA. When automobiles first appeared on the scene, they were indeed a novelty, and certainly nothing worthy of considerable regulatory control. They shared the road with pedestrians, horses and buggies, and bicycles, and as I understand it were quite amicable about it (for the most part). Automobiles were built according to manufacturer specifications which were fully controlled by the manufacturers, and they were generally built with aesthetics, functionality, and profits as the key drivers.

As we all know, the automobile industry quickly made the transition from novelty to way of life within the blink of an eye. It did not take long until the automotive industry become the cornerstone of American manufacturing, and consequently an enormous influencer of economic dominance.

In other words, it got very damn big really damn fast.

As this growth progressed, it soon became apparent that we had to make major changes in the way we, as Americans, did things. It was no longer prudent to assume that everyone would share the road in a somewhat Utopian fashion. While automobiles created ENORMOUS benefits for society, they also introduced SIGNIFICANT challenging issues, many of which were indeed quite dangerous. I am certain I do not need to articulate all of these, since we are all quite aware of the many dangers associated with an anarchistic automotive culture, especially since we are all quite aware of the significant dangers we all face in an automotive society where most people do indeed (by and large) follow the rules. This is quite evident when a solitary driver loses control of a vehicle on a busy highway and causes massive multi-car collisions. It is truly amazing that we all manage to avoid such chaos as well as we do, but we do indeed manage.

...and why is that?

Well, it is because we have learned to do so the hard way. Most of the rules of the automotive ecosystem have come about as a direct result of lots of people dying, or lots of people being forced to live in a society where quality of life is negatively impacted. Seat belts, for example, where not always required in cars. In the early days of seat belts they were an option. They appeared in cars in the early 1900's, but were not required in US cars until the 1970's. In fact, most of the safety standards that exist today for the automotive industry did not appear until the 1970's, and NOT because the US Automotive industry decided it was a good idea, but mostly as a result of the crusades of the venerable Ralph Nader. Seat belts became standard equipment as a direct result of Federal legislation mandating that all carmakers include them. However, it was not until laws were enacted mandating their use by passengers that we began realizing a decline in traffic related deaths. To quote a section of an article from The Prevention Institute:

Mandatory laws have proved effective both in increasing seatbelt usage and decreasing traffic fatalities. The Centers for Disease Control and Prevention reports that seatbelt use nationwide increased from 11% in 1981 to 68% in 1997. NHTSA reports that the motor vehicle fatality rate as measured per 100,000 population decreased from 21.49 in 1981 to 15.69 in 1997, and also decreased as measured by 100 million vehicle miles traveled, from 3.2 in 1981 to 1.6 in 1997. While these decreases cannot be attributed to the use of seatbelts alone, seatbelts are credited with playing a significant role in these advancements.

Enter California. Being the state with both the largest population of people, the largest economy, and the most cars, California is always a good place to go for significant statistical samplings. California is somewhat notoriously well know for being a first mover on many initiatives that tend to serve the public interest over corporate interests, often at the behest of corporate interests. This, as it turns out, is a very good thing more often than not. When the rest of the USA was (and still is) debating the efficacy of environmental controls, and what should and should not be enforced to preserve our environment, California simply forged ahead and passed what many industry leaders deemed punitive measures to prevent (among other things) automobiles that emit large amounts of pollutants. Because the State of California represents the largest customer base for car manufacturers (1 in 11 cars is in California), the automotive industry was left with two choices. Either they could conform to California emissions laws, or they could find another less stringent customer base.

We all know which way they went, and now that I live in California I am extremely thankful. Anyone who has lived in California's most populous areas knows darn well that there are A LOT of cars here and they are on the road all the time. Prior to the enactment of state mandated emissions standards California regularly had days where the pollution was so severe that people were warned to stay indoors with their windows shut. While California does indeed continue to face air quality challenges in populous areas, the attention this has gotten at the state level has resulted in significant improvements in air quality. Perhaps most importantly, the significant changes automotive manufacturers had to make to their products in order to do business in California has led to a fundamental change in manufacturing which the entire US (and the world) now benefits from.

Let's go back to seat belts for a moment. Despite the mandate that seat belts must be worn when driving, compliance increases only as enforcement increases. This is true with nearly every rule of society, and certainly with rules where the manifestation of ignoring the rule rarely leads to any negative consequences. In other words, since most people who do not were a seatbelt when driving do not experience any negative consequence from not wearing one (they normally do not die or bash their heads against the dashboard or steering wheel), it is quite easy to forego this preventative measure. Compliance with seat belt laws got better as police officers began issuing citations to those who failed to use them. This generally came about as a result of officers pulling over drivers for a non-seatbelt related violation, and issuing a citation for not wearing a seat belt in addition to whatever they had been pulled over for. This is known as a "Secondary Enforcement" law. California, however, is a "Primary Enforcement" state with respect to seat belt laws. In California, an officer can pull a passenger over for the sole reason of failure to wear a seat belt, and issue a citation accordingly. As controversial as this law has become, it has led to a SIGNIFICANT decrease in fatalities. From the same article cited earlier:

California, a primary enforcement state, currently reports 91% usage -- the highest in the country. After the passage of a mandatory seatbelt law in 1986, California's usage rate went from 26% to approximately 45%. By 1992, California's usage had increased to 71%. With the passage of the primary enforcement law in 1993, California's usage rate jumped to 83%, steadily climbing to the current rate. According to the National Safety Council, California's fatality rate has decreased by over 34% since the passage of the primary enforcement law.

Wow! A decrease in fatality rate by over 34%. I would say that is pretty darn significant. I would also say that it is probably NOT all solely due to the primary enforcement, since California also spends a significant amount on programs to continually educate the population on the importance of safety.

As a staunch libertarian by nature (notice the small "l") I generally oppose government intervention. In fact, prior to moving to California I was convinced I was going to find the imposition of such a notoriously intrusive set of rules for my well being to be intolerable. In fact, however, I find it quite nice. I was a smoker when I moved to California, but the nearly militant anti-smoking sentiment coupled with the heavy handed enforcement of anti-smoking laws found throughout the state have forced me to rethink my addiction, and led to a lifestyle which I find far more appealing, since hacking my lungs out every morning was not something I looked forward to every day. I am now living a smoke free life for going on 7 years! The FACT is that smoking is a horribly dangerous health hazard that I am better off not taking part in. Surely I can still buy cigarettes in California (and many millions still do), and there are plenty of places that I could smoke them, but there are plenty more places where I cannot, and those places (such as restaurants) also smell like food, flowers, fresh air, and everything else except stale tobacco smoke residue. In fact, I am often taken aback by the smell of tobacco when I do encounter it today, since it is such a rarity. I am nearly overwhelmed when I go to a state/country where smoking is prevalent, and literally smile from ear to ear when I return home to California and its tobacco averse culture.

So this (finally) brings me to the smart grid, and specifically smart grid security. California, being the typical first mover in nearly all things technology related in the USA is now rapidly deploying a smart grid. As I understand it, we are now approximately 50% rolled out with our AMI products (smart meters and such), and are continuing to move forward. This all began with PG&E at around the year 2000, and has been joined by (among others) SCE. Being at the front line, PG&E has had the dubious pleasure of being the first to experience the challenges of a smart grid rollout, and has had to take action to fix issues as they arose. Security challenges identified in the early days of deployment were certainly not nearly as prevalent as they are today. Security challenges tend to rise as deployments of technology expand (for various reasons), and we learn as we go. Generally the fixes we put in place are reactive in nature (we discover an exploit and fix it), with a more proactive approach to security arising out of parallels that can be drawn by examining proof of concept exploits. Because of our growing understanding of security challenges, both PG&E and SCE have taken a VERY proactive approach to addressing security challenges in the smart grid, and have expended significant resources in cyber security. PG&E has a very competent cyber security team working very hard at addressing these issues, and SCE has teamed up with meter manufacturer Itron to implement an entire AMI solution with security being a major focal point. In fact, Itron has emerged as a leader in AMI security space as a result of this partnership. I applaud this extraordinarily proactive approach taken by both PG&E and SCE, and am quite certain that this show of leadership will serve as a template for the entire US to follow.

...yet this does not address some significant issues.

As it turns out, each and every organization involved in addressing cyber security as it relates to AMI is operating within a walled environment. I am not referring to the higher level issues, which are being addressed by NERC, FERC, NIST, DHS, DOD, DOE, and MANY OTHERS, but specifically at the application level (where the rubber meets the road). In other words, the vendors making the products that go into the grid are all implementing security as they see fit (based on a collection of "best practices"). In the case of SCE, senior director Paul DeMartini told me (at the CPUC public hearing on March 18th, 2010) that SCE insisted that Itron implement security as a requirement. Being a large customer for Itron, this was quite an incentive to move forward with security at the application level. Yet Itron is not the only vendor in the AMI space (although they are perhaps the largest single meter manufacturer). There are MANY other vendors, and quite a number of them have a significant presence. More importantly, each and every component that all of these vendors make for the grid are all subject to security challenges (some more than others, of course), and all make up a part of the "security chain". A chain, as we all know, is only as strong as the weakest link. While it may indeed be both reasonable and fair to assume that some (if not most) of these AMI products have addressed security in a manner that adequately creates a strong link, it is entirely imprudent to assume that ALL links are adequately strong enough.

So what have we done with respect to security at this application level? Well, we are working on putting together some national standards, auditing, and enforcement policies (NIST,DHS,FERC,NERC, etc.), but we are still quite a way off from finalization. One can surmise that once the Federal rules are agreed on (which is a significant challenge in and of itself), it will take quite a bit of time before enforcement has any significant positive impact. Let's face it, seatbelt laws were first enacted in the 1970's, but enforcement of such laws did not have any significant presence (or impact) for DECADES. The same holds true for EPA laws, wherein California EPA laws still set the high bar for standards, and in fact trump national laws because they are so much more restrictive, and many states simply live under less environmentally friendly conditions.

California, however, must act in a more proactive manner simply because the choices California makes have such a massive impact on so many people. With 36 million people (as of July 2009) in California alone, bad choices (or simply inaction) affects a huge number of people simply within our borders. When you consider, however, that California is the 8th largest economy IN THE WORLD, the choices California makes has a much greater global impact. Did you know, for example, that California produces 12.8 % of ALL agricultural products in the US. Surprisingly, we manage to do this with less than 4% of our nation's farms and ranches (talk about efficiency). Couple this with what California produces for the health care industry (drugs, medical devices, systems, etc.), for our defense industry, and for our financial system and it quickly becomes apparent that California is not just one of the 50 states. It is THE STATE our global existence is most reliant on.

So when things go wrong in California, things go wrong in lots of places. California is no stranger to things going wrong as a result of state level bad decisions and inaction. Perhaps the most recent failure, due to bad security related decisions, was with electronic voting machines.

Do you remember that nightmare?

Let me refresh all of our memories for a moment. California decided that getting rid of paper based voting systems was a good idea back in the early part of this decade (for various reasons), and this led to an enormous groundswell of activity among several companies to create electronic voting machines that would help California get rid of the tyranny of paper, and consequently bring enormous amounts of money to the manufacturers of such systems. One thing led to another, and California ended up spending billions of dollars on electronic voting machines, and so followed the rest of the nation. However, California failed to adequately audit the security of such systems, and consequently the security was audited by hackers and independent security professionals after they were in place. As it turned out, the security flaws were so significant that nearly all electronic voting systems ended up being trashed, both in California and on a national level. The cost of this failure was, of course, borne by the taxpayers. As it turns out, the machine manufacturers were not held liable for these flaws because the systems were in fact CERTIFIED by California (and other states) and given the seal of approval. Simply put, it was The State of California's fault for failing to perform due diligence as far as security was concerned. Moreover, I had the dubious pleasure of working with several voting machine manufacturers after the fact to try to help them fix these problems, and as it turns out some of them had indeed addressed these issues far more adequately than suspected by those who chose to vilify them, but simply did not include better security features in their systems because the customers (i.e. The State of California) simply would not pay for them! California simply chose to inquire about what security features existed, signed off on the agreements without adequately auditing the security, and the rest is history.

As we move forward we learn hard lessons, and hopefully get better because of the lessons we learn. California recently received a big chunk of Federal stimulus money to implement electronic health records (EHR's), and one of the provisions from the Federal government is that the implementation must include security as prescribed by the HITECH act. Since there are few specific Federal guidelines in place AT THE APPLICATION LEVEL, the California Office of Health Information Integrity (CalOHII) has taken the initiative in creating a security committee and has drafted a set of security guidelines AT THE STATE LEVEL. I have had the pleasure of contributing to discussions with this committee (and have indeed been invited to participate, and have agreed to do so), and one of the main reasons why California has taken a state level initiative in security at the application level is because of the enormously critical nature of this initiative, and the enormous cost of failure of a lack of adequate security. Imagine, if you will, a hacker having the ability to alter a medical record, and imagine a malicious reason for doing so, such as altering a record of an enemy to indicate that he or she is not allergic to penicillin (for example), which could lead to death in the event he or she is given penicillin without prior knowledge. A lack of security here is indeed a life or death problem.

So this finally brings me to the California Public Utility Commission (the CPUC), who is now faced with some pretty tough decisions in light of the fact that the entire power infrastructure of the 8th largest economy is potentially vulnerable to significant cybersecurity related attacks (which could effectively shut down our power generation/distribution systems). The CPUC is in place to serve the public interest first and foremost. The vendors in the AMI space are there to serve corporate interests first and foremost. So much so, in fact, that the most significant vendors in the AMI space live behind a rock solid wall of NDA's and refuse to discuss security architectures and applications in any way even resembling a transparent and collaborative nature. I can certainly understand this from a competitive perspective since I too must adhere to such NDA's (and indeed do adhere to them), yet this forces us to live under an environment of corporate self regulation, which, as well all know, does not always seek to serve the public interest in an adequate manner. Again, as a libertarian I am okay with this as a basis for capitalist endeavors, but I am less okay with this for matters of life and death, and to a somewhat lesser degree (perhaps) for matters where I am forced to bear the cost of failure.

Who do you think is going to pay for the parts of the smart grid that must be scrapped in the event of a security failure? I can tell you with certainty that the US taxpayers did indeed bear an enormous amount of the cost of a failed electronic voting system. I can tell you that the California rate payers are bearing the cost of the AMI rollout (either directly or through Federal taxes). If it fails to deliver what we expect it to deliver (security related or otherwise) we cannot simply scrap it and go back to the way it was in the old days without a SIGNIFICANT cost (if that is even on the table). Moreover, if we are forced to replace vendor products due to security flaws, does it not strike you as somewhat interesting that the vendors may in fact directly benefit from the "double dip" nature of this scenario?

In a conversation I had with Aloke Gupta, who is a Senior Energy Analyst with the CPUC and is currently working on energy policy, he informed me that the CPUC has traditionally not been in the "verification business" with respect to public utility deployments. Fair enough, but this does not mean that they shouldn't be. Because the public simply has no choice whatsoever with respect to smart grid deployment in California (or elsewhere, for that matter), we must now rely on security choices which are being made by corporations who are tasked with (as a matter of legal due diligence to stockholders) maximize their bottom lines. The cost of secure components and design simply goes up the more you improve it, and the return on investment is nearly impossible to realize. In fact, the best security tends to completely obfuscate ROI. If there is no security failure, how can one know how much security measures helped? So the public simply cannot hope that the security choices a vendor makes are going to primarily center on what is best for the public interest. It simply does not translate to a bigger bottom line UNLESS everyone must comply with a set of enforced standards, which levels the playing field, and prevents loss of market share due to competition with another vendor who decides to take the "cheap" way out. Ideally, the deployment of the smart grid would have occurred only after security standards, auditing policies, and enforcement procedures were in place, but that did not happen, and may not happen for quite some time. From a national perspective, one can make the argument that with the US being approximately 5% rolled out with AMI, things are moving along at a reasonable pace at the Federal level. However, when you consider that fact that most of that rollout is in California, it now quickly becomes apparent that it is incumbent upon the CPUC to take firm and decisive action well in advance of national standards.

After all, they are here to serve the public.

The Smart Grid Privacy Smoke Screen

2010-03-13T06:52:00.001-08:00

Whenever I watch news on network media I view everything being said with quite a bit of cynicism. Heck! Security professionals are NOTORIOUSLY cynical. The security professional mindset is designed to quickly wade through layers of what can be seen on the surface and find that which cannot be seen, which tends to tell THE REAL STORY.

Back to the news for a moment. When I see a major topic wrapped with lots of sensationalistic coverage splattered all over the airwaves and news sources, I immediately ask "Okay, what is REALLY going on." Why is everyone talking about who does or does not have the right to use the word "retard" (for example). What is the real agenda, or what are they trying to prevent us from paying attention to.

I know it may sound conspiratorial, but I see this a lot with security, and I assume it happens everywhere else.

Let us discuss security for a moment.

There are some things in the world of security that are complex, and some that are not so complex. There are good ways to protect systems using low cost, medium cost, and high cost components and procedures. When making a determination about what is the best choice (from a financial perspective) organizations that must implement security must always balance the risks with the costs. This is simply how it is done. Many of the risks associated with security are driven by compliance. If an organization does not comply with "the rules", they can be held liable for a failure to perform due diligence. This is, by far, the biggest driver (and headache) for any organization. Security generates no ROI in this case, it simply acts as insurance. Nobody I know likes to pay their insurance premiums, but they all must.

The other way security ends up in systems is when it has been attacked. Generally the more significant the attack (i.e. the more costly the attack), the better the security solution. I do not want to spend too much time on this particular topic, but it does warrant a mention. This is the holy grail of security professionals, by the way.

In cases where security becomes a topic of discussion, and consequently a major bone of contention among vendors who are subject to security mandates, what frequently happens is that the conversation takes a direction that serves the lowest common denominator. Rather than talk about the "real" issues, we tend to talk about issues that seem to be of utmost concern, but really do not matter nearly as much as the "real" issues. This is often because the more important issues are quite a bit more complex (and consequently more costly) to deal with. By shifting the focus to the less complex issues, organizations tend to appear as if they are solving a problem (and consequently performing due diligence), but they are actually avoiding the bigger issues.

For the last several days I have been reading through piles of comments submitted to the California Public Utilities Commission (CPUC) regarding Smart Grid deployment. Within these documents there are quite a few comments regarding Smart Grid security, but the overwhelming language talks about security as it relates to privacy (i.e protection of consumer usage information).

Okay, I do indeed believe privacy is important, and hold it near and dear. California was one of the first states to enact privacy laws, and has definitely led the pack in this arena. I definitely get it. Privacy is indeed important.

Sadly, however, it is a smoke screen. The focus on privacy takes our focus off of the real security challenges we face as we deploy the Smart Grid. Privacy, as it turns out, is not as challenging an issue as preventing large scale attacks of the Smart Grid which could theoretically bring down large SCADA systems. Why do I believe this? Because it simply does not have the WOW effect from a hacker community (and media) perspective. You see, EVERYTHING that is computer/network/system related can be hacked at some point. In an ideal world, the good guys try to keep ahead of the bad guys. The bad guys are always working on taking down what the good guys have built, and the most interesting things to take down are the ones which have the most impact. Hacking my meter (or any one's meter) to see how much power I use just does not get you very much attention these days in the world of hacking. Taking down a generator, however, does.

So as I read through countless pontifications about how crucial it is to ensure our privacy, and consider the extraordinarily low risk of a breach of privacy causing our lives to change in any considerable way (let's face it, how many of us truly feel we have any privacy these days?), I cannot help but think about what an effective smoke screen this is when we consider Smart Grid security. NISTIR 7628 is fully aware of where privacy sits on the scale of things to watch out for, and the February 2010 draft clearly points this out, listing privacy as a tertiary concern as it relates to security.

Yet the public comments floating around the CPUC seem to indicate that privacy is "what it is all about". I certainly do NOT see any discussions of any value indicating otherwise. Nearly every security professional I have spoken to about Smart Grid security finds this focus a bit absurd in light of both the know (non-theoretical) and assumed (theoretical) security dangers.

I think the public should consider this as they strive to educate themselves about security and the Smart Grid.

The Cyber Warrior Mentality - The Security Warrior

2010-02-24T05:53:00.000-08:00

“The basic difference between an ordinary man and a warrior is that a warrior takes everything as a challenge while an ordinary man takes everything either as a blessing or a curse.”

-Carlos Casteneda, American author 1925-1998

I have been thinking about the warrior mentality a lot lately. It started several weeks ago when someone I was speaking to (yes you, Stewart) about cyber security referred to something I said as being indicative of having a warrior mentality. It struck me as interesting because my business partner talks about having a warrior mentality a lot, and as I had this discussion I was more than a little taken aback by the uncanny parallels between myself, my business partner, and this complete stranger I was discussing security with. Partway through our conversation I began predicting what he was going to say, based on my understanding of the situation, and it was dead on every time.

It was like he was reading my mind.

Yet this was not what I found strangest of all. As I began "gathering intelligence" in my attempt to better understand the vendor space in the cyber security landscape (needs, requirements, activities) as it relates to The Smart Grid, I consistently ran into two distinct types of people. One was the more marketing oriented type, who simply discussed security in a manner that was indeed befitting of the vendor (a security apologist if you will), and the other was the security contempory - or the "Security Warrior" as I now like to call it.

Okay, I know this may sound odd to some, but for those who fit into the category I am sure it makes perfect sense.

As a security professional who began his security career as an administrator who was thrown into the battle due to outside attacks on the company network, I was charged with fixing the problem, and I was given very few tools (and even less time) to do so. My boss did not want to hear anything about expensive firewall hardware, or outside consulting, or anything like that. I was in charge of IT, so it was my job to fix the problem, and to do so within the confines of the limited budget I had available to me.

Oddly enough, I did not view this directive with frustration or with disdain. I simply took it as my marching orders and did the best I could with it. I had been sent out to the jungle with a book of matches and a pocket knife, and it was my duty to survive with those tools, and my wits. Come to think of it, I loved it!

Having less to work with really makes some people think hard and "outside of the box". Not all people, however. Some people simply cannot cope with the situation, and give up. Others pretend that things are going to miraculously work out through some sort of cosmic intervention, and simply wait for things to change. Sometimes this inaction mentality works out for them, but it is not because of divine intervention (although I do believe in God, but that is another discussion), but it is often because someone else picks up the slack.

When given a limited toolset, the warrior does not fret. He (or she) simply takes inventory, and then begins studying the enemy, beginning with the enemy within. Fear, shame, guilt, doubt, and other such feelings and mental states are identified for what they are and dealt with promptly and effectively. The warrior studies the landscape and determines where the danger zones lie at every given moment (because they are always changing), and what to do to stay out of danger. The warrior immediately determines what threats are real, what threats are not real (but are actually more perceptions than real threats), and what threats may come, and prepares accordingly. If the threats come from other people (the biggest threat of all), then the warrior does all he can to study the perceived enemy to determine both the level of the threat and the mental state of the potential enemy. If the warrior determines that the enemy is indeed real, he does NOT rush to kill the enemy. The warrior then studies the enemy and determines if the enemy himself is indeed a true warrior as well.

THIS IS IMPORTANT !!!!

The most effective players in the cyber battle are those with a warrior mentality, ON BOTH SIDES OF THE BATTLE. A warrior views the most effective enemy as a CONTEMPORARY, who is fully capable of being just as strategic and calculating as he is (if not more so). A warrior will watch his enemy take control of a battle and marvel at the strategic nature of the enemy with deference, and then file that away as another tool in the arsenal. He may never use this tool, but he will understand it enough to know when it is being used (or about to be used) again, and will know exactly what to do to either prevent it from being effective, or prevent the enemy from using it to begin with.

Let me return back to Earth for a moment and discuss my first major security breach as a CIO. I was in charge of a medium sized retail operation, with lots of remote retail locations logging the point of sale system into our corporate serves to perform transactions (using the Remote Desktop Protocol). It was an elegant solution that worked marvelously...most of the time. One afternoon I began receiving lots of complaints from the store managers because the system had slowed to a crawl. I asked one of my team members to look into it, and he found nothing out of the ordinary, but did indeed notice a spike in outgoing traffic that seemed to be sustained. Outgoing traffic spiking was not unusual, but sustained spikes were indeed out of the ordinary.

As I shifted to the hands-on approach, I noticed the traffic was on one particular port (I do not remember the port number), and it was not one of our known ports (21, 80, etc.), so I knew something was up. I was completely fascinated at this point (although I knew someone uninvited had entered our network), and began investigating. I narrowed the culprit down to one particular server, and a careful study of the server logs revealed activity going on in the recycle bin.

Huh!!!

As it turns out, someone from Korea (I traced the IP address to Korea) had installed an FTP server in the recycle bin on the Windows server, and was serving pirated Hollywood movies from my network. ABSOLUTELY BRILLIANT !!!

I was literally tickled pink by this feat of trickery. Why had I never thought of that back in my early days of "file sharing" cat and mouse gamesmanship? Touché indeed my Korean enemy. Well played!

Of course, I knew simply getting rid of the server was not the solution to my problem. It simply treated the symptoms (slow traffic). I had to then discover the weakness in my network "armor" that had allowed the infiltration to begin with. As it turns out, it was one of the many recently patched security holes common to Windows based systems (at the time), and I had been hit prior to the patch. I fixed the patch, and then made it my mission to very carefully monitor system changes on a very granular level, which led me to the discovery that attacks on my network were happening on a very regular basis (port probing, hammering, etc.). It allowed me to study and learn my enemies' tactics, and I soon discovered that there were a lot more attempts than there were victories. Yet when the victories happened (and they did indeed happen), I learned how to stop them, and they did not happen in the same way ever again.

You see, dear reader, I placed an ENORMOUS value on the victories, because they exposed my weaknesses. One cannot effectively determine a correct strategy unless one clearly understands their weaknesses. However, a warrior who has the ability to swallow his pride can significantly reduce the number of victories the enemy has if the warrior is willing to take a step back and enlist the advice of contemporaries who have already lived through the battles, and especially the ones who have the battle scars to prove it.

As I speak to more of the people who have enlisted in the battlefront in the cyber war, I consistently run into those who have been forced to deal with security as an additional headache to deal with, and those who have a true warrior mentality. What I have found is that warriors are very good at spotting other warriors, and can usually do so almost immediately. The conversation, at this point, takes a completely different tone. Even though I am coming to them (in part) as a consultant who is trying to win brownie points with my client, who is trying to determine market opportunities (after all, everyone needs to pay bills), we immediately move past that as the discussion now becomes far more temporal. We begin discussing the evolving threat landscape, the strategic nature of the environment, and the tenacity of the enemy. We laugh heartily (yet respectfully) at the hyperbole, and focus on the true threats to our mission, which we often determine, in part, come from our side of the battle (cost constraints, time sensitivity, corporate politics, lack of transparency, etc.). We sometimes venture off into discussions that have nothing to do with our current positions, and recount tales of battles past, and battles yet to come.

It is, I surmise, much like soldiers getting together on leave or after a war. Not having been an actual soldier I cannot say this from an experiential viewpoint, but I have known enough soldiers and watched them interact with each other to know that the similarities are indeed valid. I can tell you that it is also much like parents discussing the battles and victories in raising children, which is something I am indeed achingly (and pleasantly) familiar with.

So this brings me around to a very positive understanding of the battle from a US Government/Military being involved perspective. We often hear that the US Government is not very good at dealing with cyber security (and they have indeed admitted their shortcomings in that space), and that we are in big trouble because of that. I would tend to agree with such fear mongering if it wasn't for the indisputable fact that the US Government is arguably the best IN THE WORLD at gathering warriors to fight its battles. The US Military has spent literally hundreds of years perfecting the art of war, and despite their lack of understanding of the artillery (bits, bytes, laptops) or the enemy tactics (hacks) in the ever evolving battlefront, they are certainly quite good at understanding the warrior mentality. That is why, for example, I believe we are well served by having Michael Assante serving as the head of cyber security for NERC. His job is to separate what is "real" from what is "not real", and to manage the ever evolving battle plan, and make sure that every soldier (i.e vendor, utility, etc.) in the battle is following the plan. Nearly everyone I have spoken to that falls under what I deem the "Security Warrior" mentality finds Mr. Assante's assertions to be dead on. Mr. Assante, by the way, served in the US Navy for 6 years.

Some may argue that our US Military is nothing to be proud of, but I think they are missing the point if they do. We do not win every military battle we fight it, nor are we always justified in what we do militarily (which is subject to opinion), but we definitely are very effective the majority of the time. Moreover, we have certainly succeeded to the point where Americans find the prospect of domestic battle so foreign that something horrific (like the 911 incident) is seemingly incomprehensible. Yet foreign countries throughout the world deal with their own versions of the 911 incident every day. Our combined defenses and offenses have indeed managed to generate some ill will both foreign and domestic, but they have also managed to make us not only feel safe, but actually BE SAFE!

So I am hopeful and indeed confident that as we engage this cyber war with more of a warrior mentality that we will indeed manage to both survive and indeed thrive and feel quite safe as the battle evolves. We will engage the enemy, learn from the enemy, and indeed prove triumphant if we all swallow our self-serving interests long enough to take some cues from those who have received the battle scars.

...and frankly, I am excited about the opportunity to be a part of it.

The Evolving Compliance Landscape Of Cyber Security

2010-02-22T09:54:00.000-08:00

Security, as it turns out, is largely about compliance. Anyone who has spent any significant amount of time working in security knows this all too well. Years ago my business partner and I worked on a very sophisticated health care project which involved cryptographically authenticated peripherals. The business model was such that a peripheral attached to a device was to be used once (and only once) on a patient, and then discarded. The doctor would then have to buy another (or have a stock of more peripherals) for another treatment.

We were brought in because the first generation product used a weak security solution, and it was hacked and counterfeited in 3 weeks (3 weeks from launch to counterfeits on the market)!

Bye bye business model.

So, having miserably lost the first battle in their own little cyber war, they decided it was a good idea to bring out the big guns, and they created a security solutions team (which essentially consisted of me, my partner, and the chipmaker we were working with) and we put together a very secure solution. It cost them more than their first solution, but they knew all too well the cost of failure, so it was easy for them to justify the increased expenditure.

Wouldn't it be nice if all security engagements worked out that way? Certainly for the security provider, I suppose.

We began focusing on health care security, feeling empowered by our previous engagement, and soon discovered that it was not a easy as we suspected. We touted the battle scars of our client as an indicator of the need to securitize their products, but failed in our attempts to generate revenue. It was frustrating, to say the least.

In all this, I learned quite a bit about the health care industry, and soon realized that everything in health care is compliance driven. Since the extent of security requirements for health care providers essentially falls under HIPAA regulations, all a health care organization is interested in doing is complying with HIPAA. Doing so essentially requires a security policy, and something as simple (and low tech) as requiring a 4 digit PIN to enter a system. It certainly is not something my company could sell into, since most of what is required by the client falls under the security audit, which is generally handled by the IT team medical clients already have. Since the requirements are so non-stringent, this is usually a 10-15 minute conversation.

Do you think I am exaggerating? I assure you I am not. One of the "side" jobs my company has been focusing on is iPhone applications for health care, and I can assure you that when the subject of HIPAA compliance comes up it NEVER lasts more than 15 minutes, and usually ends up with an agreement to enforce a 4 digit PIN (and a few other minor security additions). As developers who have worked in security development for quite some time, we do indeed build our software with an eye on security from the beginning, and our clients do indeed get a lot of security "freebies" because of that, but it is not because we are compelled to do so by any forces outside of our own need to not fall on our own swords as security professionals. In other words, we have chosen to self regulate our process, and our clients benefit from that.

While this is all good, and certainly makes us feel like we are doing the right thing from a due diligence perspective, it is only a small dent in the underlying battle. We know this from even further forays into working with cyber security challenges. The most interesting, perhaps, is with voting machines. Despite the bad press voting machine vendors received after many security professionals discovered gaping security flaws, they were indeed complying with requirements set forth by the election commissions in the states they sold into. In fact, the states themselves "ate" the cost of the insecure machines because of this, while the vendors got the black eye. Some vendors we worked with were indeed QUITE aware of how to build secure voting systems, and told us that such systems were unsellable because the states simply did not want to pay for them. All the vendors had to do was comply and then offer the best ROI to the clients in order to win the bid, and they did just that. It was not until after the security exploits were discovered that the US Elections Assistance Commission began taking security seriously.

Imagine that!

That does not mean that security was not considered...it certainly was. It is just that the deployment of security was weakened by a low threshold for compliance. What is even more important to realize is that the threshold is nearly impossible to determine without a proof of concept (moving from theory to reality).

Fast forward to cyber security, and specifically as it relates to the Smart Grid. Early deployments on the smart grid did indeed include requirements for security, and ALL vendors took it quite seriously (some more than others, as it appears). But this was not necessarily due to compliance issues, it was because (as one utility security expert put it), no utility company in their right mind is going to deploy something that is not secure.

...and yet, we now know that the grid has some fairly major security issues. How can this be?

Well, we have to consider security in the context of the perceived threat. For example, I can easily be shot while walking down the street of the bad part of town in any city, but that does not prompt me to invest in body armor (or even a bulletproof vest). Body armor is expensive and not very comfortable to wear, and since I have never been shot at while walking in the bad part of town, I am more than willing to rely on simpler and more cost effective security solutions (such as perhaps walking on better lit streets) to keep me safe. Moreover, I may very well get shot at and STILL decide not to get body armor. It simply takes a certain level of perceived danger for anyone to elevate their security requirements, and we really do not know what that level is until it happens.

So we stand in the presence of a smart grid deployment that is going pretty strong in the USA (on the order of 10's of millions of meters), and we have not borne witness to any major catastrophes yet. We have indeed proven that the threat is very real, and we are now working towards lowering the risk. The Department of Homeland Security, NIST, and NERC have enlisted the public and private sectors in the activities (and I have indeed joined in on the fun), and there are many smart people working on the challenges at every level you can imagine. Michael Assante of NERC co-authored an EXCELLENT article in the January/February 2010 issue of IEEE Security and Privacy magazine titled "No Grid Left Behind", and he methodically lays out the challenges and proposed solutions, and everything in the article is quite cogent. I have personally spoken to MANY members of the security community ranging from vendors, utility companies, PhD Scientists, meter manufacturers, crypto algorithm providers, and everyone in between. Everyone is working hard on the project.

Yet we have to understand that despite all the efforts to win the war (something that will never happen, as the war will never end) and prevent casualties, we are not going to come through this unscathed. Smart Grid deployments are vital to our existence because the energy savings have been proven. In a conversation I had with Echelon (a maker of AMI products) they have shown an energy savings on the order of 70% in some cases!!! That is a VERY significant number. When we consider the impact of energy savings on anywhere near that level, it certainly makes the case for Smart Grids a no brainer. I mean, think about that for a moment...saving energy by simply being smarter about where and when it is being used.

To me, as an energy consumer who spends over $500 per month to meet my energy needs (when you include fossil fuels), that hits home.

So what we have to understand that sometimes the missteps on the battlefront do indeed lead to things getting better. We have to understand that the fact that the various regulatory entities that are working towards solidifying and continuously evolving the standards (which vendors are indeed paying close attention to) are well aware that they are stakeholders in this ecology, and an insecure smart grid affects them on a very personal level. Michael Assante of NERC is a security whiz, but he is also well aware of the fact that the decisions he makes affect the outcome of this nation AND his personal life in a VERY profound way. Just like America came together to fight the enemy during World War II, after we had felt the attack at Pearl Harbor, we too can expect cooperation as we fight the ever evolving cyber security enemy.

After all, we are all on the front line.

Coordinating Efforts In Cyber Security

2010-02-15T05:14:00.000-08:00

I have truly never seen a security initiative quite as interesting and massive as the cyber security effort as it relates to Smart Grid. The more I peel back the layers, the more I discover how massive the effort is.

Just a few days ago I was invited to attend a meeting of the American Bar Association (yes, the lawyer group), and the entire meeting (2 full days) is packed with presentation after presentation and working groups dedicated to cyber security. Yes! You heard that right. Lawyers are interested in understanding cyber security on a very detailed level. Fascinating!

I have spoken to utility companies, meter manufacturers, chipmakers, HSM vendors, cypto stack vendors, regulatory bodies...the list goes on and on and on. Everyone is contributing something to the effort of fighting the current cyber war we are all involved in.

Oh...you do not think we are at war? Well please reconsider. We may indeed be at somewhat of a stalemate at times, or even in a cold war, but we are indeed at war. The enemy is constantly hard at work trying to find new ways to break into our vast cyber network, and on every level you can imagine. It ranges from Aunt Judy's Facebook page all the way to our Missile Defense system, and the attackers are RELENTLESS (and also have a lot of fun doing what they are doing).

But enough fear mongering for now. We, as humans, are completely used to being vulnerable. We fashion synthetic skins from cloth and the pelts of animals to protect ourselves from freezing to death. If we no longer had clothing WE WOULD DIE! OH MY GOD! WE MUST PROTECT OUR TEXTILE INDUSTRY!!!!

You see how ridiculous hyperbole can get...time to switch to decaf.

Being subject to attack is something we are well aware of and we have simply coordinated our efforts to a level where we all understand what it is going to take to protect ourselves from bad things. We invent penicillin to prevent ourselves from bacterial infections, and we coordinate efforts to get it into the hands of all who need it. We make vaccines to stop the spread of deadly flu. We organize our defenses to prevent the bad guys from breaking down our walls and fortresses.

We all have a pretty good idea of how and when to coordinate efforts to keep ourselves protected, and we make improvements along the way. That is what we do.

Fast forward to Smart Grid security. The exploding cyber security industry as it relates to Smart Grid seemed to come out of nowhere. Sure, I was talking about it 2 years ago with some vendors, but it wasn't until quite recently that it became an area of more intense focus with nearly every technology company in existence (and those who are not in the game yet will be in soon). I have to say that I began seeing more activity happen in this area after the 60 Minutes episode on cyber security, where they showed a transformer (I believe it was a transformer) being destroyed remotely by a simulated cyber attack. Even I, who has worked in security long enough to know that there are A LOT of unresolved security issues in this world, was taken aback by this proof of concept.

Nonetheless, the troops went into action. We are now seeing a massive land grab as everyone in the world of security reaches for their piece of the pie, and many come up with big handfuls of job security as a result. I was talking to a cyber security engineer a few days ago who told me that he was out of work 2 years before landing a sweet job working for a utility. My own company was stagnating for the last 2 years as well, despite having worked on some very large security projects. In fact, we started writing iPhone applications as a side job, and were brought back into security because of Smart Grid. It is a huge project with a dire need for security expertise on EVERY level imaginable. Definitely some promising times for the cyber warriors of the world.

Yet I see some issues popping up that I believe are indeed quite counterintuitive to meeting this war with the extreme sense of urgency it deserves. One clear obstacle to cyber security excellence is our woefully luddite government (in the USA). They make no bones about the fact that they are bordering on the dark ages where it comes to providing expertise on cyber security. They are damn good at providing expertise on blowing things up, rebuilding them, and blowing things up again, but the only asset the government can contribute to the cyber security effort at this point is $$$$$$$...and they are...billions of $$$$ in fact.

Where the government becomes a bit of an issue and perhaps a bit of a hindrance is in the effort to mandate standards for cyber security. This is where things begin to get a bit annoying. Although NIST is hard at work at finalizing NISTIR 7628, and NERC is hard at work building a nice compendium of auditing documents for cyber security, and the CPUC is hard at work scoping their Smart Grid security initiatives, nobody can tell you what the COMPLETE rules of the game are at this point. Anyone who claims to know all the rules of the game is lying, because the rules are still being written as we speak.

So what rules are the players following in the absence of a coordinated set of rules from up above? Well, I can tell you that California is indeed following (to a degree) its own rules regarding privacy as it relates to Smart Grid (SB 1386), since California law requires privacy protection. I can tell you with certainty that there are a lot of VERY smart minds from the private sector working very hard to make sure that they can build the best security with the resources they are given, and that the resources are definitely getting better (albeit slowly). I can tell you that the guys I talked to at the major utility company are being funded by the utility to do some absolutely brilliant work, and that the cyber security engineer at one of the meter manufacturing companies that the utility sources meters from has some absolutely great ideas about what it would take to improve security. I can tell you that the utility guys would love to let their vendors know what they believe would be beneficial on the front lines.

Ahhhh...that is where things get a bit unraveled. Imaging my surprise when I spoke to the utility company cyber security guys and they told me they have never met or spoken to the cyber security architect at the meter company. What the ????

Lets get back to my original assertion. We are at war. Sure, we see no blood and body parts flying (ala Saving Private Ryan), but we must accept the fact that the warriors have, to a degree, laid down their rifles and booted up their laptops, and the pending cyber battles are going to take out a lot more troops than any rail gun ever could. While guerilla warfare tactics are indeed effective, a coordinated battle effort generally wins in the end. A mixture of both is perhaps the most effective, but only in the sense that the guerilla warriors are on the same page as the well regulated troops, AND (this is the big point) that we eliminate redundancy and activities that cause a regression in the efforts. If I, as an intelligence officer, decide that a great strategic tactic is to befriend an enemy and gather information, and manage to get a few cups of coffee, some cold beers, and perhaps a nice meal with the enemy in my effort to butter him up, it is rather annoying if the guerilla warrior decides to put a bullet in his head because he has a clear shot.

As a security professional who is interested in helping the cause, I constantly run into two very different scenarios. The first scenario is the fortress of silence and dismissiveness that many (but not all) of the players in cyber security put up when I try to get some information that would help me do my job better. The second is when I reach the guy (or gal) at the organization that truly cares about security and truly understands the need for teamwork, and appreciates those of us who have fought on the front lines of the ongoing security battle, who CLEARLY see the enemy for what he is...A PEER! No, I am not talking about the REAL enemy (the guy at the top of the food chain), I am talking about the security expert the guy at the top of the food chain has hired to do his bidding.

I am unsure of how coordinated the efforts of the bad guys are at this point. I am assuming it is fairly good because hackers seem to love collaborating (take BlackHat, for instance), and I am also assuming it is going to get better as the bad guys start getting better funding.

So are we going to take a cue from that...or are we going to continue setting up our own battle fronts as we wait from the orders from above?

As Goes California...

2010-02-09T07:46:00.000-08:00

With over 10 million smart meters deployed to date, California (as the pilot for the US) has essentially acted as the pioneer of Smart Grid "movement" in the US.

Being a pioneer is not a new thing for California. We can thank California for equal rights, organic food, and much more. We can also thank California for being the first state to enact laws guaranteeing the privacy of data for its constituents, with the formation of the California Office of Privacy Protection in the year 2000.

I can remember when it happened. I was a CIO of a retail company in California when some of the first reports of major violations started to hit the airwaves. The CEO of the company started asking some questions, and I thankfully had the answers he wanted to hear. I had long considered data privacy and important issue, so I took the extra small steps to make it happen. He was pleased, and I think I got a raise.

Interestingly enough, the protection of data has always been THE most important consideration for those who practice the art of IT Security. Information Technology really began as a means of securing financial data, and in the old days the IT department did not fall under the guidance of a CIO, but was a sub-department of the office of the CFO. Once paper was replaced by bits and bytes, protecting the bits and bytes became a very important job, and IT Security became a cottage industry.

Fast forward to the new game in town - Smart Grid Security. Things are a bit different now. While the protection of data is indeed still important, it is NOT the primary focus. Smart Grid Security is focused on making sure that security breeches do not cause the system to slow down or (most importantly) stop functioning altogether. It does not take a lot of thought to understand why this is the way it is. Someone knowing how much electricity Mr. Jones is using and for what is not nearly as devastating as someone having the ability to shut Mr. Jones' electricity off. So one would surmise that the focus of vendors should be on availability and sustainability, and the rules of the game (i.e NISTIR-7628) certainly seem to point that way...

...but wait a minute. Let's examine NISTIR-7628 for a moment. This is where things start to get a bit interesting.

The September 2009 draft of NISTIR-7628, in section 3.2, discusses impact levels of 3 areas where security is an issue. From the document:

3.2 IMPACT LEVELS

The IAC impact levels are low, moderate and high. The levels are defined in Federal Information Processing Standards (FIPS) Publication 199, Standards for Security Categorization of Federal Information and Information Systems, February 2004. Following are the definitions for confidentiality, integrity and availability, as defined in statute and a table that defines low, moderate, and high impact.

CONFIDENTIALITY - “Preserving authorized restrictions on information access and disclosure, including means for protecting personal privacy and proprietary information...” [44 U.S.C., Sec. 3542]

A loss of confidentiality is the unauthorized disclosure of information.

INTEGRITY - “Guarding against improper information modification or destruction, and includes ensuring information non-repudiation and authenticity...” [44 U.S.C., Sec. 3542]

A loss of integrity is the unauthorized modification or destruction of information.

AVAILABILITY - “Ensuring timely and reliable access to and use of information...” [44 U.S.C., SEC. 3542]

A loss of availability is the disruption of access to or use of information or an information system.

A table below this section of the document categorizes the impact of security breeches, placing Confidentiality in the medium impact category.

...yet things change a bit with the release of the February 2010 version of NISTIR-7628. From the document, section 3.1:

3.1 CYBER SECURITY OBJECTIVES

In general for IT systems, the priority for the security objectives is confidentiality first, then integrity and availability. For industrial control systems, including power systems, the priorities of the security objectives are availability first, integrity second, and then confidentiality.

Availability is the most important security objective. The time latency associated with availability can vary:

␣ 4 ms for protective relaying; ␣ Sub-seconds for transmission wide-area situational awareness monitoring;

␣ Seconds for substation and feeder supervisory control and data acquisition (SCADA) data;

␣ Minutes for monitoring non-critical equipment and some market pricing information; ␣ Hours for meter reading and longer term market pricing information; and ␣ Days/weeks/months for collecting long term data such as power quality information.

Integrity for power system operations includes assurance that: ␣ Data has not been modified without authorization; ␣ Source of data is authenticated; ␣ Timestamp associated with the data is known and authenticated; and ␣ Quality of data is known and authenticated.

Confidentiality is the least critical for power system reliability. However, confidentiality is becoming more important, particularly with the increasing availability of customer information online.

As the document continues on, the impact level table becomes more granular, and discuss the impact by logical interface, confidentiality coming in 3rd place (as the opening statement indicates).

So what am I saying here? I am trying to make an important point. The protection of data privacy is not a primary consideration at the Federal level (or so it would appear from the February 2010 document), and I have to agree that this shift in thinking certainly seems to make perfect sense when you consider the impact of the system shutting down when compared to the impact of a breech in confidentiality. However, California, remember has their own privacy laws that, essentially, make privacy a PRIMARY consideration. In fact, I believe that is why the September 2009 document put confidentiality higher on the priority list.

So one can perhaps safely surmise that vendors have been forced to build privacy into the systems they are deploying in California in the absence of any solid standards from the Federal government. I say this because, as anyone who has been following the development of smart grid standards is keenly aware, the standards are currently in a state of flux (as can be seen by simply comparing versions of NISTIR-7628).

So even if the final standards place confidentiality dead last in consideration, with California leading the way AND having their own rules to follow, it stands to reason that vendors are going to have to build privacy into their systems, or potentially face elimination as they attempt to grab a piece of the pie. It also stands to reason that privacy will become a much bigger consideration as the deployment grows, and as data management becomes a cottage industry (i.e. 3rd party companies providing services to consumers based on their usage data).

As the saying goes "As Goes California, So Goes The Nation"...like it or not!

Smart Grid Security Performance Standards

2010-02-08T07:15:00.000-08:00

I have just gone through the somewhat laborious process of reviewing

February 2010 DRAFT NISTIR 7628

Smart Grid Cyber Security Strategy and Requirements

which I found through Smart Grid News (http://www.smartgridnews.com/artman/uploads/1/nist_cyber_security.pdf).

Section 3.1 touches on crypto, and the general tone of the document would suggest that logical (software) security is the method of choice due to the performance hit a system must take when implementing more secure levels of crypto (i.e. secure microcontrollers). Since performance is so important, one would logically conclude that hardware based security introduces challenges. Correct me if I am wrong, but that is how I interpret this.

That being said, what seems to be lacking is any sort of reference for performance. Since hardware based security is, in many ways, superior to logic based security (not always, but the BEST hardware based security chips are far more secure than the best logic based counterparts), then we really need a frame of reference here. Granted, no hardware based security solution will ever be able to match the performance of a logic based system (in fact, you can get maximum performance by simply using buzz words to describe a part of the system - like 256 bit encryption), but the best secure IC's deliver some pretty good performance while offering some very solid security. After all, banks rely on hardware based security (i.e smart card based security) for their most critical systems, and system availability and reliability are directly tied to the very high performance requirements vendors must adhere to in order to sell to the banking industry.

The one issue that does come into focus, of course, is budgetary constraints. Vendors of AMI systems must compete to sell their products, and the increased cost of implementing secure microcontrollers that deliver the requisite level of performance cuts into everyone's bottom line (which ultimately is the TRUE deciding factor). Logic based security can be implemented for anywhere from $0 to fractions of a cent, while high performance and high security hardware costs more.

Nonetheless, if cyber security is such a major concern (as it should be) in the implementation of the Smart Grid, then we should perhaps seek to create some target objectives for vendors of hardware based security, including performance and cost. We should also view the total cost in a systemic manner, taking into account the risk of relying on logic based security, and the cost of failure.

After all, if we are building an infrastructure that is expected to remain a part of our critical energy infrastructure for MANY years to come before replacing it with the next best thing, we should probably create solid, tactical objectives as well as higher level objectives.

It really takes both to succeed.

Auditing Smart Grid Security

2010-02-02T07:24:00.000-08:00

In my quest to better understand Smart Grid security initiatives, I have managed to gather quite a bit of useful information regarding emerging standards, layered security, and real world deployment of security in the Smart Grid. It is still a work in progress, but it is progressing, and a lot of smart minds are fueling that progress. This is all very good.

What seems to be missing, and perhaps remains as a great opportunity for those making the "land grab" on the smart grid is a cogent auditing and control methodology specifically targeted towards Smart Grid security. What I am talking about is a set of auditing requirements similar to what we see with the FDA.

Okay! Granted, the FDA is not perfect. They are not even close (in the opinions of many, I am sure). Nonetheless, the rigorous auditing procedures that health care organizations who fall under their requirements must adhere to does indeed serve two very important purposes. The first purpose is in forcing organizations to follow best practices and keep very good records (some better than others, I am sure). The second is instilling confidence in consumers globally. Did you know, for example, that FDA clearance/approval in the USA essentially guarantees clearance/approval anywhere else in the world?

At the heart of this process is a set of great auditing procedures that have been hammered out for last 100 plus years the FDA has been in existence, and many organizations have made a cottage industry out of providing auditing services for the organizations that need to fall in line. Organizations such as the American Society for Quality (ASQ) have created certification programs for auditors for the health care and essentially every other industry.

...but I am trying to get to a point with all this. I am trying to deal with a somewhat significant challenge. The standards being proposed by the NIST Interoperability Document Version 1.o alone encompass somewhere around 75 different standards. This can serve to create quite a bit of consternation and confusion for some, but bear in mind that there are literally hundreds (if not thousands) of standards used in the health care industry to "secure" our well being. Rather than focus on what specific standards are followed, auditors look at the big picture created during an audit, and determine if it passes the sniff test. If it does not, then its back to the drawing board.

This is not a joyful occasion for anyone involved (including the auditors), but it does lead to better products and systems (as well as annoying bureaucratic messes). Yet what makes this work is properly designed and well-vetted auditing guidelines and procedures.

We do not, unfortunately, have 100 years to iron out the inevitable wrinkles of Smart Grid security deployment, but we do have quite a few great auditing professionals (quality professionals, in fact) hard at work every day, and many are still looking for work in this down economy. With a 100 years of well documented procedures in place for the FDA, one could indeed surmise that the application of the same (or at least similar) methodologies could SIGNIFICANTLY curtail the development of a workable Smart Grid security auditing procedure.

...I'm just saying.

Smart Grid Security

2010-01-31T06:07:00.000-08:00

We have been working on Smart Grid security lately, and it is indeed quite interesting. It is essentially the single largest global technology project mankind has ever witnessed. The intent of the the project is to place every single node electricity touches on the Smart Grid. In stage 1 it is limited to Smart Meters replacing the classic analog meters found throughout the world. In stage x it will extend to everything in the home (ostensibly).

There are several technologies and protocols being used for communications on the Smart Grid, and all present interesting security challenges. The good news is that security is indeed a part of the discussion, and perhaps being given more attention than I have seen in the past with many large initiatives. The bad news is that the rollout is moving forward despite not having standards solidified.

This does not mean security is not being implemented. It is indeed being implemented and the key players in the ecosystem are learning as they go. Hey! This is usually how it is done. Nothing is new here. What I find interesting is that with standards not locked down yet (NIST Guidelines do not get very specific), it may mean that systems will have to be retrofitted in the event of a major security flaw. Since the minds behind the architecture seem to be aware of this, the security design seems to be quite layered, which is a good thing. This should help stop scalable hacks.

The lingering question I keep asking is "Why are equipment manufacturers not concerned with standards that may render their products obsolete?"

I would posit that this is because of 1 (or more) of these reasons:

Major manufacturers are influencing the standards to fit into their roadmaps.
Major manufacturers are planning a "version 2" with better security, and are likely to get paid again when they must replace "version 1"
Major manufacturers are not liable for non-obvious flaws, and have performed enough due diligence in "version 1" to safely hold onto the massive revenues the project brings in.

There are perhaps more reasons, but I am sure the 3 I mention are close to the mark. As a security professional, I am excited about the possibilities this project brings. Much of the information I have gathered has shown me that there are a lot of great security minds working on the Smart Grid, and the collaborative nature of the project is likely to lead to good security that can be built and managed collectively. This project is a huge "pie" with lots of wedges to share, and security is large enough to share with many security professionals, who are all likely to walk away quite full regardless of how many choose to join the project.

Exciting times indeed!

Less Compliance and More Reliance

2009-04-04T12:41:00.000-07:00

One of the most frustrating situations a "true" security expert has to face is organizations who approach us who obviously are merely interesting in implement security as a means of complying with "something". That "something" can be a legal mandate, a corporate policy, or a reaction to something that made the news. It is more or less security theater.

In the current tough economy we are now facing, I have seen a slight drop in organizations wanting security theater, since those interested in compliance are currently more interested in keeping the doors open. Organizations wastefully spend budget for what amounts to nearly useless security as a means of checking an item off of a list, but the item is not even on the list of some organizations right now. In some companies the item may be at the bottom of the list, but the items above it are so enormous, that it may as well not be on the list at all.

What this really means is that organizations looking at security are either being reactive (reacting to a real problem), or are proactively looking at security because they are convinced that a lack of security will lead to huge problems. These organizations do not want a checkoff security item, they want REAL SECURITY.

What this means for the industry is that security organizations who are providing little more than great buzzwords and complicated jargon to their potential customers are going to end up bankrupt, if this continues.

...at least I hope that is what happens

Lots of companies get away with pulling the wool over our eyes all the time, but they eventually get exposed for what they are. I hope we all become more intelligent about security and can all work together to weed out the trash.

Cyber "Street Smarts"

2008-08-13T06:54:00.000-07:00

Yesterday I was helping my next door neighbor set up a new laptop for his son. I am frequently called to service by my neighbors when they are in need of computer assistance, and gladly help them when I can.

As I was finishing up some work on his computer, he inquired about a computer he had seen me carrying into my house a few weeks earlier. He asked me if I had been able to rebuild it after the viruses had "messed it up". I told him that the reason I had to rebuild it was because the motherboard had died, and that my computers never get viruses. This obviously took him by surprise for a moment , and then he said "Oh, that's because you know what to put on your computer to protect yourself." I told him that I did not use any other "protective" software other than a virus scanner. This surprised him even more, since he too uses a good virus scanner, and since his computers, and the computers of most people he knows (and most people I know, for that matter) are constantly getting "infected". How do I do it, he wondered?

I gave him a simple analogy. If you take a civilian, arm him with a gun, and put him in the middle of high crime neighborhood, and do the same with an experienced plainclothes police officer, who do you think has the highest likelihood of not getting killed? "The police officer" was his answer. "Why?" I asked him. "Because he knows what to do to keep from getting killed.", my neighbor replied. "Exactly!" I said.

The officer knows how to avoid getting killed because the officer understands the threat landscape. I avoid getting infected, because I understand the cyber threat landscape. I simply never let my guard down in cyberspace, and despite the fact that I spend at least 5 times longer on a computer than my neighbor, I do not get infected by malware, viruses, popups, and any of the other annoyances that others I know must constantly deal with. I have taken the time to understand where the threats are coming from, and how to avoid becoming a victim of the threats. Sure, I use and recommend tools such as popup blockers and a good virus scanner, but those are there as my "backup". Most cops rarely have to as much as draw their sidearms, let alone have to use them. They take the time to understand the threat landscape, and go forth with that knowledge.

I wish that we could get the corporate customers we, as security experts, work with to buy into this notion. Nearly every one of them are more interested in what "product" they need to "get secure", and not in having us help them understand the threat landscape or "Threat Model".
Perhaps one day they will learn.

The Relationship Between Parenting, Voting Machines, Mortgage Meltdowns, and Pharmaceutical ePedigree

2008-08-06T15:39:00.000-07:00

Humans are generally well intentioned beings. We do not, however, begin life that way. Any of you who have children can certainly relate to this. A child is perhaps the most self centered being in the world. Children will fly into tantrums, hit, kick, bite, steal, and do whatever it takes to get what they both need and want. This is not because children are inherently evil. They simply do not know any other way to survive. When we attempt to teach our children how to do better it is no surprise that they do not welcome this gift of wisdom. After all, their method works to achieve the results they want, and changing gears is just too much work.

Some parents persevere in the endeavor to make their children understand the importance of being well mannered, sharing with others, and honesty. These parents are generally rewarded for their efforts in the long term, yet are often left biting their nails in frustration in the short term. It is, by any stretch of the imagination, no easy task, and many parents seek the assistance of others as they endeavor to stay the course in raising their children while attempting to avoid the pitfalls of frustration which so often force even the most determined to give in. We ask those we trust and love for assistance. We hire professionals into our homes to help us build better offspring. We send them off to schools to learn what they need to know to be all they can be. We buy books and study them, hoping to glean some insight on how to do better.

At times, however, we end up with children who don’t seem to reach maximum potential, and they grow into adults who struggle to make it in an often difficult world, and who frequently wreak havoc on a seemingly well designed sociological master plan. There is no need to expound on this; we all know what I am talking about. As Ayn Rand so eloquently illustrated in “Atlas Shrugged”, there are those that exploit and there are those that are exploited. Remarkably, the “exploiter” often begins life as the “exploited”. This is not always true, but it is true often enough to be noteworthy.

Why is it that we sometimes fail at this project? Volumes have been written with so many reasons and theories that it has created a multi-billion dollar industry for writers, doctors, psychologists, and the list goes on. Some suggest it may be diet. Others suggest it is the way we teach our children. Some feel it the music they listen to, the TV they watch, or the games they play. The way we raise our children changes with each generation, based on who is considered the expert of the day. One common thread which seems to remain intact, however, is perhaps the most valuable information of all. Parents who truly CARE about how they are raising their children seem to achieve success.

I need to stop for a moment and define what I mean by CARE. Truly caring about someone or something is, at its core, an unselfish act. It is about recognizing and setting aside personal gains, ego, fears, and barriers in order to focus on the achievement of an initiative which can stand as a testament to excellence. It is not about forcing your child to go to medical school so you can proudly boast to your friends associates that your child is a doctor. It is about doing what it takes to raise a child that can stand on his (her) own and proudly proclaim that all that he has become, whatever that may be, is in large part because you cared enough to guide him to find his passion and reach his maximum potential. To succeed at this, however, requires commitment and good judgment. The kind of commitment and judgment I am referring to is of the type that comes from careful introspective analysis in a non-egotistical manner. This is the type of commitment that considers the wisdom of others who have faced such challenges and have risen above them despite the obstacles they faced. This is the type of commitment which does not hand the task at hand over to someone else to do, while stepping back, only to lay blame on someone else when the outcome is not what was expected. This is the type of commitment and good judgment which is not afraid to question the judgment of others and raise the difficult questions, despite the fear associated with “rocking the boat” or questioning “common wisdom”. This is also the type of commitment and good judgment that leads to perhaps the most difficult task of all: The ability to admit when you have made a mistake and to change direction to fix the mistake and get back on track.

So what does this have to do with voting machines, our national mortgage crisis, and the current ePedigree solutions being proposed for ensuring authenticity of drugs from the global supply chain in the Pharmaceutical industry?

As we made the move into the modern age we live in, replete with technological marvels only a true Luddite would not embrace, we found ourselves with an ever-growing need to shed ourselves of many old ways. Voting on paper seemed to make no more sense than filling out withdrawal slips at a bank or writing checks at the grocery store. Sure, there are still those among us that embrace the old-fashioned way of performing these tasks. By and large, however, they are a dying breed. Paper-based voting systems required too much space, time, and money to tally the votes. It was clearly time to digitize the system. Voting machine companies and election committees from various states got together and began hammering out the details of the project, and the voting machines hit the ground running. Then disaster struck. Academics, reporters, and whitehat hackers discovered that the security of these systems was entirely inadequate for the purpose they were designed for. State election officials began decertifying these machines, and the court of public opinion pointed at the voting machine manufacturers and accused them of everything short of treason for their lack of attention to security. Being a security company, we decided it would be a good idea to study this situation and perhaps offer some assistance. As we discovered, the level of security of the voting machines was not a major concern for nearly all state certifying bodies at the time that these machines were first certified by the State. Some voting machine companies clearly understood what it would take to build a secure system, yet the requirements did not dictate a need for a secure system, and the voting machine companies couldn’t justify spending the money for security as it would make them uncompetitive.

Who is at fault here? Is it the election committee’s fault for not validating the security of the system? Is it the voting machine company’s fault for not insisting that the system had to be more secure and spending a little more money to make the security at least reasonable? Is it the fault of the American public for not seeing this coming? These are tough questions, but one question is easily answered: Who ended up paying for the failure? Yes, dear reader, we did.

Then there is the mortgage crisis we are all now quite familiar with. Almost everyone in the financial world knew of the enormous risks associated with sub-prime mortgages. Economists, academics, realtors, and simply sensible people tried to warn us of the dangers of what was happening in the market. Still, countless people continued to play this dangerous game, hoping to avoid being burned. Many people deluded themselves into believing those who characterized the experts that were warning us as “fear mongers” and “out of touch financially”. Hindsight is 20/20. We are paying the cost for this failure.

Now we come to the enormous ePedigree initiative. Counterfeit drugs are an enormous problem. Some estimates claim as much as 30% of drugs coming from some nations are counterfeit. Counterfeiting drugs has become a multi-billion dollar industry worldwide. Many operations which once dealt in illegal narcotics and other illegal drugs have turned to counterfeiting due to the enormity of the market and the relative ease with which those who deal in counterfeit drugs can operate (compared to those who produce illegal drugs). Clearly, something had to be done to combat this growing menace. The United States government, in cooperation with governments all over the world, decided to take action by requiring a pedigree for each and every drug produced and/or sold in the United States. By requiring a traceable pedigree for these drugs from producer to consumer, and every step along the way, in the event of a problem the point of breakdown could be detected, isolated and addressed. Initially, the rollout for this system was slated for 2010 (2009 for California), and has been pushed back to 2011. This is, without a doubt, a huge project with an enormous number of complexities involved in implementation. One of the first steps in this process that stakeholder have focused on is determining what technologies and methods would be employed to track these drugs. Will it be 2D barcodes, RFID, security chips, databases, auditing & legal resource? The list goes on. How will the information be shared? The complexity is staggering.

As a security expert, I thought it would be prudent to get involved in this process. Surely, I speculated, the organizations tasked with implementing such systems would be extremely interested in making sure that the security of the system was validated. I was perhaps a bit naïve in my zeal. Organizations involved in the Pharmaceutical manufacture and supply chain are clearly focused on compliance with a law which failure to comply with will lead to a complete inability to do business. I have witnessed a great deal of activity at the tactical level – putting together the components to comply with the law, but have yet to see any activity at the solution security level. The law simply does not call for validation of system security at any level that a counterfeiter could sidestep – these organizations are not allocating resources and mindshare to anything other than compliance. Hackers and perpetrators are much more determined, sophisticated, and resilient than government regulations around compliance. We all intuitively know this, yet where is the duty of Care to do something about it. Will this “Care” only emerge after enough people have died, or enough money has been wasted on a broken system, where people will be then be galvanized to be the hero and fix the problem, once the appropriate resources and attention has been allocated. What kind of “Caring” is this? Can a company afford to care if nobody else does?

So then I need to ask the same questions I asked earlier. Whose responsibility is it to validate the security of the system? Who is expected to CARE enough and demonstrate commitment and good judgment? Whose fault is it when the Pharmaceutical industry spends billions of dollars implementing a system that, if implemented without careful consideration of the security issues surrounding the deployment, is doomed to fail as did electronic voting systems and the mortgage markets? Only this time, people’s lives are directly at stake. Who is going to pay to implement the system, then pay to fix it when it fails, not to mention pay for the recourse to remedy wrongful deaths?

You and I will, of course.

So whose responsibility is it? Who will step up to the plate? Who can step up to the plate?

Avoiding Techno-Psychology

2008-07-03T09:55:00.000-07:00

In a highly popular international thriller novel, the villain kills a brilliant physicist and pokes his eye out to get into his highly top secret lab by using the dead physicist’s eye to open the large steel door to the lab by activating the retinal scanner. Why secure the door with a retinal scanner when a simple secret code would have been much more secure? They had it right in the James Bond Movie, Casino Royale, where a hundred million dollars was protected by a password, which they could not get from Bond, despite beating our favorite brave spy hero while he was chained to a chair. Yet biometric technology is widely deployed for security, when in reality it’s more for convenience or the perception of security. Biometrics are only more secure if they also require a password (most systems will accept a password as a backup if the biometric scanner doesn’t work). It doesn’t take an advanced degree in theoretical physics to figure this out, yet most of us don’t see this because we are infected by Techno-Psychology.

The problems that prevent us from achieving excellence and integrity in our technology driven world are those which we all have intuitively known and understood since we were children. The problem is not a lack of knowledge or skill in technology, for this can be learned by one with intelligence, determination, and resources. It is our intuition often being overwhelmed by a strong current of Techno-Psychology in the river of our business life.

In the business of securing information and products, we have a myriad of powerful security technologies available to achieve our objectives. These technologies are very complex, and understood by few. But what is much more complex than these technologies, is understanding how to apply them in practice, the most complicated aspect being at the most senior management levels.

Failure to achieve management excellence in security has lead to security failures that have cost billions of dollars, and in some cases lives. For example the failure of DVD and electronic voting machine security was caused by the sloppy deployment of secure technologies. There was no analysis of failure at the whole system level, focus was on deployment of the technology – much like putting a steel front door and lock on your house while leaving the keys under the doormat or while you still have a sliding glass door in the back. This is clearly not caused by lack of technical skills, these are management problems caused by the way risk is analyzed, communicated, and managed. The cost of failures in security goes on – companies losing over half their revenue from clones and hacks, medical equipment (e.g. defibrillators) being reprogrammed over wireless connections, credit card numbers being skimmed from fake readers, newly issued electronic passports being compromised, tracking of food and drugs, not to mention 911.

Excellence and integrity stop where caring and accountability stop. This is true with baking a cake and is equally true in deploying security. The most important issues in security, in descending order of importance, are understanding the placement of liability, the true objectives of the organization, the impact on the overall system/business processes, and the way success is measured. Understanding the technology is the easiest part of this business, by far.

Traditional management practices can be an impediment to excellence and integrity. For example:

The pressure to show quick results and measure success based on money spent or technology deployed
The lack of transparency in complex issues
The lack of understanding of key drivers for success and how success is defined
The lack of resources applied to planning and understanding the impact of solutions on people and processes before deployment begins

.. all while the villains are determined to find a solution to their problem

It is understandable why we are dominated by Technology-Psychology in a global, complex world. Our society has been driven by growth for millennium, caveat emptor. However, we are reaching a cross roads, where our collective DNA which has been growth focused may drive us into a wall.

Excellence and integrity in security start with excellence and integrity in management. This is true in security and is equally true elsewhere. It is especially important in domains that are complex, critical, require significant resources, are hard to measure success, and are long term focused – domains such as Environmental technologies and programs (e.g. Ethanol), Charitable Donations, Safety, and Education. The management skills, or rather culture, required for excellence is similar across these diverse domains.

Why "Security Theater" Is Always Standing Room Only

2008-06-18T09:57:00.000-07:00

The venerable Bruce Schneier (http://www.schneier.com/blog/) is widely accredited for coining the term "Security Theater". Many of you have had the pleasure of experiencing this show of shows in your everyday lives. This is the person at the checkout counter that asks to see your driver's license when you use a credit card, glances at it, and lets you through. I have pointed out to more than one cashier that my name on my credit card does not match the name on my license, just to see what the reaction would be. Most of the time it is something like "We just have to ask to see it.".

Other examples are not nearly as innocuous. Our company has been asked, on more than one occasion, to implement a security chip on a system at the lowest possible cost, and then generate a report for upper level management which would lead them to believe that the system is far more secure than truth would indicate. We, of course, are happy to implement any chip they want. What we will not do is generate a misleading report. In theater lingo it goes something like this: We will build the set, but we will not write the script.

As security consultants, our reputation hinges on the fact that we will not pander to this mentality. It has forced us to walk away from many business deals. What is perhaps the most alarming outcome of our actions is that someone else is inevitably given the job who is more than happy to direct and produce the theatrical production. Perhaps most importantly, technology companies who make claims that their products are secure are NOT HELD LIABLE for failing to deliver on the promise. Just try to sue a company who makes a security tool or appliance that fails, allowing a hacker to get through. Your time (and money) would be better spent trying to get the Middle East to live in peace.

The time and money an organization needs to invest in creating a fantastic "Security Theater" production is absolutely trivial compared to the massive amounts of money collected from unsuspecting theater attendees. What is perhaps the most alarming nuance of all is that once the public finds out that all is not as it appears in the grand production, the organization (or "Theater Company") merely has to create another episode of "Security Theater", perhaps this time bigger, brighter, and throw in some free popcorn (i.e. "Two years of free updates."). Crowds are guaranteed to come in droves.

The late PT Barnum would be so proud.

Medical Device Security In The News

2008-04-08T10:42:00.001-07:00

Recently, the Medical Device Security Center published a paper describing an attack on Pacemakers and Implantable Cardiac Defibrillators which would allow the wireless reprogramming of these devices by an unauthorized entity, with the potential of dire consequences to the person having one of these devices implanted in them. News of the exploit made its way into several online news sources, and the medical device community was quick to point out that such exploits have never been reported in the real world.

While this may indeed be true, it is important to note that, almost infallibly, news of such exploits often leads to a "me too" mentality among hackers with the intention of proving it can be done in the real world. Hackers often view exploits as an art form, and strive to create more "elegant" versions of the exploits in an attempt to "one-up" the last hacker. While this may be an annoyance when it comes to consumer electronics, digital television, or computer systems, it is much more than an annoyance when it comes to medical devices.

The time has come for medical device companies to take a more proactive stance about device security.

With Victory Comes Responsibility

2008-02-22T07:23:00.000-08:00

The Medical Device Community has been handed a victory by the Supreme Court with their decision in Riegel vs. Medtronic . Without going into too much detail, I will summarize the essence of the article by stating that The Supreme Court ruled in favor of Medtronic in a case where faulty wiring on an implanted medical device Medtronic manufactured failed, and the reason they ruled in favor of Medtronic is due to the fact that the FDA approved of the design.

As the writers of the blog Drug and Device Law very eloquently point out, this now means that medical device manufacturers and the FDA must now step up to the plate and prove to the world in general, that patients are best served by backing off with litigation and letting the professionals do their jobs. Please allow me a little poetic license.

What it all boils down to is this:

Medical Device Manufacturers Must Now Take An Even Greater Integrity Based Approach To Insuring Safety Of Medical Devices
The FDA Must Step Up Their Efforts In Insuring Devices Are Safe

If the Medical Device Community and the FDA fail to deliver from this point forward, Congress will inevitably be granted the power to step in and "fix" the problem. This is rarely a good thing.

Incorrectly Defining The Problem Based On The Solution

2008-02-21T05:35:00.000-08:00

Imagine this. You walk into your Doctor's office because you feel you may have a problem, and he reaches into his drug cabinet and pulls out a few bottles of drugs, tells you what they treat, and then gives them to you and says "That should take care of the problem!" He does not look at your health history, ask you any questions, check any vital signs. He just hands you some drugs (mind you, they are very effective drugs for the ailments they treat), and wishes you well. Oh, and he also delivers a hefty bill for his services. The doctor has a nice office and plenty of framed certificates on the wall, so he must be good. Time to start popping those pills!

You might get lucky. Maybe your health problem is treatable with one or more of those drugs, and it all works out. Then again, maybe your problem has nothing to do with what those drugs are meant to treat. Of course, you don't know this until it is too late. After all, you paid good money for the advice of the doctor with the fancy office and certificates all over the wall. Who are you to question the validity of his judgement ?

Of course, most of us are a little more careful with our health than this. We have come to expect a little more due diligence from our healthcare providers. We expect to have our vitals checked, records looked at, and some sort of sensible diagnosis before receiving treatment.

Security, however, is often handled in the manner first described. Organizations often blindly trust the security vendor's suggestions with almost no understanding of the problem and, more often than not, no discussion of requirements. Security vendors love to talk bits, bytes, standards, and certifications in an attempt to establish credibility. Sadly, this is often quite effective as a sales technique. Sometimes the security vendor's products solve some, or even most of the problem. Sometimes it just ends up being a very expensive mistake, which leads to a false sense of security, which is worse than no security at all. If you have no security, and know it, at least it forces you to pay attention. Blindly trusting a security system which does not deliver on the promise is a sure pathway to destruction, much like taking drugs for an illness you don't have.

The iClone

2008-02-19T07:05:00.000-08:00

Growing up in the late 1960's and into the early 1970's, a common snide comment we would make when something performed on a less than stellar level was "Made in Japan". It was a holdover from the early part of the twentieth century, when Japan was known as th country that produced cheap junk. Today, Japan is certainly not known for its junk. Made In Japan is now considered, by nearly everyone, to be a mark of quality.

Japan achieved this lofty status through a combination of hard work, assistance from other nations, perseverance, the amazing capability to copy the work of others, and a market that would buy their products. The market was perhaps the toughest barrier, as Japan had to compete with products from the United States, Germany, Italy, and many other long-established producers of higher-quality consumables. Nonetheless, they did it.

Enter China, which is essentially Japan on mega-steroids. Japan currently has a polulation of 127,433,494 (https://www.cia.gov/library/publications/the-world-factbook/print/ja.html) and China has a population of 1,321,851,888 (https://www.cia.gov/library/publications/the-world-factbook/geos/ch.html). Yes, that is a little more than 10 times the population of Japan. China is also already well established in the world market. The Chinese are also quite advanced and many of them have lot of time on their hands; time which can be used to tear apart any Western technology they find and figure out how to clone it. What makes it even easier for them is the simple fact that many of the Western technological products sold in the world are built in China. Every screw, screen, case, logo, all the way to the package it ships in. To a worker in China, often earning barely enough to make ends meet, an extra few dollars "on the side" earned for turning a blind eye when someone wants to review some design plans, or borrow a machine, or buy a few extra parts off the line is the difference between three square meals of rice and three square meals of rice and chicken. I think you get the point.

Enter the iClone , China's amazing take on the popular phone (also made in China) of a similar name. Not being content to merely copy the other guy, the makers of the iClone have decided to improve upon it. Please read the article for a full picture.

Medical device companies produce devices with very high cost consumables attached to them, or simply make single-use standalone consumable devices. Some of these high cost consumables sell for thousands of dollars (yes, for a single-use item). How long do you think it will be before China starts cloning these items? They are already doing it with drugs, food, and even toothpaste.

Makers of consumer electronic devices who have their items cloned take it in the teeth financially, and the customer sometimes benefits from this, sometimes not. Makers of consumables for the medical device industry who have their items cloned take it in the teeth financially, and the customer (patient) can benefit from this as well, OR end up dying. Does anyone see an issue with this?

I here the train coming....

2008-02-17T16:01:00.000-08:00

Security is a tough sell. Nobody really wants to pay for security anymore than they want to pay for insurance. We recently became involved with a security project for a medical device company. In essence, they wanted us to help design a system which would enforce a single-use policy for an electronic consumable. We designed a rather elegant system which encorporates cryptographic modules which authenticate via the use of cryptographic keys. This charged us, and we decided to move forward in pursuit of medical device manufacturers who we suspected would surely love this capability. Surprisingly, despite the interest, very few medical device companies have considered security in medical devices in any noteworthy manner.

To most companies, and medical device companies are no exception, security is only considered as a solution to problem. Once a company has had to deal with the pain of not having security, then they are willing to spend the time and money to fix the problem. Unfortunately, security cannot be added to a design very easily. It has to be built in from the beginning. Voting machine companies are feeling this pain right now.

Medical device companies are, in essence, waiting for the "train wreck" to happen. Then, and (in most cases) only then will they decide to build security into their products. Most of our business comes from clients who have either been on the train when it wrecked, caused the train wreck, or are close to someone who has been involved in a train wreck.

An ounce of prevention is worth a pound of cure.

Welcome To Our Blog

2008-02-17T06:52:00.000-08:00

Security as it applies to a system is perhaps one of the most misunderstood concepts in the world of technology. We choose to frame this weblog with this statement because security as it applies to many other areas of our lives is often handled in a much more sensible manner. For example, if your goal is to not lose your children when you go to a crowded theme park, you might agree to a system with your significant other (and perhaps your children) to insure that does not happen. You might also agree on what you might do if "Plan A" fails. One procedure you would probably not use is buying an "Antiloss Child Securomatic System" and implicitly trusting it to do the job. You might decide, however, that it would be a good idea to give your children cell phones, or put a business card with your phone numbers in their pockets or shoes. In other words, the "technology" deployed would naturally fall out of the requirement of not losing track of your children. Surprisingly, the "technology" does not have to be very high tech at all, and can end up doing a far better job at securing your children than the "Antiloss Child Securomatic System", and cost far less.

This may be a somewhat silly analogy, yet this is one of many issues we face. It is easier for some organizations to simply trust a consultant or a technology to do the job than to work towards understanding the problem and working with the consultant and technology to achieve the desired result. What do you think?