Sunday, March 27, 2011

The Limitations Of Voluntary Efforts

James Lewis of the Center for Strategic and International Studies (CSIS) is one of my favorite figures in the world of cybersecurity.  I would venture that both Mr. Lewis and the very wise Michael Assante truly get at the heart of the issues we face in the world of cybersecurity like no others.  They do not cover everything, but the issues they do discuss are profound in nature.

Both Mr. Lewis and Mr. Assante like to use analogies as part of their discussions, and I like that as well.  Michael Assante co-wrote a great piece on Roman Aqueducts with current NERC CSO Mark Weatherford (Assante is the former CSO), where they use analogies to compare Roman Aqueducts to the Smart Grid.  I do not want to go into detail about the article, so make sure you read it.  I can assure you it is quite well done.

James Lewis likes to keep things a bit closer to home, and uses a somewhat Socratic method to stimulate critical thinking.  This is extremely evident in a recent testimony to the House Committee on Homeland Security.  In his testimony, Mr. Lewis makes a number of absolutely wonderful points, but the part that resounded with me was the following:

"There is no other area of national security were we rely on voluntary action reinforced by incentives. A policy of voluntary efforts for better cybersecurity reinforced by incentives is not a serious effort to protect national security against real damage and a growing threat. These proposals are best seen as intended to block reform rather than to promote cybersecurity."

In order to understand this, we need a little background.  Nearly all progress being made on cybersecurity in the USA is due to voluntary efforts.  I personally volunteer my time to participate in NIST, OpenSG, and DHS working groups to address cybersecurity for the Smart Grid, as well as additional work I do for California for health care security.  This is not an altruistic endeavor.  My incentive to do so is either because I am paid to do so by a client, or because I want to develop a skill and become a subject matter expert for the purpose of exploring opportunities for commerce.  This works out well in my case, because my efforts have helped me pay my bills.  I am not getting rich doing this, but I am also not eating ramen noodles for dinner every night.

As it turns out, I am not alone.  There are literally hundreds (if not thousands) of people volunteering their efforts to the cause for the same exact reasons.  Of those, about 5% to 10% regularly contribute something besides attending a meeting or a conference call.  There is some progress being made, but it is slow, and most of it has no teeth whatsoever.  Why is that?  Because nobody is in charge, and since there is no consideration for efforts, there is nothing anyone in charge could do anyway.

Don't misinterpret what I am saying here.  I do not believe voluntary efforts are a bad thing.  Heck!  Our US Voluntary Militia of the 1700's did a fine job whooping some butt back in the day.  Yet one has to understand that once the threat turned into a battle, it was no longer a good idea to sit around and hope that the local blacksmith was going to show up for an attack against the Redcoats.

What our Congress seems to not "get" is that we are currently fighting a daily battle against the bad guys, and the bad guys are winning.  We are not fighting a war in the "classic" sense, but we are definitely getting our butts kicked more often than we would like to admit it.  It leads one to question just what does Congress and our President use as a determining factor for pouring money into a national security effort.  In fact, I wonder if national security really has anything to do with it at all.  When we light up skies overseas with bombs, and take down villages on the ground, are we doing this to protect anything, or are we simply trying to show the world that we still have plenty of firepower to go around?  Are we protecting our interests, or are we trying to get the President re-elected, or is a member of Congress trying to get re-elected, or more campaign contributions, or whatever?

Of course it could simply be that Congress and Mr. President simply do not feel the battle, because most of what goes on in the world of cyber attacks is not broadcast on the nightly news, and even if it was it is not likely to have the impact of bombs dropping and villagers screaming with blood running from their temples.   It could also be tradition.  Our Congress is all about tradition in many ways, and one of them is the long tradition of spending a lot of money overseas fighting battles, and NOT fighting domestic cybercrime.  Sure, there have been some token payments, but nothing approaching the billion dollars per day we are now spending fighting wars overseas.

At my last Smart Grid Security Conference I had two people from state public utility commissions as speakers.  Bill Hunteman from the US Department of Energy (a great and intelligent person, by the way) told me he was pleased to see that the PUC's were willing to send them to events such as mine where they can interact with others and discuss cybersecurity issues.  When I told Mr. Hunteman that the only reason they came was because I paid for their travel expenses, he was surprised to hear that state PUC's had no budget for such events.  This is particularly alarming when you consider that state PUC's are tasked with making decisions about cybersecurity for all of the distribution and "user" portion of the Smart Grid (i.e. Smart Meters, Advanced Metering Infrastructure, Home Area Networking), and nobody at the Federal level has any authority over this.

Think about this for a moment.  We have 50 states with public utility commissions who have little or no budget allocated for cybersecurity expertise being tasked to come up with rules for Smart Grid cybersecurity.  That is perhaps as effective as gong to your state Department of Motor vehicles and asking them to come up with some rules for earthquake proofing their spare time, with no budget for hiring bridge experts.

So hopefully, once we are done starting new wars with countries, and have wrapped up some of the other wars we are fighting, our government will consider funding some of the efforts to deal with our coming cyberwar, because once the daily battles turn into a full blown war, we are not going to be ready for it.

Not at this rate.

Wednesday, March 23, 2011

RSA SecurID and The Smart Grid

The compromise of RSA's SecurID system is one of several security-related hot topics this week.  I am still not sure how significant the compromise is (some say it is not overly significant, others claim it is a massive problem), but one thing is quite clear to me and many others I know in the security world - It is not a major surprise that their security has been compromised.

Security gets compromised.  That is what happens with security.  Some organizations have great track records.  As I understand it RSA SecurID has a 20 year track record.  Cryptography Research claims an 8 year perfect record for their CryptoFirewall product (used to protect, among other technologies, pay tv).  DES and SHA1 had their days in the sun, but all good things come to an end, and the newer and (hopefully) better technologies take their place.  Heck!  There was a time when the Mac loving world believed that Macs were immune to security compromises.  Think again!

RSA will fix whatever is broken.  They are a good company with a long history of knowing what they are doing.  Sure, they make mistakes along the way, but they are a good provider of security products.  The fact is that a compromise helps build better security.  As users of security technologies, we should EXPECT compromise at some point, and be prepared for it.  I am a careful driver, who does not text as I drive, and stays within the speed limit (more or less), and I wear my seatbelt and back up out of parking spots nice and slow.  Nonetheless, I am fully aware that operating my motor vehicle puts me at high risk for an automobile accident.  I can avoid automobile accidents entirely only by never getting anywhere near an automotive vehicle.  In the world of technology it is the same story.  If I play in the cyber world I am going to face cybersecurity incidents.

That brings me to the Smart Grid, and perhaps more specifically Smart Meters (or AMI in general).  Utility Commissions throughout the US (and perhaps the world) are hoping that rate cases cover AMI products that are going to be "good to go" for somewhere around 15 or 20 years.  This strikes me (and others I have spoken to) as somewhat of a pipe dream.  RSA SecurID is built to function as a sort of "Fort Knox" of security systems, and it lasted about 20 years.  AMI products are simply not designed that way today, and it may be a while before they are.  Simply put, utilities do not require that level of security and do not want to pay for it...and neither do consumers today (who will ultimately foot the bill due to recovery rules).  It is more likely that AMI systems rolling out TODAY (and not 3 years ago) may remain "secure" for a maximum of 10 years.

Think about this for a minute.  They are sitting on the outside of homes.  There are no set requirements for security.  The protocols, designs, and general security knowledge of vendors vary.  This is new territory and we are in the earliest stages of deployment.  We have to expect that we are not likely to get this right on the first few tries.  We have to also expect that we are going to learn (and have learned) some valuable security lessons as we proceed.

This creates a bit of an issue for utility commissions and consumers, because we have to pay for replacing devices that fail to remain secure over time.  As consumers, we are used to having to upgrade technology as time progresses.  Who keeps a computer or cell phone for 10 years today?  Okay, there are some who do, but not many.  Technology simply moves too fast and a few years down the line anyone who uses technology to get things done simply accepts that in order to continue reaping the benefits of technology, upgrades are a given.

Ahhh...and that is the key!  The consumer needs to experience the benefits.

I have a Smart Meter on my house and since I switched to off-peak pricing I have seen a drop in my power bill of approximately 30%.  I simply do not use power very much in the middle of the day.  Sure, it goes up a bit in the summer when I use my AC more, but it plummets in the winter.  This is significant for me because my winter bills used to eclipse my summer bills.  As a consumer, I am happy to have a Smart Meter on my house because these savings would not be possible without a Smart Meter (or so I am told).  As a consumer, if I see a benefit, I will pay to play.

As a consumer I am also aware that I have not experienced any security-related issues...yet.  It could be a long time before I do, but I am aware that the high tech nature of the Smart Grid opens up a gaping hole from a security perspective.  That is the nature of technology.  As a user of pocket computers (that is what we should be calling mobile phones and devices), I am aware of the major security holes that tag along with the technology, and also pray that bad things will not happen on the mobile device front.  Nonetheless, I know that it is indeed quite likely that somewhere within the next decade an enterprising attacker will figure out a way to exploit our favorite new technologies, and we are not likely to go back to doing things the old fashioned way as a result of the compromise.

Let us accept the fact that this is the world we live in, continue working to build better security, avoid freaking out when things go wrong, and reap the benefits that inevitably come with new technologies.

We live and we learn.

Sunday, March 20, 2011

Congress, Mr. President, You Are About To Get Served

I was enjoying lunch yesterday afternoon with my family when my iPhone sent me a push notification from CNN that the US had mounted an attack on Libya.  My brow immediately became furrowed at this news, since we are all keenly aware of the billions we are spending on wars today, and Libya now represents more opportunities to spend untold billions fighting yet another war (spare me the correction for using the term "war").

At my recent Smart Grid Security East Conference, I had a great panel with representatives from the US Department of Energy (DOE), The Federal Energy Regulatory Commission (FERC), and The North American Electric Reliability Corporation (NERC).  I called this the "Super Panel", since we had all the major Federal decision making organizations on stage at once discussing Smart Grid security.  I asked a simple question "From a distribution perspective, meaning the part that deals with Smart Meters and the consumer, who is in charge of security?"  The answer was the same across the board.  The Federal government is not in charge.  It is up to the States, meaning the Public Utility Commissions.

I then brought up a point that I continuously keep bringing up whenever anyone will listen.  State PUCs do not have any resources to address security.  The California Public Utility Commission, which is one of the largest in the country, has no staff dedicated to Smart Grid security, and very little in the way of knowledge of security.  I know this because I have spent quite a bit of time working with the CPUC (voluntarily) in the past year.  Some of them are eager to learn, to be sure, but they are a long way from being able to make decisions that will adequately address security issues.

I brought this up to Bill Hunteman, who is the Senior Advisor for Cyber Security for the US Department of Energy (and a very wise man), and he told me he is well aware of this issue, and the DOE is well aware of this issue, but (for now) there is nothing they can do about it, because they simply do not have the funding to address this issue.  I asked about where the funding needs to come from and he (and the rest of the panel) told me that it has to come from Congress.

This seems simple enough.  Congress likes to spend money to fight wars, and we are being attacked on a constant and consistent basis on the cyber front, so what's the problem?

I am not sure what is going on here.  Mr. Hunteman also mentioned (on another panel) that the Federal government is considering pulling some of the DOE funds they had allocated for addressing the Smart Grid, which was in response to a question I had asked regarding the likelihood that Congress would release additional funding to address cyber security for the Smart Grid.  This made me feel more than a bit concerned, since we are currently addressing the security of our critical infrastructure through the voluntary efforts of a bunch of people who are essentially only doing so in hopes of future opportunities for commerce.  Make NO MISTAKE ABOUT IT!

Congress and our President have authorized somewhere close to a billion dollars a day for our ongoing war efforts.  I have heard estimates of between $700 million and a billion a day.  If we took 1 billion dollars out of the budget for fighting wars, that would mean that we could give each state $20 million in financial resources to address cyber security.  I know that the California Public Utility Commission could certainly use the money.

This past week the big security news was about a compromise of RSA's SecureID system.  This is used extensively in both enterprise and government, and has caused a great deal of concern in the security industry.  We are still not sure how severe the damage was, but we nobody I know in the security world is particularly surprised that the exploit occurred.  We EXPECT exploits of this nature to happen, and we KNOW they are only going to get bigger and more sophisticated.

It is really only a matter of time before our critical infrastructure is hit with the mother of all attacks, and when that happens I am not sure Congress and Mr. President are going to be able to offer anything to us.

Again, just my opinion.

Tuesday, March 15, 2011

The Smart Grid Brass Ring For Consumers

Since I live and breathe Smart Grid these days (even though I am focused solely on Smart Grid security), I am constantly dumbfounded at the lack of awareness of the Smart Grid by most people I speak to "on the street".  I go to parties, social engagements, or wherever I may travel,  and am asked what I do for a living, and when I tell whoever is inquiring that I work in Smart Grid Cyber Security, the inevitable question is nearly always "What is a smart grid?"

It is particularly interesting when one of my neighbors asks me and I take him (or her) to the meter on the side of their house and point to the recently installed Smart Meter and explain what it is and how it works.  Sometimes I hear comments like "Oh yeah, I heard some people have had their bills go up." or "I hear these cause cancer."

So we are essentially with either no perception of the Smart Grid, or a collection of sound bites that fail to tell the story.

...but what is the story?

What I mean by this is "What is the story from a consumer perspective?"  Why do I want a Smart Meter, or a Smart Grid anyway?

Years ago, before I had a computer, I heard of this "Internet".  It was all over the place, this Internet discussion.  Internet this, Internet that, email, surfing, World Wide Web, @this, .com that, blah, blah, blah.  It was a big joke to those of us who did not partake in the least for a little while.

Then the internet became cool.  No, I don't mean cool in a hipster with gel in his hair sense.  I mean that the Internet became an environment where everyone could get something out of it beyond the interesting technology that makes it all work.  It became a fixture.  It started to take off.

The Internet did not really take off until it became something that delivered value to those who partook in it.  Once we realized that we could save money shopping online, save stamps with online banking, save fuel with online shopping, download music, movies, news, entertainment, etc. we all took to it like ants to picnics.  We are hooked.

Then came the whole smart phone revolution.  We discovered that we could not only carry this cool Internet with us, but we could also download interesting "apps" that we could do interesting things with, and suddenly discovered that these apps were something we could no longer live without.

Okay...maybe that is a stretch, but those of us who live happily in the app using world will probably agree that apps are a great thing, and they endear us to our devices.

So that brings me around to the Smart Grid, and consumer adoption.  This morning I saw an article about a new iPhone application called JouleBug, which makes a game out of saving energy.  The application is rather low tech, but it is sort of interesting with good graphics.  It illustrates a conversation I have had with some people in the Smart Grid space, where I insist that what it will take for consumers to "buy into" the Smart Grid is a combination of some sort of savings on power consumption AND an interesting way to interact with the ecosystem.  Sound bites on news programs and lower bills alone will not win us over.  An cool iPhone application (as an example) with cool graphics, push notifications, easy to use, and a general fun feeling may indeed be a winner.  If it gets us to change our energy usage in a positive way...even better.  After all, changing consumer behavior is really what the Smart Grid is about.

We, as humans, are not far removed from the creatures that flock towards bright and shiny things.  Perhaps we are a bit shallow in that regard, but if it gets us to save energy, then so be it.

Just my opinion.

Thursday, March 10, 2011

Knowing What To Ask For

One of my favorite conference speakers has to be Robert Former of Itron.  Robert is the in-house penetration tester at Itron.  He is paid to break things, and it is a job he thoroughly enjoys.  Like many vulnerability testers, he is essentially a no nonsense guy that shoots from the hip.  If you ask Robert a question about security, you are likely to get a very practical (and honest) answer.

One of the comments Robert has made (on more than one occasion) is that AMI vendors deliver what their customers ask for.  Early deployments of Smart Meters lacked many of the security features of today's Smart Meters for two reasons.  One reason was that many of the security concerns we face today simply did not exist in our conscience back then.  Sure, some will argue that we should have known better, and we should have learned our lesson from blah blah blah, but the reality of how we deal with security is only partially pro-active.  Think about this for a moment.  You are not going to walk about town in a bulletproof vest until you realize that there are bullets flying.  You may take a few precautions if you hear news of some people getting shot, but barricading yourself in body armor is not likely unless you are living in a war zone.

So with AMI, despite all the glorious hype we sometimes see, rest assured we are nowhere near a war zone.  Yes, there have been a few shots fired over the bow, but I have yet to hear of any casualties (or, for that matter, any injuries whatsoever).

The other reason why early meters lacked security found in today's meters is because utilities simply did not demand it.  Utilities wanted (and still want) inexpensive, reliable, and easy to manage meters.  Adding security to a meter can directly impact all three of these criteria.  Early deployments were focused on just getting everything to work, and many still are.

Yet we live and we learn...or so we hope.  The fact is that utilities are now keenly aware of the need for security, and they are now beginning to demand it from vendors.

However, this is not necessarily working out as well as it should.

I have had conversations with several vendors who have told me that some potential customers have essentially copied and pasted the entire NIST IR 7628 final report (which is 3 volumes) and said something akin to "do this" to their vendors.  As someone who is currently working on developing testing and certification guidelines for Smart Grid security as part of the NIST Testing and Certification CSWG working sub-group, I can assure you that this is not a good idea.  This is like handing a copy of "Larousse Gastronomique" to a caterer and saying "cook this".

Knowing what to ask for is crucially important for a utility.  Without some type of guidance, utilities are not going to be very effective at making demands.  In fact, without knowledge of what they ask for, utilities are likely to accept anything they are given as a response to their demands.  I mean, how are they going to verify anything anyway?

The work being done in OpenSG is seeking to rectify this.  There are a number of prominent utilities working in OpenSG, but the majority of utilities in the USA are not active members of OpenSG.  There is a wealth of information available to anyone who wants it, and anyone (utility or not) can participate in the work.  By educating themselves about security, utilities can create RFP's in an informed manner, and they can also take advantage of the tools and people available to help them verify that they are getting what they demand.  Getting involved is easy.  Send an email to (who is the current chair), or (the current co-chair) and you will be on your way.

The answers are out there.

Sunday, March 6, 2011

Travis Goodspeed Outside The Box

You are not likely to forget your first encounter with the very neighborly Travis Goodspeed.  He is a rather lanky young man (age 24 as of this posting) hailing from the Knoxville, TN area, who speaks with the slightest Southern drawl, and sports a rather impressive crop of dreadlocks.  Travis is an extraordinarily polite, easygoing, and friendly person who is very sociable and is quite fond of West Coast Style IPA beer, which he longs for when he visits Germany (where malty lager is the brew of choice).

Travis likes to challenge security assertions.  He likes to shave, etch, probe, and otherwise infiltrate computer chips in his quest to discover what secrets lie within.  He insists he does this for fun, and after sitting with him for a bit and listening to his exploits, I am convinced he must be having the time of his life. I am also glad he is not one of "the bad guys".

I had the pleasure of having Travis join us at my Smart Grid Security East conference, as a panelist and as a fixture in my expo area, where he set up shop with some of his tools and toys (some homegrown, some off the shelf) and proceeded to show the crowd how he managed to hack the "security" of a Microsoft Wireless Keyboard.  Mind you, this was not an old keyboard, but one he had recently purchased.  Apparently Microsoft decided to use the MAC address as the key for this keyboard communication scheme.  Travis showed how, using a rather interesting badge he had created, he was able to monitor every keystroke typed into the keyboard, and display it on a monitor.

I find this particularly intriguing, because for the last year or so I have been working on security guidelines for the State of California Office of Health Information Integrity as part of the Security Steering Committee for the Privacy and Security Advisory Board.  We have been creating guidelines addressing security for health information exchanges in order to help ensure that health care organizations in California align themselves with requirement under the HIPAA HITECH privacy and security regulations.  While we have done all we can to deal with issues such as how people should interface with systems, and how data should be handled in the system (mind you, it is not perfect, but we are working hard on the issues), something like a wireless keyboard communication protocol is so far out of scope it may as well be a discussion on the topic of corn pads.

We still live in a world where, from a security perspective, device manufacturers are essentially exempt from any liability for making silly choices.  Microsoft has enough money and brain trust to address this issue properly.  They could easily implement a design that transcends this level of silliness, but they choose not to do so.  Yet a health care organization that decides to replace their keyboards with the cool wireless ones available from their hardware supplier is one Travis Goodspeed Hope Badge away from having everything they type into the electronic health record becoming publicly available information.

People like Travis (and there are a lot of people like him, both good and not so good) think way outside of the box.  Organizations that spend millions (and even billions) of dollars trying to secure their systems who fail to understand this should prepare for lots of sleepless nights, and many sour looks as they face their boards of directors.

Friday, March 4, 2011

The Greatest Outcome

Well, 6 months of hard work, planning, endless phone calls, emails, accolades, assaults, cancellations, headaches, strong cups of coffee, sponsorship groveling, blogging, writing, and finally making it all happen are over.

My Smart Grid Security East conference was a success.  With the help of all the wonderful speakers, sponsors, and my own team, we made it happen.   We have a bunch of videos to edit and post on the site, and some papers, presentations, and slide shows to upload, but the lion's share of the work is done.

It is quite overwhelming when so many brilliant people tell you how wonderful something you built is.  These are all people I respect TREMENDOUSLY, and I cannot help feeling elated by their approval.  It is both exhilarating and humbling.  I would love to list them all in this blog posting, but you would probably stop reading halfway through the list, and I want to bring up a far more important hang on.

As my conference drew to a close, I reflected on all the moments I felt defined my sense of accomplishment for this event.  There was the opportunity to sit down to lunch with Bill Hunteman, Senior Advisor for Cyber Security for the US Department of Energy, who chatted openly about the DOE's roll and challenges.  Then there was the opportunity to enjoy breakfast with Matt Carpenter and Michael Assante.  I had an opportunity to converse extensively with the young, brilliant, and very "neighborly" Travis Goodspeed, who exposes security flaws in between pints of his favorite IPA's.  I enjoyed countless meals and moments with Daniel Thanos of GE Energy, and Bobby Brown of EnerNex, and Erich Gunther (who wears so many hats...well, lets just say he is everywhere).

Then there were all the wonderful AMI security minds.  There was Ed Beroset of Elster, and Stephen Chasko of Landis+Gyr, and Ido Dubrawsky and Robert Former of Itron.  All brilliant people working hard to build the products that we will rely on to securely manage our energy infrastructure going forward.

I literally could go on for several pages, but suffice it to say I had many "moments" with some great people.

Yet, life has a way of showing you what really matters right at the moment when you think you have it all figured out.  As I was leaving the conference hall at the final moments of the event, I was approached by a lovely young lady, who goes by the name of Summer.  She had won a free pass to my conference through a contest Andy Bochman of the Smart Grid Security Blog held.

She had contacted me by email after she won, and was thrilled because she is focusing her studies on Smart Grid cyber security, and she was attending school at the nearby Tennessee Tech University.  I welcomed her and told her she could bring along someone else from the school as a guest of the conference.

As Summer approached me, her young face broke out into a huge smile, and she profusely thanked me for the event and the opportunity to hear from the brightest minds in the world of Smart Grid security.  She then told me that the person she had brought with her had decided to change his thesis topic to Smart Grid security.

It was at that moment that I felt truly humbled.  As I get older (and hopefully wiser), and raise my children to be the best that they can be, the things that matter the most to me are maturing.  There was a time when I felt it would be great to be remarkably wealthy (okay, I still think that would be great), or achieve great fame (granted, that wouldn't be so bad either), but what matters the most is when you find a way to be the change you want to see.  I look at so much of what our youth has to deal with today, and often wonder how they can possibly cope with the mess they have been given.  I wonder where they can look for any guidance that will in any way affect them in a positive manner, and at that moment realized that something I had created served to positively influence at least 2 young minds.

Wouldn't we all like to do that?

Be the change you want to see...