Smart Grid Security and the CPUC

I am sure everyone can cite an example in life where various entities (including oneself) are forced into accepting responsibilities they may or may not be prepared to accept. This generally comes about in a somewhat organic manner, and sometimes hits a point where the duty of care rises in a nearly exponential manner, which creates a situation where all activity is then reactive in nature. This is not necessarily the best position to be in, but it happens quite frequently.

An example of this would be the extremely rapid growth of the automotive age in the USA. When automobiles first appeared on the scene, they were indeed a novelty, and certainly nothing worthy of considerable regulatory control. They shared the road with pedestrians, horses and buggies, and bicycles, and as I understand it were quite amicable about it (for the most part). Automobiles were built according to manufacturer specifications which were fully controlled by the manufacturers, and they were generally built with aesthetics, functionality, and profits as the key drivers.

As we all know, the automobile industry quickly made the transition from novelty to way of life within the blink of an eye. It did not take long until the automotive industry become the cornerstone of American manufacturing, and consequently an enormous influencer of economic dominance.

In other words, it got very damn big really damn fast.

As this growth progressed, it soon became apparent that we had to make major changes in the way we, as Americans, did things. It was no longer prudent to assume that everyone would share the road in a somewhat Utopian fashion. While automobiles created ENORMOUS benefits for society, they also introduced SIGNIFICANT challenging issues, many of which were indeed quite dangerous. I am certain I do not need to articulate all of these, since we are all quite aware of the many dangers associated with an anarchistic automotive culture, especially since we are all quite aware of the significant dangers we all face in an automotive society where most people do indeed (by and large) follow the rules. This is quite evident when a solitary driver loses control of a vehicle on a busy highway and causes massive multi-car collisions. It is truly amazing that we all manage to avoid such chaos as well as we do, but we do indeed manage.

...and why is that?

Well, it is because we have learned to do so the hard way. Most of the rules of the automotive ecosystem have come about as a direct result of lots of people dying, or lots of people being forced to live in a society where quality of life is negatively impacted. Seat belts, for example, where not always required in cars. In the early days of seat belts they were an option. They appeared in cars in the early 1900's, but were not required in US cars until the 1970's. In fact, most of the safety standards that exist today for the automotive industry did not appear until the 1970's, and NOT because the US Automotive industry decided it was a good idea, but mostly as a result of the crusades of the venerable Ralph Nader. Seat belts became standard equipment as a direct result of Federal legislation mandating that all carmakers include them. However, it was not until laws were enacted mandating their use by passengers that we began realizing a decline in traffic related deaths. To quote a section of an article from The Prevention Institute:

Mandatory laws have proved effective both in increasing seatbelt usage and decreasing traffic fatalities. The Centers for Disease Control and Prevention reports that seatbelt use nationwide increased from 11% in 1981 to 68% in 1997. NHTSA reports that the motor vehicle fatality rate as measured per 100,000 population decreased from 21.49 in 1981 to 15.69 in 1997, and also decreased as measured by 100 million vehicle miles traveled, from 3.2 in 1981 to 1.6 in 1997. While these decreases cannot be attributed to the use of seatbelts alone, seatbelts are credited with playing a significant role in these advancements.

Enter California. Being the state with both the largest population of people, the largest economy, and the most cars, California is always a good place to go for significant statistical samplings. California is somewhat notoriously well know for being a first mover on many initiatives that tend to serve the public interest over corporate interests, often at the behest of corporate interests. This, as it turns out, is a very good thing more often than not. When the rest of the USA was (and still is) debating the efficacy of environmental controls, and what should and should not be enforced to preserve our environment, California simply forged ahead and passed what many industry leaders deemed punitive measures to prevent (among other things) automobiles that emit large amounts of pollutants. Because the State of California represents the largest customer base for car manufacturers (1 in 11 cars is in California), the automotive industry was left with two choices. Either they could conform to California emissions laws, or they could find another less stringent customer base.

We all know which way they went, and now that I live in California I am extremely thankful. Anyone who has lived in California's most populous areas knows darn well that there are A LOT of cars here and they are on the road all the time. Prior to the enactment of state mandated emissions standards California regularly had days where the pollution was so severe that people were warned to stay indoors with their windows shut. While California does indeed continue to face air quality challenges in populous areas, the attention this has gotten at the state level has resulted in significant improvements in air quality. Perhaps most importantly, the significant changes automotive manufacturers had to make to their products in order to do business in California has led to a fundamental change in manufacturing which the entire US (and the world) now benefits from.

Let's go back to seat belts for a moment. Despite the mandate that seat belts must be worn when driving, compliance increases only as enforcement increases. This is true with nearly every rule of society, and certainly with rules where the manifestation of ignoring the rule rarely leads to any negative consequences. In other words, since most people who do not were a seatbelt when driving do not experience any negative consequence from not wearing one (they normally do not die or bash their heads against the dashboard or steering wheel), it is quite easy to forego this preventative measure. Compliance with seat belt laws got better as police officers began issuing citations to those who failed to use them. This generally came about as a result of officers pulling over drivers for a non-seatbelt related violation, and issuing a citation for not wearing a seat belt in addition to whatever they had been pulled over for. This is known as a "Secondary Enforcement" law. California, however, is a "Primary Enforcement" state with respect to seat belt laws. In California, an officer can pull a passenger over for the sole reason of failure to wear a seat belt, and issue a citation accordingly. As controversial as this law has become, it has led to a SIGNIFICANT decrease in fatalities. From the same article cited earlier:

California, a primary enforcement state, currently reports 91% usage -- the highest in the country. After the passage of a mandatory seatbelt law in 1986, California's usage rate went from 26% to approximately 45%. By 1992, California's usage had increased to 71%. With the passage of the primary enforcement law in 1993, California's usage rate jumped to 83%, steadily climbing to the current rate. According to the National Safety Council, California's fatality rate has decreased by over 34% since the passage of the primary enforcement law.

Wow! A decrease in fatality rate by over 34%. I would say that is pretty darn significant. I would also say that it is probably NOT all solely due to the primary enforcement, since California also spends a significant amount on programs to continually educate the population on the importance of safety.

As a staunch libertarian by nature (notice the small "l") I generally oppose government intervention. In fact, prior to moving to California I was convinced I was going to find the imposition of such a notoriously intrusive set of rules for my well being to be intolerable. In fact, however, I find it quite nice. I was a smoker when I moved to California, but the nearly militant anti-smoking sentiment coupled with the heavy handed enforcement of anti-smoking laws found throughout the state have forced me to rethink my addiction, and led to a lifestyle which I find far more appealing, since hacking my lungs out every morning was not something I looked forward to every day. I am now living a smoke free life for going on 7 years! The FACT is that smoking is a horribly dangerous health hazard that I am better off not taking part in. Surely I can still buy cigarettes in California (and many millions still do), and there are plenty of places that I could smoke them, but there are plenty more places where I cannot, and those places (such as restaurants) also smell like food, flowers, fresh air, and everything else except stale tobacco smoke residue. In fact, I am often taken aback by the smell of tobacco when I do encounter it today, since it is such a rarity. I am nearly overwhelmed when I go to a state/country where smoking is prevalent, and literally smile from ear to ear when I return home to California and its tobacco averse culture.

So this (finally) brings me to the smart grid, and specifically smart grid security. California, being the typical first mover in nearly all things technology related in the USA is now rapidly deploying a smart grid. As I understand it, we are now approximately 50% rolled out with our AMI products (smart meters and such), and are continuing to move forward. This all began with PG&E at around the year 2000, and has been joined by (among others) SCE. Being at the front line, PG&E has had the dubious pleasure of being the first to experience the challenges of a smart grid rollout, and has had to take action to fix issues as they arose. Security challenges identified in the early days of deployment were certainly not nearly as prevalent as they are today. Security challenges tend to rise as deployments of technology expand (for various reasons), and we learn as we go. Generally the fixes we put in place are reactive in nature (we discover an exploit and fix it), with a more proactive approach to security arising out of parallels that can be drawn by examining proof of concept exploits. Because of our growing understanding of security challenges, both PG&E and SCE have taken a VERY proactive approach to addressing security challenges in the smart grid, and have expended significant resources in cyber security. PG&E has a very competent cyber security team working very hard at addressing these issues, and SCE has teamed up with meter manufacturer Itron to implement an entire AMI solution with security being a major focal point. In fact, Itron has emerged as a leader in AMI security space as a result of this partnership. I applaud this extraordinarily proactive approach taken by both PG&E and SCE, and am quite certain that this show of leadership will serve as a template for the entire US to follow.

...yet this does not address some significant issues.

As it turns out, each and every organization involved in addressing cyber security as it relates to AMI is operating within a walled environment. I am not referring to the higher level issues, which are being addressed by NERC, FERC, NIST, DHS, DOD, DOE, and MANY OTHERS, but specifically at the application level (where the rubber meets the road). In other words, the vendors making the products that go into the grid are all implementing security as they see fit (based on a collection of "best practices"). In the case of SCE, senior director Paul DeMartini told me (at the CPUC public hearing on March 18th, 2010) that SCE insisted that Itron implement security as a requirement. Being a large customer for Itron, this was quite an incentive to move forward with security at the application level. Yet Itron is not the only vendor in the AMI space (although they are perhaps the largest single meter manufacturer). There are MANY other vendors, and quite a number of them have a significant presence. More importantly, each and every component that all of these vendors make for the grid are all subject to security challenges (some more than others, of course), and all make up a part of the "security chain". A chain, as we all know, is only as strong as the weakest link. While it may indeed be both reasonable and fair to assume that some (if not most) of these AMI products have addressed security in a manner that adequately creates a strong link, it is entirely imprudent to assume that ALL links are adequately strong enough.

So what have we done with respect to security at this application level? Well, we are working on putting together some national standards, auditing, and enforcement policies (NIST,DHS,FERC,NERC, etc.), but we are still quite a way off from finalization. One can surmise that once the Federal rules are agreed on (which is a significant challenge in and of itself), it will take quite a bit of time before enforcement has any significant positive impact. Let's face it, seatbelt laws were first enacted in the 1970's, but enforcement of such laws did not have any significant presence (or impact) for DECADES. The same holds true for EPA laws, wherein California EPA laws still set the high bar for standards, and in fact trump national laws because they are so much more restrictive, and many states simply live under less environmentally friendly conditions.

California, however, must act in a more proactive manner simply because the choices California makes have such a massive impact on so many people. With 36 million people (as of July 2009) in California alone, bad choices (or simply inaction) affects a huge number of people simply within our borders. When you consider, however, that California is the 8th largest economy IN THE WORLD, the choices California makes has a much greater global impact. Did you know, for example, that California produces 12.8 % of ALL agricultural products in the US. Surprisingly, we manage to do this with less than 4% of our nation's farms and ranches (talk about efficiency). Couple this with what California produces for the health care industry (drugs, medical devices, systems, etc.), for our defense industry, and for our financial system and it quickly becomes apparent that California is not just one of the 50 states. It is THE STATE our global existence is most reliant on.

So when things go wrong in California, things go wrong in lots of places. California is no stranger to things going wrong as a result of state level bad decisions and inaction. Perhaps the most recent failure, due to bad security related decisions, was with electronic voting machines.

Do you remember that nightmare?

Let me refresh all of our memories for a moment. California decided that getting rid of paper based voting systems was a good idea back in the early part of this decade (for various reasons), and this led to an enormous groundswell of activity among several companies to create electronic voting machines that would help California get rid of the tyranny of paper, and consequently bring enormous amounts of money to the manufacturers of such systems. One thing led to another, and California ended up spending billions of dollars on electronic voting machines, and so followed the rest of the nation. However, California failed to adequately audit the security of such systems, and consequently the security was audited by hackers and independent security professionals after they were in place. As it turned out, the security flaws were so significant that nearly all electronic voting systems ended up being trashed, both in California and on a national level. The cost of this failure was, of course, borne by the taxpayers. As it turns out, the machine manufacturers were not held liable for these flaws because the systems were in fact CERTIFIED by California (and other states) and given the seal of approval. Simply put, it was The State of California's fault for failing to perform due diligence as far as security was concerned. Moreover, I had the dubious pleasure of working with several voting machine manufacturers after the fact to try to help them fix these problems, and as it turns out some of them had indeed addressed these issues far more adequately than suspected by those who chose to vilify them, but simply did not include better security features in their systems because the customers (i.e. The State of California) simply would not pay for them! California simply chose to inquire about what security features existed, signed off on the agreements without adequately auditing the security, and the rest is history.

As we move forward we learn hard lessons, and hopefully get better because of the lessons we learn. California recently received a big chunk of Federal stimulus money to implement electronic health records (EHR's), and one of the provisions from the Federal government is that the implementation must include security as prescribed by the HITECH act. Since there are few specific Federal guidelines in place AT THE APPLICATION LEVEL, the California Office of Health Information Integrity (CalOHII) has taken the initiative in creating a security committee and has drafted a set of security guidelines AT THE STATE LEVEL. I have had the pleasure of contributing to discussions with this committee (and have indeed been invited to participate, and have agreed to do so), and one of the main reasons why California has taken a state level initiative in security at the application level is because of the enormously critical nature of this initiative, and the enormous cost of failure of a lack of adequate security. Imagine, if you will, a hacker having the ability to alter a medical record, and imagine a malicious reason for doing so, such as altering a record of an enemy to indicate that he or she is not allergic to penicillin (for example), which could lead to death in the event he or she is given penicillin without prior knowledge. A lack of security here is indeed a life or death problem.

So this finally brings me to the California Public Utility Commission (the CPUC), who is now faced with some pretty tough decisions in light of the fact that the entire power infrastructure of the 8th largest economy is potentially vulnerable to significant cybersecurity related attacks (which could effectively shut down our power generation/distribution systems). The CPUC is in place to serve the public interest first and foremost. The vendors in the AMI space are there to serve corporate interests first and foremost. So much so, in fact, that the most significant vendors in the AMI space live behind a rock solid wall of NDA's and refuse to discuss security architectures and applications in any way even resembling a transparent and collaborative nature. I can certainly understand this from a competitive perspective since I too must adhere to such NDA's (and indeed do adhere to them), yet this forces us to live under an environment of corporate self regulation, which, as well all know, does not always seek to serve the public interest in an adequate manner. Again, as a libertarian I am okay with this as a basis for capitalist endeavors, but I am less okay with this for matters of life and death, and to a somewhat lesser degree (perhaps) for matters where I am forced to bear the cost of failure.

Who do you think is going to pay for the parts of the smart grid that must be scrapped in the event of a security failure? I can tell you with certainty that the US taxpayers did indeed bear an enormous amount of the cost of a failed electronic voting system. I can tell you that the California rate payers are bearing the cost of the AMI rollout (either directly or through Federal taxes). If it fails to deliver what we expect it to deliver (security related or otherwise) we cannot simply scrap it and go back to the way it was in the old days without a SIGNIFICANT cost (if that is even on the table). Moreover, if we are forced to replace vendor products due to security flaws, does it not strike you as somewhat interesting that the vendors may in fact directly benefit from the "double dip" nature of this scenario?

In a conversation I had with Aloke Gupta, who is a Senior Energy Analyst with the CPUC and is currently working on energy policy, he informed me that the CPUC has traditionally not been in the "verification business" with respect to public utility deployments. Fair enough, but this does not mean that they shouldn't be. Because the public simply has no choice whatsoever with respect to smart grid deployment in California (or elsewhere, for that matter), we must now rely on security choices which are being made by corporations who are tasked with (as a matter of legal due diligence to stockholders) maximize their bottom lines. The cost of secure components and design simply goes up the more you improve it, and the return on investment is nearly impossible to realize. In fact, the best security tends to completely obfuscate ROI. If there is no security failure, how can one know how much security measures helped? So the public simply cannot hope that the security choices a vendor makes are going to primarily center on what is best for the public interest. It simply does not translate to a bigger bottom line UNLESS everyone must comply with a set of enforced standards, which levels the playing field, and prevents loss of market share due to competition with another vendor who decides to take the "cheap" way out. Ideally, the deployment of the smart grid would have occurred only after security standards, auditing policies, and enforcement procedures were in place, but that did not happen, and may not happen for quite some time. From a national perspective, one can make the argument that with the US being approximately 5% rolled out with AMI, things are moving along at a reasonable pace at the Federal level. However, when you consider that fact that most of that rollout is in California, it now quickly becomes apparent that it is incumbent upon the CPUC to take firm and decisive action well in advance of national standards.

After all, they are here to serve the public.

The Smart Grid Privacy Smoke Screen

Whenever I watch news on network media I view everything being said with quite a bit of cynicism. Heck! Security professionals are NOTORIOUSLY cynical. The security professional mindset is designed to quickly wade through layers of what can be seen on the surface and find that which cannot be seen, which tends to tell THE REAL STORY.

Back to the news for a moment. When I see a major topic wrapped with lots of sensationalistic coverage splattered all over the airwaves and news sources, I immediately ask "Okay, what is REALLY going on." Why is everyone talking about who does or does not have the right to use the word "retard" (for example). What is the real agenda, or what are they trying to prevent us from paying attention to.

I know it may sound conspiratorial, but I see this a lot with security, and I assume it happens everywhere else.

Let us discuss security for a moment.

There are some things in the world of security that are complex, and some that are not so complex. There are good ways to protect systems using low cost, medium cost, and high cost components and procedures. When making a determination about what is the best choice (from a financial perspective) organizations that must implement security must always balance the risks with the costs. This is simply how it is done. Many of the risks associated with security are driven by compliance. If an organization does not comply with "the rules", they can be held liable for a failure to perform due diligence. This is, by far, the biggest driver (and headache) for any organization. Security generates no ROI in this case, it simply acts as insurance. Nobody I know likes to pay their insurance premiums, but they all must.

The other way security ends up in systems is when it has been attacked. Generally the more significant the attack (i.e. the more costly the attack), the better the security solution. I do not want to spend too much time on this particular topic, but it does warrant a mention. This is the holy grail of security professionals, by the way.

In cases where security becomes a topic of discussion, and consequently a major bone of contention among vendors who are subject to security mandates, what frequently happens is that the conversation takes a direction that serves the lowest common denominator. Rather than talk about the "real" issues, we tend to talk about issues that seem to be of utmost concern, but really do not matter nearly as much as the "real" issues. This is often because the more important issues are quite a bit more complex (and consequently more costly) to deal with. By shifting the focus to the less complex issues, organizations tend to appear as if they are solving a problem (and consequently performing due diligence), but they are actually avoiding the bigger issues.

For the last several days I have been reading through piles of comments submitted to the California Public Utilities Commission (CPUC) regarding Smart Grid deployment. Within these documents there are quite a few comments regarding Smart Grid security, but the overwhelming language talks about security as it relates to privacy (i.e protection of consumer usage information).

Okay, I do indeed believe privacy is important, and hold it near and dear. California was one of the first states to enact privacy laws, and has definitely led the pack in this arena. I definitely get it. Privacy is indeed important.

Sadly, however, it is a smoke screen. The focus on privacy takes our focus off of the real security challenges we face as we deploy the Smart Grid. Privacy, as it turns out, is not as challenging an issue as preventing large scale attacks of the Smart Grid which could theoretically bring down large SCADA systems. Why do I believe this? Because it simply does not have the WOW effect from a hacker community (and media) perspective. You see, EVERYTHING that is computer/network/system related can be hacked at some point. In an ideal world, the good guys try to keep ahead of the bad guys. The bad guys are always working on taking down what the good guys have built, and the most interesting things to take down are the ones which have the most impact. Hacking my meter (or any one's meter) to see how much power I use just does not get you very much attention these days in the world of hacking. Taking down a generator, however, does.

So as I read through countless pontifications about how crucial it is to ensure our privacy, and consider the extraordinarily low risk of a breach of privacy causing our lives to change in any considerable way (let's face it, how many of us truly feel we have any privacy these days?), I cannot help but think about what an effective smoke screen this is when we consider Smart Grid security. NISTIR 7628 is fully aware of where privacy sits on the scale of things to watch out for, and the February 2010 draft clearly points this out, listing privacy as a tertiary concern as it relates to security.

Yet the public comments floating around the CPUC seem to indicate that privacy is "what it is all about". I certainly do NOT see any discussions of any value indicating otherwise. Nearly every security professional I have spoken to about Smart Grid security finds this focus a bit absurd in light of both the know (non-theoretical) and assumed (theoretical) security dangers.

I think the public should consider this as they strive to educate themselves about security and the Smart Grid.

The Cyber Warrior Mentality - The Security Warrior

“The basic difference between an ordinary man and a warrior is that a warrior takes everything as a challenge while an ordinary man takes everything either as a blessing or a curse.”

-Carlos Casteneda, American author 1925-1998

I have been thinking about the warrior mentality a lot lately. It started several weeks ago when someone I was speaking to (yes you, Stewart) about cyber security referred to something I said as being indicative of having a warrior mentality. It struck me as interesting because my business partner talks about having a warrior mentality a lot, and as I had this discussion I was more than a little taken aback by the uncanny parallels between myself, my business partner, and this complete stranger I was discussing security with. Partway through our conversation I began predicting what he was going to say, based on my understanding of the situation, and it was dead on every time.

It was like he was reading my mind.

Yet this was not what I found strangest of all. As I began "gathering intelligence" in my attempt to better understand the vendor space in the cyber security landscape (needs, requirements, activities) as it relates to The Smart Grid, I consistently ran into two distinct types of people. One was the more marketing oriented type, who simply discussed security in a manner that was indeed befitting of the vendor (a security apologist if you will), and the other was the security contempory - or the "Security Warrior" as I now like to call it.

Okay, I know this may sound odd to some, but for those who fit into the category I am sure it makes perfect sense.

As a security professional who began his security career as an administrator who was thrown into the battle due to outside attacks on the company network, I was charged with fixing the problem, and I was given very few tools (and even less time) to do so. My boss did not want to hear anything about expensive firewall hardware, or outside consulting, or anything like that. I was in charge of IT, so it was my job to fix the problem, and to do so within the confines of the limited budget I had available to me.

Oddly enough, I did not view this directive with frustration or with disdain. I simply took it as my marching orders and did the best I could with it. I had been sent out to the jungle with a book of matches and a pocket knife, and it was my duty to survive with those tools, and my wits. Come to think of it, I loved it!

Having less to work with really makes some people think hard and "outside of the box". Not all people, however. Some people simply cannot cope with the situation, and give up. Others pretend that things are going to miraculously work out through some sort of cosmic intervention, and simply wait for things to change. Sometimes this inaction mentality works out for them, but it is not because of divine intervention (although I do believe in God, but that is another discussion), but it is often because someone else picks up the slack.

When given a limited toolset, the warrior does not fret. He (or she) simply takes inventory, and then begins studying the enemy, beginning with the enemy within. Fear, shame, guilt, doubt, and other such feelings and mental states are identified for what they are and dealt with promptly and effectively. The warrior studies the landscape and determines where the danger zones lie at every given moment (because they are always changing), and what to do to stay out of danger. The warrior immediately determines what threats are real, what threats are not real (but are actually more perceptions than real threats), and what threats may come, and prepares accordingly. If the threats come from other people (the biggest threat of all), then the warrior does all he can to study the perceived enemy to determine both the level of the threat and the mental state of the potential enemy. If the warrior determines that the enemy is indeed real, he does NOT rush to kill the enemy. The warrior then studies the enemy and determines if the enemy himself is indeed a true warrior as well.

THIS IS IMPORTANT !!!!

The most effective players in the cyber battle are those with a warrior mentality, ON BOTH SIDES OF THE BATTLE. A warrior views the most effective enemy as a CONTEMPORARY, who is fully capable of being just as strategic and calculating as he is (if not more so). A warrior will watch his enemy take control of a battle and marvel at the strategic nature of the enemy with deference, and then file that away as another tool in the arsenal. He may never use this tool, but he will understand it enough to know when it is being used (or about to be used) again, and will know exactly what to do to either prevent it from being effective, or prevent the enemy from using it to begin with.

Let me return back to Earth for a moment and discuss my first major security breach as a CIO. I was in charge of a medium sized retail operation, with lots of remote retail locations logging the point of sale system into our corporate serves to perform transactions (using the Remote Desktop Protocol). It was an elegant solution that worked marvelously...most of the time. One afternoon I began receiving lots of complaints from the store managers because the system had slowed to a crawl. I asked one of my team members to look into it, and he found nothing out of the ordinary, but did indeed notice a spike in outgoing traffic that seemed to be sustained. Outgoing traffic spiking was not unusual, but sustained spikes were indeed out of the ordinary.

As I shifted to the hands-on approach, I noticed the traffic was on one particular port (I do not remember the port number), and it was not one of our known ports (21, 80, etc.), so I knew something was up. I was completely fascinated at this point (although I knew someone uninvited had entered our network), and began investigating. I narrowed the culprit down to one particular server, and a careful study of the server logs revealed activity going on in the recycle bin.

Huh!!!

As it turns out, someone from Korea (I traced the IP address to Korea) had installed an FTP server in the recycle bin on the Windows server, and was serving pirated Hollywood movies from my network. ABSOLUTELY BRILLIANT !!!

I was literally tickled pink by this feat of trickery. Why had I never thought of that back in my early days of "file sharing" cat and mouse gamesmanship? Touché indeed my Korean enemy. Well played!

Of course, I knew simply getting rid of the server was not the solution to my problem. It simply treated the symptoms (slow traffic). I had to then discover the weakness in my network "armor" that had allowed the infiltration to begin with. As it turns out, it was one of the many recently patched security holes common to Windows based systems (at the time), and I had been hit prior to the patch. I fixed the patch, and then made it my mission to very carefully monitor system changes on a very granular level, which led me to the discovery that attacks on my network were happening on a very regular basis (port probing, hammering, etc.). It allowed me to study and learn my enemies' tactics, and I soon discovered that there were a lot more attempts than there were victories. Yet when the victories happened (and they did indeed happen), I learned how to stop them, and they did not happen in the same way ever again.

You see, dear reader, I placed an ENORMOUS value on the victories, because they exposed my weaknesses. One cannot effectively determine a correct strategy unless one clearly understands their weaknesses. However, a warrior who has the ability to swallow his pride can significantly reduce the number of victories the enemy has if the warrior is willing to take a step back and enlist the advice of contemporaries who have already lived through the battles, and especially the ones who have the battle scars to prove it.

As I speak to more of the people who have enlisted in the battlefront in the cyber war, I consistently run into those who have been forced to deal with security as an additional headache to deal with, and those who have a true warrior mentality. What I have found is that warriors are very good at spotting other warriors, and can usually do so almost immediately. The conversation, at this point, takes a completely different tone. Even though I am coming to them (in part) as a consultant who is trying to win brownie points with my client, who is trying to determine market opportunities (after all, everyone needs to pay bills), we immediately move past that as the discussion now becomes far more temporal. We begin discussing the evolving threat landscape, the strategic nature of the environment, and the tenacity of the enemy. We laugh heartily (yet respectfully) at the hyperbole, and focus on the true threats to our mission, which we often determine, in part, come from our side of the battle (cost constraints, time sensitivity, corporate politics, lack of transparency, etc.). We sometimes venture off into discussions that have nothing to do with our current positions, and recount tales of battles past, and battles yet to come.

It is, I surmise, much like soldiers getting together on leave or after a war. Not having been an actual soldier I cannot say this from an experiential viewpoint, but I have known enough soldiers and watched them interact with each other to know that the similarities are indeed valid. I can tell you that it is also much like parents discussing the battles and victories in raising children, which is something I am indeed achingly (and pleasantly) familiar with.

So this brings me around to a very positive understanding of the battle from a US Government/Military being involved perspective. We often hear that the US Government is not very good at dealing with cyber security (and they have indeed admitted their shortcomings in that space), and that we are in big trouble because of that. I would tend to agree with such fear mongering if it wasn't for the indisputable fact that the US Government is arguably the best IN THE WORLD at gathering warriors to fight its battles. The US Military has spent literally hundreds of years perfecting the art of war, and despite their lack of understanding of the artillery (bits, bytes, laptops) or the enemy tactics (hacks) in the ever evolving battlefront, they are certainly quite good at understanding the warrior mentality. That is why, for example, I believe we are well served by having Michael Assante serving as the head of cyber security for NERC. His job is to separate what is "real" from what is "not real", and to manage the ever evolving battle plan, and make sure that every soldier (i.e vendor, utility, etc.) in the battle is following the plan. Nearly everyone I have spoken to that falls under what I deem the "Security Warrior" mentality finds Mr. Assante's assertions to be dead on. Mr. Assante, by the way, served in the US Navy for 6 years.

Some may argue that our US Military is nothing to be proud of, but I think they are missing the point if they do. We do not win every military battle we fight it, nor are we always justified in what we do militarily (which is subject to opinion), but we definitely are very effective the majority of the time. Moreover, we have certainly succeeded to the point where Americans find the prospect of domestic battle so foreign that something horrific (like the 911 incident) is seemingly incomprehensible. Yet foreign countries throughout the world deal with their own versions of the 911 incident every day. Our combined defenses and offenses have indeed managed to generate some ill will both foreign and domestic, but they have also managed to make us not only feel safe, but actually BE SAFE!

So I am hopeful and indeed confident that as we engage this cyber war with more of a warrior mentality that we will indeed manage to both survive and indeed thrive and feel quite safe as the battle evolves. We will engage the enemy, learn from the enemy, and indeed prove triumphant if we all swallow our self-serving interests long enough to take some cues from those who have received the battle scars.

...and frankly, I am excited about the opportunity to be a part of it.

The Evolving Compliance Landscape Of Cyber Security

Security, as it turns out, is largely about compliance. Anyone who has spent any significant amount of time working in security knows this all too well. Years ago my business partner and I worked on a very sophisticated health care project which involved cryptographically authenticated peripherals. The business model was such that a peripheral attached to a device was to be used once (and only once) on a patient, and then discarded. The doctor would then have to buy another (or have a stock of more peripherals) for another treatment.

We were brought in because the first generation product used a weak security solution, and it was hacked and counterfeited in 3 weeks (3 weeks from launch to counterfeits on the market)!

Bye bye business model.

So, having miserably lost the first battle in their own little cyber war, they decided it was a good idea to bring out the big guns, and they created a security solutions team (which essentially consisted of me, my partner, and the chipmaker we were working with) and we put together a very secure solution. It cost them more than their first solution, but they knew all too well the cost of failure, so it was easy for them to justify the increased expenditure.

Wouldn't it be nice if all security engagements worked out that way? Certainly for the security provider, I suppose.

We began focusing on health care security, feeling empowered by our previous engagement, and soon discovered that it was not a easy as we suspected. We touted the battle scars of our client as an indicator of the need to securitize their products, but failed in our attempts to generate revenue. It was frustrating, to say the least.

In all this, I learned quite a bit about the health care industry, and soon realized that everything in health care is compliance driven. Since the extent of security requirements for health care providers essentially falls under HIPAA regulations, all a health care organization is interested in doing is complying with HIPAA. Doing so essentially requires a security policy, and something as simple (and low tech) as requiring a 4 digit PIN to enter a system. It certainly is not something my company could sell into, since most of what is required by the client falls under the security audit, which is generally handled by the IT team medical clients already have. Since the requirements are so non-stringent, this is usually a 10-15 minute conversation.

Do you think I am exaggerating? I assure you I am not. One of the "side" jobs my company has been focusing on is iPhone applications for health care, and I can assure you that when the subject of HIPAA compliance comes up it NEVER lasts more than 15 minutes, and usually ends up with an agreement to enforce a 4 digit PIN (and a few other minor security additions). As developers who have worked in security development for quite some time, we do indeed build our software with an eye on security from the beginning, and our clients do indeed get a lot of security "freebies" because of that, but it is not because we are compelled to do so by any forces outside of our own need to not fall on our own swords as security professionals. In other words, we have chosen to self regulate our process, and our clients benefit from that.

While this is all good, and certainly makes us feel like we are doing the right thing from a due diligence perspective, it is only a small dent in the underlying battle. We know this from even further forays into working with cyber security challenges. The most interesting, perhaps, is with voting machines. Despite the bad press voting machine vendors received after many security professionals discovered gaping security flaws, they were indeed complying with requirements set forth by the election commissions in the states they sold into. In fact, the states themselves "ate" the cost of the insecure machines because of this, while the vendors got the black eye. Some vendors we worked with were indeed QUITE aware of how to build secure voting systems, and told us that such systems were unsellable because the states simply did not want to pay for them. All the vendors had to do was comply and then offer the best ROI to the clients in order to win the bid, and they did just that. It was not until after the security exploits were discovered that the US Elections Assistance Commission began taking security seriously.

Imagine that!

That does not mean that security was not considered...it certainly was. It is just that the deployment of security was weakened by a low threshold for compliance. What is even more important to realize is that the threshold is nearly impossible to determine without a proof of concept (moving from theory to reality).

Fast forward to cyber security, and specifically as it relates to the Smart Grid. Early deployments on the smart grid did indeed include requirements for security, and ALL vendors took it quite seriously (some more than others, as it appears). But this was not necessarily due to compliance issues, it was because (as one utility security expert put it), no utility company in their right mind is going to deploy something that is not secure.

...and yet, we now know that the grid has some fairly major security issues. How can this be?

Well, we have to consider security in the context of the perceived threat. For example, I can easily be shot while walking down the street of the bad part of town in any city, but that does not prompt me to invest in body armor (or even a bulletproof vest). Body armor is expensive and not very comfortable to wear, and since I have never been shot at while walking in the bad part of town, I am more than willing to rely on simpler and more cost effective security solutions (such as perhaps walking on better lit streets) to keep me safe. Moreover, I may very well get shot at and STILL decide not to get body armor. It simply takes a certain level of perceived danger for anyone to elevate their security requirements, and we really do not know what that level is until it happens.

So we stand in the presence of a smart grid deployment that is going pretty strong in the USA (on the order of 10's of millions of meters), and we have not borne witness to any major catastrophes yet. We have indeed proven that the threat is very real, and we are now working towards lowering the risk. The Department of Homeland Security, NIST, and NERC have enlisted the public and private sectors in the activities (and I have indeed joined in on the fun), and there are many smart people working on the challenges at every level you can imagine. Michael Assante of NERC co-authored an EXCELLENT article in the January/February 2010 issue of IEEE Security and Privacy magazine titled "No Grid Left Behind", and he methodically lays out the challenges and proposed solutions, and everything in the article is quite cogent. I have personally spoken to MANY members of the security community ranging from vendors, utility companies, PhD Scientists, meter manufacturers, crypto algorithm providers, and everyone in between. Everyone is working hard on the project.

Yet we have to understand that despite all the efforts to win the war (something that will never happen, as the war will never end) and prevent casualties, we are not going to come through this unscathed. Smart Grid deployments are vital to our existence because the energy savings have been proven. In a conversation I had with Echelon (a maker of AMI products) they have shown an energy savings on the order of 70% in some cases!!! That is a VERY significant number. When we consider the impact of energy savings on anywhere near that level, it certainly makes the case for Smart Grids a no brainer. I mean, think about that for a moment...saving energy by simply being smarter about where and when it is being used.

To me, as an energy consumer who spends over $500 per month to meet my energy needs (when you include fossil fuels), that hits home.

So what we have to understand that sometimes the missteps on the battlefront do indeed lead to things getting better. We have to understand that the fact that the various regulatory entities that are working towards solidifying and continuously evolving the standards (which vendors are indeed paying close attention to) are well aware that they are stakeholders in this ecology, and an insecure smart grid affects them on a very personal level. Michael Assante of NERC is a security whiz, but he is also well aware of the fact that the decisions he makes affect the outcome of this nation AND his personal life in a VERY profound way. Just like America came together to fight the enemy during World War II, after we had felt the attack at Pearl Harbor, we too can expect cooperation as we fight the ever evolving cyber security enemy.

After all, we are all on the front line.

Coordinating Efforts In Cyber Security

I have truly never seen a security initiative quite as interesting and massive as the cyber security effort as it relates to Smart Grid. The more I peel back the layers, the more I discover how massive the effort is.

Just a few days ago I was invited to attend a meeting of the American Bar Association (yes, the lawyer group), and the entire meeting (2 full days) is packed with presentation after presentation and working groups dedicated to cyber security. Yes! You heard that right. Lawyers are interested in understanding cyber security on a very detailed level. Fascinating!

I have spoken to utility companies, meter manufacturers, chipmakers, HSM vendors, cypto stack vendors, regulatory bodies...the list goes on and on and on. Everyone is contributing something to the effort of fighting the current cyber war we are all involved in.

Oh...you do not think we are at war? Well please reconsider. We may indeed be at somewhat of a stalemate at times, or even in a cold war, but we are indeed at war. The enemy is constantly hard at work trying to find new ways to break into our vast cyber network, and on every level you can imagine. It ranges from Aunt Judy's Facebook page all the way to our Missile Defense system, and the attackers are RELENTLESS (and also have a lot of fun doing what they are doing).

But enough fear mongering for now. We, as humans, are completely used to being vulnerable. We fashion synthetic skins from cloth and the pelts of animals to protect ourselves from freezing to death. If we no longer had clothing WE WOULD DIE! OH MY GOD! WE MUST PROTECT OUR TEXTILE INDUSTRY!!!!

You see how ridiculous hyperbole can get...time to switch to decaf.

Being subject to attack is something we are well aware of and we have simply coordinated our efforts to a level where we all understand what it is going to take to protect ourselves from bad things. We invent penicillin to prevent ourselves from bacterial infections, and we coordinate efforts to get it into the hands of all who need it. We make vaccines to stop the spread of deadly flu. We organize our defenses to prevent the bad guys from breaking down our walls and fortresses.

We all have a pretty good idea of how and when to coordinate efforts to keep ourselves protected, and we make improvements along the way. That is what we do.

Fast forward to Smart Grid security. The exploding cyber security industry as it relates to Smart Grid seemed to come out of nowhere. Sure, I was talking about it 2 years ago with some vendors, but it wasn't until quite recently that it became an area of more intense focus with nearly every technology company in existence (and those who are not in the game yet will be in soon). I have to say that I began seeing more activity happen in this area after the 60 Minutes episode on cyber security, where they showed a transformer (I believe it was a transformer) being destroyed remotely by a simulated cyber attack. Even I, who has worked in security long enough to know that there are A LOT of unresolved security issues in this world, was taken aback by this proof of concept.

Nonetheless, the troops went into action. We are now seeing a massive land grab as everyone in the world of security reaches for their piece of the pie, and many come up with big handfuls of job security as a result. I was talking to a cyber security engineer a few days ago who told me that he was out of work 2 years before landing a sweet job working for a utility. My own company was stagnating for the last 2 years as well, despite having worked on some very large security projects. In fact, we started writing iPhone applications as a side job, and were brought back into security because of Smart Grid. It is a huge project with a dire need for security expertise on EVERY level imaginable. Definitely some promising times for the cyber warriors of the world.

Yet I see some issues popping up that I believe are indeed quite counterintuitive to meeting this war with the extreme sense of urgency it deserves. One clear obstacle to cyber security excellence is our woefully luddite government (in the USA). They make no bones about the fact that they are bordering on the dark ages where it comes to providing expertise on cyber security. They are damn good at providing expertise on blowing things up, rebuilding them, and blowing things up again, but the only asset the government can contribute to the cyber security effort at this point is $$$$$$$...and they are...billions of $$$$ in fact.

Where the government becomes a bit of an issue and perhaps a bit of a hindrance is in the effort to mandate standards for cyber security. This is where things begin to get a bit annoying. Although NIST is hard at work at finalizing NISTIR 7628, and NERC is hard at work building a nice compendium of auditing documents for cyber security, and the CPUC is hard at work scoping their Smart Grid security initiatives, nobody can tell you what the COMPLETE rules of the game are at this point. Anyone who claims to know all the rules of the game is lying, because the rules are still being written as we speak.

So what rules are the players following in the absence of a coordinated set of rules from up above? Well, I can tell you that California is indeed following (to a degree) its own rules regarding privacy as it relates to Smart Grid (SB 1386), since California law requires privacy protection. I can tell you with certainty that there are a lot of VERY smart minds from the private sector working very hard to make sure that they can build the best security with the resources they are given, and that the resources are definitely getting better (albeit slowly). I can tell you that the guys I talked to at the major utility company are being funded by the utility to do some absolutely brilliant work, and that the cyber security engineer at one of the meter manufacturing companies that the utility sources meters from has some absolutely great ideas about what it would take to improve security. I can tell you that the utility guys would love to let their vendors know what they believe would be beneficial on the front lines.

Ahhhh...that is where things get a bit unraveled. Imaging my surprise when I spoke to the utility company cyber security guys and they told me they have never met or spoken to the cyber security architect at the meter company. What the ????

Lets get back to my original assertion. We are at war. Sure, we see no blood and body parts flying (ala Saving Private Ryan), but we must accept the fact that the warriors have, to a degree, laid down their rifles and booted up their laptops, and the pending cyber battles are going to take out a lot more troops than any rail gun ever could. While guerilla warfare tactics are indeed effective, a coordinated battle effort generally wins in the end. A mixture of both is perhaps the most effective, but only in the sense that the guerilla warriors are on the same page as the well regulated troops, AND (this is the big point) that we eliminate redundancy and activities that cause a regression in the efforts. If I, as an intelligence officer, decide that a great strategic tactic is to befriend an enemy and gather information, and manage to get a few cups of coffee, some cold beers, and perhaps a nice meal with the enemy in my effort to butter him up, it is rather annoying if the guerilla warrior decides to put a bullet in his head because he has a clear shot.

As a security professional who is interested in helping the cause, I constantly run into two very different scenarios. The first scenario is the fortress of silence and dismissiveness that many (but not all) of the players in cyber security put up when I try to get some information that would help me do my job better. The second is when I reach the guy (or gal) at the organization that truly cares about security and truly understands the need for teamwork, and appreciates those of us who have fought on the front lines of the ongoing security battle, who CLEARLY see the enemy for what he is...A PEER! No, I am not talking about the REAL enemy (the guy at the top of the food chain), I am talking about the security expert the guy at the top of the food chain has hired to do his bidding.

I am unsure of how coordinated the efforts of the bad guys are at this point. I am assuming it is fairly good because hackers seem to love collaborating (take BlackHat, for instance), and I am also assuming it is going to get better as the bad guys start getting better funding.

So are we going to take a cue from that...or are we going to continue setting up our own battle fronts as we wait from the orders from above?

As Goes California...

With over 10 million smart meters deployed to date, California (as the pilot for the US) has essentially acted as the pioneer of Smart Grid "movement" in the US.

Being a pioneer is not a new thing for California. We can thank California for equal rights, organic food, and much more. We can also thank California for being the first state to enact laws guaranteeing the privacy of data for its constituents, with the formation of the California Office of Privacy Protection in the year 2000.

I can remember when it happened. I was a CIO of a retail company in California when some of the first reports of major violations started to hit the airwaves. The CEO of the company started asking some questions, and I thankfully had the answers he wanted to hear. I had long considered data privacy and important issue, so I took the extra small steps to make it happen. He was pleased, and I think I got a raise.

Interestingly enough, the protection of data has always been THE most important consideration for those who practice the art of IT Security. Information Technology really began as a means of securing financial data, and in the old days the IT department did not fall under the guidance of a CIO, but was a sub-department of the office of the CFO. Once paper was replaced by bits and bytes, protecting the bits and bytes became a very important job, and IT Security became a cottage industry.

Fast forward to the new game in town - Smart Grid Security. Things are a bit different now. While the protection of data is indeed still important, it is NOT the primary focus. Smart Grid Security is focused on making sure that security breeches do not cause the system to slow down or (most importantly) stop functioning altogether. It does not take a lot of thought to understand why this is the way it is. Someone knowing how much electricity Mr. Jones is using and for what is not nearly as devastating as someone having the ability to shut Mr. Jones' electricity off. So one would surmise that the focus of vendors should be on availability and sustainability, and the rules of the game (i.e NISTIR-7628) certainly seem to point that way...

...but wait a minute. Let's examine NISTIR-7628 for a moment. This is where things start to get a bit interesting.

The September 2009 draft of NISTIR-7628, in section 3.2, discusses impact levels of 3 areas where security is an issue. From the document:

3.2 IMPACT LEVELS

The IAC impact levels are low, moderate and high. The levels are defined in Federal Information Processing Standards (FIPS) Publication 199, Standards for Security Categorization of Federal Information and Information Systems, February 2004. Following are the definitions for confidentiality, integrity and availability, as defined in statute and a table that defines low, moderate, and high impact.

CONFIDENTIALITY - “Preserving authorized restrictions on information access and disclosure, including means for protecting personal privacy and proprietary information...” [44 U.S.C., Sec. 3542]

A loss of confidentiality is the unauthorized disclosure of information.

INTEGRITY - “Guarding against improper information modification or destruction, and includes ensuring information non-repudiation and authenticity...” [44 U.S.C., Sec. 3542]

A loss of integrity is the unauthorized modification or destruction of information.

AVAILABILITY - “Ensuring timely and reliable access to and use of information...” [44 U.S.C., SEC. 3542]

A loss of availability is the disruption of access to or use of information or an information system.

A table below this section of the document categorizes the impact of security breeches, placing Confidentiality in the medium impact category.

...yet things change a bit with the release of the February 2010 version of NISTIR-7628. From the document, section 3.1:

3.1 CYBER SECURITY OBJECTIVES

In general for IT systems, the priority for the security objectives is confidentiality first, then integrity and availability. For industrial control systems, including power systems, the priorities of the security objectives are availability first, integrity second, and then confidentiality.

Availability is the most important security objective. The time latency associated with availability can vary:

␣ 4 ms for protective relaying; ␣ Sub-seconds for transmission wide-area situational awareness monitoring;

␣ Seconds for substation and feeder supervisory control and data acquisition (SCADA) data;

␣ Minutes for monitoring non-critical equipment and some market pricing information; ␣ Hours for meter reading and longer term market pricing information; and ␣ Days/weeks/months for collecting long term data such as power quality information.

Integrity for power system operations includes assurance that: ␣ Data has not been modified without authorization; ␣ Source of data is authenticated; ␣ Timestamp associated with the data is known and authenticated; and ␣ Quality of data is known and authenticated.

Confidentiality is the least critical for power system reliability. However, confidentiality is becoming more important, particularly with the increasing availability of customer information online.

As the document continues on, the impact level table becomes more granular, and discuss the impact by logical interface, confidentiality coming in 3rd place (as the opening statement indicates).

So what am I saying here? I am trying to make an important point. The protection of data privacy is not a primary consideration at the Federal level (or so it would appear from the February 2010 document), and I have to agree that this shift in thinking certainly seems to make perfect sense when you consider the impact of the system shutting down when compared to the impact of a breech in confidentiality. However, California, remember has their own privacy laws that, essentially, make privacy a PRIMARY consideration. In fact, I believe that is why the September 2009 document put confidentiality higher on the priority list.

So one can perhaps safely surmise that vendors have been forced to build privacy into the systems they are deploying in California in the absence of any solid standards from the Federal government. I say this because, as anyone who has been following the development of smart grid standards is keenly aware, the standards are currently in a state of flux (as can be seen by simply comparing versions of NISTIR-7628).

So even if the final standards place confidentiality dead last in consideration, with California leading the way AND having their own rules to follow, it stands to reason that vendors are going to have to build privacy into their systems, or potentially face elimination as they attempt to grab a piece of the pie. It also stands to reason that privacy will become a much bigger consideration as the deployment grows, and as data management becomes a cottage industry (i.e. 3rd party companies providing services to consumers based on their usage data).

As the saying goes "As Goes California, So Goes The Nation"...like it or not!

Smart Grid Security Performance Standards

I have just gone through the somewhat laborious process of reviewing

February 2010 DRAFT NISTIR 7628

Smart Grid Cyber Security Strategy and Requirements

which I found through Smart Grid News (http://www.smartgridnews.com/artman/uploads/1/nist_cyber_security.pdf).

Section 3.1 touches on crypto, and the general tone of the document would suggest that logical (software) security is the method of choice due to the performance hit a system must take when implementing more secure levels of crypto (i.e. secure microcontrollers). Since performance is so important, one would logically conclude that hardware based security introduces challenges. Correct me if I am wrong, but that is how I interpret this.

That being said, what seems to be lacking is any sort of reference for performance. Since hardware based security is, in many ways, superior to logic based security (not always, but the BEST hardware based security chips are far more secure than the best logic based counterparts), then we really need a frame of reference here. Granted, no hardware based security solution will ever be able to match the performance of a logic based system (in fact, you can get maximum performance by simply using buzz words to describe a part of the system - like 256 bit encryption), but the best secure IC's deliver some pretty good performance while offering some very solid security. After all, banks rely on hardware based security (i.e smart card based security) for their most critical systems, and system availability and reliability are directly tied to the very high performance requirements vendors must adhere to in order to sell to the banking industry.

The one issue that does come into focus, of course, is budgetary constraints. Vendors of AMI systems must compete to sell their products, and the increased cost of implementing secure microcontrollers that deliver the requisite level of performance cuts into everyone's bottom line (which ultimately is the TRUE deciding factor). Logic based security can be implemented for anywhere from $0 to fractions of a cent, while high performance and high security hardware costs more.

Nonetheless, if cyber security is such a major concern (as it should be) in the implementation of the Smart Grid, then we should perhaps seek to create some target objectives for vendors of hardware based security, including performance and cost. We should also view the total cost in a systemic manner, taking into account the risk of relying on logic based security, and the cost of failure.

After all, if we are building an infrastructure that is expected to remain a part of our critical energy infrastructure for MANY years to come before replacing it with the next best thing, we should probably create solid, tactical objectives as well as higher level objectives.

It really takes both to succeed.