The Smart Grid Privacy Smoke Screen

Whenever I watch news on network media I view everything being said with quite a bit of cynicism. Heck! Security professionals are NOTORIOUSLY cynical. The security professional mindset is designed to quickly wade through layers of what can be seen on the surface and find that which cannot be seen, which tends to tell THE REAL STORY.

Back to the news for a moment. When I see a major topic wrapped with lots of sensationalistic coverage splattered all over the airwaves and news sources, I immediately ask "Okay, what is REALLY going on." Why is everyone talking about who does or does not have the right to use the word "retard" (for example). What is the real agenda, or what are they trying to prevent us from paying attention to.

I know it may sound conspiratorial, but I see this a lot with security, and I assume it happens everywhere else.

Let us discuss security for a moment.

There are some things in the world of security that are complex, and some that are not so complex. There are good ways to protect systems using low cost, medium cost, and high cost components and procedures. When making a determination about what is the best choice (from a financial perspective) organizations that must implement security must always balance the risks with the costs. This is simply how it is done. Many of the risks associated with security are driven by compliance. If an organization does not comply with "the rules", they can be held liable for a failure to perform due diligence. This is, by far, the biggest driver (and headache) for any organization. Security generates no ROI in this case, it simply acts as insurance. Nobody I know likes to pay their insurance premiums, but they all must.

The other way security ends up in systems is when it has been attacked. Generally the more significant the attack (i.e. the more costly the attack), the better the security solution. I do not want to spend too much time on this particular topic, but it does warrant a mention. This is the holy grail of security professionals, by the way.

In cases where security becomes a topic of discussion, and consequently a major bone of contention among vendors who are subject to security mandates, what frequently happens is that the conversation takes a direction that serves the lowest common denominator. Rather than talk about the "real" issues, we tend to talk about issues that seem to be of utmost concern, but really do not matter nearly as much as the "real" issues. This is often because the more important issues are quite a bit more complex (and consequently more costly) to deal with. By shifting the focus to the less complex issues, organizations tend to appear as if they are solving a problem (and consequently performing due diligence), but they are actually avoiding the bigger issues.

For the last several days I have been reading through piles of comments submitted to the California Public Utilities Commission (CPUC) regarding Smart Grid deployment. Within these documents there are quite a few comments regarding Smart Grid security, but the overwhelming language talks about security as it relates to privacy (i.e protection of consumer usage information).

Okay, I do indeed believe privacy is important, and hold it near and dear. California was one of the first states to enact privacy laws, and has definitely led the pack in this arena. I definitely get it. Privacy is indeed important.

Sadly, however, it is a smoke screen. The focus on privacy takes our focus off of the real security challenges we face as we deploy the Smart Grid. Privacy, as it turns out, is not as challenging an issue as preventing large scale attacks of the Smart Grid which could theoretically bring down large SCADA systems. Why do I believe this? Because it simply does not have the WOW effect from a hacker community (and media) perspective. You see, EVERYTHING that is computer/network/system related can be hacked at some point. In an ideal world, the good guys try to keep ahead of the bad guys. The bad guys are always working on taking down what the good guys have built, and the most interesting things to take down are the ones which have the most impact. Hacking my meter (or any one's meter) to see how much power I use just does not get you very much attention these days in the world of hacking. Taking down a generator, however, does.

So as I read through countless pontifications about how crucial it is to ensure our privacy, and consider the extraordinarily low risk of a breach of privacy causing our lives to change in any considerable way (let's face it, how many of us truly feel we have any privacy these days?), I cannot help but think about what an effective smoke screen this is when we consider Smart Grid security. NISTIR 7628 is fully aware of where privacy sits on the scale of things to watch out for, and the February 2010 draft clearly points this out, listing privacy as a tertiary concern as it relates to security.

Yet the public comments floating around the CPUC seem to indicate that privacy is "what it is all about". I certainly do NOT see any discussions of any value indicating otherwise. Nearly every security professional I have spoken to about Smart Grid security finds this focus a bit absurd in light of both the know (non-theoretical) and assumed (theoretical) security dangers.

I think the public should consider this as they strive to educate themselves about security and the Smart Grid.

The Cyber Warrior Mentality - The Security Warrior

“The basic difference between an ordinary man and a warrior is that a warrior takes everything as a challenge while an ordinary man takes everything either as a blessing or a curse.”

-Carlos Casteneda, American author 1925-1998

I have been thinking about the warrior mentality a lot lately. It started several weeks ago when someone I was speaking to (yes you, Stewart) about cyber security referred to something I said as being indicative of having a warrior mentality. It struck me as interesting because my business partner talks about having a warrior mentality a lot, and as I had this discussion I was more than a little taken aback by the uncanny parallels between myself, my business partner, and this complete stranger I was discussing security with. Partway through our conversation I began predicting what he was going to say, based on my understanding of the situation, and it was dead on every time.

It was like he was reading my mind.

Yet this was not what I found strangest of all. As I began "gathering intelligence" in my attempt to better understand the vendor space in the cyber security landscape (needs, requirements, activities) as it relates to The Smart Grid, I consistently ran into two distinct types of people. One was the more marketing oriented type, who simply discussed security in a manner that was indeed befitting of the vendor (a security apologist if you will), and the other was the security contempory - or the "Security Warrior" as I now like to call it.

Okay, I know this may sound odd to some, but for those who fit into the category I am sure it makes perfect sense.

As a security professional who began his security career as an administrator who was thrown into the battle due to outside attacks on the company network, I was charged with fixing the problem, and I was given very few tools (and even less time) to do so. My boss did not want to hear anything about expensive firewall hardware, or outside consulting, or anything like that. I was in charge of IT, so it was my job to fix the problem, and to do so within the confines of the limited budget I had available to me.

Oddly enough, I did not view this directive with frustration or with disdain. I simply took it as my marching orders and did the best I could with it. I had been sent out to the jungle with a book of matches and a pocket knife, and it was my duty to survive with those tools, and my wits. Come to think of it, I loved it!

Having less to work with really makes some people think hard and "outside of the box". Not all people, however. Some people simply cannot cope with the situation, and give up. Others pretend that things are going to miraculously work out through some sort of cosmic intervention, and simply wait for things to change. Sometimes this inaction mentality works out for them, but it is not because of divine intervention (although I do believe in God, but that is another discussion), but it is often because someone else picks up the slack.

When given a limited toolset, the warrior does not fret. He (or she) simply takes inventory, and then begins studying the enemy, beginning with the enemy within. Fear, shame, guilt, doubt, and other such feelings and mental states are identified for what they are and dealt with promptly and effectively. The warrior studies the landscape and determines where the danger zones lie at every given moment (because they are always changing), and what to do to stay out of danger. The warrior immediately determines what threats are real, what threats are not real (but are actually more perceptions than real threats), and what threats may come, and prepares accordingly. If the threats come from other people (the biggest threat of all), then the warrior does all he can to study the perceived enemy to determine both the level of the threat and the mental state of the potential enemy. If the warrior determines that the enemy is indeed real, he does NOT rush to kill the enemy. The warrior then studies the enemy and determines if the enemy himself is indeed a true warrior as well.

THIS IS IMPORTANT !!!!

The most effective players in the cyber battle are those with a warrior mentality, ON BOTH SIDES OF THE BATTLE. A warrior views the most effective enemy as a CONTEMPORARY, who is fully capable of being just as strategic and calculating as he is (if not more so). A warrior will watch his enemy take control of a battle and marvel at the strategic nature of the enemy with deference, and then file that away as another tool in the arsenal. He may never use this tool, but he will understand it enough to know when it is being used (or about to be used) again, and will know exactly what to do to either prevent it from being effective, or prevent the enemy from using it to begin with.

Let me return back to Earth for a moment and discuss my first major security breach as a CIO. I was in charge of a medium sized retail operation, with lots of remote retail locations logging the point of sale system into our corporate serves to perform transactions (using the Remote Desktop Protocol). It was an elegant solution that worked marvelously...most of the time. One afternoon I began receiving lots of complaints from the store managers because the system had slowed to a crawl. I asked one of my team members to look into it, and he found nothing out of the ordinary, but did indeed notice a spike in outgoing traffic that seemed to be sustained. Outgoing traffic spiking was not unusual, but sustained spikes were indeed out of the ordinary.

As I shifted to the hands-on approach, I noticed the traffic was on one particular port (I do not remember the port number), and it was not one of our known ports (21, 80, etc.), so I knew something was up. I was completely fascinated at this point (although I knew someone uninvited had entered our network), and began investigating. I narrowed the culprit down to one particular server, and a careful study of the server logs revealed activity going on in the recycle bin.

Huh!!!

As it turns out, someone from Korea (I traced the IP address to Korea) had installed an FTP server in the recycle bin on the Windows server, and was serving pirated Hollywood movies from my network. ABSOLUTELY BRILLIANT !!!

I was literally tickled pink by this feat of trickery. Why had I never thought of that back in my early days of "file sharing" cat and mouse gamesmanship? Touché indeed my Korean enemy. Well played!

Of course, I knew simply getting rid of the server was not the solution to my problem. It simply treated the symptoms (slow traffic). I had to then discover the weakness in my network "armor" that had allowed the infiltration to begin with. As it turns out, it was one of the many recently patched security holes common to Windows based systems (at the time), and I had been hit prior to the patch. I fixed the patch, and then made it my mission to very carefully monitor system changes on a very granular level, which led me to the discovery that attacks on my network were happening on a very regular basis (port probing, hammering, etc.). It allowed me to study and learn my enemies' tactics, and I soon discovered that there were a lot more attempts than there were victories. Yet when the victories happened (and they did indeed happen), I learned how to stop them, and they did not happen in the same way ever again.

You see, dear reader, I placed an ENORMOUS value on the victories, because they exposed my weaknesses. One cannot effectively determine a correct strategy unless one clearly understands their weaknesses. However, a warrior who has the ability to swallow his pride can significantly reduce the number of victories the enemy has if the warrior is willing to take a step back and enlist the advice of contemporaries who have already lived through the battles, and especially the ones who have the battle scars to prove it.

As I speak to more of the people who have enlisted in the battlefront in the cyber war, I consistently run into those who have been forced to deal with security as an additional headache to deal with, and those who have a true warrior mentality. What I have found is that warriors are very good at spotting other warriors, and can usually do so almost immediately. The conversation, at this point, takes a completely different tone. Even though I am coming to them (in part) as a consultant who is trying to win brownie points with my client, who is trying to determine market opportunities (after all, everyone needs to pay bills), we immediately move past that as the discussion now becomes far more temporal. We begin discussing the evolving threat landscape, the strategic nature of the environment, and the tenacity of the enemy. We laugh heartily (yet respectfully) at the hyperbole, and focus on the true threats to our mission, which we often determine, in part, come from our side of the battle (cost constraints, time sensitivity, corporate politics, lack of transparency, etc.). We sometimes venture off into discussions that have nothing to do with our current positions, and recount tales of battles past, and battles yet to come.

It is, I surmise, much like soldiers getting together on leave or after a war. Not having been an actual soldier I cannot say this from an experiential viewpoint, but I have known enough soldiers and watched them interact with each other to know that the similarities are indeed valid. I can tell you that it is also much like parents discussing the battles and victories in raising children, which is something I am indeed achingly (and pleasantly) familiar with.

So this brings me around to a very positive understanding of the battle from a US Government/Military being involved perspective. We often hear that the US Government is not very good at dealing with cyber security (and they have indeed admitted their shortcomings in that space), and that we are in big trouble because of that. I would tend to agree with such fear mongering if it wasn't for the indisputable fact that the US Government is arguably the best IN THE WORLD at gathering warriors to fight its battles. The US Military has spent literally hundreds of years perfecting the art of war, and despite their lack of understanding of the artillery (bits, bytes, laptops) or the enemy tactics (hacks) in the ever evolving battlefront, they are certainly quite good at understanding the warrior mentality. That is why, for example, I believe we are well served by having Michael Assante serving as the head of cyber security for NERC. His job is to separate what is "real" from what is "not real", and to manage the ever evolving battle plan, and make sure that every soldier (i.e vendor, utility, etc.) in the battle is following the plan. Nearly everyone I have spoken to that falls under what I deem the "Security Warrior" mentality finds Mr. Assante's assertions to be dead on. Mr. Assante, by the way, served in the US Navy for 6 years.

Some may argue that our US Military is nothing to be proud of, but I think they are missing the point if they do. We do not win every military battle we fight it, nor are we always justified in what we do militarily (which is subject to opinion), but we definitely are very effective the majority of the time. Moreover, we have certainly succeeded to the point where Americans find the prospect of domestic battle so foreign that something horrific (like the 911 incident) is seemingly incomprehensible. Yet foreign countries throughout the world deal with their own versions of the 911 incident every day. Our combined defenses and offenses have indeed managed to generate some ill will both foreign and domestic, but they have also managed to make us not only feel safe, but actually BE SAFE!

So I am hopeful and indeed confident that as we engage this cyber war with more of a warrior mentality that we will indeed manage to both survive and indeed thrive and feel quite safe as the battle evolves. We will engage the enemy, learn from the enemy, and indeed prove triumphant if we all swallow our self-serving interests long enough to take some cues from those who have received the battle scars.

...and frankly, I am excited about the opportunity to be a part of it.

The Evolving Compliance Landscape Of Cyber Security

Security, as it turns out, is largely about compliance. Anyone who has spent any significant amount of time working in security knows this all too well. Years ago my business partner and I worked on a very sophisticated health care project which involved cryptographically authenticated peripherals. The business model was such that a peripheral attached to a device was to be used once (and only once) on a patient, and then discarded. The doctor would then have to buy another (or have a stock of more peripherals) for another treatment.

We were brought in because the first generation product used a weak security solution, and it was hacked and counterfeited in 3 weeks (3 weeks from launch to counterfeits on the market)!

Bye bye business model.

So, having miserably lost the first battle in their own little cyber war, they decided it was a good idea to bring out the big guns, and they created a security solutions team (which essentially consisted of me, my partner, and the chipmaker we were working with) and we put together a very secure solution. It cost them more than their first solution, but they knew all too well the cost of failure, so it was easy for them to justify the increased expenditure.

Wouldn't it be nice if all security engagements worked out that way? Certainly for the security provider, I suppose.

We began focusing on health care security, feeling empowered by our previous engagement, and soon discovered that it was not a easy as we suspected. We touted the battle scars of our client as an indicator of the need to securitize their products, but failed in our attempts to generate revenue. It was frustrating, to say the least.

In all this, I learned quite a bit about the health care industry, and soon realized that everything in health care is compliance driven. Since the extent of security requirements for health care providers essentially falls under HIPAA regulations, all a health care organization is interested in doing is complying with HIPAA. Doing so essentially requires a security policy, and something as simple (and low tech) as requiring a 4 digit PIN to enter a system. It certainly is not something my company could sell into, since most of what is required by the client falls under the security audit, which is generally handled by the IT team medical clients already have. Since the requirements are so non-stringent, this is usually a 10-15 minute conversation.

Do you think I am exaggerating? I assure you I am not. One of the "side" jobs my company has been focusing on is iPhone applications for health care, and I can assure you that when the subject of HIPAA compliance comes up it NEVER lasts more than 15 minutes, and usually ends up with an agreement to enforce a 4 digit PIN (and a few other minor security additions). As developers who have worked in security development for quite some time, we do indeed build our software with an eye on security from the beginning, and our clients do indeed get a lot of security "freebies" because of that, but it is not because we are compelled to do so by any forces outside of our own need to not fall on our own swords as security professionals. In other words, we have chosen to self regulate our process, and our clients benefit from that.

While this is all good, and certainly makes us feel like we are doing the right thing from a due diligence perspective, it is only a small dent in the underlying battle. We know this from even further forays into working with cyber security challenges. The most interesting, perhaps, is with voting machines. Despite the bad press voting machine vendors received after many security professionals discovered gaping security flaws, they were indeed complying with requirements set forth by the election commissions in the states they sold into. In fact, the states themselves "ate" the cost of the insecure machines because of this, while the vendors got the black eye. Some vendors we worked with were indeed QUITE aware of how to build secure voting systems, and told us that such systems were unsellable because the states simply did not want to pay for them. All the vendors had to do was comply and then offer the best ROI to the clients in order to win the bid, and they did just that. It was not until after the security exploits were discovered that the US Elections Assistance Commission began taking security seriously.

Imagine that!

That does not mean that security was not considered...it certainly was. It is just that the deployment of security was weakened by a low threshold for compliance. What is even more important to realize is that the threshold is nearly impossible to determine without a proof of concept (moving from theory to reality).

Fast forward to cyber security, and specifically as it relates to the Smart Grid. Early deployments on the smart grid did indeed include requirements for security, and ALL vendors took it quite seriously (some more than others, as it appears). But this was not necessarily due to compliance issues, it was because (as one utility security expert put it), no utility company in their right mind is going to deploy something that is not secure.

...and yet, we now know that the grid has some fairly major security issues. How can this be?

Well, we have to consider security in the context of the perceived threat. For example, I can easily be shot while walking down the street of the bad part of town in any city, but that does not prompt me to invest in body armor (or even a bulletproof vest). Body armor is expensive and not very comfortable to wear, and since I have never been shot at while walking in the bad part of town, I am more than willing to rely on simpler and more cost effective security solutions (such as perhaps walking on better lit streets) to keep me safe. Moreover, I may very well get shot at and STILL decide not to get body armor. It simply takes a certain level of perceived danger for anyone to elevate their security requirements, and we really do not know what that level is until it happens.

So we stand in the presence of a smart grid deployment that is going pretty strong in the USA (on the order of 10's of millions of meters), and we have not borne witness to any major catastrophes yet. We have indeed proven that the threat is very real, and we are now working towards lowering the risk. The Department of Homeland Security, NIST, and NERC have enlisted the public and private sectors in the activities (and I have indeed joined in on the fun), and there are many smart people working on the challenges at every level you can imagine. Michael Assante of NERC co-authored an EXCELLENT article in the January/February 2010 issue of IEEE Security and Privacy magazine titled "No Grid Left Behind", and he methodically lays out the challenges and proposed solutions, and everything in the article is quite cogent. I have personally spoken to MANY members of the security community ranging from vendors, utility companies, PhD Scientists, meter manufacturers, crypto algorithm providers, and everyone in between. Everyone is working hard on the project.

Yet we have to understand that despite all the efforts to win the war (something that will never happen, as the war will never end) and prevent casualties, we are not going to come through this unscathed. Smart Grid deployments are vital to our existence because the energy savings have been proven. In a conversation I had with Echelon (a maker of AMI products) they have shown an energy savings on the order of 70% in some cases!!! That is a VERY significant number. When we consider the impact of energy savings on anywhere near that level, it certainly makes the case for Smart Grids a no brainer. I mean, think about that for a moment...saving energy by simply being smarter about where and when it is being used.

To me, as an energy consumer who spends over $500 per month to meet my energy needs (when you include fossil fuels), that hits home.

So what we have to understand that sometimes the missteps on the battlefront do indeed lead to things getting better. We have to understand that the fact that the various regulatory entities that are working towards solidifying and continuously evolving the standards (which vendors are indeed paying close attention to) are well aware that they are stakeholders in this ecology, and an insecure smart grid affects them on a very personal level. Michael Assante of NERC is a security whiz, but he is also well aware of the fact that the decisions he makes affect the outcome of this nation AND his personal life in a VERY profound way. Just like America came together to fight the enemy during World War II, after we had felt the attack at Pearl Harbor, we too can expect cooperation as we fight the ever evolving cyber security enemy.

After all, we are all on the front line.

Coordinating Efforts In Cyber Security

I have truly never seen a security initiative quite as interesting and massive as the cyber security effort as it relates to Smart Grid. The more I peel back the layers, the more I discover how massive the effort is.

Just a few days ago I was invited to attend a meeting of the American Bar Association (yes, the lawyer group), and the entire meeting (2 full days) is packed with presentation after presentation and working groups dedicated to cyber security. Yes! You heard that right. Lawyers are interested in understanding cyber security on a very detailed level. Fascinating!

I have spoken to utility companies, meter manufacturers, chipmakers, HSM vendors, cypto stack vendors, regulatory bodies...the list goes on and on and on. Everyone is contributing something to the effort of fighting the current cyber war we are all involved in.

Oh...you do not think we are at war? Well please reconsider. We may indeed be at somewhat of a stalemate at times, or even in a cold war, but we are indeed at war. The enemy is constantly hard at work trying to find new ways to break into our vast cyber network, and on every level you can imagine. It ranges from Aunt Judy's Facebook page all the way to our Missile Defense system, and the attackers are RELENTLESS (and also have a lot of fun doing what they are doing).

But enough fear mongering for now. We, as humans, are completely used to being vulnerable. We fashion synthetic skins from cloth and the pelts of animals to protect ourselves from freezing to death. If we no longer had clothing WE WOULD DIE! OH MY GOD! WE MUST PROTECT OUR TEXTILE INDUSTRY!!!!

You see how ridiculous hyperbole can get...time to switch to decaf.

Being subject to attack is something we are well aware of and we have simply coordinated our efforts to a level where we all understand what it is going to take to protect ourselves from bad things. We invent penicillin to prevent ourselves from bacterial infections, and we coordinate efforts to get it into the hands of all who need it. We make vaccines to stop the spread of deadly flu. We organize our defenses to prevent the bad guys from breaking down our walls and fortresses.

We all have a pretty good idea of how and when to coordinate efforts to keep ourselves protected, and we make improvements along the way. That is what we do.

Fast forward to Smart Grid security. The exploding cyber security industry as it relates to Smart Grid seemed to come out of nowhere. Sure, I was talking about it 2 years ago with some vendors, but it wasn't until quite recently that it became an area of more intense focus with nearly every technology company in existence (and those who are not in the game yet will be in soon). I have to say that I began seeing more activity happen in this area after the 60 Minutes episode on cyber security, where they showed a transformer (I believe it was a transformer) being destroyed remotely by a simulated cyber attack. Even I, who has worked in security long enough to know that there are A LOT of unresolved security issues in this world, was taken aback by this proof of concept.

Nonetheless, the troops went into action. We are now seeing a massive land grab as everyone in the world of security reaches for their piece of the pie, and many come up with big handfuls of job security as a result. I was talking to a cyber security engineer a few days ago who told me that he was out of work 2 years before landing a sweet job working for a utility. My own company was stagnating for the last 2 years as well, despite having worked on some very large security projects. In fact, we started writing iPhone applications as a side job, and were brought back into security because of Smart Grid. It is a huge project with a dire need for security expertise on EVERY level imaginable. Definitely some promising times for the cyber warriors of the world.

Yet I see some issues popping up that I believe are indeed quite counterintuitive to meeting this war with the extreme sense of urgency it deserves. One clear obstacle to cyber security excellence is our woefully luddite government (in the USA). They make no bones about the fact that they are bordering on the dark ages where it comes to providing expertise on cyber security. They are damn good at providing expertise on blowing things up, rebuilding them, and blowing things up again, but the only asset the government can contribute to the cyber security effort at this point is $$$$$$$...and they are...billions of $$$$ in fact.

Where the government becomes a bit of an issue and perhaps a bit of a hindrance is in the effort to mandate standards for cyber security. This is where things begin to get a bit annoying. Although NIST is hard at work at finalizing NISTIR 7628, and NERC is hard at work building a nice compendium of auditing documents for cyber security, and the CPUC is hard at work scoping their Smart Grid security initiatives, nobody can tell you what the COMPLETE rules of the game are at this point. Anyone who claims to know all the rules of the game is lying, because the rules are still being written as we speak.

So what rules are the players following in the absence of a coordinated set of rules from up above? Well, I can tell you that California is indeed following (to a degree) its own rules regarding privacy as it relates to Smart Grid (SB 1386), since California law requires privacy protection. I can tell you with certainty that there are a lot of VERY smart minds from the private sector working very hard to make sure that they can build the best security with the resources they are given, and that the resources are definitely getting better (albeit slowly). I can tell you that the guys I talked to at the major utility company are being funded by the utility to do some absolutely brilliant work, and that the cyber security engineer at one of the meter manufacturing companies that the utility sources meters from has some absolutely great ideas about what it would take to improve security. I can tell you that the utility guys would love to let their vendors know what they believe would be beneficial on the front lines.

Ahhhh...that is where things get a bit unraveled. Imaging my surprise when I spoke to the utility company cyber security guys and they told me they have never met or spoken to the cyber security architect at the meter company. What the ????

Lets get back to my original assertion. We are at war. Sure, we see no blood and body parts flying (ala Saving Private Ryan), but we must accept the fact that the warriors have, to a degree, laid down their rifles and booted up their laptops, and the pending cyber battles are going to take out a lot more troops than any rail gun ever could. While guerilla warfare tactics are indeed effective, a coordinated battle effort generally wins in the end. A mixture of both is perhaps the most effective, but only in the sense that the guerilla warriors are on the same page as the well regulated troops, AND (this is the big point) that we eliminate redundancy and activities that cause a regression in the efforts. If I, as an intelligence officer, decide that a great strategic tactic is to befriend an enemy and gather information, and manage to get a few cups of coffee, some cold beers, and perhaps a nice meal with the enemy in my effort to butter him up, it is rather annoying if the guerilla warrior decides to put a bullet in his head because he has a clear shot.

As a security professional who is interested in helping the cause, I constantly run into two very different scenarios. The first scenario is the fortress of silence and dismissiveness that many (but not all) of the players in cyber security put up when I try to get some information that would help me do my job better. The second is when I reach the guy (or gal) at the organization that truly cares about security and truly understands the need for teamwork, and appreciates those of us who have fought on the front lines of the ongoing security battle, who CLEARLY see the enemy for what he is...A PEER! No, I am not talking about the REAL enemy (the guy at the top of the food chain), I am talking about the security expert the guy at the top of the food chain has hired to do his bidding.

I am unsure of how coordinated the efforts of the bad guys are at this point. I am assuming it is fairly good because hackers seem to love collaborating (take BlackHat, for instance), and I am also assuming it is going to get better as the bad guys start getting better funding.

So are we going to take a cue from that...or are we going to continue setting up our own battle fronts as we wait from the orders from above?

As Goes California...

With over 10 million smart meters deployed to date, California (as the pilot for the US) has essentially acted as the pioneer of Smart Grid "movement" in the US.

Being a pioneer is not a new thing for California. We can thank California for equal rights, organic food, and much more. We can also thank California for being the first state to enact laws guaranteeing the privacy of data for its constituents, with the formation of the California Office of Privacy Protection in the year 2000.

I can remember when it happened. I was a CIO of a retail company in California when some of the first reports of major violations started to hit the airwaves. The CEO of the company started asking some questions, and I thankfully had the answers he wanted to hear. I had long considered data privacy and important issue, so I took the extra small steps to make it happen. He was pleased, and I think I got a raise.

Interestingly enough, the protection of data has always been THE most important consideration for those who practice the art of IT Security. Information Technology really began as a means of securing financial data, and in the old days the IT department did not fall under the guidance of a CIO, but was a sub-department of the office of the CFO. Once paper was replaced by bits and bytes, protecting the bits and bytes became a very important job, and IT Security became a cottage industry.

Fast forward to the new game in town - Smart Grid Security. Things are a bit different now. While the protection of data is indeed still important, it is NOT the primary focus. Smart Grid Security is focused on making sure that security breeches do not cause the system to slow down or (most importantly) stop functioning altogether. It does not take a lot of thought to understand why this is the way it is. Someone knowing how much electricity Mr. Jones is using and for what is not nearly as devastating as someone having the ability to shut Mr. Jones' electricity off. So one would surmise that the focus of vendors should be on availability and sustainability, and the rules of the game (i.e NISTIR-7628) certainly seem to point that way...

...but wait a minute. Let's examine NISTIR-7628 for a moment. This is where things start to get a bit interesting.

The September 2009 draft of NISTIR-7628, in section 3.2, discusses impact levels of 3 areas where security is an issue. From the document:

3.2 IMPACT LEVELS

The IAC impact levels are low, moderate and high. The levels are defined in Federal Information Processing Standards (FIPS) Publication 199, Standards for Security Categorization of Federal Information and Information Systems, February 2004. Following are the definitions for confidentiality, integrity and availability, as defined in statute and a table that defines low, moderate, and high impact.

CONFIDENTIALITY - “Preserving authorized restrictions on information access and disclosure, including means for protecting personal privacy and proprietary information...” [44 U.S.C., Sec. 3542]

A loss of confidentiality is the unauthorized disclosure of information.

INTEGRITY - “Guarding against improper information modification or destruction, and includes ensuring information non-repudiation and authenticity...” [44 U.S.C., Sec. 3542]

A loss of integrity is the unauthorized modification or destruction of information.

AVAILABILITY - “Ensuring timely and reliable access to and use of information...” [44 U.S.C., SEC. 3542]

A loss of availability is the disruption of access to or use of information or an information system.

A table below this section of the document categorizes the impact of security breeches, placing Confidentiality in the medium impact category.

...yet things change a bit with the release of the February 2010 version of NISTIR-7628. From the document, section 3.1:

3.1 CYBER SECURITY OBJECTIVES

In general for IT systems, the priority for the security objectives is confidentiality first, then integrity and availability. For industrial control systems, including power systems, the priorities of the security objectives are availability first, integrity second, and then confidentiality.

Availability is the most important security objective. The time latency associated with availability can vary:

␣ 4 ms for protective relaying; ␣ Sub-seconds for transmission wide-area situational awareness monitoring;

␣ Seconds for substation and feeder supervisory control and data acquisition (SCADA) data;

␣ Minutes for monitoring non-critical equipment and some market pricing information; ␣ Hours for meter reading and longer term market pricing information; and ␣ Days/weeks/months for collecting long term data such as power quality information.

Integrity for power system operations includes assurance that: ␣ Data has not been modified without authorization; ␣ Source of data is authenticated; ␣ Timestamp associated with the data is known and authenticated; and ␣ Quality of data is known and authenticated.

Confidentiality is the least critical for power system reliability. However, confidentiality is becoming more important, particularly with the increasing availability of customer information online.

As the document continues on, the impact level table becomes more granular, and discuss the impact by logical interface, confidentiality coming in 3rd place (as the opening statement indicates).

So what am I saying here? I am trying to make an important point. The protection of data privacy is not a primary consideration at the Federal level (or so it would appear from the February 2010 document), and I have to agree that this shift in thinking certainly seems to make perfect sense when you consider the impact of the system shutting down when compared to the impact of a breech in confidentiality. However, California, remember has their own privacy laws that, essentially, make privacy a PRIMARY consideration. In fact, I believe that is why the September 2009 document put confidentiality higher on the priority list.

So one can perhaps safely surmise that vendors have been forced to build privacy into the systems they are deploying in California in the absence of any solid standards from the Federal government. I say this because, as anyone who has been following the development of smart grid standards is keenly aware, the standards are currently in a state of flux (as can be seen by simply comparing versions of NISTIR-7628).

So even if the final standards place confidentiality dead last in consideration, with California leading the way AND having their own rules to follow, it stands to reason that vendors are going to have to build privacy into their systems, or potentially face elimination as they attempt to grab a piece of the pie. It also stands to reason that privacy will become a much bigger consideration as the deployment grows, and as data management becomes a cottage industry (i.e. 3rd party companies providing services to consumers based on their usage data).

As the saying goes "As Goes California, So Goes The Nation"...like it or not!

Smart Grid Security Performance Standards

I have just gone through the somewhat laborious process of reviewing

February 2010 DRAFT NISTIR 7628

Smart Grid Cyber Security Strategy and Requirements

which I found through Smart Grid News (http://www.smartgridnews.com/artman/uploads/1/nist_cyber_security.pdf).

Section 3.1 touches on crypto, and the general tone of the document would suggest that logical (software) security is the method of choice due to the performance hit a system must take when implementing more secure levels of crypto (i.e. secure microcontrollers). Since performance is so important, one would logically conclude that hardware based security introduces challenges. Correct me if I am wrong, but that is how I interpret this.

That being said, what seems to be lacking is any sort of reference for performance. Since hardware based security is, in many ways, superior to logic based security (not always, but the BEST hardware based security chips are far more secure than the best logic based counterparts), then we really need a frame of reference here. Granted, no hardware based security solution will ever be able to match the performance of a logic based system (in fact, you can get maximum performance by simply using buzz words to describe a part of the system - like 256 bit encryption), but the best secure IC's deliver some pretty good performance while offering some very solid security. After all, banks rely on hardware based security (i.e smart card based security) for their most critical systems, and system availability and reliability are directly tied to the very high performance requirements vendors must adhere to in order to sell to the banking industry.

The one issue that does come into focus, of course, is budgetary constraints. Vendors of AMI systems must compete to sell their products, and the increased cost of implementing secure microcontrollers that deliver the requisite level of performance cuts into everyone's bottom line (which ultimately is the TRUE deciding factor). Logic based security can be implemented for anywhere from $0 to fractions of a cent, while high performance and high security hardware costs more.

Nonetheless, if cyber security is such a major concern (as it should be) in the implementation of the Smart Grid, then we should perhaps seek to create some target objectives for vendors of hardware based security, including performance and cost. We should also view the total cost in a systemic manner, taking into account the risk of relying on logic based security, and the cost of failure.

After all, if we are building an infrastructure that is expected to remain a part of our critical energy infrastructure for MANY years to come before replacing it with the next best thing, we should probably create solid, tactical objectives as well as higher level objectives.

It really takes both to succeed.

Auditing Smart Grid Security

In my quest to better understand Smart Grid security initiatives, I have managed to gather quite a bit of useful information regarding emerging standards, layered security, and real world deployment of security in the Smart Grid. It is still a work in progress, but it is progressing, and a lot of smart minds are fueling that progress. This is all very good.

What seems to be missing, and perhaps remains as a great opportunity for those making the "land grab" on the smart grid is a cogent auditing and control methodology specifically targeted towards Smart Grid security. What I am talking about is a set of auditing requirements similar to what we see with the FDA.

Okay! Granted, the FDA is not perfect. They are not even close (in the opinions of many, I am sure). Nonetheless, the rigorous auditing procedures that health care organizations who fall under their requirements must adhere to does indeed serve two very important purposes. The first purpose is in forcing organizations to follow best practices and keep very good records (some better than others, I am sure). The second is instilling confidence in consumers globally. Did you know, for example, that FDA clearance/approval in the USA essentially guarantees clearance/approval anywhere else in the world?

At the heart of this process is a set of great auditing procedures that have been hammered out for last 100 plus years the FDA has been in existence, and many organizations have made a cottage industry out of providing auditing services for the organizations that need to fall in line. Organizations such as the American Society for Quality (ASQ) have created certification programs for auditors for the health care and essentially every other industry.

...but I am trying to get to a point with all this. I am trying to deal with a somewhat significant challenge. The standards being proposed by the NIST Interoperability Document Version 1.o alone encompass somewhere around 75 different standards. This can serve to create quite a bit of consternation and confusion for some, but bear in mind that there are literally hundreds (if not thousands) of standards used in the health care industry to "secure" our well being. Rather than focus on what specific standards are followed, auditors look at the big picture created during an audit, and determine if it passes the sniff test. If it does not, then its back to the drawing board.

This is not a joyful occasion for anyone involved (including the auditors), but it does lead to better products and systems (as well as annoying bureaucratic messes). Yet what makes this work is properly designed and well-vetted auditing guidelines and procedures.

We do not, unfortunately, have 100 years to iron out the inevitable wrinkles of Smart Grid security deployment, but we do have quite a few great auditing professionals (quality professionals, in fact) hard at work every day, and many are still looking for work in this down economy. With a 100 years of well documented procedures in place for the FDA, one could indeed surmise that the application of the same (or at least similar) methodologies could SIGNIFICANTLY curtail the development of a workable Smart Grid security auditing procedure.

...I'm just saying.