Sorry Health Care...Game Over!

Back in the day when pinball machines were all the rage at game arcades (and the only video game was pong), one could bear witness to a room full of pre-pubescent boys and girls (mostly boys as I recall) pumping quarters into machines and batting steel balls with rubber coated "flippers" in order to prevent the ball from falling into the shoot. The longer you could keep batting that ball the more points you would score. If you were particularly well versed in the art of pinball machines you could shake and tilt the machine to a degree (avoiding the inevitable "tilt" caused by being over-zealous) and perhaps score more points by making the ball move where you wanted it to move.

Nonetheless, this mastery of batting the ball and shaking and gyrating the machine came to an end for even the most skilled of pinball wizards. Many players would continue to shake and gyrate the machine after their last steel ball (back than you got at least 3 balls) had fallen, but there was simply no denying the reality of what the scoreboard prominently displayed in bold letters...GAME OVER !

Sure, those with paper routes or other sources of quarters could keep pumping legal tender into the system to give it another go around, but the result was inevitably the same. Eventually you have to give into reality. You can't bat the ball around forever.

The idea of health care organizations having to take responsibility for security and privacy in an ever expanding digital age is certainly not new. The first HIPAA regulations passed in 1996. I am no math genius, but that is about 14 years by my calculations. In 14 years, however, health care organizations and providers have been lax in dealing with security. I currently serve on the CalPSAB security steering committee, and that seems to be something we all agree on (actually, we seem to agree on a lot more than that). Having just returned from the Safeguarding Health Information: Building Assurance through HIPAA Security conference in Washington DC, it seems quite clear that the Office of Civil Rights (OCR) and Federal Trade Commission (FTC) are also aware that a lack of due diligence on the part of health care practitioners (and business associates) with respect to security and privacy has gone on long enough.

Nonetheless, we still see organizations (such as The American Medical Association, American Osteopathic Association and Medical Society of the District of Columbia) fishing for more quarters to pump into the machine.

Hey! Why not? They have plenty.

In an article published on the excellent Health Data Management Blog, the author references a lawsuit filed by the aforementioned entities. The essence of the lawsuit is that health care organizations do not want to fall under the authority of the FTC with respect to the "Red Flags" rule the FTC currently requires creditors to abide by. The (ridiculous) claim being made by the filers of the lawsuit is that (from the article):

Among other factors, the medical associations argue that physicians are not commonly referred to as "creditors," nor are patients ordinarily thought of as "account holders" or "customers."

Wow! Am I understanding this correctly? This is coming down to a definition of what a "creditor" or "customer" is?

At the Washington DC meeting one of my takeaways was that OCR is really putting the hammer down, and perhaps they should consider less "stick" and more "carrot" in dealing with organizations that have to comply with the rules. However, when I witness the equivalent of a bunch of pinball wizards banging on a machine as they fish for more chances to avoid the inevitability of owning up to the fact that batting balls around eventually loses its charm, I shed some of my sympathy.

The Health Care Industry simply cannot keep playing this game forever. It is time to focus their energy on ways to address security and privacy concerns in a meaningful way, and stop fighting what is inevitable.

GAME OVER!

Privacy: A Prescription For Disaster

I have been watching the world of cyber security unfold for the last several years in a manner I would best describe as divergently focused. As a security professional who frequently engages in deep (almost philosophical) discussions with other security professionals (or more appropriately, Security Warriors) I am constantly amused at the frustrations we seem to share about privacy being the biggest driver of security in emerging technological initiatives.

I recently wrote about this on the topic of Smart Grid security, and Gib Sorebo of SAIC followed up on his blog with his opinion. Gib and I have had some long and great discussions about the issue of privacy in a world of security vulnerabilities, and the one area that seems to get us both preaching is the issue of privacy as it relates to health care. Simply put, this is a good time to shift the security discussion to something that really matters, and I am sorry to say that privacy needs to leave the room for a while.

Okay, I am sure the entire world of privacy evangelists are probably going to want to send me that nasty fruitcake (or worse) they have been holding onto for the last 20 years after reading that last statement, but please hear me out before you head over to the post office. Privacy IS important and DOES MATTER to me and probably every security professional in this world. I am a strong supporter of privacy, and consistently do all I can to protect my privacy. I refuse to give my address and phone number at stores that ask for it when I pay with cash or ask me for that information for any reason whatsoever. I refuse to share ANY information with ANY entity that requests it that I deem is not on a need-to-know list, and have held up lines in stores, banks, and other places (sorry to all of you who stood behind me) defending my rights to my own information. Privacy is indeed very important in the digital world we are now completely enveloped in.

...but it has got to stop being a part of health care security discussions, or we are probably going to end up with a lot of dead people as a result.

In fact, we already are ending up with seriously damaged patients in the age of digital health care. I read an article on The Huffington Post this morning titled "Electronic Medical Record Shift: Signs Of Harm Emerge As Doctors Move From Paper" which pointed out how either bad information or a failure in software has led to patient trauma (heart attacks, seizures). The article did not speak of security issues that led to failures in these systems, yet the failures found in these systems serve to illustrate what I have been talking about for years. If a system is vulnerable to penetration and compromise by an attacker, the attacker can cause a lot more harm than a patient would suffer as a result of a privacy breach.

Let me specifically paint a scenario based upon the Huffington Post article. The first sentence of the article speaks of hospital workers misreading medical dosage information and dispensing 10 times the normal dose of a medication, leading to a patient heart attack. Under HIPAA HITECH, if an attacker should enter a system and change a single patient record (perhaps a patient who is a political figure) for a medical dosage to purposely cause a heart attack or death, the health care organization would be in violation of a privacy law, but could not be held liable for the death of the patient due to a failure in data integrity. In my opinion (and the opinion of others I have spoken to about this issue), there is something very wrong with this.

The problem becomes even more complicated when you add medical devices to the system. Medical devices have become increasingly "smart" and are now trusted devices on health care networks. Devices perform many functions in health care, and the information some devices are trusted with gathering is often used to make life or death decisions. A device which automates the process of typing blood and then sends the information to a patient record database is indeed one type of device that would fall into this category description. If an attacker could spoof such a device he could then populate the database with incorrect information that could kill a patient. Moreover, some medical devices have firmware that can be updated (and in some cases over a network connection), which opens up the possibility of rogue firmware that could be purposely introduced to cause havoc.

I bring this up because the cost of failure due to a privacy breach simply pales in comparison to the potential cost of failure due to a failure to deliver correct information to the system. One leads to embarrassment and potential financial headaches, the other leads to death. Why is this distinction important? Well, except for obvious reasons, it is important because in a world where risks are mitigated based on costs of failure from a LEGAL perspective (i.e. a finable offense), the actual cost of failure due to a privacy breach is infinitesimally small compared to to somebody dying. A good lawyer can potentially turn a $1.5 million dollar fine (the maximum fine for a single instance under HITECH) down considerably if he or she could convince a judge/jury that the punishment does not really fit the crime.

It happens all the time, in fact, in other industries. At one time I worked for a company that dealt in motor oil who faced millions of dollars in fines from the EPA for statutory violations, but the fine was reduced to a few thousand dollars because the violation simply did not lead to anyone being harmed. The potential for harm was very high (as is true with medical record breaches), but if nobody is actually harmed then a slap on the wrist is a common punishment (especially if you have a good lawyer). Sure, it may cost you in legal fees, but if you already have a staff of lawyers anyway it is not so hard to stomach.

HITECH is a good step in the right direction for better security, but it still completely fails to address the bigger issues. As we continue to build out our "internet of health care" and interconnect data sources at a national (and eventually global) level, the security risks grow at a nearly exponential rate. This is because attackers like to attack systems more as they get bigger simply because it has a bigger impact. We should not wait for theoretical dangers to manifest themselves before we address these issues. Security vulnerabilities of large infrastructures are well known enough today that a failure to pro-actively address them is simply nothing more than negligence, and the health care industry should act more responsibly.

They know better.

The Grid Reliability and Infrastructure Defense Act- Better Late Than Never

As I have discussed many times in the past, security is primarily driven by compliance.

Wait...let me back up for a moment.

While many organizations (and particularly those who are involved in The Smart Grid) are indeed elevating security on the priority scale of "things we gotta do", we can be certain that any organization that has felt the pain of an attack will do more to secure their deployments than one who has not had the displeasure of being "owned" by an attacker. While some may argue that this is not the best way to get security into a system, I will argue that it is indeed the most effective driver of security. It is human nature to react to known dangers rather than proactively defend themselves against them. Moreover, we tend to proactively secure ourselves only if the known threats are directly experienced. Simply knowing someone who has been mugged in "the city" is not enough to get most people to become exceedingly aware of their surroundings, after all.

So with the Smart Grid we are in a situation where vulnerabilities have been discovered, and many more have been theorized. While nearly everyone who is involved in Smart Grid is indeed paying attention to security, turning that into "action items" remains a bit nebulous. Utilities who are actively deploying AMI (such as PG&E and SCE) are indeed focusing what I believe are tremendous (and competent) resources on Smart Grid security, and others are paying close attention (as I have gathered from various Smart Grid groups I am involved in). Vendors have created cyber security specific positions and departments. Security consultants are now specializing in smart grid security consulting. The US Government has several groups addressing the issues (FERC, NERC, NIST, DHS, DoD) in various capacities, and the list goes on and on.

The reason I say this is all a bit nebulous is because so far we have been lacking an authoritative mandate for Smart Grid security. Sure, NERC has been working on compliance and auditing standards (NERC-CIP 002-009), but neither NERC nor any other entity has the CLEAR authority to "lay down the law" as far as Smart Grid security is concerned. Each individual state has the power to halt Smart Grid deployments (I would surmise) for any reason whatsoever, but at a national level it is still very laissez-faire. The unfortunate negative consequence of this is that states (such as California) have adopted a bit of a "hurry up and wait" mentality about security (despite the fact that California doing this with voting machines was an epic disaster). This is never a good thing, because if (and when) security issues manifest themselves, the typical response is to halt progress until a resolution is reached (again, such as happened with voting machines). This is, to say the least, very irresponsible, because as far as the Smart Grid is concerned we NEED to have it deployed NOW in order to deal with the ever increasing demand for electricity. Consider electric cars, for example. Exactly how do we expect to manage load if California has millions of electric cars plugged in and charging on a hot summer day? Our current system can barely manage the load with no electric cars on the road, with high peak air conditioning usage days leading to power outages. We NEED the Smart Grid.

I was happy to see an article on TheHill.com that spoke of the House passing the Grid Reliability and Infrastructure Defense Act (GRID) which seeks to up the ante on FERC to take control of security issues affecting the Smart Grid. I am not generally fond of Congress passing laws that serve to penalize those who do not comply, as this generally leads to more consternation and less solution (in my opinion). So I was happy to see a section of this bill which seem to instead focus on providing resources to entities that are deploying the Smart Grid. From the bill:

COST RECOVERY.—If the Commission determines that owners, operators, or users of the bulk-power system or of defense critical electric infrastructure have incurred substantial costs to comply with an order under this subsection and that such costs were prudently incurred and cannot reasonably be recovered through regulated rates or market prices for the electric energy or services sold by such owners, operators, or users, the Commission shall, after notice and an opportunity for comment, establish a mechanism that permits such owners, operators, or users to recover such costs.

Now I know this is not very specific, but it does seem to address perhaps the biggest concern businesses involved in Smart Grid deployment may have in addressing security - COST $$$$.

It is not a law yet, and it may indeed go through some changes (perhaps not for the better) as it makes its way towards becoming a law, but I have high hopes.

...and hope springs eternal.

The Need For A Security Paradigm Shift

I remember years ago, when Stephen Covey's bestseller The Seven Habits Of Highly Effective People was making its rounds throughout the business world, the introduction of the word "paradigm" in my vocabulary. I was working in a resort way back then and our director of operations used to love walking around and tossing the term out like Rockefeller gave away dimes to the poor. He was a great operations director, and certainly was not deserving of the gentle ribbing he took for the liberal use of a term that nobody in my world seemed to want to care about. Frankly, most of us cared more about changes in our scheduled shifts more than we cared about "paradigm shifts".

Still, I did indeed listen intently to what he had to say. I liked him a lot, and he liked me. He convinced me to read Covey's book, and I gained a better understanding of several concepts, most importantly the concept of the paradigm shift.

To summarize my understanding of it in as few words as possible is perhaps something I am incapable of, so I defer to a definition I found while perusing the venerable Wikipedia. Here is the section I found best describes it:

the historian of science Thomas Kuhn gave paradigm its contemporary meaning when he adopted the word to refer to the set of practices that define a scientific discipline at any particular period of time. Kuhn himself came to prefer the terms exemplar and normal science, which have more precise philosophical meanings. However in his book The Structure of Scientific Revolutions Kuhn defines a scientific paradigm as:

• what is to be observed and scrutinized
• the kind of questions that are supposed to be asked and probed for answers in relation to this subject
• how these questions are to be structured
• how the results of scientific investigations should be interpreted

The bullet points capture the essence of what I believe is absolutely critical as we continue to discuss the topic of securing the smart grid.

Anyone who knows me knows that I am generally a very positive person, and generally give most people the benefit of a doubt. However, you also know that I tend to not suffer foolishness lightly. I call things like I see them, and although I am sometimes way off base, I am on target often enough to cause those I target (using my Socratic methods) to feel a bit uncomfortable. To those of you who I have made uncomfortable, my apologies for making you feel uncomfortable. My intention is not to get you to dislike me. My intention is to get you to see things differently, or to get you to shift your paradigm.

What led me to this blog posting was an article I read titled Securing The Smart Grid by Elinor Mills. This article is a combination of what I believe is sound information layered with generous doses of conjecture. I am not going to get into what I believe is conjecture at this point, since that will indeed take more time than I have this morning. What I did find worthy of calling out, however, was a quote made by Jesse Berst of Smart Grid News:

Jesse Berst, managing director of the Global Smart Energy consultancy and founder of Smart Grid News, said he didn't see any reason why the energy industry wouldn't be able to secure the infrastructure as it modernizes.

"The physical security concerns me more than the cyber security because we've solved the cyber (security issues) for other big consequential infrastructures (like financial and Internet) and I think we can solve it to that same degree of safety for this one," Berst said.

Now let me preface this by saying that I believe Jesse's contributions to the entire world of Smart Grid are indeed beyond admirable. I read Smart Grid News on a daily basis, and find it to be a wealth of information. I will also be the first to admit that he is light years ahead of me (and perhaps a lot of people) in his understanding of The Smart Grid.

...however, his statement "...we've solved the cyber (security issues) for other big consequential infrastructures (like financial and Internet) and I think we can solve it to that same degree of safety for this one," truly left my mouth hanging open.

Are you kidding me?

Okay, maybe CNet took that out of context. God knows the media seems to do that with more frequency than we would like to see. So I will indeed work with the assumption that this may be the case, and dissect this statement as one that may indeed be put forth by someone who may not be aware of how a security professional (such as myself) might view it.

Let's start with the first part of the statement "..we've solved the cyber (security issues) for other big consequential infrastructures (like financial and Internet)".

Really?

I am not entirely sure where to start with this one. Let's just take financial to begin with, and describe how we have "solved" those issues. Despite having "solved" the cyber security issues with respect to the financial world, the financial industry still loses billions per year due to cyber attacks, and then passes these losses on to the consumer. One "solution" the financial industry put forth several years ago was PCI Compliance, which simply shifts losses to merchants, who then are forced to raise prices to cover the losses. Another "solution" is to jack up credit card fees and interest rates ("risk management" as they like to call it) to cover the losses that the financial industry cannot pass on to the merchants. Sadly, there is no way any consumer can avoid falling into this abyss. If I do not want to use credit cards I am hampered by having to write checks or use cash for anything and everything. I also must live with the cost of failure that cyber crimes impose on my merchants through higher prices.

Such is life. Do I get by despite this mess? Certainly! Is the problem "solved"? Nope! In fact, I am not sure it can ever be solved. What I am sure of is that we seem to be able to live with this particular cost of failure in cyber security, and that may indeed be good enough (for now).

It is the second part of the statement "I think we can solve it to that same degree of safety for this one", however, that got me to bolt out of bed and start writing. Here is where the entire world of Smart Grid security "apologists" need to go through the mother of all paradigm shifts. Solving the security issues to "the same degree of safety" where The Smart Grid is concerned does not quite seem to cut it, now does it? Let me explain.

Let's consider the cost of failure.

While we live in a world of hyperbole in the world of cyber insecurity, we now also live in a world where some of the inherent weaknesses in the Smart Grid security arena have made the transition from theoretical to proof of concept. Perhaps the most famous of these is the infamous Aurora Attack seen on 60 Minutes (that was when my phone started ringing). What that showed us was that the cost of failure in security could lead to power being shut down in some areas for months. Now I know that some of you are going to want to attack this by telling me that we can simply redirect power from elsewhere, and that is indeed true, but what you would probably leave out of that statement is the fact that redirecting power during the middle of a blistering summer heat wave is next to impossible, and I am quite sure that a malicious attacker (not just a script kiddie) is keenly aware of that.

...and there is more. A lot more in fact. Utilities are keenly aware of the issues, and so is our government, and they do indeed care A LOT about security. Way more than the media is willing to give them credit for. In fact, I have never seen an industry embrace the importance of security with such fervor as the power industry has in the last year. In California both PG&E and SCE have invested considerable resources in dealing with these issues. One member of the cyber security team at PG&E sent me a message at 10:50 PM several nights ago in response to a question I had (regarding a conference I am planning). I was surprised that he replied so late and he informed me he was still at work! When I asked him why, he told me that he (and others on his team) often work late. When I see this level of dedication from security professionals I do indeed feel quite comfortable about the work being done towards securing our grid, and so should others (in my opinion).

Nonetheless, there is a dire need for a paradigm shift in the discussions surrounding Smart Grid security. We cannot use examples where the cost of failure truly pales in comparison to the cost of failure when we have no electricity. Imagine, if you will, a scenario where someone hacks your bank account and takes all of your money. You contact that bank and it can take several weeks to get your money back. I know this to be true because I know someone who went through such a nightmare. Nonetheless, she did not go hungry or die. She had food in her house, and credit cards, and family and friends. It really amounted to a nasty inconvenience, and she got all of her money back eventually. The "degree of safety" built into the system was indeed more than adequate to deal with this situation, but nobody who has their nose to the grindstone in the world of Smart Grid security would consider this to be a valid presumption of having the situation under control. We are fortunate enough to have only a small (yet significant) fraction of our power infrastructure on The Smart Grid, and everyone involved is working hard to deal with the issues at hand.

Let us avoid out of context statements making their way into the public consciousness, which will inevitably lead to a loss in credibility for those who are working hard to resolve these issues. The public loves to dwell on the negative even more than the media loves to talk about it.

In other words, let's not fuel the naysayers.

Bravo PG&E! The Proactive Approach Always Wins

I am once again making my way through comments to the CPUC, and was extremely pleased to find a comment made by PG&E:

PG&E AGREES WITH THE CUSTOMER PRIVACY GOALS AND POLICIES RECOMMENDED BY CONSUMER PRIVACY ADVOCATES

A number of consumer groups have provided specific recommendations regarding customer privacy goals and standards in this proceeding, including the Center for Democracy and Technology, Electronic Frontier Foundation, Consumer Federation of California, TURN and the Division of Ratepayer Advocates.5 These recommendations generally urge the Commission to move cautiously and very carefully in updating or revising its existing rules on customer privacy, particularly third-party access to customer data. In addition, the consumer privacy advocates point out that there are certain sources of national “best practices” for protecting consumer privacy in all industries that the Commission should consider and endorse, such as the “Fair Information Practices Principles” developed over the years and cited by various federal agencies, such as the Department of Homeland Security.6

As PG&E pointed out in its opening comments, we adhere to existing, strict and precise Commission and statutory rules and standards providing for protection of customer privacy, and do not see the need to dilute or reduce those protections. However, after reviewing the comments and presentations by consumer privacy advocates, PG&E agrees that it is timely in

this proceeding for the Commission and all parties to “benchmark” the existing customer privacy protections in the Public Utilities Code and utility tariffs against the national consumer privacy standards and goals applicable to other industries and consumer services. This is particularly important to the extent that the Commission will need to establish and be able to enforce these privacy protections and cyber-security protocols against third-parties who may be granted access by customers to sensitive or confidential personal information. For this purpose, we agree that the Commission and interested parties should start with review of the “Fair Information Practices Principles” and other national consumer privacy laws and guidelines, and evaluate whether enhancements or improvements in current Commission and utility practices should be considered in light of those national guidelines. PG&E recommends that this “benchmarking” effort be an integral element of the utilities’ SB 17 deployment plans.

My take on this is that PG&E is indeed committed to not only dealing with security issues head on, but is also sensitive to the concerns of the various agencies who are voicing their concerns. Moreover, they are willing to voice their commitment directly with the CPUC and essentially tell the CPUC that the ball is in their court.

I look forward to other key players in Smart Grid coming forward with this level of commitment! PG&E has been a leader in Smart Grid since day one, and leadership is what drives excellence.

I anxiously await the CPUC's decision on this.

Smart Grid Security and the CPUC

I am sure everyone can cite an example in life where various entities (including oneself) are forced into accepting responsibilities they may or may not be prepared to accept. This generally comes about in a somewhat organic manner, and sometimes hits a point where the duty of care rises in a nearly exponential manner, which creates a situation where all activity is then reactive in nature. This is not necessarily the best position to be in, but it happens quite frequently.

An example of this would be the extremely rapid growth of the automotive age in the USA. When automobiles first appeared on the scene, they were indeed a novelty, and certainly nothing worthy of considerable regulatory control. They shared the road with pedestrians, horses and buggies, and bicycles, and as I understand it were quite amicable about it (for the most part). Automobiles were built according to manufacturer specifications which were fully controlled by the manufacturers, and they were generally built with aesthetics, functionality, and profits as the key drivers.

As we all know, the automobile industry quickly made the transition from novelty to way of life within the blink of an eye. It did not take long until the automotive industry become the cornerstone of American manufacturing, and consequently an enormous influencer of economic dominance.

In other words, it got very damn big really damn fast.

As this growth progressed, it soon became apparent that we had to make major changes in the way we, as Americans, did things. It was no longer prudent to assume that everyone would share the road in a somewhat Utopian fashion. While automobiles created ENORMOUS benefits for society, they also introduced SIGNIFICANT challenging issues, many of which were indeed quite dangerous. I am certain I do not need to articulate all of these, since we are all quite aware of the many dangers associated with an anarchistic automotive culture, especially since we are all quite aware of the significant dangers we all face in an automotive society where most people do indeed (by and large) follow the rules. This is quite evident when a solitary driver loses control of a vehicle on a busy highway and causes massive multi-car collisions. It is truly amazing that we all manage to avoid such chaos as well as we do, but we do indeed manage.

...and why is that?

Well, it is because we have learned to do so the hard way. Most of the rules of the automotive ecosystem have come about as a direct result of lots of people dying, or lots of people being forced to live in a society where quality of life is negatively impacted. Seat belts, for example, where not always required in cars. In the early days of seat belts they were an option. They appeared in cars in the early 1900's, but were not required in US cars until the 1970's. In fact, most of the safety standards that exist today for the automotive industry did not appear until the 1970's, and NOT because the US Automotive industry decided it was a good idea, but mostly as a result of the crusades of the venerable Ralph Nader. Seat belts became standard equipment as a direct result of Federal legislation mandating that all carmakers include them. However, it was not until laws were enacted mandating their use by passengers that we began realizing a decline in traffic related deaths. To quote a section of an article from The Prevention Institute:

Mandatory laws have proved effective both in increasing seatbelt usage and decreasing traffic fatalities. The Centers for Disease Control and Prevention reports that seatbelt use nationwide increased from 11% in 1981 to 68% in 1997. NHTSA reports that the motor vehicle fatality rate as measured per 100,000 population decreased from 21.49 in 1981 to 15.69 in 1997, and also decreased as measured by 100 million vehicle miles traveled, from 3.2 in 1981 to 1.6 in 1997. While these decreases cannot be attributed to the use of seatbelts alone, seatbelts are credited with playing a significant role in these advancements.

Enter California. Being the state with both the largest population of people, the largest economy, and the most cars, California is always a good place to go for significant statistical samplings. California is somewhat notoriously well know for being a first mover on many initiatives that tend to serve the public interest over corporate interests, often at the behest of corporate interests. This, as it turns out, is a very good thing more often than not. When the rest of the USA was (and still is) debating the efficacy of environmental controls, and what should and should not be enforced to preserve our environment, California simply forged ahead and passed what many industry leaders deemed punitive measures to prevent (among other things) automobiles that emit large amounts of pollutants. Because the State of California represents the largest customer base for car manufacturers (1 in 11 cars is in California), the automotive industry was left with two choices. Either they could conform to California emissions laws, or they could find another less stringent customer base.

We all know which way they went, and now that I live in California I am extremely thankful. Anyone who has lived in California's most populous areas knows darn well that there are A LOT of cars here and they are on the road all the time. Prior to the enactment of state mandated emissions standards California regularly had days where the pollution was so severe that people were warned to stay indoors with their windows shut. While California does indeed continue to face air quality challenges in populous areas, the attention this has gotten at the state level has resulted in significant improvements in air quality. Perhaps most importantly, the significant changes automotive manufacturers had to make to their products in order to do business in California has led to a fundamental change in manufacturing which the entire US (and the world) now benefits from.

Let's go back to seat belts for a moment. Despite the mandate that seat belts must be worn when driving, compliance increases only as enforcement increases. This is true with nearly every rule of society, and certainly with rules where the manifestation of ignoring the rule rarely leads to any negative consequences. In other words, since most people who do not were a seatbelt when driving do not experience any negative consequence from not wearing one (they normally do not die or bash their heads against the dashboard or steering wheel), it is quite easy to forego this preventative measure. Compliance with seat belt laws got better as police officers began issuing citations to those who failed to use them. This generally came about as a result of officers pulling over drivers for a non-seatbelt related violation, and issuing a citation for not wearing a seat belt in addition to whatever they had been pulled over for. This is known as a "Secondary Enforcement" law. California, however, is a "Primary Enforcement" state with respect to seat belt laws. In California, an officer can pull a passenger over for the sole reason of failure to wear a seat belt, and issue a citation accordingly. As controversial as this law has become, it has led to a SIGNIFICANT decrease in fatalities. From the same article cited earlier:

California, a primary enforcement state, currently reports 91% usage -- the highest in the country. After the passage of a mandatory seatbelt law in 1986, California's usage rate went from 26% to approximately 45%. By 1992, California's usage had increased to 71%. With the passage of the primary enforcement law in 1993, California's usage rate jumped to 83%, steadily climbing to the current rate. According to the National Safety Council, California's fatality rate has decreased by over 34% since the passage of the primary enforcement law.

Wow! A decrease in fatality rate by over 34%. I would say that is pretty darn significant. I would also say that it is probably NOT all solely due to the primary enforcement, since California also spends a significant amount on programs to continually educate the population on the importance of safety.

As a staunch libertarian by nature (notice the small "l") I generally oppose government intervention. In fact, prior to moving to California I was convinced I was going to find the imposition of such a notoriously intrusive set of rules for my well being to be intolerable. In fact, however, I find it quite nice. I was a smoker when I moved to California, but the nearly militant anti-smoking sentiment coupled with the heavy handed enforcement of anti-smoking laws found throughout the state have forced me to rethink my addiction, and led to a lifestyle which I find far more appealing, since hacking my lungs out every morning was not something I looked forward to every day. I am now living a smoke free life for going on 7 years! The FACT is that smoking is a horribly dangerous health hazard that I am better off not taking part in. Surely I can still buy cigarettes in California (and many millions still do), and there are plenty of places that I could smoke them, but there are plenty more places where I cannot, and those places (such as restaurants) also smell like food, flowers, fresh air, and everything else except stale tobacco smoke residue. In fact, I am often taken aback by the smell of tobacco when I do encounter it today, since it is such a rarity. I am nearly overwhelmed when I go to a state/country where smoking is prevalent, and literally smile from ear to ear when I return home to California and its tobacco averse culture.

So this (finally) brings me to the smart grid, and specifically smart grid security. California, being the typical first mover in nearly all things technology related in the USA is now rapidly deploying a smart grid. As I understand it, we are now approximately 50% rolled out with our AMI products (smart meters and such), and are continuing to move forward. This all began with PG&E at around the year 2000, and has been joined by (among others) SCE. Being at the front line, PG&E has had the dubious pleasure of being the first to experience the challenges of a smart grid rollout, and has had to take action to fix issues as they arose. Security challenges identified in the early days of deployment were certainly not nearly as prevalent as they are today. Security challenges tend to rise as deployments of technology expand (for various reasons), and we learn as we go. Generally the fixes we put in place are reactive in nature (we discover an exploit and fix it), with a more proactive approach to security arising out of parallels that can be drawn by examining proof of concept exploits. Because of our growing understanding of security challenges, both PG&E and SCE have taken a VERY proactive approach to addressing security challenges in the smart grid, and have expended significant resources in cyber security. PG&E has a very competent cyber security team working very hard at addressing these issues, and SCE has teamed up with meter manufacturer Itron to implement an entire AMI solution with security being a major focal point. In fact, Itron has emerged as a leader in AMI security space as a result of this partnership. I applaud this extraordinarily proactive approach taken by both PG&E and SCE, and am quite certain that this show of leadership will serve as a template for the entire US to follow.

...yet this does not address some significant issues.

As it turns out, each and every organization involved in addressing cyber security as it relates to AMI is operating within a walled environment. I am not referring to the higher level issues, which are being addressed by NERC, FERC, NIST, DHS, DOD, DOE, and MANY OTHERS, but specifically at the application level (where the rubber meets the road). In other words, the vendors making the products that go into the grid are all implementing security as they see fit (based on a collection of "best practices"). In the case of SCE, senior director Paul DeMartini told me (at the CPUC public hearing on March 18th, 2010) that SCE insisted that Itron implement security as a requirement. Being a large customer for Itron, this was quite an incentive to move forward with security at the application level. Yet Itron is not the only vendor in the AMI space (although they are perhaps the largest single meter manufacturer). There are MANY other vendors, and quite a number of them have a significant presence. More importantly, each and every component that all of these vendors make for the grid are all subject to security challenges (some more than others, of course), and all make up a part of the "security chain". A chain, as we all know, is only as strong as the weakest link. While it may indeed be both reasonable and fair to assume that some (if not most) of these AMI products have addressed security in a manner that adequately creates a strong link, it is entirely imprudent to assume that ALL links are adequately strong enough.

So what have we done with respect to security at this application level? Well, we are working on putting together some national standards, auditing, and enforcement policies (NIST,DHS,FERC,NERC, etc.), but we are still quite a way off from finalization. One can surmise that once the Federal rules are agreed on (which is a significant challenge in and of itself), it will take quite a bit of time before enforcement has any significant positive impact. Let's face it, seatbelt laws were first enacted in the 1970's, but enforcement of such laws did not have any significant presence (or impact) for DECADES. The same holds true for EPA laws, wherein California EPA laws still set the high bar for standards, and in fact trump national laws because they are so much more restrictive, and many states simply live under less environmentally friendly conditions.

California, however, must act in a more proactive manner simply because the choices California makes have such a massive impact on so many people. With 36 million people (as of July 2009) in California alone, bad choices (or simply inaction) affects a huge number of people simply within our borders. When you consider, however, that California is the 8th largest economy IN THE WORLD, the choices California makes has a much greater global impact. Did you know, for example, that California produces 12.8 % of ALL agricultural products in the US. Surprisingly, we manage to do this with less than 4% of our nation's farms and ranches (talk about efficiency). Couple this with what California produces for the health care industry (drugs, medical devices, systems, etc.), for our defense industry, and for our financial system and it quickly becomes apparent that California is not just one of the 50 states. It is THE STATE our global existence is most reliant on.

So when things go wrong in California, things go wrong in lots of places. California is no stranger to things going wrong as a result of state level bad decisions and inaction. Perhaps the most recent failure, due to bad security related decisions, was with electronic voting machines.

Do you remember that nightmare?

Let me refresh all of our memories for a moment. California decided that getting rid of paper based voting systems was a good idea back in the early part of this decade (for various reasons), and this led to an enormous groundswell of activity among several companies to create electronic voting machines that would help California get rid of the tyranny of paper, and consequently bring enormous amounts of money to the manufacturers of such systems. One thing led to another, and California ended up spending billions of dollars on electronic voting machines, and so followed the rest of the nation. However, California failed to adequately audit the security of such systems, and consequently the security was audited by hackers and independent security professionals after they were in place. As it turned out, the security flaws were so significant that nearly all electronic voting systems ended up being trashed, both in California and on a national level. The cost of this failure was, of course, borne by the taxpayers. As it turns out, the machine manufacturers were not held liable for these flaws because the systems were in fact CERTIFIED by California (and other states) and given the seal of approval. Simply put, it was The State of California's fault for failing to perform due diligence as far as security was concerned. Moreover, I had the dubious pleasure of working with several voting machine manufacturers after the fact to try to help them fix these problems, and as it turns out some of them had indeed addressed these issues far more adequately than suspected by those who chose to vilify them, but simply did not include better security features in their systems because the customers (i.e. The State of California) simply would not pay for them! California simply chose to inquire about what security features existed, signed off on the agreements without adequately auditing the security, and the rest is history.

As we move forward we learn hard lessons, and hopefully get better because of the lessons we learn. California recently received a big chunk of Federal stimulus money to implement electronic health records (EHR's), and one of the provisions from the Federal government is that the implementation must include security as prescribed by the HITECH act. Since there are few specific Federal guidelines in place AT THE APPLICATION LEVEL, the California Office of Health Information Integrity (CalOHII) has taken the initiative in creating a security committee and has drafted a set of security guidelines AT THE STATE LEVEL. I have had the pleasure of contributing to discussions with this committee (and have indeed been invited to participate, and have agreed to do so), and one of the main reasons why California has taken a state level initiative in security at the application level is because of the enormously critical nature of this initiative, and the enormous cost of failure of a lack of adequate security. Imagine, if you will, a hacker having the ability to alter a medical record, and imagine a malicious reason for doing so, such as altering a record of an enemy to indicate that he or she is not allergic to penicillin (for example), which could lead to death in the event he or she is given penicillin without prior knowledge. A lack of security here is indeed a life or death problem.

So this finally brings me to the California Public Utility Commission (the CPUC), who is now faced with some pretty tough decisions in light of the fact that the entire power infrastructure of the 8th largest economy is potentially vulnerable to significant cybersecurity related attacks (which could effectively shut down our power generation/distribution systems). The CPUC is in place to serve the public interest first and foremost. The vendors in the AMI space are there to serve corporate interests first and foremost. So much so, in fact, that the most significant vendors in the AMI space live behind a rock solid wall of NDA's and refuse to discuss security architectures and applications in any way even resembling a transparent and collaborative nature. I can certainly understand this from a competitive perspective since I too must adhere to such NDA's (and indeed do adhere to them), yet this forces us to live under an environment of corporate self regulation, which, as well all know, does not always seek to serve the public interest in an adequate manner. Again, as a libertarian I am okay with this as a basis for capitalist endeavors, but I am less okay with this for matters of life and death, and to a somewhat lesser degree (perhaps) for matters where I am forced to bear the cost of failure.

Who do you think is going to pay for the parts of the smart grid that must be scrapped in the event of a security failure? I can tell you with certainty that the US taxpayers did indeed bear an enormous amount of the cost of a failed electronic voting system. I can tell you that the California rate payers are bearing the cost of the AMI rollout (either directly or through Federal taxes). If it fails to deliver what we expect it to deliver (security related or otherwise) we cannot simply scrap it and go back to the way it was in the old days without a SIGNIFICANT cost (if that is even on the table). Moreover, if we are forced to replace vendor products due to security flaws, does it not strike you as somewhat interesting that the vendors may in fact directly benefit from the "double dip" nature of this scenario?

In a conversation I had with Aloke Gupta, who is a Senior Energy Analyst with the CPUC and is currently working on energy policy, he informed me that the CPUC has traditionally not been in the "verification business" with respect to public utility deployments. Fair enough, but this does not mean that they shouldn't be. Because the public simply has no choice whatsoever with respect to smart grid deployment in California (or elsewhere, for that matter), we must now rely on security choices which are being made by corporations who are tasked with (as a matter of legal due diligence to stockholders) maximize their bottom lines. The cost of secure components and design simply goes up the more you improve it, and the return on investment is nearly impossible to realize. In fact, the best security tends to completely obfuscate ROI. If there is no security failure, how can one know how much security measures helped? So the public simply cannot hope that the security choices a vendor makes are going to primarily center on what is best for the public interest. It simply does not translate to a bigger bottom line UNLESS everyone must comply with a set of enforced standards, which levels the playing field, and prevents loss of market share due to competition with another vendor who decides to take the "cheap" way out. Ideally, the deployment of the smart grid would have occurred only after security standards, auditing policies, and enforcement procedures were in place, but that did not happen, and may not happen for quite some time. From a national perspective, one can make the argument that with the US being approximately 5% rolled out with AMI, things are moving along at a reasonable pace at the Federal level. However, when you consider that fact that most of that rollout is in California, it now quickly becomes apparent that it is incumbent upon the CPUC to take firm and decisive action well in advance of national standards.

After all, they are here to serve the public.

The Smart Grid Privacy Smoke Screen

Whenever I watch news on network media I view everything being said with quite a bit of cynicism. Heck! Security professionals are NOTORIOUSLY cynical. The security professional mindset is designed to quickly wade through layers of what can be seen on the surface and find that which cannot be seen, which tends to tell THE REAL STORY.

Back to the news for a moment. When I see a major topic wrapped with lots of sensationalistic coverage splattered all over the airwaves and news sources, I immediately ask "Okay, what is REALLY going on." Why is everyone talking about who does or does not have the right to use the word "retard" (for example). What is the real agenda, or what are they trying to prevent us from paying attention to.

I know it may sound conspiratorial, but I see this a lot with security, and I assume it happens everywhere else.

Let us discuss security for a moment.

There are some things in the world of security that are complex, and some that are not so complex. There are good ways to protect systems using low cost, medium cost, and high cost components and procedures. When making a determination about what is the best choice (from a financial perspective) organizations that must implement security must always balance the risks with the costs. This is simply how it is done. Many of the risks associated with security are driven by compliance. If an organization does not comply with "the rules", they can be held liable for a failure to perform due diligence. This is, by far, the biggest driver (and headache) for any organization. Security generates no ROI in this case, it simply acts as insurance. Nobody I know likes to pay their insurance premiums, but they all must.

The other way security ends up in systems is when it has been attacked. Generally the more significant the attack (i.e. the more costly the attack), the better the security solution. I do not want to spend too much time on this particular topic, but it does warrant a mention. This is the holy grail of security professionals, by the way.

In cases where security becomes a topic of discussion, and consequently a major bone of contention among vendors who are subject to security mandates, what frequently happens is that the conversation takes a direction that serves the lowest common denominator. Rather than talk about the "real" issues, we tend to talk about issues that seem to be of utmost concern, but really do not matter nearly as much as the "real" issues. This is often because the more important issues are quite a bit more complex (and consequently more costly) to deal with. By shifting the focus to the less complex issues, organizations tend to appear as if they are solving a problem (and consequently performing due diligence), but they are actually avoiding the bigger issues.

For the last several days I have been reading through piles of comments submitted to the California Public Utilities Commission (CPUC) regarding Smart Grid deployment. Within these documents there are quite a few comments regarding Smart Grid security, but the overwhelming language talks about security as it relates to privacy (i.e protection of consumer usage information).

Okay, I do indeed believe privacy is important, and hold it near and dear. California was one of the first states to enact privacy laws, and has definitely led the pack in this arena. I definitely get it. Privacy is indeed important.

Sadly, however, it is a smoke screen. The focus on privacy takes our focus off of the real security challenges we face as we deploy the Smart Grid. Privacy, as it turns out, is not as challenging an issue as preventing large scale attacks of the Smart Grid which could theoretically bring down large SCADA systems. Why do I believe this? Because it simply does not have the WOW effect from a hacker community (and media) perspective. You see, EVERYTHING that is computer/network/system related can be hacked at some point. In an ideal world, the good guys try to keep ahead of the bad guys. The bad guys are always working on taking down what the good guys have built, and the most interesting things to take down are the ones which have the most impact. Hacking my meter (or any one's meter) to see how much power I use just does not get you very much attention these days in the world of hacking. Taking down a generator, however, does.

So as I read through countless pontifications about how crucial it is to ensure our privacy, and consider the extraordinarily low risk of a breach of privacy causing our lives to change in any considerable way (let's face it, how many of us truly feel we have any privacy these days?), I cannot help but think about what an effective smoke screen this is when we consider Smart Grid security. NISTIR 7628 is fully aware of where privacy sits on the scale of things to watch out for, and the February 2010 draft clearly points this out, listing privacy as a tertiary concern as it relates to security.

Yet the public comments floating around the CPUC seem to indicate that privacy is "what it is all about". I certainly do NOT see any discussions of any value indicating otherwise. Nearly every security professional I have spoken to about Smart Grid security finds this focus a bit absurd in light of both the know (non-theoretical) and assumed (theoretical) security dangers.

I think the public should consider this as they strive to educate themselves about security and the Smart Grid.