An example of this would be the extremely rapid growth of the automotive age in the USA. When automobiles first appeared on the scene, they were indeed a novelty, and certainly nothing worthy of considerable regulatory control. They shared the road with pedestrians, horses and buggies, and bicycles, and as I understand it were quite amicable about it (for the most part). Automobiles were built according to manufacturer specifications which were fully controlled by the manufacturers, and they were generally built with aesthetics, functionality, and profits as the key drivers.
As we all know, the automobile industry quickly made the transition from novelty to way of life within the blink of an eye. It did not take long until the automotive industry become the cornerstone of American manufacturing, and consequently an enormous influencer of economic dominance.
In other words, it got very damn big really damn fast.
As this growth progressed, it soon became apparent that we had to make major changes in the way we, as Americans, did things. It was no longer prudent to assume that everyone would share the road in a somewhat Utopian fashion. While automobiles created ENORMOUS benefits for society, they also introduced SIGNIFICANT challenging issues, many of which were indeed quite dangerous. I am certain I do not need to articulate all of these, since we are all quite aware of the many dangers associated with an anarchistic automotive culture, especially since we are all quite aware of the significant dangers we all face in an automotive society where most people do indeed (by and large) follow the rules. This is quite evident when a solitary driver loses control of a vehicle on a busy highway and causes massive multi-car collisions. It is truly amazing that we all manage to avoid such chaos as well as we do, but we do indeed manage.
...and why is that?
Well, it is because we have learned to do so the hard way. Most of the rules of the automotive ecosystem have come about as a direct result of lots of people dying, or lots of people being forced to live in a society where quality of life is negatively impacted. Seat belts, for example, where not always required in cars. In the early days of seat belts they were an option. They appeared in cars in the early 1900's, but were not required in US cars until the 1970's. In fact, most of the safety standards that exist today for the automotive industry did not appear until the 1970's, and NOT because the US Automotive industry decided it was a good idea, but mostly as a result of the crusades of the venerable Ralph Nader. Seat belts became standard equipment as a direct result of Federal legislation mandating that all carmakers include them. However, it was not until laws were enacted mandating their use by passengers that we began realizing a decline in traffic related deaths. To quote a section of an article from The Prevention Institute:
Mandatory laws have proved effective both in increasing seatbelt usage and decreasing traffic fatalities. The Centers for Disease Control and Prevention reports that seatbelt use nationwide increased from 11% in 1981 to 68% in 1997. NHTSA reports that the motor vehicle fatality rate as measured per 100,000 population decreased from 21.49 in 1981 to 15.69 in 1997, and also decreased as measured by 100 million vehicle miles traveled, from 3.2 in 1981 to 1.6 in 1997. While these decreases cannot be attributed to the use of seatbelts alone, seatbelts are credited with playing a significant role in these advancements.
Enter California. Being the state with both the largest population of people, the largest economy, and the most cars, California is always a good place to go for significant statistical samplings. California is somewhat notoriously well know for being a first mover on many initiatives that tend to serve the public interest over corporate interests, often at the behest of corporate interests. This, as it turns out, is a very good thing more often than not. When the rest of the USA was (and still is) debating the efficacy of environmental controls, and what should and should not be enforced to preserve our environment, California simply forged ahead and passed what many industry leaders deemed punitive measures to prevent (among other things) automobiles that emit large amounts of pollutants. Because the State of California represents the largest customer base for car manufacturers (1 in 11 cars is in California), the automotive industry was left with two choices. Either they could conform to California emissions laws, or they could find another less stringent customer base.
We all know which way they went, and now that I live in California I am extremely thankful. Anyone who has lived in California's most populous areas knows darn well that there are A LOT of cars here and they are on the road all the time. Prior to the enactment of state mandated emissions standards California regularly had days where the pollution was so severe that people were warned to stay indoors with their windows shut. While California does indeed continue to face air quality challenges in populous areas, the attention this has gotten at the state level has resulted in significant improvements in air quality. Perhaps most importantly, the significant changes automotive manufacturers had to make to their products in order to do business in California has led to a fundamental change in manufacturing which the entire US (and the world) now benefits from.
Let's go back to seat belts for a moment. Despite the mandate that seat belts must be worn when driving, compliance increases only as enforcement increases. This is true with nearly every rule of society, and certainly with rules where the manifestation of ignoring the rule rarely leads to any negative consequences. In other words, since most people who do not were a seatbelt when driving do not experience any negative consequence from not wearing one (they normally do not die or bash their heads against the dashboard or steering wheel), it is quite easy to forego this preventative measure. Compliance with seat belt laws got better as police officers began issuing citations to those who failed to use them. This generally came about as a result of officers pulling over drivers for a non-seatbelt related violation, and issuing a citation for not wearing a seat belt in addition to whatever they had been pulled over for. This is known as a "Secondary Enforcement" law. California, however, is a "Primary Enforcement" state with respect to seat belt laws. In California, an officer can pull a passenger over for the sole reason of failure to wear a seat belt, and issue a citation accordingly. As controversial as this law has become, it has led to a SIGNIFICANT decrease in fatalities. From the same article cited earlier:
California, a primary enforcement state, currently reports 91% usage -- the highest in the country. After the passage of a mandatory seatbelt law in 1986, California's usage rate went from 26% to approximately 45%. By 1992, California's usage had increased to 71%. With the passage of the primary enforcement law in 1993, California's usage rate jumped to 83%, steadily climbing to the current rate. According to the National Safety Council, California's fatality rate has decreased by over 34% since the passage of the primary enforcement law.
Wow! A decrease in fatality rate by over 34%. I would say that is pretty darn significant. I would also say that it is probably NOT all solely due to the primary enforcement, since California also spends a significant amount on programs to continually educate the population on the importance of safety.
As a staunch libertarian by nature (notice the small "l") I generally oppose government intervention. In fact, prior to moving to California I was convinced I was going to find the imposition of such a notoriously intrusive set of rules for my well being to be intolerable. In fact, however, I find it quite nice. I was a smoker when I moved to California, but the nearly militant anti-smoking sentiment coupled with the heavy handed enforcement of anti-smoking laws found throughout the state have forced me to rethink my addiction, and led to a lifestyle which I find far more appealing, since hacking my lungs out every morning was not something I looked forward to every day. I am now living a smoke free life for going on 7 years! The FACT is that smoking is a horribly dangerous health hazard that I am better off not taking part in. Surely I can still buy cigarettes in California (and many millions still do), and there are plenty of places that I could smoke them, but there are plenty more places where I cannot, and those places (such as restaurants) also smell like food, flowers, fresh air, and everything else except stale tobacco smoke residue. In fact, I am often taken aback by the smell of tobacco when I do encounter it today, since it is such a rarity. I am nearly overwhelmed when I go to a state/country where smoking is prevalent, and literally smile from ear to ear when I return home to California and its tobacco averse culture.
So this (finally) brings me to the smart grid, and specifically smart grid security. California, being the typical first mover in nearly all things technology related in the USA is now rapidly deploying a smart grid. As I understand it, we are now approximately 50% rolled out with our AMI products (smart meters and such), and are continuing to move forward. This all began with PG&E at around the year 2000, and has been joined by (among others) SCE. Being at the front line, PG&E has had the dubious pleasure of being the first to experience the challenges of a smart grid rollout, and has had to take action to fix issues as they arose. Security challenges identified in the early days of deployment were certainly not nearly as prevalent as they are today. Security challenges tend to rise as deployments of technology expand (for various reasons), and we learn as we go. Generally the fixes we put in place are reactive in nature (we discover an exploit and fix it), with a more proactive approach to security arising out of parallels that can be drawn by examining proof of concept exploits. Because of our growing understanding of security challenges, both PG&E and SCE have taken a VERY proactive approach to addressing security challenges in the smart grid, and have expended significant resources in cyber security. PG&E has a very competent cyber security team working very hard at addressing these issues, and SCE has teamed up with meter manufacturer Itron to implement an entire AMI solution with security being a major focal point. In fact, Itron has emerged as a leader in AMI security space as a result of this partnership. I applaud this extraordinarily proactive approach taken by both PG&E and SCE, and am quite certain that this show of leadership will serve as a template for the entire US to follow.
...yet this does not address some significant issues.
As it turns out, each and every organization involved in addressing cyber security as it relates to AMI is operating within a walled environment. I am not referring to the higher level issues, which are being addressed by NERC, FERC, NIST, DHS, DOD, DOE, and MANY OTHERS, but specifically at the application level (where the rubber meets the road). In other words, the vendors making the products that go into the grid are all implementing security as they see fit (based on a collection of "best practices"). In the case of SCE, senior director Paul DeMartini told me (at the CPUC public hearing on March 18th, 2010) that SCE insisted that Itron implement security as a requirement. Being a large customer for Itron, this was quite an incentive to move forward with security at the application level. Yet Itron is not the only vendor in the AMI space (although they are perhaps the largest single meter manufacturer). There are MANY other vendors, and quite a number of them have a significant presence. More importantly, each and every component that all of these vendors make for the grid are all subject to security challenges (some more than others, of course), and all make up a part of the "security chain". A chain, as we all know, is only as strong as the weakest link. While it may indeed be both reasonable and fair to assume that some (if not most) of these AMI products have addressed security in a manner that adequately creates a strong link, it is entirely imprudent to assume that ALL links are adequately strong enough.
So what have we done with respect to security at this application level? Well, we are working on putting together some national standards, auditing, and enforcement policies (NIST,DHS,FERC,NERC, etc.), but we are still quite a way off from finalization. One can surmise that once the Federal rules are agreed on (which is a significant challenge in and of itself), it will take quite a bit of time before enforcement has any significant positive impact. Let's face it, seatbelt laws were first enacted in the 1970's, but enforcement of such laws did not have any significant presence (or impact) for DECADES. The same holds true for EPA laws, wherein California EPA laws still set the high bar for standards, and in fact trump national laws because they are so much more restrictive, and many states simply live under less environmentally friendly conditions.
California, however, must act in a more proactive manner simply because the choices California makes have such a massive impact on so many people. With 36 million people (as of July 2009) in California alone, bad choices (or simply inaction) affects a huge number of people simply within our borders. When you consider, however, that California is the 8th largest economy IN THE WORLD, the choices California makes has a much greater global impact. Did you know, for example, that California produces 12.8 % of ALL agricultural products in the US. Surprisingly, we manage to do this with less than 4% of our nation's farms and ranches (talk about efficiency). Couple this with what California produces for the health care industry (drugs, medical devices, systems, etc.), for our defense industry, and for our financial system and it quickly becomes apparent that California is not just one of the 50 states. It is THE STATE our global existence is most reliant on.
So when things go wrong in California, things go wrong in lots of places. California is no stranger to things going wrong as a result of state level bad decisions and inaction. Perhaps the most recent failure, due to bad security related decisions, was with electronic voting machines.
Do you remember that nightmare?
Let me refresh all of our memories for a moment. California decided that getting rid of paper based voting systems was a good idea back in the early part of this decade (for various reasons), and this led to an enormous groundswell of activity among several companies to create electronic voting machines that would help California get rid of the tyranny of paper, and consequently bring enormous amounts of money to the manufacturers of such systems. One thing led to another, and California ended up spending billions of dollars on electronic voting machines, and so followed the rest of the nation. However, California failed to adequately audit the security of such systems, and consequently the security was audited by hackers and independent security professionals after they were in place. As it turned out, the security flaws were so significant that nearly all electronic voting systems ended up being trashed, both in California and on a national level. The cost of this failure was, of course, borne by the taxpayers. As it turns out, the machine manufacturers were not held liable for these flaws because the systems were in fact CERTIFIED by California (and other states) and given the seal of approval. Simply put, it was The State of California's fault for failing to perform due diligence as far as security was concerned. Moreover, I had the dubious pleasure of working with several voting machine manufacturers after the fact to try to help them fix these problems, and as it turns out some of them had indeed addressed these issues far more adequately than suspected by those who chose to vilify them, but simply did not include better security features in their systems because the customers (i.e. The State of California) simply would not pay for them! California simply chose to inquire about what security features existed, signed off on the agreements without adequately auditing the security, and the rest is history.
As we move forward we learn hard lessons, and hopefully get better because of the lessons we learn. California recently received a big chunk of Federal stimulus money to implement electronic health records (EHR's), and one of the provisions from the Federal government is that the implementation must include security as prescribed by the HITECH act. Since there are few specific Federal guidelines in place AT THE APPLICATION LEVEL, the California Office of Health Information Integrity (CalOHII) has taken the initiative in creating a security committee and has drafted a set of security guidelines AT THE STATE LEVEL. I have had the pleasure of contributing to discussions with this committee (and have indeed been invited to participate, and have agreed to do so), and one of the main reasons why California has taken a state level initiative in security at the application level is because of the enormously critical nature of this initiative, and the enormous cost of failure of a lack of adequate security. Imagine, if you will, a hacker having the ability to alter a medical record, and imagine a malicious reason for doing so, such as altering a record of an enemy to indicate that he or she is not allergic to penicillin (for example), which could lead to death in the event he or she is given penicillin without prior knowledge. A lack of security here is indeed a life or death problem.
So this finally brings me to the California Public Utility Commission (the CPUC), who is now faced with some pretty tough decisions in light of the fact that the entire power infrastructure of the 8th largest economy is potentially vulnerable to significant cybersecurity related attacks (which could effectively shut down our power generation/distribution systems). The CPUC is in place to serve the public interest first and foremost. The vendors in the AMI space are there to serve corporate interests first and foremost. So much so, in fact, that the most significant vendors in the AMI space live behind a rock solid wall of NDA's and refuse to discuss security architectures and applications in any way even resembling a transparent and collaborative nature. I can certainly understand this from a competitive perspective since I too must adhere to such NDA's (and indeed do adhere to them), yet this forces us to live under an environment of corporate self regulation, which, as well all know, does not always seek to serve the public interest in an adequate manner. Again, as a libertarian I am okay with this as a basis for capitalist endeavors, but I am less okay with this for matters of life and death, and to a somewhat lesser degree (perhaps) for matters where I am forced to bear the cost of failure.
Who do you think is going to pay for the parts of the smart grid that must be scrapped in the event of a security failure? I can tell you with certainty that the US taxpayers did indeed bear an enormous amount of the cost of a failed electronic voting system. I can tell you that the California rate payers are bearing the cost of the AMI rollout (either directly or through Federal taxes). If it fails to deliver what we expect it to deliver (security related or otherwise) we cannot simply scrap it and go back to the way it was in the old days without a SIGNIFICANT cost (if that is even on the table). Moreover, if we are forced to replace vendor products due to security flaws, does it not strike you as somewhat interesting that the vendors may in fact directly benefit from the "double dip" nature of this scenario?
In a conversation I had with Aloke Gupta, who is a Senior Energy Analyst with the CPUC and is currently working on energy policy, he informed me that the CPUC has traditionally not been in the "verification business" with respect to public utility deployments. Fair enough, but this does not mean that they shouldn't be. Because the public simply has no choice whatsoever with respect to smart grid deployment in California (or elsewhere, for that matter), we must now rely on security choices which are being made by corporations who are tasked with (as a matter of legal due diligence to stockholders) maximize their bottom lines. The cost of secure components and design simply goes up the more you improve it, and the return on investment is nearly impossible to realize. In fact, the best security tends to completely obfuscate ROI. If there is no security failure, how can one know how much security measures helped? So the public simply cannot hope that the security choices a vendor makes are going to primarily center on what is best for the public interest. It simply does not translate to a bigger bottom line UNLESS everyone must comply with a set of enforced standards, which levels the playing field, and prevents loss of market share due to competition with another vendor who decides to take the "cheap" way out. Ideally, the deployment of the smart grid would have occurred only after security standards, auditing policies, and enforcement procedures were in place, but that did not happen, and may not happen for quite some time. From a national perspective, one can make the argument that with the US being approximately 5% rolled out with AMI, things are moving along at a reasonable pace at the Federal level. However, when you consider that fact that most of that rollout is in California, it now quickly becomes apparent that it is incumbent upon the CPUC to take firm and decisive action well in advance of national standards.
After all, they are here to serve the public.
No comments:
Post a Comment