Be wary of rating agencies

Ruv Cohen, over at Elastic Vapor, proposed an interesting idea for a Cloud Service Rating Agency. The idea was further defined as a "Cloud Performance Ability (CPA) that estimates it's ability to meet certain service levels"; similar in intent to Standard & Poor's Claims-Paying Ability rating for an insurance provider explained as the "financial capacity to meet its insurance obligations".

I love the concept of some standard, some metric that allows us all to look at a complex issue and agree what we're looking at, but there are a few problems:

• Metrics and ratings hide nuance, by design, which may be a relevant factor in your personal evaluation of a provider
• Every single rating agency has shown themselves vulnerable to the introduction of complex artefacts - look at what the introduction of CDO's ushered in and how credit scoring behaved
• Most rating agencies are for-profit entities, which means that while integrity is a priority in their branding, it is almost certainly not the topmost priority in their business objectives

(more issues with credit rating agencies can be found on wikipedia)

Now, before you think I pick on rating agencies unfairly, other public trusts (such as public audit firms) have suffered from conflict of interest problems, that have led to bad decision making (Arthur Andersen's involvement in Enron is a canonical example and one of the reasons for the existence of Sarbanes-Oxley legislation).

So bottom line, if you establish for-profit providing rating services, ultimately the integrity (intentionally or otherwise) will come into question.

Experience has also taught us that self-administered assessments - unless exceptionally detailed - are at best somewhat informative, at worst theatre (the early days of PCI-DSS come to mind).

If we were to build a Cloud Service Rating Agency, what we would really need is an independent, non-profit entity, something like the North American Electric Reliability Corporation (NERC). An entity with claws and a focus on assurance, so while I agree with James Urquhart that data is not electricity, I think it's an interesting industry to draw lessons from.