Frequent readers know metrics are a passion of mine. They provide key stories to answer:
Should we invest more or less in security
Are we performing to commitments
Which groups are top performers
Recently SIRA started a project to define metrics based on the NIST Cyber Security Framework (CSF). I applaud and support the effort! Currently they’re looking for volunteers so please check out their site. SIRA approaches metrics with a construct I love and also use: Goal, Question, Metric to help define metrics and ensure they align with the business. Related, Alex Hutton and David Mortman cover GQM very well in their RSA presentation.
In my experience I’ve had success with some approaches that are a bit different than SIRA’s. I think the following are key to successful metrics:
Every metric must have a defined target. This value is negotiated across stakeholders with the control owner ultimately accountable.
Whenever possible, the unit of measure should be a % e.g. % systems in CMDB compared to discovery scans. I found when most metrics are percentages it’s easier for stakeholders to review the collection. Some metrics should be counts e.g. # of H,M,L incidents per quarter.
Whenever possible, metrics should be defined where higher values are considered “good” e.g. % vulnerabilities mitigated per SLA. This really improves visual communication and makes it easier to set targets consistently. Of course this doesn’t apply to incident counts!
The SIRA team took the approach to define a metric for every CSF control objective. This may be the best, most comprehensive approach however a lot of time will be spent defining metrics that aren’t used.
An important aspect when defining metrics across a control framework is to embrace that many objectives don’t lend themselves to sustained, target-based metrics e.g. “Priorities for organizational mission, objectives, and activities are established and communicated.” Many objectives should just be assessed in regular risk assessments.
So I took a couple hours with the CSF and for each control objective (98 I believe), assigned a Metric Priority rating of 1,2, RA, or covered:
Priority 1: most value, implement first
P2: consider implementing second
RA: control objective better suited for periodic assessment
Covered: when multiple control objectives are covered by one metric
I then defined metrics for P1 and P2 control objectives. Here are the metric counts across the CSF:
For some organizations, I think many of the RA’s can be good metrics. They just didn’t pass my internal GQM bar. Also, when I compared my CSF metrics to our catalog, I found 9 missing. My first draft can be downloaded here (including the 9 extra): Caliber_CSF_metricsdraft.
If there’s interest, I’ll spend more time refining the metrics and revisit my ratings. With some refinement they might make for a good template in Metrics Manager. Please let me know your thoughts.