April 30, 2015

April 10, 2015

Please reload

Recent Posts

I'm busy working on my blog posts. Watch this space!

Please reload

Featured Posts

What matters to you?

February 2, 2013

 

Lots has been written why measuring current control performance contributes to the answer of "How much security do we need?" If you measure what matters, does tactical control performance matter? Maybe it matters to you but does it matter to the business? Associating technical controls to business performance is difficult. I almost always hear, "how does patching this server vuln within 90 days help the business win?" Great question, and if your business doesn't know the answer, it's definitely IT Security's fault.

Okay, off my soapbox. I've written about techniques to present metrics, define targets, and visualize performance. Perhaps I obsess over measurement too much but I'm comfortable with my obsessions.  Over the last couple years I've noted what folks measure and what they wish they could.

Before I get to the list, I have to include some guidelines:

  • Each metric must have a reporting duration e.g. quarterly, annually
  • Each metric must have a defined target per reporting duration
  • Significant sampling may be used where appropriate
  • Metrics should address coverage or performance e.g. % monitored or # incidents
  • Every metric must be associated to business objectives
  • Executive rollups are % difference from actual vs. target values

I should also define categories mapped to business priorities e.g.

  • Increase Revenue: develop new applications or support business initiatives
  • Protect Brand: minimize PII loss, availability, public breaches, law suits
  • Support Business: protect IP, keep existing revenue streams working
  • Reduce Costs: improve the bottom line
  • Comply Efficiently: optimize regulatory compliance costs

Executive Metrics (count 24)

Since managers shouldn't see everything, what stories are important for your business success? The following are cherry picked from the larger Operational Metrics list later in the post.

Security performance supporting availability and IT responsiveness goals         

This one may be too tactical for the CEO but relevant to an audit committee interested in control performance.

Weighted average of % difference of actual vs. target values of the operational metric group:

  • Access
  • Device
  • Monitoring
  • Vendor
  • Change Control

Security performance supporting Business Line growth               

  • % of projects integrating SDL checkpoints
  • # of High impact incidents
  • # of Medium impact incidents
  • # of Low impact incidents
  • % of strategic risks with a treatment decision
  • (If applicable) application development: avg. # hours at risk: from notification of P1 security bugs to production update
  • # of applications with one or more P1 security bugs identified in production

Employee performance

  • # of High rated incidents related to Employee behavior
  • % of Employees receiving role based awareness activities

Compliance - all from below

Program - all from below

Operational Metrics

Now the big list (please forgive the cut'n'paste spacing below)

Application Security Team Metrics          

  • % of projects integrating SDL checkpoints and deliverables          Increase Revenue
  • % developers with security training         Increase Revenue
  • % of applications risk ranked       Increase Revenue
  • % of high impact applications with production pen tests                Increase Revenue
  • # of applications with one or more critical vulns in production     Increase Revenue

Application Development Team Metrics              

  • # of static analysis warnings        Increase Revenue
  • # of static analysis false positives              Increase Revenue
  • % of false positives         Increase Revenue
  • # of repeat security bugs              Increase Revenue
  • # of p1 security bugs in code-complete drops to QA        Increase Revenue
  • # of p1 security bugs pre-prod   Increase Revenue

DevOps  Security Metrics             

  • % of teams with automated deployment processes (dev environments = production)    Increase Revenue
  • # of environment differences found in dev from production during deployment               Increase Revenue
  • avg. # hours at risk: from notification of P1 bug to production     Increase Revenue

Access  

  • % employees de-provisioned within policy          Protect Brand
  • % file shares with appropriate access control      Protect Brand
  • % user permissions or roles verified appropropriate        Protect Brand
  • % servers using directory services or centrally managed         Protect Brand
  • % privileged accounts reviewed for appropriateness       Protect Brand

Data       

  • # of senstive data pattern matches via email e.g.  PII, PCI, business defined         Protect Brand
  • # of PII or PCI pattern matches on unapproved file servers          Protect Brand

Employees         

  • % of Employees receiving role based awareness activities            Protect Brand

Change Control                

  • # of unauthorized changes          Support Business
  • # emergency changes         Support Business
  • # emergency changes related to security              Support Business

Corporate Devices          

  • % of servers with owners and classified              Support Business
  • % of servers enrolled in vulnerability management process         Support Business
  • % of high impact servers compliant to minimum baseline standards         Support Business
  • % of vulns fixed within within SLA (may be a dupe of the above)                Support Business
  • % difference of enumerated servers vs. cmdb            Support Business

Workstations/Laptops/Tablets 

  • % of workstations enrolled in vulnerability management process              Support Business
  • % of workstations compliant to minimum baseline standards      Support Business

Mobile 

  • % mobile devices enrolled in management platform       Support Business

Policy & Standards          

  • % of policies reviewed per schedule            Support Business
  • % of minimum baseline standards reviewed per schedule            Support Business

Monitoring & Response               

  • % production servers monitored              Support Business
  • # of tier 2 investigations                  Reduce Costs
  • # of H,M,L incidents          Reduce Costs
  • # hours, mean time to recover      Reduce Costs
  • # repeat cause incidents                 Reduce Costs

Risk Management           

  • % of strategic risks with a treatment decision      Increase Revenue

Vendor Management    

  • % vendors with risk ranking         Support Business
  • % of vendors assessed <per schedule> Support Business
  • # of overdue vendor issues         Support Business
  • % of new contracts following security process    Support Business
  • % of outsourced hosting (cloud) solutions complying with security policies

Compliance        

  • # of  audits with no significant findings   Comply Efficiently
  • # of overdue findings     Comply Efficiently
  • # of repeat findings        Comply Efficiently
  • # of policy exceptions    Comply Efficiently
  • # of overdue exceptions              Comply Efficiently

Program               

  • % projects completed on time and budget           Reduce Costs
  • % budget Plan to Forecast           Reduce Costs
  • % of processes defined in catalog, RACI, metrics               Reduce Costs
  • plus-minus change to security service satisfaction            Reduce Costs
  • % SLA’s met or exceeded             Reduce Costs

Quite the list! Count breakdown by business relevance:

  • Increase Revenue: 15
  • Protect Brand: 8
  • Support Business: 17
  • Reduce Costs: 6
  • Comply Efficiently: 5

Total: 51

Please let me know if you'd like the xlsx of the above. Also, please push back if you think all this measurement is too onerous. If measurement wasn't built in when the control was implemented, metrics can be really expensive. Deciding if you should even measure performance is a great indicator of your company culture. Deploying a new control without including the expense to measure is also telling...

As always, feel free to contact me if you'd like to discuss measurement or need assistance getting it done. I have to leave you with some metrics eye candy:

Share on Facebook
Share on Twitter
Please reload

Follow Us

I'm busy working on my blog posts. Watch this space!

Please reload

Search By Tags