Why Your Security Analysis Metrics And KPIs Are Wrong

By Dan Jodarski

Your Security Analysis Metrics, Measurements And Key Performance Indicators Are All WrongYour federal agency relies on a number of metrics, measures and key performance indicators (KPIs) when it comes to your organizational security. But without measuring the correct metrics and KPIs, your agency is less secure than ever. 

The organizational security and counter-intelligence community within the U.S. government has broadly adopted the concepts of data, metrics and measurement as applied to security analysis. As with any broad adoption, however, a number of mistakes have now become mainstream – and mistakes in government security have major consequences. 

Not all measurements matter. Some metrics keep you informed and abreast with evolving threats (including insider threats), but many are just wastes of your time, giving you a feeling of false confidence in your security efforts.

Here are a few tips to help you shore up your security analysis with meaningful metrics and KPIs – so your organizational security efforts more effectively protect information, assets and human lives. 

Tip #1: Balance Leading And Lagging Measures 

Most data dashboards used in security analysis are filled with lagging metrics – measurements that report results but don’t predict future actions. While many lagging metrics are required for federal agency compliance, they’re not useful when it comes to making decisions. 

Leading metrics, on the other hand, are proven predictors of results. Also known as key performance indicators (KPIs), leading metrics should be the focus of your daily and weekly work, since they prepare you to take preventative – rather than reactive – action.

For example, reporting the number of known security violations is a lagging indicator, while calculating a general security awareness score is a leading indicator. 

Tip #2: Stop Using “Gut” Metrics 

Government security and counter-intelligence officials have a long history of using “gut instinct” metrics to complete their security analysis. If you’re serious about agency security, stop using these metrics immediately. 

Many decision-by-gut measurements are often based on holdover policies and behaviors with no basis in solid data science. Especially when it comes to employee screening and background checks, there are far too many measures still being used that have never effectively predicted misbehavior or an insider threat incident.

Chasing false positives due to legacy metrics is a waste of time, taxpayer money and your limited capacity to focus on the most probable threats. Insurance companies and even casinos have tapped into the predictive power of data science to detect insider threats; it’s about time your security analysis did as well. 

Tip #3: Define Tolerance Ranges (And Their Consequences) In Advance 

In counter-intelligence and security analysis, tolerance ranges define the upper and lower limit of a particular metric. If that measure rises above or drops below the tolerance range, then it’s time to take the appropriate action. 

However, many federal agency leaders haven’t defined the tolerance ranges of their security KPIs. Or, if they have defined a tolerance range, they haven’t mapped out which actions should be taken once the metric leaves the given range. Without these definitions, your measurements are a waste of time. 

Especially for continuous improvement evaluations, it’s essential that you define the acceptable limits of your security metrics well in advance and follow up if they leave your tolerance range. As you learn the rhythm of your security metrics and their defined tolerance ranges, you should always adjust them based on past experiences. 

Tip #4: Insist On Data Science Literacy 

Perhaps the biggest issue facing information security and counter-intelligence teams is their inexperience with data science. In the past, statistics and data science were not a requirement for most agency security positions, but now, these data skills are more important than ever. 

With so many emerging and evolving threats facing federal agencies, only security personnel steeped in data science are able to effectively manage and mitigate each one. Without a firm understanding of statistics and data science, your organizational security efforts are forever catching up to criminals and insider threats – instead of preventing them.  

Security analysis must begin with solid metrics and measurements. But unless your agency is measuring the correct KPIs and leveraging them to make informed decisions, your security efforts fail.  

Does your federal agency need to improve security processes within a limited schedule and budget? Click below to download this e-book from Big Sky Associates and discover how to make continuous improvement efforts that are cost-effective for your organizational security budget. 
Download Your Free Report: The Ultimate Process Improvement Guide From Initial Data Analysis To Final Implementation Plan