When an organization identifies a new technology threat, the first response is often to look for a piece of technology or a service that can mitigate or remove the threat. This response is justified, and often achieves the desired results. However, these results come at a cost, typically thousands –if not hundreds of thousands of dollars, along with a substantial time cost to IT personnel. This time cost comes in both the initial implementation of new tools, but also the ongoing support of those tools and necessary training to IT personnel. While some threats necessitate the purchase of new tools, organizations can frequently leverage existing tools to achieve the same threat mitigation at a fraction of the financial and resource cost. Frequently when working with clients, I have encountered situations where the client was looking to purchase a new tool to fulfill a specific need, but they already had a tool that could have solved the problem. The issue here is that many organizations only utilize a small subset of their IT tools and may not even realize the full or updated capabilities of what they already have. To leverage existing tools and services, an organization must be confident in the performance of those tools. It is imperative for organizations to develop better ways to measure the effectiveness of existing tools and services before continuing to purchase additional solutions.
Difficulties in Measuring IT Service Success
Outside of IT departments, the successes of individual business units are typically defined by return on investment (ROI). While every CFO would love to see ROI on every technology spend, this is a very difficult metric for IT departments, which are typically a cost driver, rather than a revenue driver. It’s not that technology doesn’t enable business in the end. It’s just usually more of an operational expense and can be hard to tie into profit margins. Further, ROI isn’t going to necessarily measure whether or not you are using your tools as effectively as you could be (which is more of a savings and efficacy issue). Thus, organizations must develop metrics to specifically track and report on the effectiveness of existing tools and services. (For this exercise, you want to look at cybersecurity efficacy, the indicators for which follow below.)
What You Should Be Measuring Instead, and Why
Capturing the right metrics serves a dual purpose. First, it identifies areas for improvement. Additionally, metrics captured can be used to identify gaps in security. These gaps can then be used by management to better allocate resources dedicated to addressing cybersecurity risk. It is important to point out here that these metrics are useful for all areas of an organization. IT needs to understand how effective existing tools are to be able to make decisions on which tools and services are useful and which can be removed, compliance departments need to understand how the tools that are in place are performing to ensure all compliance obligations are met and to identify any gaps in compliance, and accounting departments need to know which tools are worth paying for and what impacts will occur if a tool or service is removed. For example, a cyber incident can have catastrophic financial impact if it occurs in one area of your business, but may have only a minimal impact in another. So, your inventory lists, data classification, and relevant compliance requirements will be helpful in this process.
While this may sound very straight-forward, identifying what you have across each department, as well as its current/potential security (and thus financial) impact, is much easier said than done. Further, it will differ for each organization due to a variance in missions and strategies from one organization to the next.
So where do you start? To begin, you typically want to identify the most important areas of your organization –those most essential to your business model, whether that includes a specific department that drives revenue, a data set, or a group of customers– and use that knowledge to fuel the creation of metrics designed to show the effectiveness of the protection technologies that are in place. Once you are able to list those key areas, you can move on to how you want to measure success for the tools associated with each one. In this context, you will be measuring how successful each of those tools are at securing the systems and data with which they interact.
Useful Metrics
There are nearly an infinite number of metrics that IT departments can generate, but the majority of these only muddy the waters surrounding the effectiveness of tools and services. For a metric to be useful it must be two things: 1) relevant to the organization mission and strategy and 2) easily actionable. Without those two things, any metrics captured are difficult to utilize. As mentioned before, IT departments generate a vast amount of data that could be reported upon, but it is important to narrow the scope of the data to only that which is relevant to business issues. Large reports do not indicate maturity, and do not provide more value than clear and focused reports. To ensure that the data captured is useful, begin by determining the desired outcome. Below is a brief list of example metrics that can be used to improve security across an organization.
- Incident Response
- Track total number of Incident Response cases over a set period, and the average lifespan of those cases.
- Identify the average lifespan of a vulnerability before detection (requires mature incident response investigations).
- Tool Improvement
- Track the number of false positives generated by security monitoring tools.
- Process Improvement
- Track the average lifespan of a vulnerability (the average time from detection to remediation/mitigation).
- Total all control execution failures over the measurement period (ex: failed patches, passwords not being changed, users not taking security awareness training, etc.).
After capturing metrics, it is vital to create individual action plans to utilize the new information. Some useful metrics, such as the average lifespan of incident response cases, will require IT controls before it is possible to capture those metrics. It may be required to generate action plans prior to capturing these metrics in order to achieve the desired results.
Identify the Gaps and Successes
Once metrics have been captured, it may become apparent that some tools are not performing as expected or that they are not being utilized as much as they could be. This could be a security monitoring tool that generates more false positives than it does real alerts, a patch management tool that is not successfully patching all devices, a vulnerability scanning tool that is not able to scan the entire network, or any number of other trends you notice. At this point, it may be a good idea to contact your vendor to discuss potential reasons for this, and to work with the vendor to develop action plans to improve the performance of these tools. Additionally, do not hesitate to share the challenges that you are facing with these tools with the vendor. It is of interest to them to help you get the most out of their tools and services. This can lead to the discovery of additional features that are not presently being utilized or may reveal vendors that make more promises than they can keep. This is vital information that can help drive action plans, whether it is the further implementation of a toolset, or the removal of a vendor altogether.
Metrics without Action are Useless
Creating action plans to respond to identified metrics is necessary to generate a comprehensive IT security strategy. By capturing the metrics, management can develop an informed response and create a clear path forward. Below are several potential action items that could be generated from the list of metrics that was provided in the previous section. These action items may be taken as a result of the metrics, or to capture more informative metrics.
- Incident Response
- Develop an incident response plan that includes incident categories and classification of incidents depending on the systems that are impacted.
- Process Testing
- Develop processes to test the Incident Response and Disaster Recovery/Business Continuity Plans.
- Perform penetration testing with explicit goals in mind such as: assessing the alerting capabilities of monitoring tools, assessing the response time of detected incidents, assessing IT staff knowledge of response processes, etc.
- Tool Improvement
- Work with vendors and service providers to tune existing tools with the goal of reducing the number of false positives generated by those tools.
- Provide additional training to any IT staff responsible for managing these tools.
- Process Improvement
- Utilize metrics to improve the processes surrounding patch management.
- Implement additional processes to identify control failures early.
It is important to ensure specific goals are set as a part of each individual action plan. These goals should align with the overall organizational mission and objectives. Once fully formed, action plans should be delivered to executive management to identify the successes and failures of existing tools and services along with a clear path forward for the IT department to improve existing tools, remove things that are not functioning well, or identify whether it is necessary to purchase additional technologies. When they see your full evaluation, you are also more likely to be able to justify spends on tools that are legitimately needed, but be prepared for the potential that the new tool you actually need could differ from your original hypothesis.
In Summary
Organizations that have a clear understanding of the effectiveness of their IT controls have an advantage when it comes to budgeting, compliance, and overall IT support. It can be overwhelming to implement, but it is well worth it. And do not be fooled by sales, no IT service is truly plug-n-play. Every new tool you implement into the environment will come with new challenges including management and maintenance duties. Organizations should seek to get as much value out of their existing toolsets as possible by capturing metrics on the performance of those tools. Organizations that do not have mature documentation practices may find it difficult to generate useful metrics. TRUE encourages those organizations to look into a tool like TrueSpeed to help document IT controls and track the execution of those controls.