Observability and security – where are the crossover points?

Developers are implementing more of their applications using microservices and containers, and this combination of application design and infrastructure will continue to grow. For example, Gartner has predicted more than 75 percent of global organizations will be running containerised applications in production by 2022, up from less than 30% in 2020. For developers, this approach makes it easier to break down applications into different components and then make changes over time.

The business result is that companies can deliver services for customers faster. However, there are some other consequences from those decisions. With so many more moving parts to look at, developers have to automate how they gather data on their infrastructure components, while security teams have to understand the new models for applications too.

SEE ALSO: An introduction to scriptless testing

Observability and security

For both developers and security teams, getting the right data around application installs is necessary to make decisions about priorities. This data takes the form of logs, metrics, events and tracing data from applications, cloud services and other infrastructure components. Together, this data provides a complete set of information around application performance. This is often referred to as observability data, where individual transaction data can help show the performance of the overall application.

For developers, observability data helps to diagnose the problem if an application is not working. Seeing where issues exist helps developers fix problems faster and get services up and running. However, the use case for security is different – for security analysts, this data can be analysed to show outlying events that represent hacking attempts or someone up to no good within the network.

While both teams can use the same data, in reality the situation can be very different. Each department may have its own sets of data gathered from their own tools, creating duplicate sets of information that still have to be stored and interrogated. Alongside this, developers and security teams have very different things to look for – for example, developers are often more reactive and only care about their log data when something is wrong, while security analysts spend their time looking for rogue events and outliers continuously in the hope of spotting issues before they become breaches.

The challenge here is that, for all the data coming in, each team ends up working in a silo and not seeing the big picture. The teams’ key performance indicators and goals are different too, which can exacerbate the problems around communicating when things do crop up. Lastly, not only is it more expensive to keep multiple copies of the same data, but it also leads to potential conflict or working at cross-purposes.

Why is this important now? It is due to the interest around software supply chain attacks. The software supply chain involves all the components that go into making an application or service – this can include components developed internally, third party and commercial tools and open source components, as well as the infrastructure that runs them and the process used to manage how these are all developed and integrated. With containerisation and microservices leading to more components – and with some of those services pulled directly from public repositories – the challenge for security is how to keep all this secure over time.

The big headlines around the attack on Solarwinds put the spotlight on how companies use software tools within their businesses to manage their own operations, and the long-term challenge is how to keep those tools safe and secure. It also put more emphasis on how developers and security teams understand these processes. In this situation, observability data can help security teams understand developers and their procedures.

Collaborating around data

The first challenge for many security and developer teams is to actually talk about this as a potential problem. For many developers, the security department is perceived as the team that comes in after all the hard work on designing and building applications and then finds faults. For security, developers are thought of as the ones who care more about shiny new toys rather than fixing the existing problems. On both sides, there can be misunderstanding and blame.

To get around this problem, it’s worth looking at how data can enable both sides to work together more effectively. Both developers and security analysts rely on data to carry out their jobs, so looking at observability can help establish common ground. Both sides care about reliability and availability of services, so you can use this as a common ground for discussion. When everyone involved understands their efforts are part of a bigger approach, it is easier to overcome problems. By sharing observability data and tying both developer productivity and security performance to reliability of services, everyone should have the same goals in mind.

Both sides can also benefit from one another’s expertise. For example, developers already have a strong level of insight into their software supply chains and how each update goes through the Continuous Integration / Continuous Deployment pipeline. This data – and the knowledge that goes with it – can and should be available to security analysts, so they can get that insight into what is changing over time and what ‘good’ behaviour looks like across the process. The developer team can also provide information to the security analysts around what expected behaviour looks like, helping them reduce false positives and be more efficient. In turn, this data can be used to spot anomalies in applications, whether this is a component not performing as expected or a connection that should not be in place.

In return, security teams can help developers see where potential issues might come up around compliance and performance. For example, if an update leads to data being exposed unnecessarily, the security team can flag this and ensure it is prioritised for fixing. The aim should be to get these changes spotted earlier in the process, then back into the development team workflow quickly. The result should be that problems can be prioritised and fixed efficiently, rather than security acting as the gatekeeper that halts production completely for massive amounts of rework.

SEE ALSO: “The impact of poor machine identity management can be devastating”

Looking ahead around observability

For both sides, observability data should provide a consistent set of information that both security and development can use for their purposes. By using the same data, there should be less opportunity for conflict around any issues that arise and the opportunity to fix things faster before they affect either performance or security. Better still, it can also provide an opportunity for both sides to consolidate the volumes of duplicate data that they are responsible for, saving on costs.

Ultimately, this approach relies on both high level analysis and alerting, and access to the raw data itself when detail is needed. Being able to drill down into the specific data points like a log file can help if something goes wrong, or if there is more investigation needed on a suspect interaction. Ideally, this should be a smooth process with no need to carry out manual work across different tools.