Testing is a best-case scenario to validate the system's correctness. But, it doesn't predict the failure cases that may occur in production. Experienced engineering teams would tell you that production environments are not uniform and full of exciting deviations.

The fun fact is – testing in production helps you test the code changes on live user traffic, catch the bugs early, and deliver a robust solution that increases customer satisfaction. But, it doesn't help you detect the root cause of the failure.

And that's why adopting observability in testing is critical. It gives you full-stack visibility inside the infrastructure and production to detect and resolve problems faster. As a result, people using observability are 2.1 times more likely to detect any issues and report a 69% better MTTR.

Symptoms of a Lack of Observability

The signs of not having proper observability showed up in the work of engineers every day. When there was a problem with production, which happened daily, the developers' attempts to find the cause of the problem would often hit a wall, and tickets would stay stuck in Jira. This happened because they didn't have enough information and details to figure out the root cause. To overcome these challenges, the developers sometimes used existing logs, which was not very helpful as they had to access logs for each service one at a time using Notepad++ and manually search through them. This made the developers feel frustrated and made it difficult for the company to clearly show customers how and when critical issues would be fixed, which could harm the company's reputation over time.

Observability: What Does It Really Mean?

For a tester, having proper observability means the ability to know what's happening within a system. This information is very valuable for testers. Although observability is commonly associated with reliability engineering, it helps testers better understand and investigate complex systems. This allows the tester and their team to enhance the system's quality, such as its security, reliability, and performance, to a greater extent.

I found out about this problem through a challenging experience. Many others might have had a similar experience. While checking a product, I had trouble understanding the complications of the product, which is common for testers. As I tried to understand the product by reading its instructions and talking to the people involved, I noticed that the information I gathered did not make sense. At the time, I was unfamiliar with the technical term for this, but in hindsight, it was evident that the system lacked observability. It was almost impossible to know what was happening inside the application.

While testing concentrates on determining if a specific functionality performs as intended, observability concentrates on the system's overall health. As a result, they paint a complete picture of your system when taken as a whole.

Traditional software testing, i.e., testing in pre-production or staging environments, focus on validating the system's correctness. However, until you run your services inside the production environment, you won't be able to cover and predict every failure that may occur.

Testing in production helps you discover all the possible failure cases of a system, thereby providing service reliability and stability. With observability, you can have an in-depth view of your infrastructure and production environments. In addition, you can predict the failure in production environments through the telemetry data, such as logs, metrics, and traces.

Observability in the production environment helps you deliver robust products to the customers.

Is Observability Really Replacing Testing?

From a tester's perspective, there's no replacement for the level of detail that a truly observable system can provide. Although on a practical level, observability has three pillars — logs (a record of an event that has happened inside a system), metrics (a value that reflects some particular behavior inside a system), and traces (a low-level record of how something has moved inside a system)  — it is also more than those three elements. "It's not about logs, metrics, or traces," software engineer Cindy Sridharan writes in Distributed Systems Observability, "but about being data-driven during debugging and using the feedback to iterate on and improve the product." In other words, to do observability well, you not only need effective metrics, well-structured logs, and extensive tracing. You also need a mindset that is inquisitive, exploratory, and eager to learn and the processes that can make all of those things meaningful and impactful.

This makes testers and observability natural allies. Testing is, after all, about asking questions about a system or application, being curious about how something works or, often, how something should work; observability is very much about all of those things. It's too bad, then, that too many testers are unaware of observability — not only will it help them do their job more effectively, but they're also exactly the sort of people in the software development lifecycle who can evangelize for building observable systems. 

To keep things simple, there are two key ways we should see observability as helping testers:

1. It helps testers uncover granular details about system issues: During exploratory testing, observability can help testers find the root cause of any issues through telemetry data such as logs, traces, and metrics, helping in better collaboration among various teams and providing faster incident resolution.
2. It helps testers ask questions and explore the system: Testers are curious and like to explore new things. With the observability tool, they can explore the system deeply and discover the issues. It helps them uncover valuable information that assists them in making informed decisions while testing.

Conclusion

Testing and observability go hand-in-hand in ensuring the robustness and reliability of a system. While traditional testing focuses on validating the system's correctness in pre-production environments, testing in production can uncover all the possible failure cases. On the other hand, Observability provides full-stack visibility into the infrastructure and production environments, helping detect and resolve problems faster. In addition, observability helps testers uncover granular details about system issues and enables them to ask questions and explore the system more deeply. Testers and observability are natural allies, and adopting observability can lead to better incident resolution, informed testing decisions, and increased customer satisfaction.