Inside DataOps: 3 Ways DevOps Analytics Can Create Better Products

Could the next great product insight be hidden in your operational data? For many companies, the answer is yes! The emerging practice of DataOps combines the best of data engineering and DevOps to provide high-quality insights to the product team. The aim is to create better products by leveraging data analytics to inform product roadmaps and features.

For example, a combination of user experience (UX) insights and product usage data can determine which features are working or not working for users. Analytics can also equip teams with intel on which features are viable and which need improvement. Using DataOps strategies, teams can gather product data to inform critical decisions that improve the customer experience (CX).

Let’s learn more about DataOps, and what types of DevOps analytics can be most useful to product teams.

What is DataOps

Successful DevOps practices involve the use of continuous integration/continuous delivery (CI/CD) pipelines to communicate efficiently and ship products quickly. Logs and event data are generated at every stage in the development lifecycle. This log data can be transformed and analyzed for a variety of purposes, including:

• Monitoring the performance of applications
• Troubleshooting systems, networks, and machines
• Measuring and understanding user behavior
• Verifying compliance with internal policies and industry-specific regulations
• Detecting and responding to security incidents
• Evaluating the root cause of a network event.

READ: Build a DataOps Foundation for Agile Data Analytics

Many teams may not be tapping into their data to its full potential. That’s where the practice of DataOps comes in. Gartner defines DataOps as “a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization.”  In a simplified Data Ops workflow for product development, data managers can include DevOps teams and data engineers, and the data consumers are usually product teams.

>The end goal is to invest in product data to inform software development and engineering decisions. Ultimately, this helps teams effectively design products for the end user, leading to increased customer satisfaction. Successful SaaS companies such as Datadog, Slack, Calendly and Zoom have used this strategy to accelerate growth and develop features that matter to customers.

Let’s take a look at three practical ways to use DataOps strategies to improve products and services.

1. Use production log data to speed up software delivery and performance

For fast-growing ed-tech SaaS company Transeo, application and networking layer log data can make a major difference to the CX. Mission-critical processes within the software have tangible, real-life impacts. For example, a student could potentially not get into college if the software didn’t deliver a transcript correctly to a university.

To prevent these issues from happening, the DevOps team needed access to long-term log data to troubleshoot its cloud services and application performance, and make critical improvements in the software development and delivery process. If a product wasn’t working as expected, it could lead to a poor CX and customer churn. Knowing where to improve certain features and functionality of the platform was critical.

However, the company’s existing DevOps tools, including observability and analytics tools like Elasticsearch, weren’t efficient or reliable enough to uncover insights hidden within terabytes of log data. This data was generated by Transeo’s applications, load balancers, Kubernetes clusters, and Nginx controllers, all of which had different formats streaming to Amazon Simple Storage Service (Amazon S3). The end result was an object data store filled with information that was extremely difficult to access and use.

As an alternative, Transeo transformed its existing cloud object storage environment into a data lake for cost-effective analytics at scale. The team leveraged its new data lake platform to streamline and automate data normalization and indexing, without having to move or transform data. This capability now helps Transeo ground future product decisions in data. In addition, the development team can act quickly on data and ensure that users are not impacted by application and system issues.

READ: Fast-growing SaaS Scales Log Analytics While Saving 30% per Month

2. Correlate user data and system events to understand user actions

Product data can also help teams understand how features are used, and how application performance impacts certain user actions. Equipped with this information, DataOps teams can build dashboards for product stakeholders to improve critical features and functionality.

From there, product stakeholders such as sales and marketing teams can tap into this user data to create more informed campaigns and close leads. For example, Slack leveraged user data to determine when activity increased, and fed this information to sales teams to close deals. To do this, the team tapped an internal administrative tool to identify its most active users by measuring things like workspace activity, messages sent, invites, and more. This data was valuable to sales, as it allowed them to tailor outreach to the user and determine who was most likely to respond as adoption accelerated.

This tight collaboration of DataOps and business users requires readily accessible data, along with strong data literacy skills to drive faster time to insights. A major part of enabling data literacy is to set up the right infrastructure and approach to data management. For many companies, a data lake architecture is the best choice for efficiently storing, securing, and accessing their data, including log files and other critical information that can reveal important business insights.

Watch: Data Lake Opportunities: Rethinking Data Analytics Optimization [VIDEO]

3. Experiment with namespace data and UX actions to drive customer satisfaction

The right access to product data can also drive customer satisfaction for existing users, especially newly onboarded customers. That’s what DevOps platform GitLab did to ensure that its new users got to the right use cases faster. By looking at a user’s namespace data, the team was able to determine how users progressed through various use cases of their platform. Four example use cases were:

• Version control and collaboration
• Continuous integration
• DevSecOps
• DevOps platform

The GitLab team then analyzed this customer journey data to create personalized onboarding for new users. This process helped users get to the right use case faster, and directed them to the appropriate features in the platform. In some instances, the team identified opportunities to “land and expand” product usage within its customer base. Many other organizations have correlated UX and CX data in a variety of ways, using everything from social media to customer service analytics to create a better product.

LISTEN: Keeping the Chaos Searchable

Understanding what’s hidden in your data lake

Using log analytics, user and UX data, DevOps and product teams can collaborate to create more innovative products driven by user demand. As mentioned above, a data lake platform unlocks access to data for DataOps teams, by empowering them to use existing, low-cost cloud storage options such as AWS S3.

From there, it’s simple to correlate various data types, and query data without data movement. Through this process, any DataOps team can discover what’s hidden in their data lake, and how that data can inform product development and future roadmaps.

Want to learn more about how to manage data overload with data lakes?

Get the 2022 Cloud Data & Analytics Survey Report

Additional Resources

Read the Blog: Talent Shortage 2022: Stretching Your Lean DevSecOps Team

Watch the Webinar: The Agile Journey in 2022: DevOps, Cloud, Automation, and Data Management

Check out the Whitepaper: Gartner® Predicts 2022 - Driving Toward Digital Infrastructure Platforms

About the Author, Dave Armlin

FOLLOW ME ON:

Dave Armlin is the VP Customer Success of ChaosSearch. In this role, he works closely with new customers to ensure successful deployments, as well as with established customers to help streamline integrating new workloads into the ChaosSearch platform. Dave has extensive experience in big data and customer success from prior roles at Hubspot, Deep Information Sciences, Verizon, and more. Dave loves technology and balances his addiction to coffee with quality time with his wife, daughter, and son as they attack whatever sport is in season. He holds a Bachelor of Science in Computer Science from Northeastern University. More posts by Dave Armlin