Image Credit: John Lund

Aiven, a cloud-data platform based in Helsinki, has fleshed out an open-source ecosystem for Apache Kafka, a popular event-streaming platform. The new offerings promise to help enterprises consolidate their Kafka infrastructure using open-source components. 

“Event streaming is transitioning toward the main stack of the IT infrastructure,” Filip Yonov, director of data streaming product management at Aiven, told VentureBeat. “At Aiven, we have witnessed the fastest growth in the event-streaming domain compared to all other products.”

Apache Kafka provides the infrastructure for wiring streams of data together from databases, apps, IoT devices, and third-party sources. Kafka helps organize raw data into event streams that reduce data size and are easier to integrate into event-driven apps and analytics. Enterprises use it to improve customer experiences, build the industrial metaverse and monitor patients. 

However, building out a Kafka infrastructure involves a lot of moving parts. Aiven has consolidated all the necessary tooling into one place to simplify this process. Key new enhancements include support for Apache Flink and data governance. These complement existing tools for connecting services, replicating data and managing schemas for Kafka deployments.

The need for simplicity

LinkedIn originally developed Kafka to integrate data across its large microservices infrastructure and open-sourced it in 2011. Over the intervening years, large enterprises have customized the tooling for their own needs, and several vendors have rolled out proprietary enhancements to fill in gaps around governance and integration. Many organizations use Kafka for various data pipeline scenarios, such as transferring data between applications in real-time or moving data from a database to a data warehouse.

Yonov told VentureBeat that as Kafka clusters become larger and more complex, they require additional tooling and governance to ensure proper operation and management. “Unlike existing Kafka solutions, Aiven’s offering does not require organizations to choose between proprietary tools and vendor lock-in or open-source technologies without support,” he said.

Improving the developer experience with event streaming

One essential aspect has been to democratize the experience for working with event-streaming data. The open-source tool, Klaw, provides a self-service interface for managing Kafka clusters. Kafkawize, which develops Klaw, recently joined Aiven’s open-source development office in September to help integrate their tools together. Now they are working together to improve self-service, simplify user management and enforce data governance. 

Another significant development was to connect streaming data to SQL queries familiar to data engineers. The new Aiven for Apache Flink tools allows teams to process larger volumes of events and run real-time analytics using SQL. Aiven provides this as a fully managed service that reduces the complexity of deploying a Flink cluster. It also simplifies the integration with Aiven for Apache Kafka to filter, enrich and aggregate events on the fly. 

Aiven hopes to replicate the success of other open-source frameworks like PostgreSQL, Kubernetes and Linux, built by a healthy mix of contributions from various communities. 

“We truly believe that fostering an open-source, community-driven and inclusive ecosystem of technologies around Apache Kafka can drive further innovations and new developments in the data-streaming domain, ensuring the long sustainment of the technology in the future,” Yonov said.