Data analytics in real time: InfluxData evolves to support new workloads, demands and data

INFRA

Accurately interpreting data in real time is critical for many industries. Mechanical infrastructure, industrial equipment, stock market analytical tools, weather forecasting and equipment monitoring sensors are just a few examples of tools that could benefit from time-series data.

Time-series data platform company InfluxData Inc. is staying ahead of the market by working on perfecting time-series databases through its InfluxDB platform. The company hopes to expand its offerings, developed with open-source software, to a broader audience in the near future.

In separate interviews during a special broadcast from theCUBE, SiliconANGLE Media’s livestreaming studio, industry experts from InfluxData explain the innovative technology they are working on. As part of the broadcast, titled “Evolving InfluxDB Into the Smart Data Platform for Time Series,” theCUBE’s industry analyst Dave Vellante talks with InfluxData’s Brian Gilmore, director of IoT and emerging technology; Anais Dotis-Georgiou, lead developer advocate; and Tim Yocum (pictured), director of engineering, about how time-series data is evolving; how the company is improving its time-series database, InfluxDB; its continued commitment to open-source programming; and time-series data in general. They also touched upon the help the company has received with Apache Parquet and Rust programming and the contributions their developments have made to the ecosystem. (* Disclosure below.)

Innovating in a competitive market

Internet of things devices are quickly gaining traction. It’s no secret that the functionality of these devices depends on continuous inputs and outputs of data. However, high latency levels have traditionally prevented data from being transmitted in real time. Most data transmission today is accompanied by some level of time lag.

Time-series data refers to any data that can be timestamped and analyzed historically or in real time. Time slices can be identified down to the millisecond and used for data analysis that will improve real-time operations and help forecast future scenarios. Organizations that have the ability to import and export large volumes of data at lightning speeds will have a competitive edge over those that do not have this ability.

InfluxDB is staying ahead of the market by providing solutions for organizations of the future, anticipating their long-term needs, according to Gilmore. The catalyst for the new enhancements to the company’s engine was prompted initially by a shift toward the cloud, leading to the InfluxDB Cloud platform. Ever since this shift, it has been able to continually innovate its services to meet the ever-expanding needs of a society increasingly dependent on big data.

“If we think about what our customers are coming to us with now … related to requests like SQL query support, things like that, we have to figure out a way to execute those for them in a way that will scale long term,” Gilmore said.

InfluxDB is leading further innovations that aim to integrate its database into a smart data platform, taking advantage of all the innovation in big data and time-series data seen over the past few years. For customers, these changes will largely be transparent but will provide impressive new capabilities, especially for those who deal with high workloads, according to Gilmore.

The shift from time-series specialist to real-time analytics, and the ability to support both, is a natural evolution for InfluxDB. 

“The time-series market is one of the markets that we’ve worked diligently to lead. But we had to ask ourselves what we could do to better handle those queries … can we get to a point where the result sets are coming back so quickly from the time of query that we can limit that window down to minutes, then seconds?” Gilmore explained. “And, now, with this new engine, we are really talking about a query window that could be returning results in milliseconds of time since it hit the ingest queue.”

Since the first version of the InfluxDB Cloud, 60,000 people have signed up. The diversity of customers has enabled InfluxData to continually improve its product via feedback from new enterprises, developers, large companies and home hobbyists who use their service, according to Gilmore. The variety of operational customers that rely on InfluxDB — such as those in aerospace, defense and manufacturing, to name a few — has helped the company improve its real-time historical time-series database.

Here’s the complete video interview with Brian Gilmore:

A purpose-built time series database

The Rust implementation of Apache Arrow, the use of Parquet, and tooling such as DataFusion are powering a new engine for InfluxDB. By increasing the granularity of time-series data, real-time data analytics are made possible. By leveraging IOx (pronounced “eye-ox”), InfluxDB is innovating its engine to achieve faster query speeds while maintaining a cost-effective approach.

During the event, Dotis-Georgiou dove deeper into the benefits of IOx, explaining how InfluxDB is utilizing Apache Parquet in the Hadoop ecosystem and DataFusion written in Rust to continually innovate its time-series database. 

“InfluxData is doing a lot of really exciting things with InfluxDB IOx,” she said. “I really encourage you, if you are interested in learning more … go to the monthly Tech Talks and Community Office Hours. They are on every second Wednesday of the month at 8:30 a.m. Pacific Time … as a developer advocate, I want to answer your questions. So if there’s a particular technology or stack that you want to dive deeper into and want more explanation about how InfluxDB leverages it to build IOx, I will be really excited to produce content on that topic for you!”

Through its continued commitment to the open-source community, InfluxDB has made significant contributions to the Apache ecosystem, including enhancements to DataFusion, such as an increase in memory optimization and supportive additional SQL features, including support for timestamp, arithmetic and exists clauses, as well as support for memory control.

Here’s the complete video interview with Anais Dotis-Georgiou:

Updating a SaaS engine mid-flight

Open-source software has enabled InfluxData to keep its technology competitive. The company utilizes OSS to deliver its own core storage engine technologies, web services, templating engines and more. However, the open-source community innovates so quickly that it’s sometimes hard to know what tomorrow’s next hot technology will be versus which one will be soon outdated or replaced.

The ability to remain versatile, try new things and adapt quickly is key to the company’s success, according to Yocum.

“Look at Kubernetes, for example,” he said. “That ecosystem is driven by thousands of intelligent developers, engineers and builders. They’re adding value every day. We really have to keep up with that. As the stack changes, we try different technologies, we try different methods, and at the end of the day, we come up with a better platform.”

The application of Kubernetes has been central to the community’s ability to run InfluxDB. The ability to run services across different cloud platforms, such as AWS, Azure and Google, has helped with overall efficiency. By building on the “shoulders of giants,” as Yoakum puts it, InfluxDB has been able to focus on its own platform engineerings. 

Through open-source platforms, InfluxData has been able to maintain the continuous deployment of new enhancements and innovations in real time and mid-flight with minimal or no disruptions to the customer experience, Yocum concluded.

Here’s the complete video interview with Tim Yocum:

And make sure to watch the complete “Evolving InfluxDB Into the Smart Data Platform for Time Series” event video below:

Photo: SiliconANGLE