Five top trends in Australia shaping data engineering in 2022

Tuesday, 22 March 2022 11:16

Five top trends in Australia shaping data engineering in 2022

By Theo Hourmouzis, ANZ regional vice president, Snowflake

Snowflake ANZ regional vice president Theo Hourmouzis

GUEST OPINION: Data pipelines are the lifeblood of the modern enterprise. Efficient data pipelines can be the difference between an architecture that delivers real value to the business or one that becomes a burden. Enterprises receive data 24 hours a day, seven days a week, from the web, enterprise applications, and mobile and IoT devices. Data engineers must collect, transform, and deliver data to different lines of business, while keeping up with the latest technology innovations to stay ahead of business demands. However, traditional legacy architectures create challenges every step of the way, making the task of the data engineer more complex.

There are five key global trends that are likely to shape the data engineering space in 2022. They are:

1. The rise of unstructured data

Unstructured data accounts for a tremendous amount of information, but it poses an interesting challenge for organisations attempting to extract value from it. It is not easy to search, analyse, or query unstructured data, especially on the fly, posing more challenges to data architectures that are already complex.

Unstructured data has typically been outside the realm of traditional data warehouses. Across the integration, transformation, aggregation, and delivery steps, organisations deploy multiple tools, with each connection point a possible point of failure. Governance, security, and silo issues are commonplace with this approach, as multiple copies of data eventually emerge to meet the varying needs of different parts of the business. In the end, organisations spend more time managing infrastructure and little time actually working with data.

At the same time, data science and analytics has evolved well beyond dealing with just structured or semi-structured data. For this reason, data pipelines within a business also need to be able to deal with unstructured data.

Take the example of a telecommunications company. The company wants to analyse trends around customer churn and so needs to combine structured data from its billing system with unstructured data such as email communication with customers or call centre recordings.

Expect tools and platforms that can process both structured and unstructured data to become more dominant during the year.

2. Breaking down the new data silos

Many organisations have tended to believe that problems they have as a result of their data being stored in multiple, siloed locations can be solved by simply shifting that data to the cloud. It's their belief this will allow easier access and enable analytics teams to locate and use any data they might need.

The reality is somewhat different. Many businesses have found cloud usage actually increases the number of silos as different groups make use of different cloud platforms. This actually perpetuates the issue rather than solves it. Governance, security, and silo issues are commonplace with this approach, as multiple copies of data eventually emerge to meet the varying needs of different parts of the business. In the end, organisations spend more time managing infrastructure and little time actually working with data.

This situation can be overcome through a strategy of data sharing. Data is shared with parties who need access to it but remains stored in its current location. This allows the issue of data silos to be resolved without requiring massive changes to the underlying architecture.

3. The rise of unified streaming

During the past few years, data streaming has become much more widely used within many organisations. It involves the continuous transfer of data at a steady rate, and is being driven by the rise of cloud computing and the internet of things (IoT).

Streaming offers benefits when it comes to data access, however it can also cause challenges when teams have to manage and maintain the streams to ensure they are consistent. Many streaming services are also very specialised which means they cannot be applied to all potential use cases.

During 2022, work will continue on new streaming tools and services that will overcome these restrictions and make the approach even more valuable for businesses.

4. Serverless infrastructures

Data engineering and processing is increasingly shifting to serverless environments, hosted on cloud platforms. This allows a business to get value from their data without having to deploy and manage an on-premise infrastructure.

Serverless strategies are gaining particular traction within businesses that are providing SaaS products to customers. These businesses will not know ahead of time the usage patterns that are likely to be seen and so need the flexibility to be able to deal with shifting volumes of data.

It's about having the ability to scale up and down as requirements change and the strategy is likely to become even more widely used in 2022.

5. Language independent pipeline runtime

If you look back 10 years, data pipelines were being written using a single language – typically Java. Now that the data engineering sector has matured, software developers are more focused on using the best possible tools for a particular job, and this results in pipelines being built using multiple languages.

For this reason, expect data engineering platforms to continue to increase the ways in which engineers and developers can readily use multiple languages as part of their workflows.

Together, these trends will help to shape the data engineering space as it continues to evolve in the year ahead. This will ensure businesses can derive maximum value from their data, and take advantage of new trends and opportunities as they emerge.

Read 131 times

PROMOTE YOUR WEBINAR ON ITWIRE

MORE INFO HERE!

INTRODUCING ITWIRE TV

SEE WHAT'S ON ITWIRE TV NOW!