Tools such as ChatGPT, as well as Midjourney AI or Lensa AI, have taken over the internet in the past few months like never before. Everyone is talking about them, posting printscreens of conversations with the AI, or AI-generated pictures, some dwelling on the inevitable rise of artificial intelligence (and possible takeover of our jobs), others pointing out the shortcomings of the technology. Nonetheless, we have entered a new era, that of artificial intelligence and machine learning, and with it, big data has sparked once again everyone’s attention.
Why is that? It is more about machine learning models rather than artificial intelligence, since these tools are trained with data – a lot of data that is processed and analyzed in a short amount of time. Sure, the latest GPT model (GPT-3.5) was fine tuned by humans; but it is based on the previous model (GPT-3), which was trained on a massive internet data set (or billions of online texts). Although ML models have been around for a while, and big data was a popular topic in 2015, now, with the rise of these tools, they are once again gaining huge visibility in all industries, and in all aspects of life.
Not only that, but the rising number of IoT devices also puts big data at the forefront of the discussion. According to Statista, by 2030, the number of IoT-connected devices is expected to reach 25.44 billion, a rise from only 10.07 billion in 2021. Given these stats and information about AI, let’s see how augmented analytics, DataOps and observability, data stewardship, synthetic data generation, data mesh, data fabric, and industry cloud platforms are changing the landscape of big data in 2023.
Since producing correct insights from data faster is one of the main goals for most companies nowadays, there’s no wonder that augmented analytics gains momentum. While traditional analytics offers insights through reports from previously collected user requirements or predefined queries, augmented analytics makes use of AI, ML and NLP processes to generate reports automatically, without the need for a data engineer or a data scientist to spend time preparing data for business intelligence, thus reducing the manual labor of traditional BI work.
Augmented analytics allows producing context-aware suggestions, automating tasks and analyzing conversations, thus, it not only augments how people locate, explore, and analyze data in BI platforms, but it also augments data scientists by automating several aspects pertaining to data science, ML, AI model development, as well as management and deployment. While this notion first appeared in Gartner in 2017, it is only now that it is gaining strength in organizations, since it allows businesses to get instant insights for specific queries from the data lake directly, without designing specific data pipelines.
The agile methodology in software or application development is no new thing. However, applying agile principles to your data processes, known as DataOps, is one of the trends that are seen growing in 2023. Data-centric organizations will start adopting the agile and iterative techniques that are common in DevOps to the whole lifecycle of data, as this provides velocity in delivering new insights.
Besides velocity, DataOps brings observability into big data as well, this framework helping with the monitoring of data health and reducing data downtime. As DevOps has risen in popularity and shown that these processes help develop and deliver more efficient software, it is now starting to influence big data processes as well. Having DevOps join forces with data scientists and data engineers will represent a plus for the ongoing development processes and the whole lifecycle of data, as opposed to having different staff handle each part of it: data generation, storage, transportation, processing, and management.
As part of the data lifecycle management, a specialization of data governance is also gaining traction lately: data stewardship. But what is the difference between the two? While data governance deals with the high-level policies, processes, and procedures, a data steward is only focusing on the implementation of procedures, making sure that the policies and the standards are followed accordingly.
Acting as a link between the IT department and the business-side of an organization, a data steward carries out data usage and security policies, while acting as both a data coordinator (tracking data transportation) and a data corrector (how data can be used). His main responsibilities include but are not limited to defining the data, identifying and maintaining data quality, optimizing workflows, monitoring data usage to assist different teams, and ensuring compliance and security of the data.
If you’re looking to manage data better, maybe a data steward is just what you need.
Data privacy concerns, as well as difficulties in obtaining real world data, has made synthetic data generation one of 2023’s popular trends. Using artificially produced data can speed up the training of AI and ML models, expand use cases, improve accuracy, and help protect data that contains sensitive information.
One of the critical topics in the last few years related to AI and ML was that the models end up with biases towards certain groups of people (race, gender, biological sex, age, and culture) due to the data that they are fed. Enter: synthetic generated data, which can help fight against these biases.
Having all your information in a central data lake, with only one data team to analyze and extract useful insights for various teams (management in different departments, product owners) within an organization was considered a great idea initially. However, it has slowly been deemed a bottleneck since the data team don’t have the time to both fix broken data pipelines and understand domain data/learn domain knowledge. That’s how domain-driven design appeared, where domain teams (or product teams) own and know their domain but need to reach to the central data team to get the necessary data-driven insights.
Coined in 2019, data mesh proposes a shift from the centralized platforms (data lake, data warehouse) to a paradigm that draws from modern distributed architecture, with the goal of democratizing and managing data at scale. We could also say a shift of the responsibility for data from the central data team to the domain teams. Thus, a data mesh architecture allows domain teams to perform cross-domain data analysis on their own.
Data fabric, which we have also discussed in last year’s article, continues its popularity, as the data fabric technology is revolutionizing the way data is collected, analyzed, and managed. The technology has evolved to be a great architecture that provides business agility and data accessibility, through seamless and real-time integration, and cross-access in the diversified data silos of a big data system. Valued at $812.6 million in 2018, the data fabric market size is projected to reach $4,546.9 million by 2026.
Cloud Migration and Hybrid Cloud
In the search for digital transformation, migrating to the cloud still represents a focus for most companies, and remains one of the most important digital transformation trends. Depending on the industry, since some data must be kept on-prem, we find that the hybrid cloud is still the go-to for most organizations.
Industry Cloud Platforms
A newer strategy in cloud is represented by industry cloud platforms, which offer a combination of SaaS, PaaS, and IaaS to support vertical industry segments, turning cloud platforms into business platforms. They accelerate cloud adoption by having a set of industry-specific sets of capabilities (modular, composable platforms) that act like building blocks offering relevant solutions for organizations in industries where generic solutions are not the best match. Gartner predicts that by 2027 more than 50% of organizations will use industry cloud platforms to provide agility and innovation, while reducing time to market and avoiding vendor lock-in.
Through this article we’ve seen why augmented analytics, DataOps and observability, data stewardship, synthetic data generation, data mesh, data fabric, and industry cloud platforms represent the main big data trends to look forward to in 2023; what are your business plans related to data for this year? As we are offering big data services to our customers, we're curious to see which of the above trends will prevail this year.
An enthusiastic writing and communication specialist, Andreea Jakab is keen on technology and enjoys writing about cloud platforms, big data, infrastructure, gaming, and more. In her role as Social Media & Content Strategist at eSolutions.tech, she focuses on creating content and developing marketing strategies for the eSolutions’ blog and social media platforms.
RO
EN
