The Shift to Small and Wide Data is Redefining AI Analytics

The days of chasing massive datasets may soon be over. For years, businesses have been enthralled by the promise of "big data," relying on enormous datasets to fuel machine learning models and uncover hidden trends. But as technology evolves and the global business landscape becomes increasingly dynamic, a new trend is taking center stage: small and wide data.

This shift, predicted to dominate data strategies by 2025, signals a profound change in how organizations approach analytics. According to a Gartner report, 70% of organizations will pivot from big data to small and wide data by then, embracing a more nimble, context-driven approach to decision-making. This evolution challenges the long-held belief that more data inherently leads to better insights.

Big data, while revolutionary in its time, has become a double-edged sword. The vast resources required to collect, store, and process large datasets often yield diminishing returns. Moreover, rapid changes in consumer behavior, like those observed during the COVID-19 pandemic, have exposed big data’s fragility. Models built on outdated information can quickly lose relevance, forcing organizations to rethink their strategies.

Ross Dawson, a noted strategic advisor, explains the shift succinctly: small and wide data leverages “diverse inputs, taking disparate sources and learning from them and their correlations without necessarily requiring the brute force of size.” This approach prioritizes relevance and adaptability over sheer volume, allowing businesses to react faster to changing circumstances.

“Big data has distracted us from a key truth: more data doesn’t always mean better insights,” says George Kailas, CEO of Prospero.ai. “Large datasets often introduce noise that obscures actionable intelligence. The future of AI lies in its ability to analyze small, precise datasets—the kind that answer specific, high-impact questions.”

Kailas likens this to choosing between a searchlight and a magnifying glass: while the former may illuminate a broad area, the latter reveals the intricate details that truly matter. For businesses, this means adopting a more targeted approach to data collection and analysis, focusing on quality over quantity.

Kailas’s perspective underscores the benefits of small and wide data: reduced costs, faster processing, and sharper insights. By forgoing the exhaustive collection of unnecessary data, organizations can streamline their analytics processes and make more informed decisions with fewer resources.

The adoption of small and wide data also aligns with the principles of agile development, emphasizing iterative, value-driven approaches to problem-solving. Instead of aggregating vast datasets into unwieldy “data lakes,” businesses are now collecting only the data necessary to address specific use cases.

For example, Kaizen Analytix demonstrated this strategy by helping a client acquire daily natural gas prices to refine company projections. Rather than building an expansive database of energy prices, Kaizen focused solely on the most relevant data, testing hypotheses before integrating the information into broader systems. This approach not only saved time and money but also delivered actionable results faster.

This shift is also changing the way AI models are designed and implemented. Traditional machine learning methods often rely on “black box” models, which demand vast amounts of data to detect patterns. However, these models can struggle to adapt to rapid changes, such as those seen during the pandemic.

In contrast, newer techniques like Bayesian modeling and ridge regression are more resilient. These methods thrive on smaller datasets and can be updated quickly to reflect changing conditions. By relying on statistical assumptions rather than brute-force computation, these models are better suited for tactical decision-making in dynamic environments.

“Small data makes AI less data-hungry,” Kailas explains. “It allows businesses to focus on what really matters—delivering value through actionable insights, not chasing unnecessary complexity.”

The pivot to small and wide data isn’t just a technological shift; it’s a cultural one. It challenges businesses to rethink how they define success in analytics. Instead of equating data volume with value, companies are now prioritizing agility, adaptability, and precision.

While big data will continue to play a role in uncovering deep, long-term trends, the future of analytics lies in leveraging smaller, smarter datasets for day-to-day decision-making. This evolution reflects a broader truth: in a world of constant change, relevance and speed are more important than scale.

As Kailas puts it, “The businesses that succeed in 2025 won’t be those with the biggest datasets—they’ll be the ones that ask the right questions and use the right data to answer them.”

By embracing small and wide data, organizations can stay ahead of the curve, unlocking new opportunities in a landscape where precision and agility are the keys to success.

Photo by Markus Spiske