The Data, The Analytics and the In Between

Analytics transformation within organizations starts with data — collecting, organizing and storing data that might be of some value to you. Temperatures from sensors, clickstream from your website, demographic data, cash flows, job performance of your employees, on and on and on.

Rob Thomas, who lead IBM cloud when I was there, co-authored a book on the “AI ladder” — a concept that IBM sales reps spent 2018–19 fruitlessly trying to interpret and regurgitate to customers. I agree with the sentiment around data collection — every company should collect as much data as possible. Who knows what kind of value might be found?

What should organizations be aiming for in analytics?

What’s easy to see two years down the road, here in 2021, is that AI is not the end goal. At least, AI is not the end goal for most organizations, now. Most organizations are trying to simply take advantage of the data that they have and make it more valuable. This is what most organizations are trying to do, and my proposal is that all organizations should be doing this.

If I were to think of a framework for taking advantage of data systematically, I would take the following approach.

You need to capture the data

First is data, and the systems that collect & store that data. Databases, data marts, warehouses, lakes, whatever your flavor and preference. Experts will holler about what the right data infrastructure is — and some are better than others — but you can make do with most. We like Snowflake, I hear cool things about Firebolt, lakes are cool and our Spark-based friends are building lakehouses…it’s all good!

Believe it or not, we’ve worked with customers who aren’t capturing core information about their business that could provide a significant ROI across a variety of functions. The above is not a given.

So what comes after the data?

“What do I do with it”, you ask? After data, there’s tools that make value of that data. Maybe you’re using your data to train a model that informs how you service customer returns, or perhaps you are creating market share visualizations that a team of analysts use to support product decisions.

Tools that make value of your data are what most people call analytics, with opposite poles of “BI” and “Data Science”. These poles might be converging, depending on who you ask, and this is where we play in the market — serving the convergence of BI & Data Science.

Why are these poles converging? We know that technical skill can be automated, but creativity cannot. The joins, the transformations, regression analysis, decision trees can be automated or organized into a repeatable framework. What can’t be automated (usually) are the minds that interpret the answers and take action.

A variety of tools are extending access to these automation technologies to more people, making anyone their own analyst. These tools are generally referred to as augmented analytics, one example being Tellius. We think that putting this automation in the hands of more people, with a simple interface, will be a gamechanger for companies all over the world as they unlock the power of their human capital.

So you’ve got the data, and you’ve got the tools…but why aren’t your business users becoming analytics superstars?

The In Between

The reason why organizations struggle even when they have the right data, and the right analytics tools, is the Great In Between. This refers to tools that help your users make sense of that data, by giving business context to the data structures that they may not be used to.

This is the missing piece — giving users a data experience that aligns with the corporate lexicon they are familiar with is key to driving adoption of analytics tools, especially augmented analytics. The integration of data definitions, robust metadata and connected concepts within augmented analytics tools is an incredibly immature field as of today, but it holds the key to a successful conversion of business user to self-serve analyst.

Luckily, there are some fast growing tools out there like our friends at Collibra, Alation and other open source knowledge graphs that make the above possible.

If you are an organization looking to transform at scale and take advantage of your human capital, you need three things:

1. The data (marts, lakes, lakehouses, who cares)
2. Augmented analytics
3. The In Between (data catalog that you integrate with #2)

If you have a better framework, I’d love to hear from you.