Ad hoc analysis: business intelligence’s fast food problem

In a world where businesses live or die on data, self-serve BI is the information super-food we’ve been waiting for. Or so the rhetoric around this popular tool would have us believe.

Also known as data exploration, exploratory data analysis, data discovery and ad hoc reporting, self-serve BI has many names, but one big promise: to radically reduce the time organizations spend on data collection, analysis, and reporting.

Central to this promise is that any employee who wants to can collect and analyze company data on their own. Regardless of training or experience, anyone is supposed to be able to quickly become proficient in data collection, organization and analysis.

Does the promise reflect reality? And do organizations benefit from employees’ widespread use of self-serve BI tools?

What self-serve BI does well

Maybe; it depends on what you want it to do. Ad hoc data analysis is:

• agile and easy to use; no one will waste much time figuring out how to use it
• readily accessible because most such tools live in the cloud
• flexible: users can interact with reports of all sorts, in many ways (e.g., drilling down or through data on customizable tables, charts, and crosstabs)
• built from reusable components, so it’s easy to create more and more reports, or have multiple reports interact

These are all good things. However, they can’t guarantee organizations will gain either full understanding of their business data or consistent, superior value from it.

Where the ad hoc promise fails

The fact is, self-serve BI’s primary attraction — making reporting both easier and more accessible — is also what makes it potentially dangerous. Why?

They don’t go far enough. Say a company selling umbrellas sees sales increase dramatically one week; using exploratory data analysis, a team learns this spike resulted from unusually rainy weather. Too often, teams will stop at explaining a past event such as this and think they’ve gained valuable information. Most of the time, this approach merely confirms something already known or easily guessed, without running a report at all.

They’re treated as official reports and widely shared. Too often, ad hoc reports are confused with official ones; they may become corporate gospel, especially if they’re nicely formatted; they may then be used to support major business initiatives. The only reports organizations should consider official are those planned by qualified experts for specific or regular reporting needs (e.g. monthly reports).

An ad-hoc approach to data analysis can be extremely subjective. If everyone in an organization can imagine and create reports, odds are some will be severely skewed by confirmation bias: that is, when a person stops researching when the information they’ve already gathered supports the result they want.

Considered in light of the DELTA Plus Model, the above describes an organization displaying level-two data maturity: Able to analyze data to explain certain observations, such companies also own much useful data. But this data is usually siloed, its disparate pieces accessed by team members with strikingly different skill sets who don’t communicate with one another.

Self-serve BI may ultimately, despite the hype, leave a bad taste in your mouth. It absolutely won’t help you build a smart, sophisticated, data-led organization.

Empty Calories

Articles addressing self-serve BI generally focus on how it helps users with little or no data analysis knowledge very quickly become data professionals; promises of fast results abound.

Fast results aren’t necessarily good results, however, just as fast food isn’t necessarily good food. Fast food may not harm one person, when eaten in moderation; but when a community consumes it often or exclusively, health issues will inevitable result. Likewise, ad hoc analysis might not harm an organization if one person does it occasionally.

But when a company’s predominant approach to mining and analyzing data is to have staff, regardless of role or expertise, create BI reports, that company will suffer. It will suffer, for the time and the financial resources wasted, as well as for the high potential for making poor decisions based on such reports.

For exploratory data analysis to be useful and accurate, a lot of work has to be done in advance, like:

• preparing and cleaning data, to make sure it’s accurate
• building a data warehouse, so all subsets and components are accessible and can interact
• establishing standard business rules, so metrics are measured consistently throughout an organization
• de-duplicating data, to save space and to prevent errors occurring if values stored in duplicated sections aren’t all consistently updated
• regular refreshes to keep data current
• creating infrastructure to maintain these safeguards’ effectiveness

Having data analysis novices perform this kind of work may be likened to thinking you can win a marathon although you eat only junk food and rarely work out.

One step towards future — and present — success

Ad hoc analysis works best when handled those who understand its strengths and limitations. Key to this understanding is knowing self-serve BI may be used to set the stage for predictive analytics, but it’s only one step in a larger journey towards becoming a data-led organization.

Companies that commit to a sophisticated and centralized BI plan that includes exploratory data analysis when appropriate will spend less time and money, get exponentially more accurate and useful data, and maintain sharper focus on their strategic business goals. These companies will be the ones that thrive.

At 3AG Systems, we support organizations ready to take a more sophisticated and mature approach to data.

Originally published at www.3agsystems.com.

Ad hoc analysis: business intelligence’s fast food problem was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.