Will the last BI vendor please turn out the lights

Will the last BI vendor please turn out the lights?

As someone who believes strongly in the potential for business intelligence (BI) to empower people and transform organizations, I have an important concern to share with every like-minded BI pro: We aren’t being disrupted, it’s already happened. Despite increasingly powerful dashboards and data exploration tools, traditional approaches to business intelligence are struggling to meet the expectations of the modern data-driven workforce.

Fortunately the core tenets of BI — that combining data makes it more valuable and that people are more powerful with data — are alive and well. What has changed is both the volume and diversity of data, as well as the expectations of end-users who depend on it. The result is that data demands are increasingly being satisfied with tools that simply aren’t BI. This should excite us all as it presents a huge opportunity to rethink our data strategies and heighten the impact of our teams. Examining three mega-trends influencing the BI market tells us that the future looks very different and helps point the way forward.

Trend 1. The data driven workforce has arrived.

It should be abundantly clear that everyone is comfortable with data nowadays. It’s embedded in our personal lives in all sorts of ways: the reviews when we shop online, fitness tracking when we jog down the street, movie recommendations when we chill on the couch, social apps feeding us the latest personalized news, and so on. This pervasiveness of data translates to the workplace as well because everyone needs data to get their job done. Not just traditional analysts. Not just quantitative marketers. Not just growth hackers. Everyone including factory floor workers, pizza delivery drivers, and even school teachers get more powerful with data.

And, although it pains my heart to say it, these modern data consumers don’t expect classic BI reports, or even fancy natural-language enabled dashboards. They expect the data to come to them tailored in an interface designed specifically for the task at hand, ideally integrated into a tool they are already familiar with.

Of course we still need great BI tools for our analysts and data jockeys. But we must also be aware that there are a rapidly increasing number of data-enabled workers who view the idea of using dashboards the same way that you or I might view using a rotary phone. And that’s okay. I’ll claim that the future of BI doesn’t look much like a BI application at all, and that you don’t need everyone to become familiar with analyst tools to have an insight-driven business. People aren’t going to go to BI, BI has to go to the people. This is already happening in a big way.

Trend 2. The proliferation of SaaS applications.

SaaS adoption has absolutely exploded. Think for a moment about the number of SaaS applications you’ve used today. I’m writing this post at 10am and I’ve already used Namely, Paylocity, DataDog, JIRA, and about 10 other tools — all of which I love, all of which are essentially polished user interfaces on top of rows of data.

There is an awesome purpose-built SaaS app for most every problem you could encounter at a company. Looker uses about 140 SaaS apps to run our business and we’re not remotely unique. Mary Meeker’s recent Internet Trends report found that the average enterprise is using about 1,000 SaaS applications, and that’s trending upward.

While this is incredibly powerful from the perspective of the average employee, it’s a nightmare challenge for IT departments and data analysts. We BI pros fundamentally believe in the power of connected data. 1,000 SaaS apps means 1,000 little data silos that should be stitched together. But let’s be honest, is that really what’s happening in our organizations today?

According to a McKinsey Digital study, only 1 percent of all the data created in the past two years has been analyzed. This should come as no surprise. You aren’t alone if you feel like your data team spends 90% of their time trying to wrangle data and keep the chaos at bay. But no amount of workbook automation, no amount of SQL enablement training, no amount of ETL code is capable of wrangling the explosion of data volume and complexity. And there is no going back, this is the world we live in (someone in your company has probably signed up for a new SaaS app while you’ve been reading this blog post). Fortunately, there is hope for data teams in the form of more powerful infrastructure.

Trend 3. Modern data infrastructure.

Modern massively parallel processing (MPP) data warehouses (e.g. Google Cloud BigQuery, Snowflake, Amazon Redshift Spectrum) have made step-function improvements in just the past few years. They can hold immense amounts of data, query it all in seconds, and even do advanced analytics directly in the database — all this at a cost which is bafflingly cheap compared to last generation technologies.

Looker is not traditional BI and we took a huge bet on MPP databases early on. We predicted that they would lead to fundamental changes in data infrastructure. In a world where we can dump as much data as we want into one place, query it fast, and pay pennies for the privilege, whole steps in the traditional data engineering workflow can be simplified.

Rather than doing the heavy lifting of creating aggregate tables, massaging data, and data prep outside of the database, we do much of it in-database by transforming data when it’s queried. And since queries can be directly executed against the data warehouse, that data is more fresh, more detailed, and more trustworthy. It’s better data for less effort. This significantly improves the lives of data engineers because they can spend less of their time building and maintaining data pipelines and more time doing what they really want: empowering end-users with data-driven experiences.

So what does this all mean for the future of BI?

One thing remains constant: the analyst is still the hero. It is their knowledge and passion for data that will provide the deepest insights to businesses.

But in a world where everyone — not just analysts — depends on tailored data apps to get their work done, where data complexity and volume is increasing at an incredible rate, and where data infrastructure is exponentially more powerful, I believe the analyst has a new set of responsibilities. In addition to being the hero who wrangles and interprets data with their own powerful set of tools, they must also learn to empower others with data in new and unique ways. This means taking a fundamentally different approach to BI:

• New infrastructure that connects SaaS apps and departmental data with a BI fabric that flexes and scales with the immense demands of the modern data ecosystem.
• New governance that provides a single source of truth for business data. One which is unified, trusted, fresh, and comprehensive.
• New purpose-built data experiences that go beyond dashboards, reports, and embedded visualizations to provide actionable insights delivered when and where they’re needed.

In doing so, analysts can unlock a new frontier in which people from all backgrounds, in any type of company or department, including those who traditionally have not worked in data, will be empowered to work smarter. In this world everyone, each in their own way, becomes a data hero.

rock on@nickcald

Curious how these three BI trends are impacting actual companies? Check back here in a few weeks as Nick will detail some real-world scenarios.

Will the last BI vendor please turn out the lights was originally published in Hacker Noon on Medium, where people are continuing the conversation by highlighting and responding to this story.