At a recent summit of Business Intelligence experts, hosted by Dashboard Insight, one of the questions posed to the panel I was sitting on was "Whither BI, what is the future of BI?" This is a question that has been steadily plaguing me over the past several years, and one of the reasons why I had consciously started to focus my attention on the dashboard side of the business. After engaging in many well-intentioned and executed BI projects that yielded mediocre results at best, I started to wonder "Is BI broken? Is there a better way?" At the time I saw a focus on dashboards solving many of the inherent problems that I had recognized in the traditional approach to BI, because a greater emphasis was put on solving user problems versus data ones. Over the course of delivering nearly a hundred unique dashboard solutions I came to adopt what I refer to as a user-centric approach to business intelligence design. Some people also refer to this as a top-down approach, which lies in stark contrast to the traditional bottom-up or data-centric approach used today in most significant business intelligence projects.
How we got to where we are:
At this point it might be helpful to talk a bit about the history of BI and the shifting user base it has been designed to target. The bottom-up (data-centric) approach to BI started when the primary goal of business intelligence was solving the engineering and architectural challenges of integrating and reporting against a company's internal data that was often siloed in a few large data repositories that were generated by their various critical business software such as the accounting, sales, and inventory systems. The focus of these efforts was creating data structures that data analysts could do rudimentary reporting on and analytics. When business users needed answers to questions like "what are our top 10 selling products versus our top 10 most profitable products" the data analysts could then design reports that tied together the accounting data with sales and inventory data. The primary users of these systems were trained data analysts who served as a human bridge between the business users and the specialized tools designed to access the back end BI systems.
In an effort to create reporting systems that gave a small specialized group of data analysts the ability to generate a myriad of reports for a diverse set of business needs, the focus was on designing highly flexible systems that anticipated a wide array of abstract business requirements. The BI industry responded accordingly by developing the necessary tooling to extract, transform, process, organize, and analyze these abstract data structures, and thus we saw the emergence of ETL tools, Data Warehouses, OLAP cubes, as well as many other specialized technologies. Up to this point, the innovation in BI was really focused on how data was managed and manipulated. Technologies were invented and methodologies developed and espoused, in some cases with an almost religious fervor. But in this early period of BI, during the emergence and development of BI as a discipline, one key ingredient was conspicuously missing: the end-user who was the ultimate beneficiary of this "intelligence."
Where we are today:
Cut to today, and we find ourselves in a surprisingly different environment. Business and technology have been radically altered in many unexpected and significant ways through humanity's adoption of the internet during the mid to late 1990's. We now find ourselves in a highly dynamic and connected environment where business moves at a much faster pace, requiring that decisions be made faster and with more accuracy. We also are faced with exponential growth in the volume of data we produce, collect, analyze and are forced to interact with. Not only has the amount of data grown significantly, it is also far more distributed and heterogeneous than it ever was. Companies no longer have their critical business data stored in just a few large systems, but they also receive important business data from many ever-changing outside sources that the company may have little or no control over.
As BI has evolved over the past twenty years and has tried to keep pace with these ever more complex set of business and technology conditions, there has been more and more attention on enabling business users with direct access to these business intelligence tools. The first of these end user tools came in the form of static or canned reports that users could access directly, followed by the invention of "ad hoc" reports that gave users WYSIWYG tools to create their own reports against pre-determined data structures, and we now have progressed to easy-to-understand highly visual dynamic dashboard displays. The phrase "BI for the masses" has come into vogue over the past couple years, but unfortunately BI as an industry is still correctly perceived by the business community as having very little success in achieving this vision. BI tools are still considered too hard to use, too long to implement, and costing too much. Why is this?
Obviously there are real technology and business process challenges that we must overcome to accommodate the volume and pace of data generated by our internet enabled global economy and business conditions, but I believe the primary challenge we are facing is the BI industry itself. At the risk of being lambasted by the cadre of established industry gurus I would like to posit the thesis that the large BI companies and the recognized "experts" are actually hindering the very innovation and processes that would most benefit business and increase the efficacy of these tools. I make this statement because I believe too much of the focus is still being placed on collecting, manipulating and managing data when it really should be put on how users interact with the data, and what business conditions they are trying to improve via this interaction.