In today’s fast moving business environment, it is clear that Business Intelligence (BI) is a mandatory part of any company’s decision making environment. The question then becomes how to build this environment in a way that keeps up with ever-changing business requirements, new and large numbers of users, all forms of analytics, and increasing volumes of data.
Forward-thinking CIOs understand the changeable nature of their BI environments but may be stymied about how to build flexible BI architectures without “breaking the bank”. Fortunately there are BI vendors specializing in new database technologies that can handle many of the pressures for quick, inexpensive and flexible BI architectures.
These new databases use a completely new approach to storing data. Instead of the traditional row-by-row mechanism of most RDBMSs, these innovative databases store data in a column-by-column fashion. The benefits for analytics from this change are enormous.
However, simply storing data in a columnar fashion is not enough to result in the vast increase in performance achieved by some of these new databases. Compression along with intelligent de-compression techniques add to the performance increase.
But even this is not enough. In determining which column-based database to use, you must also look into the maintenance burden on the IT staff. Look for technologies that no longer require the creation and backbreaking maintenance of indices or partitions.
Finally, look for a database technology that truly does seamlessly scale. Its loading timeframes should not be affected by increasing volumes of data or by increasing numbers of tables.
Business Intelligence (BI) started more than a decade ago as homegrown, nice-to-have, decision support mechanisms, mostly used by statisticians and financial analysts. The ability to analyze month-overmonth financials or study past market performances was on the list of needed BI information – just not at the top.
We’ve come a long way since then. BI is now a mandatory part of the enterprise IT environment. Its usage has spread to every corner of the organization, and to every level of the organization chart. It is now a mission-critical part of operations, supporting not only traditional analytics but also daily operational decision making, supporting rapid fraud detection mechanisms, risk mitigation analytics, behavioral and market predictions, etc.
With these significant requirements come massive changes to the underlying infrastructures sustaining BI environments. These changes include:
- Significantly increasing volumes of data. Now that BI has permeated the business, the amount of data that must be collected to satisfy these needs is enormous. And it is not just more of the same data. Click stream data from websites, RFID tracking data, and other new sources have greatly added to the volumes needed for analytics. Further increasing storage requirements, many business users require that the data be collected several times during the day for operational BI purposes. It is not unusual for an enterprise data warehouse to contain many terabytes or even a petabyte, of historical data for analytical and reporting purposes.
- Fast response times for queries and analyses. With the advent of operational BI or decision support for front line decision makers, the response time for queries must match those commonly found in other operational systems. That is, less than 2 seconds. An already angry customer will not take kindly to waiting while the rep “pulls up” their lifetime value score or next-best product offer. This means that BI environments must differentiate between traditional strategic or even tactical queries from those of an operational nature. This “mixed workload” capability is at the heart of the new technologies and data warehouse appliances built specifically for BI.
- Seamless scalability. Most data warehouses don’t start in the terabyte storage range, but they tend to get there quickly. This means that the hardware and software must scale up, with minimal impact on both the IT staff and the existing technological environment. Going from 1 TB to 100 TB should be as painless and trouble-free as going from 1 GB to 1 TB. Again this is the focus of the new data warehouse appliance vendors.
- Much broader audiences. BI started out supporting business analysts, statisticians, advanced market researchers and others having a need for and education in analytical capabilities. Over the years, more and more employees throughout the enterprise began to see the value of BI but did not necessarily have the skills or education in its usage. BI vendors had to change their interfaces, making utilization much simpler to understand, navigation easier to perform, and comprehension effortless through visualization techniques. From an infrastructure standpoint, it meant the technology had to support massive increases in the numbers of business users, while still supporting the traditional audience of sophisticated analysts.
- More innovative analytics by business users. As the usage increased, so did the complexity of the analytics. The evolution of BI utilization in most organizations proceeds from the creation of simple reports, to time series comparisons, to complex models of fraud detection and risk mitigation, to intricate predictive analyses of customer and market behaviors. The time frame of the data for these analytics also changed. Daily, weekly and monthly snapshots of data were no longer sufficient. Enterprises required BI support for
intraday or operational decision making. Operational systems are now being fitted with streaming and embedded analytics to help front line workers make better decisions throughout the day.
Savvy CIOs understand these changes and are examining their BI environments to determine the role that innovative technologies can play in the new world order of analytics. They understand that a new reality has come to their businesses and that reality is called change. They recognize that their technologies must support the new pressures on their enterprises for rapid change to satisfy the BI stakeholders.