Article written by Claudia Imhoff, Ph.D.
What a world we live in today – Fickle customers, demanding government regulations, economic insecurity, and thriving competitors coming from all corners of the globe. A company’s success or failure depends on all of its employees making good decisions. Business Intelligence (BI) is an important ingredient in the decision making recipe.
No longer relegated to the ivory towers of highly technical and specialized power users, BI is permeating all aspects of the enterprise, opening its doors to a much larger community of users. BI has been very successful in supporting traditional strategic analytics like customer buying habits, product profitability, market trending, etc.
Now it is being asked to support more pervasive/operational needs such as analyzing real time supply chains with demand chains, detecting fraud immediately from transactions, and alerting employees to sudden changes in markets, customer behaviors, and even competitor actions.
These new users and new forms of BI have put great stress on the underlying technologies. Specifically, some of the challenges that are intrinsic to a comprehensive BI environment are:
- The more the merrier – maybe? BI, traditionally relegated to the back room analysts and statisticians is now front and center with front line workers. By rolling out BI to more users, any problems occurring with the BI technology – performance issues, availability concerns, unplanned delays, and so on – have much bigger ramifications. Certainly, more people will be affected but, more importantly, the operations of the organization could be affected.
- Sub-second response times – These new, operational users cannot wait long for answers to their queries. In fact, many implementations of operational BI are proactive; the application can analyze transactions as they are streaming into the enterprise and send out alerts to appropriate personnel when an unexpected result occurs. BI technologies must perform as if they were supporting operational systems, that is, return results in sub-second to a few seconds timeframe.
- Pervasive BI should not degrade the query performance of traditional strategic BI users – BI environments must now handle a mixed workload. It must perform well for both the short, fast transaction-like queries from operational users while still supportingthe longer-running, more resource-intensive strategic queries. Many technologies have found this one characteristic of the comprehensive BI environments to be very difficult to accommodate. The technology must be robust enough and have an underlying architecture that supports the needs of mixed workloads, ensuring each type of query gets the appropriate amount of resources to satisfy the needs of the users.
- More volumes of data – Perhaps the biggest change coming from the addition of operational BI to a BI environment comes from the increased quantity of required data. Operational BI not only increases the frequency of data loads; it also means that new kinds of data will be required. The data warehouse will begin growing at unheard of rates if the first operational BI application is successful. These new users will ask for more and different kinds of data when they see the value of the environment.
The result is an explosion in query volume and increasing query complexity. Unfortunately, many BI implementers are faced with technological infrastructures that can’t keep up in the face of this maelstrom. You can go only so far by adding more indices, more partitioning, more hardware, and more caching to improve throughput. Business users are outstripping such mitigating activities quickly with increasingly complex, unplanned, ad hoc queries that require larger and larger amounts of data.
What’s IT to do?
Fortunately, help for IT is available from innovative BI vendors who have stepped up to resolve these problems. A bright spot comes from the innovative BI technology vendors, such as Kognitio. These groundbreaking architectures represent the advances needed to support complex and ad-hoc queries but, with so many now on the market, what attributes should you look for to cope with your query storm?