In today’s world, having intelligent business information is critical. Everyone talks about it, but connecting the dots and knowing how to deliver a capable system can be daunting. And many times businesses and IT go down the “proverbial rat holes”, spending a lot of money and time and still have not achieved their goals.
Some Suggestions That May Help
Pick the right database – Ideally you want to pick the right database that enables high scalability, parallel processing, long running queries, innovative indexes, and built-in analytical and aggregation services. SQL Server is an excellent and includes built-in analysis services and has incorporated SPSSfor statistics. Oracle is excellent and DB2 is strong. MySQL and PostreSQL would be limited in their ability to handle the volume of transactions and the aggregation and analysis. Columnar databases are very interesting and emerging for fast speed using a very interesting model to store and access the data. The business intelligence database should be implemented on its own server and avoid disrupting systems on other servers.
Plan for a lot of storage – The analytical or intelligence database will accumulate details over a long time frame, and grows by the number of facts and dimensions, aggregation, and continuing flow of information. Terabytes is within the realm and depends upon how much data you are getting and how often.
Define the sources – Identify where the source information is coming from, including internal IT systems, external systems, RSS feeds, web sites, etc.
Identify what information you want to apply intelligence – The concept has not changed over the years. Information is organized into facts (information that is measurable…increases, decreases; e.g. sales, work orders, claims, incidents, orders, invoices, etc.) and dimensions (“by geography, time, clients, products, etc.). Generally, a fact is constructed for each type of measurable data and is a very long record with all data and is de-normalized. And dimensions are organized into structures. For example, geography might be organized into country, region, sub-region, and state. Defining this structure is very important for maintaining the information storage.
Define the mapping and flow of data from source systems to the intelligence system – What is often called extraction, transformation, and loading (ETL) is essential to getting information into the intelligence. There are many good third party tools available that can make this effort much easier including Informatica.
Perform trial uploads – Once everything is setup, perform trial uploads to test and validate the processing of data from source systems, the transformation, and the loading into the business intelligence system.
Select the Reporting Tool – The reporting tool accesses the business intelligence system and enables ad-hoc and pre-defined queries and reports with drill-down to different levels of the fact-and dimensions. There are many tools available, including JasperSoft, Business Objects, Dundas, and Component Art. At times, additional development is required to achieve the level of flexibility required by the business.
Implement background pattern and data gathering algorithms – The intent here is to have background programs implemented that are continuously evaluating the aggregated data and surfacing patterns and relationships that are often missed by people due to the volume of information and changing. There are many different algorithms that can be applied here: apriori algorithm, naive bayes classifier, Boyer-Moore style algorithm, DNA pattern evaluation, etc.
Implement learning model –By combining the algorithmic results and storing the information, the system will maintain a learning structure to apply for future evaluation.
Implement the notification model – Targeted users will be notified of findings as they occur and present access to the user interface with the reporting details.
Automate the entire flow – Implement automation scripts to enable all processes to occur automatically.
The business can achieve an automated business intelligence system that is adaptive, learning, and continuously providing insight in actionable business intelligence and recommendations on how to proceed forward.