Article written by Claudia Imhoff, Ph.D., Intelligent Solutions.

Summary

In today’s dynamic environment, organizations are faced with new and increasingly critical decisions which can affect their very survival. Decision makers, in turn, have placed more and more demands on their Business Intelligence (BI) environments. Their demands are increasing the pressure on IT to deliver the right data at the right time, and faster than ever before.

“The right data” means well-documented, reliable, and consistent data.

“The right time” means delivering the appropriate data in a timeframe appropriate for the particular decision. That is, delivery with enough lead time to allow the effect(s) of the decision(s) to make a difference.

“The appropriate data” means the combination of historical data found in data warehouses and data marts, low latency data found in an operational data store, and real time data from operational systems. Bringing the appropriate data together requires sophisticated data
integration infrastructures that go way beyond simple data combinations and displays. So, data integration requires a mixture of techniques and technologies – each specific to the type of decisionmaking being performed.

In this paper, the two most popular techniques for data integration (virtual data federation and physical data consolidation) are described, and the data integration technologies that support each technique (EII and ETL) are reviewed. Through eight BI examples, advice is offered to help implementers determine when to use each technique and technology.

A Strategic Architecture for BI

Today’s enterprises must be fueled by good decisions – about their customers, products, employees, partners, and more. Business Intelligence (BI) has long helped the company’s decision makers by supplying them with reliable versions of historical data – snapshots used to determine trends, patterns, comparisons over time, etc.

Strategic decisions are certainly critical to the ongoing well being of the organization, but are not the only type of decisions being made. Today’s BI architecture must also support a myriad of operational decisions, driven by today’s dynamic business environment.

The public domain, logical architecture that I advocate for BI is the Corporate Information Factory¹ (CIF). Over the years, the CIF has earned its stripes as the most reliable and most implemented architecture for sustainable BI.

Figure 1: The Corporate Information Factory

Several CIF components – the data warehouse, data marts, and operational data store – have remained fairly stable over the years. The data warehouse is still the main source of generic, integrated, historical snapshots of data. The operational data store is used to store low latency or near-real time integrated data. The specialized analytical data marts are dependent on the data warehouse or operational data store for their supply of data and contain the analytical applications. Each serves a specific purpose in handling planned and unplanned queries and reports.

Although these components have endured, the way we create them has changed dramatically. While there is no question of the value of historical snapshots, the time latency involved in the preparation of the data makes them unsuitable for intraday or operational decision
making. So, the CIF evolved to support not only traditional, strategic BI, but also tactical and operational BI.

Data Integration Techniques & Technologies

Today, BI implementers are fortunate to have two data integration techniques available to help them create world class environments – CIF or otherwise. These are Physical Data Consolidation and Virtual Data Federation:

• Physical Data Consolidation – This technique uses processes that capture, cleanse, integrate, transform, and load data into a target data store. Typically data is consolidated using extraction, transformation, and load (ETL) technologies, which obtain data
  from operational data sources, transform it to the corporate standard, and load it into physical data stores. Informatica’s PowerCenter is an example of an ETL product.
• Virtual Data Federation – This technique uses processes that provide a real-time integrated view of disparate data types from multiple sources, providing a universal data access layer. Data federation uses enterprise information integration (EII) technologies to create virtual stores of data from data warehouses, marts, operational data stores, and operational systems. Composite Software’s Information Server is an example of an EII product.

The biggest question for implementers is not whether to use these techniques, but when to use them. Each has its place within a data integration infrastructure but it is not always clear when to use one or the other. To help implementers make the “when” decision, eight examples are presented below, including the rationale, the benefits, and the drawbacks of choosing one over the other.