I'm on a long and narrow road,
I walk all day, I walk all night,
I cannot tell what is my plight,
I walk all day, I walk all night.
Soon as I came into the world,
That moment I began my fight,
I'm in an inn with double gates,
I walk all day, I walk all night.
These two verses are by Asik Veysel, one of Turkey’s foremost poets and folklorists. Though blind for most of his life, he created many evocative poems and songs that still inspire people today. The two verses quoted above remind me of some of the difficulties and challenges we all face when we try to journey on the information highway. Do we walk alone on narrow roads between locked and isolated oases of information, or can we break free and efficiently use all the information we have amassed?
As soon as societies began to coalesce around a ruler, the control and management of information became increasingly important. Our information journey started with basic data entry scratched on baked clay tablets, then papyrus and paper. Amassing the data was one thing, but storing it and then being able to retrieve it efficiently, was another. Even as technology progressed and we were able to store ever-increasing amounts of information in an electronic format, the age-old problem was still apparent: how do we manage and maximise the potential of all that data?
From Data to Databases and Business Applications
Databases hold different types of information depending on the organization’s needs and for the purposes of this article, we will focus on business databases. Every business transaction, whether it be an inventory movement, an invoice or information about a customer, is recorded for later use.
An electronic database is designed to store and manage all the data collected by an organization in a structured way so that the information can be retrieved quickly and efficiently. As the amount of information increases, it becomes apparent that simple data storage will no longer be adequate - something much more complex and robust will be needed. Thus, the database management system (DBMS) was born, which manages the database’s contents and provides maintenance tasks.
There are several kinds of DBMS created by different technology vendors that range in sophistication, but all are designed to manage operational business data. While the DBMS improves the management of data, there is still a need for applications that will speed up and save the operational transactions upon which all the data is based. There is a strong connection between business applications and databases. While the main goal of these applications is to allow business users to save transactions in the fastest and the most accurate way, they should also provide basic planning and listing outputs.
Initially, business applications such as accounting, inventory management, MRPI, MRPII and ERP operated with propriety databases; however, over time they moved to open database management systems called relational databases. As business applications matured and data management became more complex, the market began to consolidate as large players swallowed smaller ones in the hope of offering customers a one-stop shop for all their data and business application needs. It is, however, important to note that though the market has been consolidating over the past few years, the idea of open inter-operability between applications and databases is still key.
Among the needs at the top of Maslow’s hierarchy is the quest for knowledge - and as database and related application technology progressed, it became apparent that this need was not being fulfilled. Business applications are operational systems that process business transactions and are not designed to provide in-depth analytical information. To further complicate the issue, information needs also vary from organization to organization. For example, to gain competitive advantage, companies need to understand and benefit from every single transaction that takes place in their business. Managers need to know who is buying what, when, why and where. They need to be able to track and monitor all the processes in their organization. In short, they need to have a clear view of their business and they need it in real time and in an easily understood format.
Business Intelligence Appears
The emergence of business intelligence (BI) was an important milestone in data management. For the first time, it became possible to work with an organization's entire body of data. Information was now ready to be transformed into knowledge that could be used in a meaningful and actionable way. Business intelligence is an umbrella term that covers many sub components, such as data warehousing, data marts, data mining, data quality, master data management (MDM) and controversial areas such as corporate performance management (CPM).
In fact, there is some controversy over the definition and scope of business intelligence with some industry experts differentiating between core business intelligence components and corporate performance management. Putting aside such recondite disputes, for the purposes of this article we can define business intelligence as software that enables a company to collect, access and analyze corporate data to aid in decision making.
Integral to the success of BI is the data warehouse, which is simply a repository for electronically storing an organization’s selected data and then cleaning, standardizing and cataloging it so users can analyze it and share it by means of easily understood reports.
With the advent of the data warehouse - and its smaller brother, the data mart - new databases appeared using column-oriented relational database systems, which provide high-volume compression with faster query response times, rather than the traditional row-based systems (which, though they can hold vast amounts of information, are slow and cumbersome). Other vendors prefer a "high-available" dedicated data warehouse database server bundled with specific hardware for maximum performance.