Table of contents
Abstract
2
Data Management Challenges
4
The Data Integration Layer
5
Choosing The Right Software Package
6
Advantages of a Data Integration Layer
7
Business Logic Separation
7
Rapid Development
7
Data Access
7
Data Quality
8
Enterprise Features
8
Considerations
9
A Real-life Example
10
Requirements and Architecture
10
Platform Independence
11
Business Logic
12
Communication Between Components
13
Web Interface
14
Performance14
Conclusion
15
3
Data Integration
[email protected]
www.cloveretl.com
Data Management Challenges
Today, every company and enterprise is compelled to manage huge amounts of data
that impacts business on a daily basis?—?data that comes from different systems and is
stored in a multitude of formats. Over time, not only do the data formats and volumes
grow, but the complexity of the business logic increases
dramatically. Here is where we begin to scope this major
As data volumes
issue in data management.
increase, management
of both the data and
business logic becomes
unmanageable.
Many data processing applications start as a simple
spreadsheet with embedded macros or formulas. This
might be adequate for small companies and their simple
business logic. However, as data volumes increase, this
approach quickly becomes expensive and hard to manage
due to significant overhead and manual work. Further, sometimes the business logic
or understanding itself is dependent on a tool, not the other way around. Surprisingly,
such practices are quite common, even in medium or large companies. This is often true
for companies that integrate many different customers with just slightly different data
formats (e.g. different banks or insurance companies who store very similar data). Using
unsuitable tools for data management increases the administrative effort required to
keep data correct and up-to-date.
Usually after the data management becomes unbearable, companies seek solutions with
better integration, higher automation, and easier maintenance and extensibility. A very
common approach is to build a custom application (often web-based) using a database
as data storage. Even though some companies do experience a measure of success, the
resulting applications are often hard to maintain and extend since they are unique and
represent a non-standard solution. Further, the business logic is often hard-coded, which
means increased costs; the processes will surely change over time and the applications
will require re-writes. What this demands is an in-house development team (or often,
the one person who knows the code) to protect the data architecture.
What we propose is a more flexible and elegant solution to the issue of managing
data architectures for the future?—?a specific way of designing such applications with
extensibility and standardization in mind. This is accomplished by adding an additional
layer to the architecture of the application?—?an independent data integration layer.
4
Data Integration
[email protected]
www.cloveretl.com
Please complete the form to gain access to this content
