Thought Leadership
In the rapidly changing IT landscape, companies need flexible data integration tools. The integration of cloud and on-premises systems in real time is becoming increasingly important. Five key functions ensure a successful, future-proof data strategy.
A ‘typical IT landscape’ is fast becoming a thing of the past in today’s businesses. It’s an ever-changing world where systems are routinely enhanced with commerce platforms, external data sources and edge applications designed to improve the customer experience.
Most organizations today operate with a mix of cloud, hybrid cloud and on-premises solutions. Many continue to run their most business-critical operations on mainframe and IBM i computers, which are respected and valued for their high security and scalability, but are not designed for integration with modern distributed systems.
Whichever approach is taken, enterprise data integration has become more strategic. The sudden popularity of cloud data platforms such as Databricks, Snowflake, Amazon Redshift, Amazon RDS, Confluent Cloud and Azure Synapse has increased the need for powerful data integration tools that can transfer large amounts of information from transactional applications to the cloud reliably, at scale and in real time.
The days when ETL (extract, transform, load) batch jobs were executed overnight are over. This does not deliver the immediate results that today’s companies need. And perhaps more importantly, it lacks the flexibility needed to adapt to the demands that arise from the constant evolution of the company’s IT landscape and customer requirements.
The new paradigm is dynamic and fluid. It is important to provide business users with data quickly, wherever they need it. To work effectively in this new environment, data integration solutions need to be future-proofed.
Below are five key features that ensure the data integration solution can keep pace in a constantly evolving environment.
1. adding new sources and destinations
Just as the value of a network depends on the number of nodes in the network, the value of an integration solution is a function of its ability to quickly and easily add new data sources and destinations.
Companies today are constantly on the lookout for innovation. This can be something as simple as adding a new application to the IT landscape or something as complex as acquiring a competitor and integrating their information systems into the company. In either case, some sort of data integration is usually required. To make matters worse, the resulting integration solutions are likely to change over time.
In the old world, every major change in the IT landscape could require a separate project involving custom code, extensive quality assurance testing and a go-live cutover at off-peak times.
Now times have changed – and rapidly. This approach is no longer viable. The integration landscape must be able to quickly and easily expand to include data sources and targets without having to launch a major new IT initiative.
The best enterprise data integration tools are able to connect to a number of different sources and destinations, including mainframe and IBM i systems. i data is inherently challenging because it does not conform to the standard used by most modern distributed computing systems.
2. real-time integration with change data capture
(Change Data Capture CDC)
As already mentioned, the old practice of batch integration is no longer sufficient. Today’s companies need real-time or near-real-time performance, depending on the application. Timing is important.
Take, for example, the challenge that credit card companies face in detecting fraudulent transactions. Artificial intelligence (AI) algorithms are trained to detect anomalies. These range from recognizing transactions originating from a country known to be a common source of fraud, to cross-referencing transaction amounts and locations to look for common patterns that could be suspicious.
However, it is of little use to recognize a fraudulent transaction if it has already been processed. This is where real-time integration makes the difference. The same applies to systems that monitor trading venues, ATM transactions, telecommunications network outages, suspicious network traffic and much more. Real-time data visibility is a competitive advantage.
Perhaps more importantly, the opposite is true: when information is delayed, it brings less value to the organization. In today’s world, real-time integration is no longer optional.
3. easy deployment in new environments
The best data integration tools enable simple and fast deployment without the need for specialized knowledge. Provisioning should be resource efficient and easily adaptable to your use cases.
This includes the ability to:
- Deploy integration solution in new environments without the need to redevelop or redesign streaming data pipelines
- The company is protected from interruptions that could occur during the further development of the IT landscape
- Make changes to data sources and targets – without coding, reconciliation or redevelopment.
The relocation of applications to the cloud should also be possible without major interruptions to the integration concept.
4. bulletproof reliability and scale
Not all integration software is the same. Low-cost solutions may work well for low transaction volumes, but can become bottlenecks as volumes increase.
That’s why it’s so important to look for software that can scale effectively as the needs of the business grow. The data integration tools should take this into account:
- Growing data volumes
- a growing number of users
- Strong consumption peaks at times of peak demand
When looking for the right data integration software, it’s important to ask vendors how they offer reliable, predictable performance and scalability.
Reliability also means being able to react quickly in the event of an error. Even a brief network outage can result in some data not being delivered to its intended destination. The best enterprise data integration tools have built-in resilience with guaranteed delivery and data integrity. This means that every record is delivered as intended, with no duplicates for omitted transactions.
5. integrated data catalog to support metadata
When building the IT ecosystem, it is important to use tools that have the capabilities to support future-oriented use cases. One notable capability that achieves this is the data catalog. A data catalog is a core component of data governance as it provides a centralized knowledge base for users across the enterprise.
A well-integrated data catalog supports the comprehensive discovery, access, use and sharing of technical and business metadata, automating and improving your data integration and operational tasks. It summarizes all metadata related to an organization’s data assets and organizes the information in a simple, easy-to-understand format.
Additional data governance capabilities that organizations should look for include those that support governance requirements – such as data quality, data lineage and policy enforcement – while processing data for specific use cases, including master data management.
Conclusion
In order to remain competitive, a company must always keep an eye on technological developments, IT processes and IT ecosystem developments. Correct and secure data integration is an important factor that must always be considered in order to identify and avert potential risks in good time.
