Posted on May 7, 2019 by Dave Partridge
Lift and shift of mission-critical applications from on-premise data centers to Oracle Cloud Infrastructure “OCI” is a major undertaking. A key success factor for migrating these important applications is to minimize disruption during the transition from the on-premise data center to the Cloud to the point that its barely noticeable to consumers of IT services. If an outage window extends too long, impacts can be in the form of financial risk, reschedule of the cutover, or tangentially the perception of the credibility of the project team.
Providing application-level continuity requires that databases supporting the applications be operational and mirror the on-premise databases at transition time, or very soon thereafter. This implies that database updates that were made 1 second, 1 minute or even 1 year ago, are reflected in the Cloud database at the time of go-live.
If you’ve been tasked with migrating your organization’s critical applications and databases to the Cloud, read on to explore how the Oracle Data Integration Platform Cloud Service “DIPC” can provide support for achieving a successful outcome. The goal of this article is to help you decide if DIPC is appropriate for your Cloud migration. To aid in that determination, we'll discuss the challenge of migrating databases to the Cloud in a timely manner and contrast DIPC against traditional database migration methods. A review of the DIPC architecture will be provided as well. Let's start by exploring the factors that make database migration logistically challenging.
Database sizes continue to grow exponentially. It is now common for an organization to have a database footprint that is multiple terabytes in size. It's no longer just Data Warehouses that are large consumers of storage. Online transaction processing "OLTP" databases are commonly joining the terabyte club. Databases of this size present a big obstacle to a quick migration due to the time required to physically move this volume of data. The reason it takes so long is because migrating a database requires three steps.
Each step is constrained by the physical limitations of the underlying hardware. For example, the unloading speed is dependent on storage controllers and disk subsystems which are often older with limited capabilities compared to newer infrastructure. Network transfer rates between data centers are subject to the underlying transport constraints and daily variations. Many databases are trapped in older data centers with 100 MB/sec network bandwidth connectivity. Adding the time together for these three activities, it is easy to see that the end to end time for migrating a 1 TB database can easily exceed 24 hours!
Differences between the on-premise and Cloud database versions and heterogeneous operating system platforms add another layer of migration complexity. These differences can limit the available migration methods. To be specific, Oracle OCI database runs on the Oracle Enterprise Linux "OEL" operating system. Multiple versions of the database are offered from 11g on up to 19c, with 188.8.131.52 being the minimum version level. Let explore how the go-live availability and database modernization requirements shape the database migration approach next.
To develop a database migration approach and plan, one must consider these factors:
The above criteria will be used to decide the best possible migration method(s). At a high level, all methods fit into either a physical or a logical category. A physical move is completed using the backup and recovery RMAN utility. Logical migration utilizes Data Pump, or the classic import and export utilities. It is not the focus of this blog to detail all of the variations of RMAN and Data Pump as migration tools, but to point out that they are the utilities that will be used and that there is either a physical or logical migration path. If the reader is interested in exploring that detail, additional information can be found in the OCI documentation, accessible online at: https://docs.cloud.oracle.com/iaas/Content/Database/Tasks/migrating.htm.
What's important in pointing out these two categories is the fact that a logical migration is possible with any database whereas a physical migration is only possible for a database pair with similar architecture, version, and operating system platform. For example, a source database running at 184.108.40.206 on a Linux or Windows operating system can be physically migrated to an 220.127.116.11 database instance on OCI. However, a 12.2 database running on AIX 7 in the on-premise data center must be migrated using a logical method due to the operating system difference even if the target version is 12.2. Considering the fact that many source databases may change the operating system, and/or version, the chances are low that a physical migration is possible. This is especially true for database modernization cases where migration from 11.2 to 12c or higher is desired, or if non-CDB databases will become pluggable databases (PDB).
The inclination for those who are familiar with Oracle database migrations is to consider Data Guard as the preferred method for synchronizing on-premise databases to the Cloud to meet a database availability goal. The most widespread use of Data Guard is the physical standby, making it a member of the physical migration category. However, we have just pointed out that the number of databases that are eligible for this option will be limited. There is a Data Guard logical standby option, however, it is not widely used and is subject to data type limitations, making its viability as a solution quite limited.
For database migration, most databases will require a logical migration. Since the logical Data Guard has limited applicability, on-premise to Cloud database synchronization turns out to be more challenging than might be expected. This is where the Oracle Data Integration Platform becomes an excellent option. Even for databases that do qualify for a physical migration, DIPC can become the option of choice.
DIPC is an Oracle Cloud PaaS service. Included in DIPC is the Oracle GoldenGate software package. GoldenGate is a comprehensive replication solution providing real-time data integration and change data capture. It utilizes a capture, forward and apply mechanism to replicate database transactions from source to target. Conceptually, it is similar to Data Guard in that it mines committed database transactions from the Oracle redo and archive logs. However, it differs in that it doesn’t replicate block changes. Rather, it transforms database transaction information into before and after row images which are stored in an operating system queue file, called a trail file. Trail file information is then assembled by GoldenGate into SQL statements that are applied against the target database to synchronize it with the source database. Prior to enabling GoldenGate replication, the target database must be instantiated as a copy of the source database. This replica, or copy, is created by means of a point in time full copy of the source database to the target. Both RMAN and Data Pump can be used to create the replica copy. The heavy lifting of the replica creation happens well in advance of the Cloud cut-over. This relieves the time pressure associated with attempting to fit this into the cut-over window. After the database instantiation, GoldenGate capture, forward and apply replication is enabled, and transactions flow from on-premise to the Cloud.
Since GoldenGate performs logical replication, it is not subject to the database version, architecture and operating system constraints associated with a physical migration method. GoldenGate provides heterogeneous replication services, across different database versions, operating system platforms, and even across different database vendors. A further benefit is that GoldenGate is a low impact solution. Source side transaction mining is performed by a log reader process which uses a client database connection which is nearly imperceptible in its operation, reducing concerns of performance impact to the source database.
At the time of switch over to OCI from the on-premise data center, the state of the replication is interrogated to determine any lag time for remaining transactions to be flushed to the target environment. When GoldenGate reports zero lag time, that means synchronization is complete. Depending on the transaction volume, the quiescence time may vary, but it is often a matter of minutes - even in high volume scenarios. When the synchronization is complete, applications are configured to utilize the target database. Since this final sync time is typically very brief, the goal of nearly immediate IT service availability is achieved.
The Oracle DIPC Cloud provides a suite of data integration services including ETL, bulk data movement, real-time replication plus data governance and data quality. The DIPC is provisioned through the OCI console and handles the Cloud billing on a GB/hour basis. The GoldenGate software package is the product, within DIPC, that delivers the real-time integration and database replication services. The GoldenGate package is manually deployed to an OCI VM, and a registration process to the DIPC service is executed for metering. It performs database synchronization using replication processes. Generally, these processes are configured to run on the GoldenGate VM with no need to deploy any GoldenGate software on either the on-premise or OCI database servers. However, it is flexible in its deployment of the capture, forward and apply processes, allowing for them to be deployed on the source or target database server to provide support for legacy Oracle versions or Standard Edition databases.
The following diagram depicts the DIPC GoldenGate architecture in a preferred deployment model with all of the processes deployed on the OCI VM.
In the DIPC GoldenGate architecture, there are four distinct process components: Capture, Forward, Apply and Manage. These are marked in the red circled numbers in the above architecture diagram and described in more detail, next.
1 - Capture
The GoldenGate “extract” process is responsible for capturing committed transactions from Oracle redo and archive logs and writing them to the source trail file. The extract process accomplishes this by establishing and maintaining a connection to the on-premise database mining through a logreader mechanism. The before and after row images are produced and written to the trail file.
There is flexibility in the architecture. It is possible to run the extract process either local to the source server, or directly on the GoldenGate Cloud Server, allowing for a push or pull of the transactions between the database and the GoldenGate hosts.
Instantiation of the target database can alternatively be accomplished by utilizing an "INITIAL LOAD" function which is implemented by temporary extract and replicat processes which are deleted after the instantiation is completed.
2 - Forward
The GoldenGate “pump” is an optional process that is responsible for forwarding data from a source trail file to a remote destination trail file. The process is optional and required only in situations where the extract process has been deployed on the source database server. In the event the extract process is hosted on the GoldenGate server, there is no need for a pump; the source and target trails can be one and the same.
3 - Apply
The “replicat” process is responsible for applying changes to the target database by reading the before and after row image data from the trail file. Using these images, replicat constructs SQL statements and applies them to the target database through a database client connection that GoldenGate establishes and maintains.
4 - Manage
The GoldenGate manager process performs a multitude of functions including management of the replication processes, trail file space management, and reporting on replication throughput and lag times.
The DIPC Cloud service in the lower right of the diagram is a serverless PaaS service that resides within the tenancy, separate from the GoldenGate VM server.
In this blog, we explored the challenge of maximizing database availability when migrating a large database portfolio. It is a labor-intensive, time-consuming process that can impact business operations during application migrations to Cloud.
Organizations that plan to migrate to the Cloud face a significant logistical challenge in migrating large volumes of data to the cloud and being able to complete the cut-over on schedule. There are many related challenges that hamper IT from delivering on business goals using traditional database migration solutions.
Oracle Data Integration Platform provides an OCI service-based approach to facilitate the heavy lifting well in advance, and keep Cloud databases up to date via a lightweight synchronization mechanism. It removes physical constraints providing Oracle version and operating system platform independence. It is certainly not a requirement for any Cloud migration effort, but it should be in the toolbox for demanding database migrations.
Mythics has helped private corporations and public sector organizations successfully execute cloud migrations, simultaneously reducing downtime and risk. We’re happy to talk to you about a winning cloud migration strategy that includes planning & design, implementation approaches, and options for database migration methods – of course, DIPC is an option. Contact us or email email@example.com to discuss your specific requirements.
Dave Partridge, Cloud Solution Consultant, Mythics, Inc.