GoldenGate 12c Setup with Oracle 12c Pluggable Multi-Tenant DB and OEM GoldenGate Plugins

I recently built out a new installation of GoldenGate using the latest versions and features, namely the Integrated Capture and Replication capabilities for Oracle-to-Oracle Database.  It also used the latest Oracle Database 12c Multi-tentant Pluggable database on both sides, which introduces some differences in how it is setup and operates.  Lastly, I also setup the Oracle Enterprise Manager GoldenGate plugins to manage and monitor the entire DB 12c and GG platform (which was the least documented part of the process). 

Here are some of my notes and experiences - hope it helps.  For more info, here's a whitepaper describing some of the new Oracle GoldenGate 12c features and 12c DB optimizations:

http://www.oracle.com/us/products/middleware/data-integration/oracle-GoldenGate-features-wp-2030476.pdf

Setting Up Oracle Database 12c Pluggable Instances

I won't go into how to setup the new Oracle DB 12c Pluggable Databases, since there are lots of blogs and tutorials on the web.  After getting past the usual steps, I setup two pluggable databases (PDB1 and PDB2) on two different servers, and used the HR sample schemas as the basis for replication.  Here's a basic tutorial for DB12c setup:

http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/GoldenGate/12c/OGG12c_Creating_an_Oracle_Instance_for_OGG/index.html

Setting up GoldenGate 12.1.2.1

For the GoldenGate setup, I installed the latest 12.1.2.1 version that is downloadable from OTN and eDelivery.  The installs went smoothly on both servers,  Just make sure you have enough memory because the new GG Integrated uses Xstreams instead of checkpoint tables, and your streams_pool_size will need to be > 1GB (increasing overall SGA and Memory reqs). GG will also want a significant amount of additional memory for it's own cache.

I basically followed a new Oracle tutorial MOS whitepaper to setup the GG initial load, Extracts, and Replicats.  These instructions are good assuming you are somewhat familiar with GoldenGate.  Here's the MOS note and supplemental the PDF provides step-by-step commands:

https://support.oracle.com/epmos/faces/DocumentDisplay?id=1960719.1

Another option is this Oracle By Example (OBE) tutorial:

http://www.oracle.com/webfolder/technetwork/tutorials/obe/fmw/GoldenGate/12c/OGG12c_Integrated_Replicat/index.html

What’s New with GoldenGate 12c and Pluggable Databases?

If you are familiar with previous releases of GoldenGate with Oracle DB, this is where things are a little different when using DB12c and PDBs rather than the non-container DB with GoldenGate. 

Unique requirements include:

• Extracts operate at the root CDB level (CDB$ROOT) and must operate in the Integrated Capture Mode (see image above).  This makes sense since all the necessary PDB logging is done at the CDB level;
• On the Extract side, GG Owner is a CDB common user/schema (i.e. C##GGOWNER) and must connect to the CDB to interact with the LogMiner services;
• On the Replicat side, GG Owner is a local PDB user/schema (i.e. GGOWNER) and connects to the PDB (i.e. GGOWNER@PDB1).   I also created GGOWNER on the Extract-side PDB to be consistent for bi-directional replication;
• A single Extract supports pulling data from multiple PDBs, but a single Replicat will only support applying data to 1 PDB. So if pulling data from multiple PDBs, multiple Replicats will need to be created to apply all PDB data;
• As you can see in the tutorial, we don't need to create GG checkpoint tables or Data Pumps due to the new Integrated Replication features;
• There are a few new DBMS, procedures, grants, and init.ora parameters, and especially important is increasing the streams_pool_size to support Integrated Replication using streams.

Setting Up the Oracle Enterprise Manager 12c GoldenGate Plugins

Now came the tricky part of the project; setting up the OEM GoldenGate Plugin framework.  I followed the usual steps to get the OEM agents deployed to the two servers, downloaded the GG plugin and deployed to agents, and then tried to follow the instructions on discovering the GG targets.  All my version of OEM CC, Agents, Plugins, etc. are latest/greatest and here's the latest doc:

http://docs.oracle.com/cd/E24628_01/install.121/e27804/toc.htm

Yikes!  Houston, we have a problem.  I successfully setup the GG targets using the steps in the doc (tedious semi-manual discovery process but just follow the docs) but it was still very flaky (statuses wrong, missing stats, logging and configuration didn't work, etc.).  After two days of deleting and redeploying, debugging, looking through logfiles, and searching MOS notes, here’s the basic situation:

• There are two agents involved with this OEM-GG integration:  The standard OEM Agent and a GoldenGate Agent called JAGENT.  Some interactions with the OEM Management Server are directly with the JAGENT and some are through the OEM Agent;
• The embedded JAGENT in the GG 12.1.2.1 download is intended for the OGGMON Monitoring utility, and doesn't really support the OEM feature set;
• The JAGENT version that is needed for OEM is the standalone JAGENT 12.1.3 available via eDelivery (not included in GG downloads yet and MOS note doesn't reference it as an OEM issue);
• How to upgrade OGG 11g or 12c jagent to OGG Monitor Agent 12.1.3 (Doc ID 1908800.1)
  • Download OGG Monitor Server installer from eDeleivery​
    • Select a Product Pack: Oracle Fusion Middleware
    • Platform: Linux x86-64  (platform independent)
    • > Oracle Fusion Middleware 12c Media Pack 12.1.0.0.0 B69735-23
    • > Download button for -> Oracle Fusion Middleware 12c (12.1.3.0.0) GoldenGate Monitor and Veridata V44427-01 415M

• I also had to upgrade the Java JRE to 1.7 build 76, and use it as the default Java stack for everything including GG and OEM Agent
  There is a supplement to the MOS note at the following link that contains steps in Chapter 4 for upgrading and creating new JAGENT: http://docs.oracle.com/GoldenGate/1213/gg-monitor/GGAIN.pdf

• Finally, the datastore has to be deleted and recreated (dirbdb deleted is necessary) and all the settings in /cfg/Configproperties.xml are just right i.e. fully qualified host names, usernames, passwords).  See MOS Note 1487633.1 for some additional steps if discovery or target statuses start looking funky.

1.  Login to ggsci prompt and stop all processes

          stop *
          stop jagent
          stop manager

2.  Remove any initial load/replicats from ggsci prompt by running

info all tasks
delete replicat/extract [name]

3. Delete $GG_HOME/dirbdb directory.

4. Delete jagent.log, ogg_agent.log,  and gsserr.log in $GG_HOME.

5. Login to ggsci prompt and run ‘create datastore’ command

6. Start all proceses, run

          start manager
          start *
          start jagent

Randy Hardee, Vice President-Technology