Managing an E-Business Suite environment often requires cloning the production environment over to Non-production environments for the purposes of training, custom development, new module implementations, patching and more.
The cloning process has traditionally been a manual process performed by DBAs. This process involves copying over the database files – preferably using RMAN DB duplicate, the tech stack, and the application tier to the non-production servers, converting them to DEV or TEST, and then performing a number of post clone steps such as disabling workflow emails from going to end users, scrambling financial information, updating the clone date and the instance name within the apps heading, editing the custom top, changing passwords, and others. Depending upon the size and configuration the cloning process can take from 12 hours to 48 hours on average.
The number of post clone steps can get to be quite numerous and even with the best of documented procedures; one or two steps can easily be missed.
For example – the following is a list of steps in one particular cloning environment:
1. Once the cloned database is renamed, add all the temp datafiles to the TEMP tablespace.
2. All key passwords need to be updated from their production values to the appropriate values for the DEV environment. (ie: sys,system, sysadmin,apps)
3. Custom database directories need to be re-created with the correct paths for the DEV server.
4. Specific Profile options need to be re-defined – like the apps heading naming the instance and the date stamp of the clone.
5. End-dating specific users so that only the desired developers have access to the instance.
6. Removing/scrambling specific private HR data.
7. Nulling out specific email addresses to prevent workflow from emailing end-users from the DEV instance.
8. Putting a hold on all regularly scheduled production concurrent requests in the DEV instance.
9. Updating all profile option values, swapping out the prod instance name for the DEV instance name. Additional updates for the server names also run.
10. Updates for all custom environment code paths and settings.
11. Updates for any and all printer setting changes.
The above list contains a sampling of the potential post clone steps required to configure a DEV or TEST instance from the production environment. Some environments require less and some more. But if one or more of the above steps were accidentally missed, the end result is a cause of frustration and extra work for the end user(s) of the clone and the support staff.
The TriCore DBA team endeavors to look for opportunities to make the DBA operations more streamlined and error free. Wherever possible, we introduce automation scripting that has the combined benefit of reducing keyboard time for the managing DBA, reducing or eliminating the chance for human error, and producing repeatable successful clones for our customers. This allows us to deliver clones, on average, in 1/2 the normal time it takes to perform a manual clone, and with repeatable, error free delivery.
We do this with a combination of Linux Shell scripting, the use of SSH (Secure Shell) keys for cross node workflow, and the use of modular, portable code that allows us to rapidly implement clone automation. The use of SSH keys allows for scripts on remote servers to be run from a base server. E-Business Suite systems are often found on multiple tier – multiple server platforms, with the database on one server, and the applications tier on another server. Therefore, in a cloning process, a DBA can be dealing with at least 4 separate servers – prod db, prod apps, dev db, dev apps. The main auto clone script gets started from one primary server – in our case, the dev db tier. However, scripts and commands need to be run against all these servers. This is where the use of SSH keys for automated login from the scripts comes into play. The tail end of the script contains email messages sent to the DBA distribution list describing the successful completion of the clone.
The above approach enables the Tricore DBA team to analyze the environments being supported, apply automation scripting to reduce the delivery time of the clone, reduce the keyboard time of the managing DBA and produce repeatable error-free clones to happy customers.