EPM 11.2 Distributed Environment - Yup, Another RCU Post!

My blog entry EPM 11.2 RCU and you apparently struck a nerve, so consider this a follow-up!

Before we get to the "do this, do that" part, let's examine how we can determine if our attempt to deploy Hyperion / Oracle EPM 11.2 web services across multiple machines is... well, botched.

Part 1 - Is my distributed deployment botched?

Let's assume you've followed the instructions within the post linked above, and you have successfully created your WebLogic domain and have deployed a few things on machine #1 (EPM Foundation, Oracle HTTP Server, and perhaps a few supporting modules such as APS, EAS, CalcMgr, etc).  Now you're in the EPM System Configurator on machine #2 and are getting the dreaded red X when you try the Deploy To Application Server task.

user_projects\EPMINSTANCE\diagnostics\logs\configtool.log on machine #2 offers nothing beyond the vague and dreaded InvocationTargetException Java stack trace error.

By the way, ERROR messages prior to this point may typically be ignored.  You will first see "ERROR: Jars manifest check failed with message "Some referenced jars do not exist" and then a little further into the log you'll see "env.isWindows is not a function".  Don't waste your time troubleshooting those, as they're unrelated to your issue.

Once you see InvocationTargetException, close out of the log and now open wlst_debug.log in the same directory.  This points us to the underlying issue!

Do you see "WLSTException: No element JDBCSystemResource was found" at the top of a lengthy Java stack trace error?  If so, look just a few lines above.  The message doesn't look like an error at first glance:

2020-02-08 13:38:53,392 INFO  [1] com.oracle.cie.domain.script.ScriptExecutor - find JDBCSystemResource!JdbcResource!JDBCDriverParams "opss-audit-DBDS!opss-audit-DBDS!NO_NAME_0" as obj5

How could this be bad?  It is an [INFO] message after all, not an error.

It turns out there's an OPSS schema that's created by the RCU utility described within my earlier post.  The WebLogic Scripting Tool (WLST) can't find the OPSS jdbc definition files on machine #2 because they don't exist.  More than likely, this folder on your machine #2 is empty.

If you find yourself nodding your head at this point, yes.... your distributed deployment is botched.  But it can quickly be rescued!

(Note: Don't be tempted to copy those files over from machine #1.  They contain references to the RCU prefix and I haven't tested what might happen if you do that -- I suspect it would be something bad?)

Part 2 - How to fix machine #2 so we can deploy web services there?

Follow these steps:

1. Login to your WebLogic Admin Console on machine #1 (http://machine1:7001/console) as epm_admin and click "Release Configuration" if you see that button is available.
2. On machine #2, blow away user_projects\domains\EPMSystem -- all files & subfolders.
3. On machine #2, launch RCU (Middleware\oracle_common\bin\rcu.bat)
4. Use the "create" option and choose a new prefix.  Pick something sensible like the EPMINSTANCE name (assuming you aren't deploying as "epmsystem1" everywhere) or machine #2's hostname.
5. Pick all checkboxes except for the Oracle Data Integrator ones.
6. Pick the option to use the same password for all schemas.  This way is much easier.
7. Let 'er rip and close the RCU utility when done.

Last steps...

I've read in the comments on the RCU blog entry that Oracle Support is telling some people to use different "schemas" for each machine's RCU database. 

When in doubt, go with Oracle's suggested approach of 1 Oracle Schema or 1 MS SQL Database per RCU Prefix (machine), just in case you run into trouble later and need to open an SR.  This means more things for your DBA to back up on a regular basis, but it may cause less confusion for Oracle Support as you work through the SR together.

Let's take a look at the back-end database.  In this example I'll use MS SQL Server 2016.

I have more machines deployed than are visible within this screenshot, but hopefully the concept comes across!

Each EPMINSTANCE ("machine" if you are not scaling vertically) has its own RCU "prefix" assigned.  In this example, "DEV" and "LANDO".  These schema users were created by the RCU utility.

I'm using same SQL database for these and so far things are working OK.

Should you want to cluster out even further for High Availability etc., you might consider either using Oracle's suggested approach (1 database/schema for each EPMINSTANCE), or at least one per each cluster....

... e.g, EPMRCU0 has the RCU prefixes for Foundation0, CalcMgr0, EAS0, etc.  Then EPMRCU1 could have the *1 servers, EPMRCU2 could have the *2 servers, and so on.


Getting back to the task at hand, copy RCUSchema.properties from machine #1's EPMSystem11R1\common\config\11.1.2.0 folder to the same location on #2.  Edit it on machine #2 and change the "schemaPrefix=" line to the prefix you created for machine #2.  You shouldn't need to edit the two SysDBA*= lines.  If you aren't using the same SQL database / Oracle schema, you can wipe out the RcuSchemaPassword= value and add the plain text password (the password will be automatically encrypted if machine #2 becomes un-botched) and also DbURL= as appropriate.

Finally, go back into the configtool and re-attempt your deployment.  So that you don't have to re-read through old log entries, you might rename EPMINSTANCE\diagnostics\logs\config to some other name before you do this.

In Closing

Yes, I did botch one of my machines on purpose... I wanted to see what would happen if I skipped the RCU step for a machine where I did NOT intend to deploy EPM Foundation.  Why did I do this?  (Aside from enjoying pain)

Hop out to the EPM 11.2 documentation landing page on Oracle's site, and review:

Deployment Options Guide
    Section 2.1 Clustering Java Web Applications
        "When scaling Oracle Hyperion Foundation Services (underline emphasis mine), 
        you must create a new schema using RCU and edit RCUSchema.properties on each
        machine in the deployment."

This specific underlined sentence fragment led me to believe RCU was only need on machines where Foundation would be deployed.  Silly me for taking the documentation seriously!

Another related point.  See:

Installation and Configuration Guide
    Section 6.2 Creating Infrastructure Schemas Using Repository Creation Utility
        "You must run RCU on each machine in your environment."

I'd add one caveat here; RCU is only needed on machines where you want to deploy WebLogic Managed Servers (Planning Web, HFM Web, CalcMgr, etc).  Essbase, Essbase Studio, DRM, HFM Java Server, and other non-WebLogic processes don't need it because they are independent from WebLogic.  So the general rule of thumb here is: if EPM System Configurator creates a user_projects\domains\EPMSystem folder on the machine, that machine needs RCU.

Time to enjoy the rest of my weekend.  Be safe out there, and don't ask me about Oracle SOA!