In-Place Upgrade Fix for 11.2.5 After Critical Patch Update

Credit for the following information goes to a commenter on this blog named Deezel.  And also, of course, to Oracle for figuring out the fix!

The issue:

In the Hyperion / Oracle EPM 11.2.x series of releases, Critical Patch Updates are available to fix significant security vulnerabilities in Oracle WebLogic Server 12c and Java 8.

Unfortunately, one of the WebLogic patches require an update to the OPatch utility, and the update cannot be rolled back successfully.  So if you're on EPM 11.2.2.0 through 11.2.4.0, you cannot redeploy your web applications after performing the in-place upgrade to 11.2.5.0.  You get a generic error in configTool and the back-end log error messages are unhelpful.

What follows are Oracle's instructions on how to work around the issue.  Again, a big thank you to commenter Deezel and I look forward to applying the fix in one of my bricked sandboxes.

I cannot emphasize enough the importance of following Oracle's advice by backing up the folders.  Best of luck!

ISSUE SUMMARY:
After Update from 11.2.4/11.2.3 to 11.2.5 deployment of products fail.
RECOVERY STEPS:
The steps listed here are to restore the Updated 11.2.5 instance with deployment failures to working state.
Back Up:
Take back up of EPMSystem11R1 and user_projects folder available under Middleware Home.
Steps:
1 Stop all services
2 Delete all the folders under MWH except EPMSystem11R1, dbclient32,dbclient64 and user_projects
3 Rename the .oracle.products available at EOH\.oracle.product as .oracle.products_11.2.5
4 Rename the .oracle.products0 as .oracle.products
5 Run Apply Update
6 Configure as per Update Guide
7 Start Services
8 Sanity test passes

What's Clobbering My Essbase Log?

Every now and then I need to educate a customer on how OPMN works.  Well, here's some knowledge I'm giving away for free.  (3 CEOs ago hated when I did this)

Here's the symptom of what's causing your on-premises Hyperion / Oracle EPM Essbase.log to keep growing 24x7, even when no end-users are logged in and no MaxL automation is running:

What's going on here?  Nobody is logged in but the log keeps getting updated once every 20 seconds or so.

This, dear reader, is OPMN.

Oracle Process Management Notification Service.

OPMN is responsible for managing active-passive Essbase clustering.  These "hits" you're seeing in the log is OPMN pinging Essbase to see if it is alive.

"But wait a moment, Dave!  I have just one Essbase server per environment and am definitely not doing Active-Passive Essbase clustering!!!  How do I turn these pings off???"

I'm glad you asked.  Let's investigate the OPMN config file, located here by default:

Make a backup copy of opmn.xml if you've never performed this exercise before!  OPMN is very particular about the XML file, and Essbase will fail to boot up if you commit a typo error in this file.

Now click Search->Find in your favorite text editor (cough cough Notepad++ cough cough) for the word "death".  It is present in the opmn.xml file exactly once:

I have comments to offer on the restart-on-death setting.  We'll get to that in a moment.  But for now, you want to insert a new line immediately after <restart retry="2" timeout="600"/>

Paste exactly this:

<ping timeout="20" interval="0"/>

Bounce the Essbase service (Windows) or use your stopEssbase.sh / startEssbase.sh scripts in Linux.  Done.  Your modified file should look like this:

The line you pasted disables the 20-second ping.  Obviously, if you ARE indeed running Essbase in an Active-Passive cluster configuration, then you need to IGNORE my advice in this case.  That 20-second ping is what allows OPMN to fail over Essbase to the Passive node, if your environment is correctly setup.

Now let's briefly touch upon the restart-on-death setting.

The setting defaults to "true", which I tend not to like.  Why?  Because I prefer to use MaxL to gracefully shutdown the Essbase Agent rather than stopping the service or running stopEssbase.sh.  Stopping the service brings down Essbase unconditionally. An ESSSVR process that's active - whether it is loading data, running a calc script, or restructuring - is now orphaned and cannot be restarted until the PID is killed.

How I bring down Essbase gracefully and send an alert to the system admins when it doesn't come down gracefully.... well, if you must know you may buy me off with a case of Guinness.

Happy Essbase'ing!

Additional Thoughts: In-Place Upgrade to 11.2.5

So far I've performed three in-place upgrades to Hyperion / Oracle EPM 11.2.5.  Each scenario was different.  One went very smoothly, one was a trainwreck until I figured out how to fix the damage, and the third can only be repaired by copying files from a working 11.2.5 system.

Scenario One: 11.2.4 (Patched) to 11.2.5

This scenario is beyond repair unless you wipe everything under \Oracle\Middleware, except for the sole exception of \Oracle\Middleware\user_projects\epmsystem1.  Then bring over the Middleware folder from a fresh 11.2.5 install (from a throwaway VM sandbox for example).

As previously discussed on this blog and others on the Internet, when you apply the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware, the WebLogic Server patch requires an update to the OPatch utility that cannot be rolled back at this time.  This botches the entire in-place upgrade to 11.2.5 because you will be unable to redeploy any of the WebLogic Managed Servers.

Scenario Two: 11.2.4 (Not Patched) to 11.2.5

This in-place upgrade went flawlessly.

Scenario Three:  11.1.2.3.7xx to 11.2.5

Seriously, don't attempt this even though Oracle's 11.2.5 README says this upgrade path is certified.  It is possible to repair it if you know the back-end Middleware folder structure extremely well.  But you will spend waaaay more time repairing the system than if you used the swing server approach instead:  Build a fresh 11.1.2.3 system in a VM, with its own database server, patch it up to 11.1.2.3.7xx to exactly match your PROD system, then perform an in-place upgrade to 11.1.2.4, and finally LCM your stuff out and import it into a fresh 11.2.5 system.

Why the in-place upgrade from 11.1.2.3/11.1.2.4 to 11.2.5 is fraught with danger:

Your Oracle HTTP Server ("OHS") folder structure now contains a mixture of OHS 11.1.1.7 and OHS 12c binaries.  OHS refuses to start and you end up spending a lot of time troubleshooting until you discover the error messages (if you run launch.exe from the command line so you may see the error messages) really mean you're deep in what many IT professions call "DLL Hell".

Your domains\EPMSystem\bin folder scripts still refer to Java 6, JRockit 6, or Java 7 depending upon the state of your 11.1.2.x system.  The WebLogic Managed Servers won't start.

Not trusting the upgrade, I blew away everything except for user_projects\epmsystem1, and then re-ran installTool and had it install 11.2.5 fresh.  I then ran configTool and told it to re-use all of my databases, so that the application and Shared Services content was preserved.  I then redeployed everything.  This approach worked and all services, save one, are functional.

I'm still struggling on what to do with the old Reporting & Analysis Framework content.   It did not get converted over to Document Repository and I don't see any back-end utilities that handle this.  Perhaps I'm just overlooking something.

Because of this issue, Document Repository isn't visible in the Shared Services Console.  We also get that odd error message about missing the default document when logging in to EPM Workspace.  If you've encountered this issue have have overcome it, please let me know in the comments!  This one will need to become an Oracle SR.

I could of course reconfigure Financial Reporting and tell it to drop & recreate the database, but then I lose all of my old RA Framework content.  I can't "just LCM it over from the old system" because the customer's RAF repository is too huge to LCM.  I've asked the customer for their assistance in cleaning up old PDFs from years past that aren't needed anymore.  I'd then have to break up the LCM into smaller chunks and pray that 11.2.5 recognizes the content.... I suspect some Notepad++ trickery will be needed to do recursive find-replace operations.

In-Place Upgrade to EPM 11.2.5: OPMN

I have much to say about the in-place upgrade to Hyperion / Oracle EPM 11.2.5.  Anybody who knows me well... knows that I pretty much despise Hyperion in-place upgrades, and moving to 11.2.5 in-place is no exception.  In fact, it may be one of the harder ones.

We've already talked on this blog previously about how applying the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware - OPatch for Weblogic in particular - currently bricks your in-place upgrade from either 11.2.3 or 11.2.4 to 11.2.5.... unless you backed up your Middleware folder prior to patching it.

But this post is about other factors that'll make you want to tear your hair out.

Let's begin with OPMN.

No matter which version of EPM you're upgrading from, if you're upgrading in-place you will need to edit your opmn.xml.  The in-place upgrade doesn't require you to reconfigure Essbase, and that is messy anyway and trying to reconfigure Essbase makes the system think you're trying to add to the Essbase Cluster or create a new one, rather than updating the one that already exists.

One thing I do like about 11.2.5 is the name of the JDK folder is now generic; Oracle finally relented and removed the reference to the version and update numbers in the folder's name.  It is now simply \Oracle\Middleware\jdk instead of jdk1.8.0_181 or jdk160_35.  But because of this, you'll need to do some surgery.

Edit opmn.xml, find the reference to the jdk folder as shown in my screenshot above, and wipe out the version and update numbers so it is exactly DRIVELETTER:\Oracle\Middleware\jdk\jre\bin\server

The 11.2.5 web version of the README indicates an in-place upgrade on top of either 11.1.2.3 or 11.1.2.4 directly to 11.2.5.0 is "certified".  Um... no.  Caesar says "thumbs down".

I've been working my way through an in-place update to 11.2.5 for a customer who lagged behind and is still on 11.1.2.3.7xx.  I really cannot recommend this approach, despite the README's claim that it is certified.  I finally got OHS, Essbase, Planning RMI, and the WebLogic managed servers online.  This upgrade path is fraught with multiple problems and should only be attempted by EPM Infrastructure experts.  

Look forward to a more detailed post on this 11.1.2.3 -> 11.2.5.0 in-place upgrade -- problems & solutions, after the dust settles.  I have one last problem to solve, and this is: why the heck did the upgrade not convert the RA Framework content over to Document Repository?  Massaging LCM files isn't the answer as their RA Framework repository is too massive for LCM to handle all in one shot.  Ugh!