Sunday, June 13, 2021

Additional Thoughts: In-Place Upgrade to 11.2.5

So far I've performed three in-place upgrades to Hyperion / Oracle EPM 11.2.5.  Each scenario was different.  One went very smoothly, one was a trainwreck until I figured out how to fix the damage, and the third can only be repaired by copying files from a working 11.2.5 system.

Scenario One: 11.2.4 (Patched) to 11.2.5

This scenario is beyond repair unless you wipe everything under \Oracle\Middleware, except for the sole exception of \Oracle\Middleware\user_projects\epmsystem1.  Then bring over the Middleware folder from a fresh 11.2.5 install (from a throwaway VM sandbox for example).

As previously discussed on this blog and others on the Internet, when you apply the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware, the WebLogic Server patch requires an update to the OPatch utility that cannot be rolled back at this time.  This botches the entire in-place upgrade to 11.2.5 because you will be unable to redeploy any of the WebLogic Managed Servers.

Scenario Two: 11.2.4 (Not Patched) to 11.2.5

This in-place upgrade went flawlessly.

Scenario Three:  11.1.2.3.7xx to 11.2.5

Seriously, don't attempt this even though Oracle's 11.2.5 README says this upgrade path is certified.  It is possible to repair it if you know the back-end Middleware folder structure extremely well.  But you will spend waaaay more time repairing the system than if you used the swing server approach instead:  Build a fresh 11.1.2.3 system in a VM, with its own database server, patch it up to 11.1.2.3.7xx to exactly match your PROD system, then perform an in-place upgrade to 11.1.2.4, and finally LCM your stuff out and import it into a fresh 11.2.5 system.

Why the in-place upgrade from 11.1.2.3/11.1.2.4 to 11.2.5 is fraught with danger:

Your Oracle HTTP Server ("OHS") folder structure now contains a mixture of OHS 11.1.1.7 and OHS 12c binaries.  OHS refuses to start and you end up spending a lot of time troubleshooting until you discover the error messages (if you run launch.exe from the command line so you may see the error messages) really mean you're deep in what many IT professions call "DLL Hell".

Your domains\EPMSystem\bin folder scripts still refer to Java 6, JRockit 6, or Java 7 depending upon the state of your 11.1.2.x system.  The WebLogic Managed Servers won't start.

Not trusting the upgrade, I blew away everything except for user_projects\epmsystem1, and then re-ran installTool and had it install 11.2.5 fresh.  I then ran configTool and told it to re-use all of my databases, so that the application and Shared Services content was preserved.  I then redeployed everything.  This approach worked and all services, save one, are functional.

I'm still struggling on what to do with the old Reporting & Analysis Framework content.   It did not get converted over to Document Repository and I don't see any back-end utilities that handle this.  Perhaps I'm just overlooking something.

Because of this issue, Document Repository isn't visible in the Shared Services Console.  We also get that odd error message about missing the default document when logging in to EPM Workspace.  If you've encountered this issue have have overcome it, please let me know in the comments!  This one will need to become an Oracle SR.

I could of course reconfigure Financial Reporting and tell it to drop & recreate the database, but then I lose all of my old RA Framework content.  I can't "just LCM it over from the old system" because the customer's RAF repository is too huge to LCM.  I've asked the customer for their assistance in cleaning up old PDFs from years past that aren't needed anymore.  I'd then have to break up the LCM into smaller chunks and pray that 11.2.5 recognizes the content.... I suspect some Notepad++ trickery will be needed to do recursive find-replace operations.

Friday, June 4, 2021

In-Place Upgrade to EPM 11.2.5: OPMN

I have much to say about the in-place upgrade to Hyperion / Oracle EPM 11.2.5.  Anybody who knows me well... knows that I pretty much despise Hyperion in-place upgrades, and moving to 11.2.5 in-place is no exception.  In fact, it may be one of the harder ones.

We've already talked on this blog previously about how applying the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware - OPatch for Weblogic in particular - currently bricks your in-place upgrade from either 11.2.3 or 11.2.4 to 11.2.5.... unless you backed up your Middleware folder prior to patching it.

But this post is about other factors that'll make you want to tear your hair out.

Let's begin with OPMN.

No matter which version of EPM you're upgrading from, if you're upgrading in-place you will need to edit your opmn.xml.  The in-place upgrade doesn't require you to reconfigure Essbase, and that is messy anyway and trying to reconfigure Essbase makes the system think you're trying to add to the Essbase Cluster or create a new one, rather than updating the one that already exists.

One thing I do like about 11.2.5 is the name of the JDK folder is now generic; Oracle finally relented and removed the reference to the version and update numbers in the folder's name.  It is now simply \Oracle\Middleware\jdk instead of jdk1.8.0_181 or jdk160_35.  But because of this, you'll need to do some surgery.

Edit opmn.xml, find the reference to the jdk folder as shown in my screenshot above, and wipe out the version and update numbers so it is exactly DRIVELETTER:\Oracle\Middleware\jdk\jre\bin\server

The 11.2.5 web version of the README indicates an in-place upgrade on top of either 11.1.2.3 or 11.1.2.4 directly to 11.2.5.0 is "certified".  Um... no.  Caesar says "thumbs down".


I've been working my way through an in-place update to 11.2.5 for a customer who lagged behind and is still on 11.1.2.3.7xx.  I really cannot recommend this approach, despite the README's claim that it is certified.  I finally got OHS, Essbase, Planning RMI, and the WebLogic managed servers online.  This upgrade path is fraught with multiple problems and should only be attempted by EPM Infrastructure experts.  

Look forward to a more detailed post on this 11.1.2.3 -> 11.2.5.0 in-place upgrade -- problems & solutions, after the dust settles.  I have one last problem to solve, and this is: why the heck did the upgrade not convert the RA Framework content over to Document Repository?  Massaging LCM files isn't the answer as their RA Framework repository is too massive for LCM to handle all in one shot.  Ugh!


Tuesday, May 18, 2021

Oracle WebLogic Admin Server - as a Windows service made EASY


"BeaService", how did I beat you?


If you have never heard about or read about NSSM before, READ ON!

When I record installation sessions for Hyperion / Oracle EPM, I tend to pause the recording when it is time to deploy NSSM to get Oracle WebLogic Admin Server setup as a Windows service.

Because NSSM is an acronym that is not entirely nice.  I won't repeat it here.  But you NEED this tool if you don't have it!  Grab it here:  NSSM download

NSSM lets you create a Windows service for any executable .bat or .exe.  It does not work for OHS 12c and I have a separate post about that topic.  But it works wonderfully for the Oracle WebLogic Admin Server ("WLS") in either 10.x or 12.x.

Simply download NSSM and unzip it in a directory of your choice.  That directory should not be deleted later.  I save my copy in D:\Scripts\NSSM on my primary EPM Foundation Server where the master version of WLS resides.

The beauty of this utility is:

  • You can customize the name of the service, as in my first screenshot above.
  • You can make it rotate the log files.
  • You can force it to make the stdout and stderr logs reside in the same folder as your EPM service stdout/stderr logs!

I won't post the spoiler screenshots or else you won't hire me.  😈  But maybe I gave you a few helpful hints here?

Sunday, May 16, 2021

Patch 30695700 - Hyperion EPM Architect to DRM Conversion Utility

This post is going to be more painful than the last one I just wrote.

Suppose you are on Hyperion / Oracle EPM 11.1.2.4 or prior, and your metadata is under EPMA's control.  Further suppose you have many applications and converting them all to Classic Mode doesn't suit your business requirements.

You essentially have 2 alternatives to Classic Mode in EPM 11.2.x due to the elimination of EPMA.

1. Switch to the Oracle DRM Limited Use license.
2. Contact Abhi at EPMWare.com.

That's pretty much it.  If you, my dear reader, have a 3rd option I have not considered for on-premises EPM, please by all means share your solution in the comments!  With upgrades heating up in Fiscal 2021, this topic is going to come up over and over!

What the above out of the way, I'm going to focus specifically on option #1.

There is a patch numbered 30695700.  It is associated with DRM 11.1.2.4 (don't ask me why, as it really applies to an EPM 11.2.x implementation).

This is not a true "Opatch" opatch.  It is a standalone Java utility, and the Java source code is included.

Halt.  Stop right there.  Do not pass Go.  Do not collect $200 USD.

Depending upon which version of EPMA you're coming from, and also depending upon how your multi-line Essbase block storage member formulas are written, the utility crashes within seconds.

Sadly, the README delivered with patch 30695700 contains some rather.... odd... language for a non-certified and unsupported utility.

After the README states the utility is not certified by Oracle and is also unsupported, it includes a prohibition from sharing code modifications/enhancements.

So, I know exactly where in the code the crash issue happens, but sadly I am prohibited from sharing the solution.  I really believe the policy needs to be re-examined by Oracle Support so the EPM community may share knowledge with each other!

Suffice to say, a single-line member formula shouldn't have any problems during the conversion process.  The problem is when a multi-line formula contains empty lines (carriage returns for readability purposes) and the EPMA File Generator _ line continuation character doesn't appear where the conversion utility expects to find it.  That's about all I can say for now.

Quick Tip: Starting OHS 12 in a distributed Hyperion environment

This ought to be a simpler post than what I usually write.

The topic of how to start Oracle HTTP Server ("OHS") 12c in Hyperion / Oracle EPM 11.2.x comes up frequently.  I've previously posted about the little trick, buried within the EPM 11.2.x install/config guide, on how to save the WebLogic password for OHS so you aren't prompted every time you try to start up OHS.

"But hey! Dave, how do we start and stop OHS in a distributed MS Windows Server environment when OHS doesn't have its own Windows service anymore?  The good old NSSM trick we use for WebLogic Admin Server doesn't work for OHS!"

An excellent question.

Login to each OHS host server in your distributed environment as your EPM network service account.  Run the "storeuserconfig" trick to save epm_admin's password in your network service account's profile on the C drive.  Search the install/config guide for the text "storeuserconfig" and it'll take you directly to the needed command syntax.  Every member on your Hyperion support team who wants to start/stop services using their own distinct userID needs to perform this step.  This is a one-time-only step until such time as your security team forces you to change the WebLogic epm_admin password.

Now on your OHS server(s) foreign to where you run your master start/stop scripts from (you are running custom start/stop scripts, yes?), create a simple little DOS batch script like this:

sc start "Oracle Weblogic ohs NodeManager (D_Oracle_Middleware_ohs_wlserver)"  

start "StartOHS" /wait /i cmd /c %EPM_INSTANCE_HOME%\httpConfig\ohs\bin\startComponent.cmd ohs_component

Replace D_Oracle in that first command with the drive letter where you installed.  e.g. E:_Oracle, F:_Oracle or whatever is appropriate.  The drive letter is unfortunately embedded within the Oracle Node Manager 12c Windows service name.

Replace %EPM_INSTANCE_HOME% with the fully qualified path to your Oracle EPM Instance, which is DRIVE:\Oracle\Middleware\user_projects\epmsystem1 by default.

Note: the final line beginning with start "StartOHS" is all one line all the way through ohs_component.  Blogger is linewrapping it in an undesirable way.

Save this script as DRIVE:\scripts\startohs.bat or whatever you want.

Replace startComponnent.cmd with stopComponent.cmd and now you have your stopohs.bat script as well!

Finally, use this to invoke it remotely from your master start/stop scripts:

wmic /node:%FDNHOST2% process call create "DRIVE:\Scripts\startohs.bat"

wmic is a built-in Windows command.  Provided your userID is an administrator on the foreign server, you may use this syntax to invoke any batch file, command, or executable you wish.

Replace the variable %FDNHOST2% with the hostname of your 2nd OHS server that you're clustering for load balancing.  Repeat as needed if you have more than 2 OHS servers.

Simple & easy.  You're welcome!  👍

Wednesday, May 12, 2021

In-Place Upgrade to EPM 11.2.5.0

The comments section for my prior post about EPM 11.2.5.0 has really lit up with a common theme:

An in-place upgrade from either Hyperion / Oracle EPM 11.2.3.0 or 11.2.4.0 (the only certified paths to get to 11.2.5.0) fails if one applied the OPatch update necessary to subsequently apply the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware - most notably WebLogic and OHS.

The commenter community seems to be in agreement that the best approaches are:
  • If Oracle Middleware, Opatch in particular, has yet to be patched, back it up!  If disk space is available, back up \Oracle\Middleware in its entirety before applying the quarterly Cracle Critical Patch Updates.
  • If the system is unpatched, it is apparently safe to perform the in-place upgrade to 11.2.5.0.
  • If the system was patched, it will be necessary to restore \Oracle\Middleware\Opatch and perhaps \Oracle\Middleware\wlserver and \Oracle\Middlware\oracle_common.  The jury is out on exactly where the bug resides with respect to patching.  All that is known for certain is Opatch is a primary culprit and the Opatch update cannot be rolled back without performing a filesystem restore for the Opatch folder structure.
Additionally, I experienced issues with missing jar files where ODI 12.2.1.4 and FDMEE 11.2.5.0 is concerned.  I have yet to determine if this is due to the issue mentioned above....  an EPM 11.2 customer-facing upgrade that just got handed over from me to the UAT team has eaten up all of my "free time".

I've installed a fresh VM and a fresh/unpatched EPM 11.2.4.0 and will perform the in-place upgrade to 11.2.5.0 to vet if ODI/FDMEE is truly broken or not in this release.  The EPM Configurator tool does not reveal the problem when FDMEE is broken, but there's a way I can see it on the back-end.

Finally, time permitting, I'll take my unpatched 11.2.4.0 Middleware backup, restore it to my botched/patched 11.2.4.0->11.2.5.0 in-place upgrade VM and see if I can put Humpty Dumpty back together again.

And here we thought RCU was difficult....

Tuesday, April 27, 2021

EPM 11.2.5 - My Week is Full

We finally get Oracle Fusion Middleware (FMW) 12.2.1.4!

SQL Server 2017!

Essbase 21c certified!

The bad news:

If you're still using Essbase Studio, migrate away from it.  Essbase Studio is slated to be retired in EPM 11.2.6 as per this Oracle Statement Of Direction:  https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=81935405509356&id=2731169.1&_afrWindowMode=0&_adf.ctrl-state=rbcbyuw9y_4

Visual Basic scripts for FDMEE need to be migrated to Jython ASAP.  VB scripting is expected to be decertified in EPM 11.2.7.  We knew this some time ago, but now we now the release number where VB goes away for FDMEE.

Interesting news:

Oracle's readme for 11.2.5 confirms what I've suspected.  In 11.2.x we aren't getting Patch Set Updates (PSUs), but rather new "dot" releases.  Here's a copy/paste directly from the 11.2.5 readme:

EPM System updates are released on a quarterly basis, generally in January,
April, July, and October.

You can directly apply an update from the previous two updates.

I won't paste the entire thing.  When you download 11.2.5 from Oracle eDelivery, you get this folder:

EPM_11.2.5\EPM 11.2.5.0.000\EPM System Installation Documentation\EPM System Installation

Stop, open this folder.  Open the readme before doing anything.  Here's the Cliff Notes.  You can apply 11.2.5 as an in-place upgrade on top of either 11.2.3 or 11.2.4.  Do not attempt to apply it on top of 11.1.2.4 or any older release; you need to do a fresh install and LCM your apps from the older release.  Only 11.1.2.4 is certified for this migration path.

If you were an early adopter like me, you need to jump to 11.2.2 before you can get to 11.2.3 or higher.  You can apply 11.2.2 on top of either 11.2.0 or 11.2.1.  You can't get to 11.2.3, .4 or .5 unless you're on at least .2.  Job security!

Migration Path documentation:  11.1.2.4.x to 11.2.x

If you're on Financial Close Management 11.1.2.3 or older, prepare for pain.  I've done it and it is not fun.  :(

It is now time to back up my sandbox and attempt the upgrade.  Wish me luck!