EPM 11.2.x and Quarterly Java Update - Be Careful!

So far I've discovered 2 issues with respect to the Quarterly Oracle "Critical Patch Update" (abbreviated "CPU" henceforth) and Hyperion / Oracle EPM 11.2.x.

The problem seems to have been introduced with the October 2021 CPU and continues to be present in the January 2022 CPU.

So far, I've observed 2 symptoms:

1. The legacy frconfig utility fails to launch.  No error message is provided.  The javaw process appears briefly in your process list and then closes.
2. The EAS Console screen that is supposed to display a graphical chart when viewing ASO Essbase aggregate information is blank.

Here's how you can definitively know if you have the issue, and what you can do about it.

With respect to item #1 above, Oracle does have a Knowledge Base article about it.  It takes a bit of digging to find.  Essentially, Oracle acknowledges that the Quarterly CPU delivers an incomplete Java 8.

So let's say you blow away \Oracle\Middleware\jdk's content and replace it with the latest Java CPU.  After all, that's what we've been doing for years, so why change our techniques?

Don't do that anymore!

Whether you are in Linux or Windows, you need to count the number of files in the jdk folder before and after you apply the CPU.  In Windows you can right-click the folder and hit Properties.  Look at the file count.  If the number of files is below 900 after applying the CPU (typically I've seen it is in the mid-300's), you got hit with the problem.

The better way forward is to follow this procedure:

1. Back up your Middleware\jdk folder.
2. Download the Java 8 quarterly CPU.  Don't run the installer.  In the case of Windows, use 7-zip to unzip the included .gz file and then again use 7-zip to unpack the .tar file included therein.  Linux nerds like me know what to do on the Linux side of the fence.
3. MERGE the files extracted from the tar into your Middleware\jdk.  This will replace your JDK binaries that WebLogic cares about, while also preserving the JRE files that include JavaFX (EAS Console wants these) and the stuff frconfig needs.

I am hoping this condition is temporary.  Oracle knows about it because they published the KB article about it, although the workaround they suggest is something I don't agree with.

The last KB article I read said to uninstall Java 8 (how? Java 8 is bundled with EPM and there's no uninstaller in the Windows Control Panel for it - I'm speaking Windows here as that is what the bulk of my customers use nowadays), reinstall the base Java 8 release, and then install the CPU on top of it as an "in-place upgrade".  How?  Old EPM Infrastructure dinosaurs like me can figure it out, but there are many folks new to EPM 11.2.x and trying to unpack all of this knowledge is a daunting task.

The above is my own opinion, so your mileage may vary.

EPM 11.2.8 - Online!

My loyal readers just KNEW this blog post was coming.  😁

Hyperion / Oracle EPM 11.2.8.0 was released on Friday, February 11, 2022.  I have it online now and here are initial observations.

(Disclaimer: the above screenshot is from the sample "Vision" application & associated financial data that ships with Hyperion Planning.)

Get Ready To Patch!

<Feb 12, 2022 4:52:31 PM CST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.2.1.4.0 Thu Sep 12 04:04:29 GMT 2019 1974621>

F:\Oracle\Middleware\jdk\bin>.\java -version

java version "1.8.0_301"

Java(TM) SE Runtime Environment (build 1.8.0_301-b09)

Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode)

Tip:  If you want to use the old "FRConfig" utility to configure the default POV prompt on/off setting or HFR batch bursting external folders, you will want to back up the pre-delivered \Oracle\Middleware\jdk\ folder structure before applying the latest Quarterly Critical Patch Update ("CPU") for Java.  

Depending upon the method you use to patch/replace JDK 8, FRconfig won't launch and you won't see an error message explaining why.  The root cause is the Java quarterly CPU may be missing hundreds of files.  This issue may have started with the October 2021 CPU.  I noticed it also applies in the January 2022 CPU.  Oracle has a Knowledge Base article about this problem, but you have to dig a little bit to find it.  I'll probably need to write a separate blog on this specific topic alone.

Essbase 64-bit  - Release 11.1.2 (ESB11.1.2.4.045B020)

F:\Oracle\Middleware\OPatch>opatch version

OPatch Version: 13.9.4.2.7

f:\oracle\middleware\jdk\bin\java -jar f:\oracle\middleware\coherence\lib\coherence.jar

Oracle Coherence Version 12.2.1.4.0 Build 74888

(File is dated Sept 12, 2019)

Planning "Vision" Sample Application is Still Old

Vision still has a start year of FY09.  The good news is it extends out to FY27, so you have 5 more years to play with before you need to add more.

Essbase Stills Logs in TRACE Mode

Edit your $ARBORPATH\bin\logging.xml to fix this.  While you're at it, edit your $EPM_ORACLE_INSTANCE\config\OPMN\opmn\opmn.xml if you, like me, tire of seeing the opmn ping messages in Essbase.log every 20 seconds... 24x7.

You have to insert the line highlighted in yellow exactly where shown.  OPMN can be picky about the order of statements in opmn.xml apparently:

    <restart retry="2" timeout="600"/>

    <ping timeout="20" interval="0"/>

    <port id="essbase-port-range" range="32768-33768"/>

    <process-set id="AGENT" numprocs="1" restart-on-death="false">

Those of you who've worked with me in the past know I prefer restart-on-death="false" rather than the default "true" setting.  This permits a graceful shutdown of Essbase via MaxL, with error notification (if your shutdown script is clever) if an application is loading data, calculating, or restructuring.

Premier Support Extended

The 11.2.8 README says EPM 11.2 Premier Support now runs through "at least 2032".  So we have an additional 2 years than we originally expected when 11.2.0 came out.  Oracle Safe Harbor applies here of course.

Log4J Updated

Log4J is now version 2.17.1.  Consult your IT Security folks to verify this satisfies their vulnerability concerns.

MSAD 2019 Certified

EPM 11.2.8 now speaks with MSAD 2019.  This should make IT a bit happier.

Upgrade Leapfrog - GONE!!!!

Quite simply: this one is huge.  Prior to 11.2.8.0, we could only upgrade in-place from an older 11.2.x release if we were, at most, 1-2 dot releases behind.  Now we may jump directly to 11.2.8 from any 11.2.x release, provided we are coming from at least 11.2.2.  If you're still on 11.1.2.4, you will be standing up new server infrastructure and you'll jump directly to 11.2.8 and do a bunch of LCM, HFMCopyApplication, and EAS Console migrations.

This screenshot from the 11.2.8 README says it clearly:

Essbase 21c Now Certified for Planning & Profitability

This is another big one.

Many of my customers have been holding back on kicking the Essbase 21c tires because they use Hyperion Planning.  Now is the time to take a serious look at Essbase 21c.  The last I looked, Essbase 11.1.2.4 (which is the default Essbase version in EPM 11.2.x) support runs through the end of 2022.

My knowledge on this Essbase 11.1.2.4 support expiration topic may be a bit out of date, so I'll need to prowl around on Oracle's website to see if they're changed their position on this specific point or not.

Happy Upgrading!

2021 was very busy for me doing non-stop upgrades to 11.2.x for many customers.  2022 is shaping up to be the same situation, with my first 11.2.7 going live in Production one week from now.

If you've never touched EPM 11.2.x before, be sure to get some professional consulting help from someone who has successfully done it and gone live in Production.  There are important differences between standing up EPM 11.2.x versus EPM 11.1.2.x.