March 2022 Update

March 2022 is shaping up to be an interesting month, and the show isn't over yet!

One customer went live a few weeks ago on Hyperion / Oracle EPM 11.2.7.  It was a complex upgrade from EPM 11.1.2.4 but we got it done on time.

This week I upgraded another customer's EPM from 11.2.3 to 11.2.8.  The in-place upgrade wasn't easy and posed challenges... mostly due to many references on the back-end to jdk1.8_181, which no longer exists as a folder starting in 11.2.5.  Now it is simply Oracle\Middleware\jdk\.  Some of my NTT colleagues have hit similar issues, so I'm helping them troubleshoot.

A very fun little project was helping one of the co-founders of EPMware by installing Essbase 21.3 on one of his AWS images.  He wants to test if his software works against the Essbase 21c Java APIs.  He gave me full admin access and I turned it around in 1 day because he didn't need to integrate it with EPM, so it was a very simple install.  I'm anxious to hear the results of his testing!

I've been working on my own sandbox and am attempting to integrate Essbase 21.3 with EPM 11.2.8.  The 11.2.8 README says this type of configuration is certified now.  So far I've managed to get Planning to talk to Essbase 21c and successfully deploy the sample "Vision" application.

When integrated with EPM's 11.2.8 Shared Services, Essbase 21.3 registers itself smoothly so we have a familiar interface to provision users and groups to it.  The problem is you can't use LCM in EPM to work with Essbase 21.3 applications... you have to use the Essbase 21c migration tools instead.  One caveat is if you want to migrate a pre-existing Essbase 11.1.2.4 / 11.2.x app into 21c, it must come over as Unicode.  So some folks who want to attempt this may have some conversions to do.

Another weird thing I'm seeing in 21c is I can create Essbase apps just fine in the 21c EAS Console (it is the thin web client), but you cannot see the Planning apps in it!  I'm rebuilding my sandbox a slightly different way to see if I can figure it out.... the 21c online guides on Oracle's documentation site don't really address this topic.

Finally, I've got an upcoming project on my calendar to move a customer from OBIEE 11.1.1.9 to OBIEE 12.2.1.4.  This version of OBIEE isn't available on eDelivery when I last checked (Oracle says they're working on tightening up a few things), and I believe know why.  It is possible to get it, though.  Just do a web search for Oracle Business Intelligence Enterprise Edition.  With a bit of clicking, you can make your way to the OBIEE 12.2.1.4 download page.

One thing I'm unhappy about is OBIEE 12.2.1.4 requires an out-of-date Fusion Middleware Infrastructure: FMW 12.2.1.3 instead of 12.2.1.4.  I hope this gets fixed soon.

My head is spinning!  Be well,

Dave

Catch-Up Post for February 2022

Have I been slacking on updating this blog?

Yes.  Yes I have.  👎

Here's a quick few updates about what I've been working on.

1. Went live with Hyperion / Oracle EPM 11.2.7 for a customer.  Hypercare ends in a few days.  Things are really looking great!  We've got end-to-end SSL, Okta SSO, standalone ODI, load balancing... the works!

2. Installed EPM 11.2.8 and integrated it with Essbase 21.3.  Crazy stuff.  This experiment is worthy of its own blog post.

3. Oracle Business Intelligence 12.2.1.4.  Not yet on Oracle eDelivery and I believe I know why this is so.  It just isn't ready for prime-time.  The thing wants us to install Oracle Fusion Middleware 12.2.1.3.  Why????  I started pounding my face to my desk.

(Disclaimer:  this blog post was originally written on March 2, 2022.  The above paragraph will eventually be invalidated once Oracle corrects the issue.)

4. Log4J.  The Oracle KB article concerning the Log4J vulnerability was updated just this week.  Here's how to find it.  You need an active Oracle Support subscription to access this:

Apache Log4j Security Alert CVE-2021-44228 also referencing CVE-2021-45046 Mitigation on Oracle Enterprise Performance Management (Doc ID 2828262.1)

I think this is just about enough.  And today's just Wednesday.....

EPM 11.2.x and Quarterly Java Update - Be Careful!

So far I've discovered 2 issues with respect to the Quarterly Oracle "Critical Patch Update" (abbreviated "CPU" henceforth) and Hyperion / Oracle EPM 11.2.x.

The problem seems to have been introduced with the October 2021 CPU and continues to be present in the January 2022 CPU.

So far, I've observed 2 symptoms:

1. The legacy frconfig utility fails to launch.  No error message is provided.  The javaw process appears briefly in your process list and then closes.
2. The EAS Console screen that is supposed to display a graphical chart when viewing ASO Essbase aggregate information is blank.

Here's how you can definitively know if you have the issue, and what you can do about it.

With respect to item #1 above, Oracle does have a Knowledge Base article about it.  It takes a bit of digging to find.  Essentially, Oracle acknowledges that the Quarterly CPU delivers an incomplete Java 8.

So let's say you blow away \Oracle\Middleware\jdk's content and replace it with the latest Java CPU.  After all, that's what we've been doing for years, so why change our techniques?

Don't do that anymore!

Whether you are in Linux or Windows, you need to count the number of files in the jdk folder before and after you apply the CPU.  In Windows you can right-click the folder and hit Properties.  Look at the file count.  If the number of files is below 900 after applying the CPU (typically I've seen it is in the mid-300's), you got hit with the problem.

The better way forward is to follow this procedure:

1. Back up your Middleware\jdk folder.
2. Download the Java 8 quarterly CPU.  Don't run the installer.  In the case of Windows, use 7-zip to unzip the included .gz file and then again use 7-zip to unpack the .tar file included therein.  Linux nerds like me know what to do on the Linux side of the fence.
3. MERGE the files extracted from the tar into your Middleware\jdk.  This will replace your JDK binaries that WebLogic cares about, while also preserving the JRE files that include JavaFX (EAS Console wants these) and the stuff frconfig needs.

I am hoping this condition is temporary.  Oracle knows about it because they published the KB article about it, although the workaround they suggest is something I don't agree with.

The last KB article I read said to uninstall Java 8 (how? Java 8 is bundled with EPM and there's no uninstaller in the Windows Control Panel for it - I'm speaking Windows here as that is what the bulk of my customers use nowadays), reinstall the base Java 8 release, and then install the CPU on top of it as an "in-place upgrade".  How?  Old EPM Infrastructure dinosaurs like me can figure it out, but there are many folks new to EPM 11.2.x and trying to unpack all of this knowledge is a daunting task.

The above is my own opinion, so your mileage may vary.

EPM 11.2.8 - Online!

My loyal readers just KNEW this blog post was coming.  😁

Hyperion / Oracle EPM 11.2.8.0 was released on Friday, February 11, 2022.  I have it online now and here are initial observations.

(Disclaimer: the above screenshot is from the sample "Vision" application & associated financial data that ships with Hyperion Planning.)

Get Ready To Patch!

<Feb 12, 2022 4:52:31 PM CST> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.2.1.4.0 Thu Sep 12 04:04:29 GMT 2019 1974621>

F:\Oracle\Middleware\jdk\bin>.\java -version

java version "1.8.0_301"

Java(TM) SE Runtime Environment (build 1.8.0_301-b09)

Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode)

Tip:  If you want to use the old "FRConfig" utility to configure the default POV prompt on/off setting or HFR batch bursting external folders, you will want to back up the pre-delivered \Oracle\Middleware\jdk\ folder structure before applying the latest Quarterly Critical Patch Update ("CPU") for Java.  

Depending upon the method you use to patch/replace JDK 8, FRconfig won't launch and you won't see an error message explaining why.  The root cause is the Java quarterly CPU may be missing hundreds of files.  This issue may have started with the October 2021 CPU.  I noticed it also applies in the January 2022 CPU.  Oracle has a Knowledge Base article about this problem, but you have to dig a little bit to find it.  I'll probably need to write a separate blog on this specific topic alone.

Essbase 64-bit  - Release 11.1.2 (ESB11.1.2.4.045B020)

F:\Oracle\Middleware\OPatch>opatch version

OPatch Version: 13.9.4.2.7

f:\oracle\middleware\jdk\bin\java -jar f:\oracle\middleware\coherence\lib\coherence.jar

Oracle Coherence Version 12.2.1.4.0 Build 74888

(File is dated Sept 12, 2019)

Planning "Vision" Sample Application is Still Old

Vision still has a start year of FY09.  The good news is it extends out to FY27, so you have 5 more years to play with before you need to add more.

Essbase Stills Logs in TRACE Mode

Edit your $ARBORPATH\bin\logging.xml to fix this.  While you're at it, edit your $EPM_ORACLE_INSTANCE\config\OPMN\opmn\opmn.xml if you, like me, tire of seeing the opmn ping messages in Essbase.log every 20 seconds... 24x7.

You have to insert the line highlighted in yellow exactly where shown.  OPMN can be picky about the order of statements in opmn.xml apparently:

    <restart retry="2" timeout="600"/>

    <ping timeout="20" interval="0"/>

    <port id="essbase-port-range" range="32768-33768"/>

    <process-set id="AGENT" numprocs="1" restart-on-death="false">

Those of you who've worked with me in the past know I prefer restart-on-death="false" rather than the default "true" setting.  This permits a graceful shutdown of Essbase via MaxL, with error notification (if your shutdown script is clever) if an application is loading data, calculating, or restructuring.

Premier Support Extended

The 11.2.8 README says EPM 11.2 Premier Support now runs through "at least 2032".  So we have an additional 2 years than we originally expected when 11.2.0 came out.  Oracle Safe Harbor applies here of course.

Log4J Updated

Log4J is now version 2.17.1.  Consult your IT Security folks to verify this satisfies their vulnerability concerns.

MSAD 2019 Certified

EPM 11.2.8 now speaks with MSAD 2019.  This should make IT a bit happier.

Upgrade Leapfrog - GONE!!!!

Quite simply: this one is huge.  Prior to 11.2.8.0, we could only upgrade in-place from an older 11.2.x release if we were, at most, 1-2 dot releases behind.  Now we may jump directly to 11.2.8 from any 11.2.x release, provided we are coming from at least 11.2.2.  If you're still on 11.1.2.4, you will be standing up new server infrastructure and you'll jump directly to 11.2.8 and do a bunch of LCM, HFMCopyApplication, and EAS Console migrations.

This screenshot from the 11.2.8 README says it clearly:

Essbase 21c Now Certified for Planning & Profitability

This is another big one.

Many of my customers have been holding back on kicking the Essbase 21c tires because they use Hyperion Planning.  Now is the time to take a serious look at Essbase 21c.  The last I looked, Essbase 11.1.2.4 (which is the default Essbase version in EPM 11.2.x) support runs through the end of 2022.

My knowledge on this Essbase 11.1.2.4 support expiration topic may be a bit out of date, so I'll need to prowl around on Oracle's website to see if they're changed their position on this specific point or not.

Happy Upgrading!

2021 was very busy for me doing non-stop upgrades to 11.2.x for many customers.  2022 is shaping up to be the same situation, with my first 11.2.7 going live in Production one week from now.

If you've never touched EPM 11.2.x before, be sure to get some professional consulting help from someone who has successfully done it and gone live in Production.  There are important differences between standing up EPM 11.2.x versus EPM 11.1.2.x.

Final WebLogic 10.3.6 Patch For EPM 11.1.2.4

A commenter on this blog, Ben, shared how to get the final WebLogic 10.3.6 patch for Hyperion / Oracle EPM 11.1.2.4.  We may download this patch without needing to get a password from Oracle Support.  The patch number is:

33471254

You need to be on an active Oracle Support contract for your EPM 11.1.2.4 in order to access the patch.

Ben also indicated we may pull up an Oracle Knowledge Base article that lists the final patches for various components of the EPM 11.1.2.4 suite.  The KB # is:

2796575.1

When you pull up this KB article, crtl-f in your browser and plug in this text to jump directly to the Hyperion section:

3.3.22 Oracle Hyperion Infrastructure Technology

If your 11.1.2.4 environment includes Essbase, we can expect more Essbase 11.1.2.4 patches to come out in 2022, because (for now) EPM 11.2.x uses Essbase 11.1.2.4 under the covers.

As of EPM 11.2.7.0, Essbase 12c is still not certified for Hyperion Planning.  Once an 11.2.x comes out that certifies Essbase 12c for Planning, I'll take a closer look at it.

As per the 11.2.7.0 README, EPM 11.2.x patches are expected to be released Quarterly according to this rough schedule (Oracle Corporation's Safe Harbor applies here):  January, April, July, and October.

On Oracle's EPM 11.2 documentation hub, click "Essbase" on the left-hand side.  Among other things, we are explicitly told to use Essbase 11.1.2.4:

"Use Essbase 11.1.2.4, which is compatible with Enterprise Performance Management Release 11.2.x"

Consult the 11.2.7.0 Install+Config Guide for more information about using Essbase 12c in an EPM 11.2.5+ environment.

Many of you are on PTO this week, so you might not notice this post until early 2022.

Have a Happy New Year!

EPM 11.1.2.x in 2022

As 2021 is rapidly drawing to a close, it is appropriate at this time to shame remind people to either move to the EPM Cloud or upgrade on-premises Hyperion / Oracle EPM to Release 11.2.x.

While the official support documents say EPM 11.1.2.4 is supported through Dec 31, 2021, the truth is it is already out of support.  Good luck getting a new patch for it.  Security patches for EPM 11.1.2.4 effectively ended with the October 2021 Critical Patch Update (OCT2021CPU).

Due to the log4j security headache, we might see some new patches trickle out, but these are targeted toward EPM 11.2.x and things seem to evolve daily (as of this writing).

Multiple people have contacted me about how difficult it is to gain access to the last Oracle WebLogic 10.3.6 security patch.  You CAN get it.  The trick is you have to convince the person working your Oracle SR that, yes, you are paying your support contact, and thus, yes, you are entitled to the WebLogic 10.3.6 patch that is included with your support.

If you are still on 11.1.2.4, or Lord forbid an earlier version, you want to wrangle your 2022 fiscal budget to accommodate an upgrade to 11.2.x.  Your IT department will thank you because you'll be moving off of Java 6 or Java 7 (assuming you did the 6->7 swap earlier in the year).

And if you are still running Hyperion Enterprise 4, I award you no points.

EPM 11.2.7 is Not Using WebLogic 10.3

Not that it matters, but the report generated by epmsys_registry in Hyperion / Oracle EPM 11.2.7 is juuuust a tad wrong where the version of Oracle Fusion Middleware's WebLogic is concerned.

We're sitting on top of WebLogic 12.2.1.4 and we recently applied the OCT2021CPU cumulative patch update in the environment pictured above.  I'm fairly confident version "10.3.2.0" shown above is Fake News.

The back-end WebLogic logs prove we are, in fact, using 12.2.1.4.

I'm tempted to use epmsys_registry utility to update the version property.  The only thing stopping me is I can hear my executive's voice in my head:  "Don't do it unless Oracle Support certifies it!"

I am, however, very similar to John McClane and break rules all the time.  Die Hard IS a Christmas film!!!!