Final WebLogic 10.3.6 Patch For EPM 11.1.2.4

A commenter on this blog, Ben, shared how to get the final WebLogic 10.3.6 patch for Hyperion / Oracle EPM 11.1.2.4.  We may download this patch without needing to get a password from Oracle Support.  The patch number is:

33471254

You need to be on an active Oracle Support contract for your EPM 11.1.2.4 in order to access the patch.

Ben also indicated we may pull up an Oracle Knowledge Base article that lists the final patches for various components of the EPM 11.1.2.4 suite.  The KB # is:

2796575.1

When you pull up this KB article, crtl-f in your browser and plug in this text to jump directly to the Hyperion section:

3.3.22 Oracle Hyperion Infrastructure Technology

If your 11.1.2.4 environment includes Essbase, we can expect more Essbase 11.1.2.4 patches to come out in 2022, because (for now) EPM 11.2.x uses Essbase 11.1.2.4 under the covers.

As of EPM 11.2.7.0, Essbase 12c is still not certified for Hyperion Planning.  Once an 11.2.x comes out that certifies Essbase 12c for Planning, I'll take a closer look at it.

As per the 11.2.7.0 README, EPM 11.2.x patches are expected to be released Quarterly according to this rough schedule (Oracle Corporation's Safe Harbor applies here):  January, April, July, and October.

On Oracle's EPM 11.2 documentation hub, click "Essbase" on the left-hand side.  Among other things, we are explicitly told to use Essbase 11.1.2.4:

"Use Essbase 11.1.2.4, which is compatible with Enterprise Performance Management Release 11.2.x"

Consult the 11.2.7.0 Install+Config Guide for more information about using Essbase 12c in an EPM 11.2.5+ environment.

Many of you are on PTO this week, so you might not notice this post until early 2022.

Have a Happy New Year!

EPM 11.1.2.x in 2022

As 2021 is rapidly drawing to a close, it is appropriate at this time to shame remind people to either move to the EPM Cloud or upgrade on-premises Hyperion / Oracle EPM to Release 11.2.x.

While the official support documents say EPM 11.1.2.4 is supported through Dec 31, 2021, the truth is it is already out of support.  Good luck getting a new patch for it.  Security patches for EPM 11.1.2.4 effectively ended with the October 2021 Critical Patch Update (OCT2021CPU).

Due to the log4j security headache, we might see some new patches trickle out, but these are targeted toward EPM 11.2.x and things seem to evolve daily (as of this writing).

Multiple people have contacted me about how difficult it is to gain access to the last Oracle WebLogic 10.3.6 security patch.  You CAN get it.  The trick is you have to convince the person working your Oracle SR that, yes, you are paying your support contact, and thus, yes, you are entitled to the WebLogic 10.3.6 patch that is included with your support.

If you are still on 11.1.2.4, or Lord forbid an earlier version, you want to wrangle your 2022 fiscal budget to accommodate an upgrade to 11.2.x.  Your IT department will thank you because you'll be moving off of Java 6 or Java 7 (assuming you did the 6->7 swap earlier in the year).

And if you are still running Hyperion Enterprise 4, I award you no points.

EPM 11.2.7 is Not Using WebLogic 10.3

Not that it matters, but the report generated by epmsys_registry in Hyperion / Oracle EPM 11.2.7 is juuuust a tad wrong where the version of Oracle Fusion Middleware's WebLogic is concerned.

We're sitting on top of WebLogic 12.2.1.4 and we recently applied the OCT2021CPU cumulative patch update in the environment pictured above.  I'm fairly confident version "10.3.2.0" shown above is Fake News.

The back-end WebLogic logs prove we are, in fact, using 12.2.1.4.

I'm tempted to use epmsys_registry utility to update the version property.  The only thing stopping me is I can hear my executive's voice in my head:  "Don't do it unless Oracle Support certifies it!"

I am, however, very similar to John McClane and break rules all the time.  Die Hard IS a Christmas film!!!!

Zero-Day log4j Exploit Affects Hyperion

BleepingComputer.com:  log4j Exploit Explained

All modern Hyperion / Oracle EPM systems use the Apache log4j library behind the scenes.  On December 10, 2021, a zero-day exploit was revealed and attackers in the wild are already scanning vulnerable systems.

Systems running Apache log4j 2.0-beta9 up to 2.14.1 are impacted.  

EPM 11.2.x uses log4j 2.13.3, according to Oracle's 3rd Party Acknowledgements document for EPM.  In truth, I'm seeing many log4j versions in my unpatched EPM 11.2.7 sandbox.  I'll want to apply the October 2021 Critical Patch Update in my sandbox to see if this changes before I smash the Panic Button.  A search for log4j*.jar in \Oracle\Middleware reveals the following:

EPM versions 11.1.2.x may use a slightly older version, but it would still be in the range of impacted versions.

The exploit allows the attacker to completely take over a server without needing credentials of any kind, other than finding their way into your network when your EPM system is behind a corporate firewall.  This "get inside the network" scenario is not as far-fetched as it sounds, unless your IT Security folks have implemented 2-factor security on the VPN.  We then get into a "social engineering" discussion, which I won't launch into here.

I can't view the code for Oracle's EPM Cloud, so I can't say for sure if the cloud is impacted.  My gut tells me the EPM Cloud is based upon on-premises EPM technology in some fashion - Apache 2.0 and Oracle WebLogic for sure (why re-invent the wheel?).  If the cloud URL is not protected (many EPM Cloud customers nowadays DO protect their URL so only folks inside their network may hit the URL, which is a GOOD thing), then there may be an issue here.

The BleepingComputer.com link which begins this article offers some suggestions.  One is to patch log4j to log4j 2.15.0.  Based upon my screenshot above, we'd have to do this in multiple places where the version is less than this.  (Which according to my scan, is all of them).  My gut tells me this effort would not be trivial, as we'd have to touch every EPM server having these directories.

A simpler fix may be to apply one of these changes suggested by BleepingComputer:

The flaw can also be mitigated in previous releases (2.10 and later) by setting system property "log4j2.formatMsgNoLookups" to "true" or removing the JndiLookup class from the classpath.

In the case of Windows-hosted EPM systems, we're talking about editing setdomainenv.bat in user_projects\domains\EPMSystem\bin so the WebLogic Admin Server has an additional -D argument for the Java options, and editing the Windows Registry on each Windows server for each Weblogic Managed Server underneath the "Hyperion Solutions" registry hive.  Fun times.  Actually not "simple", but you're not going out and patching multiple directories across multiple servers.

I suspect many of my readers won't have an appetite for this level of "surgery".  The next Oracle Critical Patch Update is due in mid-January 2022, but only for those on EPM 11.2.x.  Those of you still on EPM 11.1.2.x with no immediate plans to upgrade or move to the EPM Cloud may consider the surgery I mentioned above.

Those of us on EPM 11.2.x can hop out to Oracle's support site and search for the various Oracle Fusion Middleware patches.  The patches will likely come out before the next Critical Patch Update is formally announced in mid-January.

Disclaimer: This blog post is speculative and may become dated after the January 2022 Critical Patch Update.  I am not an employee of Oracle Corporation and thus do not speak for them.

Dec 12, 2021 Update:

The Apache Foundation has released a patched Java .jar file for log4j that claims to fix the issue.  I'll try it in my sandbox later today to verify it doesn't brick things.

log4j Patch