Monday, December 27, 2021

Final WebLogic 10.3.6 Patch For EPM 11.1.2.4

A commenter on this blog, Ben, shared how to get the final WebLogic 10.3.6 patch for Hyperion / Oracle EPM 11.1.2.4.  We may download this patch without needing to get a password from Oracle Support.  The patch number is:

33471254

You need to be on an active Oracle Support contract for your EPM 11.1.2.4 in order to access the patch.

Ben also indicated we may pull up an Oracle Knowledge Base article that lists the final patches for various components of the EPM 11.1.2.4 suite.  The KB # is:

2796575.1

When you pull up this KB article, crtl-f in your browser and plug in this text to jump directly to the Hyperion section:

3.3.22 Oracle Hyperion Infrastructure Technology

If your 11.1.2.4 environment includes Essbase, we can expect more Essbase 11.1.2.4 patches to come out in 2022, because (for now) EPM 11.2.x uses Essbase 11.1.2.4 under the covers.

As of EPM 11.2.7.0, Essbase 12c is still not certified for Hyperion Planning.  Once an 11.2.x comes out that certifies Essbase 12c for Planning, I'll take a closer look at it.

As per the 11.2.7.0 README, EPM 11.2.x patches are expected to be released Quarterly according to this rough schedule (Oracle Corporation's Safe Harbor applies here):  January, April, July, and October.

On Oracle's EPM 11.2 documentation hub, click "Essbase" on the left-hand side.  Among other things, we are explicitly told to use Essbase 11.1.2.4:

"Use Essbase 11.1.2.4, which is compatible with Enterprise Performance Management Release 11.2.x"

Consult the 11.2.7.0 Install+Config Guide for more information about using Essbase 12c in an EPM 11.2.5+ environment.

Many of you are on PTO this week, so you might not notice this post until early 2022.

Have a Happy New Year!

Tuesday, December 21, 2021

EPM 11.1.2.x in 2022

As 2021 is rapidly drawing to a close, it is appropriate at this time to shame remind people to either move to the EPM Cloud or upgrade on-premises Hyperion / Oracle EPM to Release 11.2.x.

While the official support documents say EPM 11.1.2.4 is supported through Dec 31, 2021, the truth is it is already out of support.  Good luck getting a new patch for it.  Security patches for EPM 11.1.2.4 effectively ended with the October 2021 Critical Patch Update (OCT2021CPU).

Due to the log4j security headache, we might see some new patches trickle out, but these are targeted toward EPM 11.2.x and things seem to evolve daily (as of this writing).

Multiple people have contacted me about how difficult it is to gain access to the last Oracle WebLogic 10.3.6 security patch.  You CAN get it.  The trick is you have to convince the person working your Oracle SR that, yes, you are paying your support contact, and thus, yes, you are entitled to the WebLogic 10.3.6 patch that is included with your support.

If you are still on 11.1.2.4, or Lord forbid an earlier version, you want to wrangle your 2022 fiscal budget to accommodate an upgrade to 11.2.x.  Your IT department will thank you because you'll be moving off of Java 6 or Java 7 (assuming you did the 6->7 swap earlier in the year).

And if you are still running Hyperion Enterprise 4, I award you no points.




Sunday, December 19, 2021

EPM 11.2.7 is Not Using WebLogic 10.3



Not that it matters, but the report generated by epmsys_registry in Hyperion / Oracle EPM 11.2.7 is juuuust a tad wrong where the version of Oracle Fusion Middleware's WebLogic is concerned.

We're sitting on top of WebLogic 12.2.1.4 and we recently applied the OCT2021CPU cumulative patch update in the environment pictured above.  I'm fairly confident version "10.3.2.0" shown above is Fake News.

The back-end WebLogic logs prove we are, in fact, using 12.2.1.4.

I'm tempted to use epmsys_registry utility to update the version property.  The only thing stopping me is I can hear my executive's voice in my head:  "Don't do it unless Oracle Support certifies it!"

I am, however, very similar to John McClane and break rules all the time.  Die Hard IS a Christmas film!!!!



Saturday, December 11, 2021

Zero-Day log4j Exploit Affects Hyperion

BleepingComputer.com:  log4j Exploit Explained

All modern Hyperion / Oracle EPM systems use the Apache log4j library behind the scenes.  On December 10, 2021, a zero-day exploit was revealed and attackers in the wild are already scanning vulnerable systems.

Systems running Apache log4j 2.0-beta9 up to 2.14.1 are impacted.  

EPM 11.2.x uses log4j 2.13.3, according to Oracle's 3rd Party Acknowledgements document for EPM.  In truth, I'm seeing many log4j versions in my unpatched EPM 11.2.7 sandbox.  I'll want to apply the October 2021 Critical Patch Update in my sandbox to see if this changes before I smash the Panic Button.  A search for log4j*.jar in \Oracle\Middleware reveals the following:














EPM versions 11.1.2.x may use a slightly older version, but it would still be in the range of impacted versions.

The exploit allows the attacker to completely take over a server without needing credentials of any kind, other than finding their way into your network when your EPM system is behind a corporate firewall.  This "get inside the network" scenario is not as far-fetched as it sounds, unless your IT Security folks have implemented 2-factor security on the VPN.  We then get into a "social engineering" discussion, which I won't launch into here.

I can't view the code for Oracle's EPM Cloud, so I can't say for sure if the cloud is impacted.  My gut tells me the EPM Cloud is based upon on-premises EPM technology in some fashion - Apache 2.0 and Oracle WebLogic for sure (why re-invent the wheel?).  If the cloud URL is not protected (many EPM Cloud customers nowadays DO protect their URL so only folks inside their network may hit the URL, which is a GOOD thing), then there may be an issue here.

The BleepingComputer.com link which begins this article offers some suggestions.  One is to patch log4j to log4j 2.15.0.  Based upon my screenshot above, we'd have to do this in multiple places where the version is less than this.  (Which according to my scan, is all of them).  My gut tells me this effort would not be trivial, as we'd have to touch every EPM server having these directories.

A simpler fix may be to apply one of these changes suggested by BleepingComputer:
The flaw can also be mitigated in previous releases (2.10 and later) by setting system property "log4j2.formatMsgNoLookups" to "true" or removing the JndiLookup class from the classpath.
In the case of Windows-hosted EPM systems, we're talking about editing setdomainenv.bat in user_projects\domains\EPMSystem\bin so the WebLogic Admin Server has an additional -D argument for the Java options, and editing the Windows Registry on each Windows server for each Weblogic Managed Server underneath the "Hyperion Solutions" registry hive.  Fun times.  Actually not "simple", but you're not going out and patching multiple directories across multiple servers.

I suspect many of my readers won't have an appetite for this level of "surgery".  The next Oracle Critical Patch Update is due in mid-January 2022, but only for those on EPM 11.2.x.  Those of you still on EPM 11.1.2.x with no immediate plans to upgrade or move to the EPM Cloud may consider the surgery I mentioned above.

Those of us on EPM 11.2.x can hop out to Oracle's support site and search for the various Oracle Fusion Middleware patches.  The patches will likely come out before the next Critical Patch Update is formally announced in mid-January.

Disclaimer: This blog post is speculative and may become dated after the January 2022 Critical Patch Update.  I am not an employee of Oracle Corporation and thus do not speak for them.

Dec 12, 2021 Update:
The Apache Foundation has released a patched Java .jar file for log4j that claims to fix the issue.  I'll try it in my sandbox later today to verify it doesn't brick things.

Wednesday, October 27, 2021

EPM 11.2.7 is Available

 EPM 11.2.7 README

The EPM 11.2.x README (linked above) has been updated for 11.2.7.  The software is available on eDelivery for both 64-bit Windows and 64-bit Linux.

I'm in the process of standing up a fresh Windows Server 2019 VM and will share my findings after I get through the Install+Config of EPM 11.2.7.  I will be doing it fresh, rather than as an in-place upgrade from a prior release.  That exercise will come later.

One of the main things I'm interested in is to see how 11.2.7 handles the October 2021 Oracle Critical Patch Update ("CPU").  In particular... WebLogic.  When doing the patch search for the October 2021 CPU, I kept seeing "this patch has been superseded".  Grabbing the latest & greatest WebLogic patch actually bricked my 11.2.6 system and so I had to roll back the patch to get things working again.

Tuesday, September 14, 2021

Essbase Studio Server and Linux

So before I dive in, I need to clear the air a bit.  If you are currently using Essbase Studio, find an alternative.

Essbase Studio was removed in Hyperion / Oracle EPM 11.2.6.0.  You need to figure out a different way to build your Essbase cubes.  As the point of this blog post is not to give free application consulting, I'm going to pass over this particular topic and instead help those who are still using Essbase Studio 11.1.2.3 through 11.2.5.0 on Linux.

There's a little bug with how Essbase Studio Server was packaged for Linux, and I first detected it when doing an 11.1.2.3 upgrade back in the day.  The problem persists in 11.1.2.4, and I suspect the 11.2.0.0 through 11.2.5.0 releases have the exact same issue.

And here is the root cause and solution!

The symptom:  You run the pre-built start script for Essbase Studio Server on Linux, in your user_projects/epmsystem1/bin/ folder.  The script doesn't complain, but you never see the Java process for Essbase Studio Server start up.  What's going on here?

The root cause:  installtool.sh doesn't set the UNIX filesystem permissions correctly for the back-end Essbase Studio Server scripts.

The fix:

cd to your EPMSystem11R1 folder
Type (you will have to tweak this depending upon the variation of UNIX you are running)
find . -name *.sh -exec chmod ug+x {} \;

Done.  You've nuked it from orbit and changed all shell scripts to be executable by the owner and the group.

Sunday, August 8, 2021

11.2.6 Doesn't Fix My Bricked In-Place Upgrade

Those of you who've been following my blog for some time know that I love to experiment.

A while back, I had attempted an in-place upgrade within the Hyperion / Oracle EPM series after applying one of the quarterly Oracle Critical Patch Updates.  I have at least 2 posts here on that very topic: my original findings, and a 2nd post offering a solution shared by one of our frequent commenters here, Deezel.

The README for EPM 11.2.6.0, the newest EPM Release as of this writing, claims to automatically roll back Middleware patches are part of the in-place upgrade process.  I decided to roll the dice without first applying the solution linked above.  Mostly, I was just curious to see what would happen!

Well..... this happened:












(Trying to redeploy other web applications, such as Calculation Manager, yields a similar error)

Given this is happening in a throwaway sandbox that isn't customer-facing, I opted not to open a Service Request on this.

Other weird errors encountered during the in-place upgrade:

  • EPMSystem11R1\diagnostics\logs\install\ gobbled up 80GB of space due to a runaway 32-bit Oracle DB Client silent install.  It ate up all remaining disk space I had available, and I ended up having to forcibly terminate the javaw.exe process for installTool.  I also ended up renaming the dbclient32 folder and deleting its entry in the global Oracle Inventory.
  • The OHS upgrade failed at 50% completion status with a vague error.  I renamed the ohs folder and installTool completed during the 2nd attempt.
  • 11.2 still has the Known Issue with respect to cancelling out of configTool; it blew away both setEnv.bat and configTool.bat in the epmsystem1 instance folder.
I need to move on to other matters of investigation now, so I destroyed the VM to free up space for the next one.

It was an interesting exercise.  I'm encouraging all my peers to back up the Middleware folder (minus user_projects) prior to applying the quarterly Oracle CPU Middleware patches.  I think I still have some of my hair remaining....