All modern Hyperion / Oracle EPM systems use the Apache log4j library behind the scenes. On December 10, 2021, a zero-day exploit was revealed and attackers in the wild are already scanning vulnerable systems.
Systems running Apache log4j 2.0-beta9 up to 2.14.1 are impacted.
EPM 11.2.x uses log4j 2.13.3, according to Oracle's 3rd Party Acknowledgements document for EPM. In truth, I'm seeing many log4j versions in my unpatched EPM 11.2.7 sandbox. I'll want to apply the October 2021 Critical Patch Update in my sandbox to see if this changes before I smash the Panic Button. A search for log4j*.jar in \Oracle\Middleware reveals the following:
EPM versions 11.1.2.x may use a slightly older version, but it would still be in the range of impacted versions.
The exploit allows the attacker to completely take over a server without needing credentials of any kind, other than finding their way into your network when your EPM system is behind a corporate firewall. This "get inside the network" scenario is not as far-fetched as it sounds, unless your IT Security folks have implemented 2-factor security on the VPN. We then get into a "social engineering" discussion, which I won't launch into here.
I can't view the code for Oracle's EPM Cloud, so I can't say for sure if the cloud is impacted. My gut tells me the EPM Cloud is based upon on-premises EPM technology in some fashion - Apache 2.0 and Oracle WebLogic for sure (why re-invent the wheel?). If the cloud URL is not protected (many EPM Cloud customers nowadays DO protect their URL so only folks inside their network may hit the URL, which is a GOOD thing), then there may be an issue here.
The BleepingComputer.com link which begins this article offers some suggestions. One is to patch log4j to log4j 2.15.0. Based upon my screenshot above, we'd have to do this in multiple places where the version is less than this. (Which according to my scan, is all of them). My gut tells me this effort would not be trivial, as we'd have to touch every EPM server having these directories.
A simpler fix may be to apply one of these changes suggested by BleepingComputer:
The flaw can also be mitigated in previous releases (2.10 and later) by setting system property "log4j2.formatMsgNoLookups" to "true" or removing the JndiLookup class from the classpath.
In the case of Windows-hosted EPM systems, we're talking about editing setdomainenv.bat in user_projects\domains\EPMSystem\bin so the WebLogic Admin Server has an additional -D argument for the Java options, and editing the Windows Registry on each Windows server for each Weblogic Managed Server underneath the "Hyperion Solutions" registry hive. Fun times. Actually not "simple", but you're not going out and patching multiple directories across multiple servers.
I suspect many of my readers won't have an appetite for this level of "surgery". The next Oracle Critical Patch Update is due in mid-January 2022, but only for those on EPM 11.2.x. Those of you still on EPM 11.1.2.x with no immediate plans to upgrade or move to the EPM Cloud may consider the surgery I mentioned above.
Those of us on EPM 11.2.x can hop out to Oracle's support site and search for the various Oracle Fusion Middleware patches. The patches will likely come out before the next Critical Patch Update is formally announced in mid-January.
Disclaimer: This blog post is speculative and may become dated after the January 2022 Critical Patch Update. I am not an employee of Oracle Corporation and thus do not speak for them.
Dec 12, 2021 Update:
The Apache Foundation has released a patched Java .jar file for log4j that claims to fix the issue. I'll try it in my sandbox later today to verify it doesn't brick things.
Thanks for post,can you provide link Apache released java.jar to fix this issue
ReplyDeletehttps://dlcdn.apache.org/logging/log4j/2.15.0/apache-log4j-2.15.0-bin.zip
DeleteExtract it, rename log4-core*.jar wherever it exists in your system so that it doesn't have a jar extension anymore, and then drop log4j-core-2.15.0.jar from the unzipped patch into those folders.
How can we set this property formatMsgNoLookups=true in epm platform or how we can patch separately Apache released patch for Log4J issue?
ReplyDeleteI'm about to try it on one of my sandbox servers. It would seem the most convenient way to do it is to set it as a global environment variable named formatMsgNoLookups and set its value to true. In Linux you'd do this in your service account's .bash_profile file. Bounce services after you do this.
DeleteI know someone on IBM's "Red Team" (people hired to simulate hacker attacks) and I believe he has the Proof-Of-Concept code someone leaked to the Internet. I want to run it against my system as a "before & after" test - to determine if either the Apache log4j-core patch or the suggested environment variable plugs the security hold.
ReplyDeleteThanks Dave, informative as ever.
ReplyDeleteI looked at the Third-Party Acknowledgments for 11.1.2.4 and it looks to be unaffected versions in use only (pre 2.0) unless i'm missing something here?
"Apache Log4j 1.2.8, 1.2.13, 1.2.14, 1.2.15, 1.3"
I note it also affects the latest version of Oracle Data Integrator with a workaround that's wrong in the Oracle KB 2827793.1 (thanks to an SR, the correct path is ODI_Home\oracle_common\modules\thirdparty\log4j-2.11.1.jar in ODI 12.2.1.4).
Looks like they've just removed ODI as a from the "applies to" section on 2827793.1, but the table still links to it on the master vlun KB 2827611.1, so guess ODI will get it's own KB soon.
DeleteBen, I believe your assessment is correct. We are out there just about every day crawling through the patch availability matrix. New information is trickling out day by day.
DeleteI haven’t seen this vulnerable log4j version in 11.1.2.2 and 1.1.2.4 but is there in 11.2.1 and 11.2.6. Surprisingly I am not seeing a reference on EPM in Oracle’s article as affected or not affected.
ReplyDeleteHi Viewer.
ReplyDeleteThey aren't calling out EPM in particular. They are instead talking about the bits and pieces of Oracle Fusion Middleware that sit underneath EPM and many other Oracle enterprise applications. So, for example, when they say "Oracle JDeveloper" is impacted, that impacts certain EPM components (HFM and Financial Close Management come to mind) because some of the stuff in your Oracle\Middleware\oracle_common folder contain JDeveloper and is used by some EPM modules.
11.2.5 version installed on Test , As we see there are few references of log4j 1.x and 2.x both versions available. So we are planning to mitigating this by adding this property -Dlog4j2.formatMsgNoLookups=true in setDomainEnv batch file.
ReplyDeleteset WL_HOME=D:\Oracle\Middleware\wlserver
ReplyDeletefor %%i in ("%WL_HOME%") do set WL_HOME=%%~fsi
set BEA_JAVA_HOME=
set DEFAULT_BEA_JAVA_HOME=
-Dlog4j2.formatMsgNoLookups=true
set SUN_JAVA_HOME=D:\Oracle\Middleware\jdk
set DEFAULT_SUN_JAVA_HOME=D:\Oracle\Middleware\jdk
Can we set property as mentioned above before starting Java ?
Hello.
ReplyDeleteThere are two notes in Oracle Support:
CVE-2021-44228 / CVE-2021-45046 Impact On Oracle Essbase (Doc ID 2828787.1)
Apache Log4j Security Alert CVE-2021-44228 also referencing CVE-2021-45046 Mitigation on Oracle Enterprise Performance Management (Doc ID 2828262.1)
Briefly: 11.1.2.4 version is not impacted by these CVEs.
As for 11.2.* version: you should remove org/apache/logging/log4j/core/lookup/JndiLookup.class from all log4j*.jar files.
Hi Vlad. Thank you for the knowledge share! We found the same info over the past few days and I was about to update this post. You beat me to it!
DeleteThings seem to be evoling almost daily, so it is a good idea to re-check Oracle's Knowledge Base for the most up-to-date information.
Hi. Does the Log4j issue affect 11.1.2.3 Hyperion? Yes I realize we need to update asap 😃. But not my decision.
ReplyDeleteThanks