Monday, December 27, 2021

Final WebLogic 10.3.6 Patch For EPM 11.1.2.4

A commenter on this blog, Ben, shared how to get the final WebLogic 10.3.6 patch for Hyperion / Oracle EPM 11.1.2.4.  We may download this patch without needing to get a password from Oracle Support.  The patch number is:

33471254

You need to be on an active Oracle Support contract for your EPM 11.1.2.4 in order to access the patch.

Ben also indicated we may pull up an Oracle Knowledge Base article that lists the final patches for various components of the EPM 11.1.2.4 suite.  The KB # is:

2796575.1

When you pull up this KB article, crtl-f in your browser and plug in this text to jump directly to the Hyperion section:

3.3.22 Oracle Hyperion Infrastructure Technology

If your 11.1.2.4 environment includes Essbase, we can expect more Essbase 11.1.2.4 patches to come out in 2022, because (for now) EPM 11.2.x uses Essbase 11.1.2.4 under the covers.

As of EPM 11.2.7.0, Essbase 12c is still not certified for Hyperion Planning.  Once an 11.2.x comes out that certifies Essbase 12c for Planning, I'll take a closer look at it.

As per the 11.2.7.0 README, EPM 11.2.x patches are expected to be released Quarterly according to this rough schedule (Oracle Corporation's Safe Harbor applies here):  January, April, July, and October.

On Oracle's EPM 11.2 documentation hub, click "Essbase" on the left-hand side.  Among other things, we are explicitly told to use Essbase 11.1.2.4:

"Use Essbase 11.1.2.4, which is compatible with Enterprise Performance Management Release 11.2.x"

Consult the 11.2.7.0 Install+Config Guide for more information about using Essbase 12c in an EPM 11.2.5+ environment.

Many of you are on PTO this week, so you might not notice this post until early 2022.

Have a Happy New Year!

Tuesday, December 21, 2021

EPM 11.1.2.x in 2022

As 2021 is rapidly drawing to a close, it is appropriate at this time to shame remind people to either move to the EPM Cloud or upgrade on-premises Hyperion / Oracle EPM to Release 11.2.x.

While the official support documents say EPM 11.1.2.4 is supported through Dec 31, 2021, the truth is it is already out of support.  Good luck getting a new patch for it.  Security patches for EPM 11.1.2.4 effectively ended with the October 2021 Critical Patch Update (OCT2021CPU).

Due to the log4j security headache, we might see some new patches trickle out, but these are targeted toward EPM 11.2.x and things seem to evolve daily (as of this writing).

Multiple people have contacted me about how difficult it is to gain access to the last Oracle WebLogic 10.3.6 security patch.  You CAN get it.  The trick is you have to convince the person working your Oracle SR that, yes, you are paying your support contact, and thus, yes, you are entitled to the WebLogic 10.3.6 patch that is included with your support.

If you are still on 11.1.2.4, or Lord forbid an earlier version, you want to wrangle your 2022 fiscal budget to accommodate an upgrade to 11.2.x.  Your IT department will thank you because you'll be moving off of Java 6 or Java 7 (assuming you did the 6->7 swap earlier in the year).

And if you are still running Hyperion Enterprise 4, I award you no points.




Sunday, December 19, 2021

EPM 11.2.7 is Not Using WebLogic 10.3



Not that it matters, but the report generated by epmsys_registry in Hyperion / Oracle EPM 11.2.7 is juuuust a tad wrong where the version of Oracle Fusion Middleware's WebLogic is concerned.

We're sitting on top of WebLogic 12.2.1.4 and we recently applied the OCT2021CPU cumulative patch update in the environment pictured above.  I'm fairly confident version "10.3.2.0" shown above is Fake News.

The back-end WebLogic logs prove we are, in fact, using 12.2.1.4.

I'm tempted to use epmsys_registry utility to update the version property.  The only thing stopping me is I can hear my executive's voice in my head:  "Don't do it unless Oracle Support certifies it!"

I am, however, very similar to John McClane and break rules all the time.  Die Hard IS a Christmas film!!!!



Saturday, December 11, 2021

Zero-Day log4j Exploit Affects Hyperion

BleepingComputer.com:  log4j Exploit Explained

All modern Hyperion / Oracle EPM systems use the Apache log4j library behind the scenes.  On December 10, 2021, a zero-day exploit was revealed and attackers in the wild are already scanning vulnerable systems.

Systems running Apache log4j 2.0-beta9 up to 2.14.1 are impacted.  

EPM 11.2.x uses log4j 2.13.3, according to Oracle's 3rd Party Acknowledgements document for EPM.  In truth, I'm seeing many log4j versions in my unpatched EPM 11.2.7 sandbox.  I'll want to apply the October 2021 Critical Patch Update in my sandbox to see if this changes before I smash the Panic Button.  A search for log4j*.jar in \Oracle\Middleware reveals the following:














EPM versions 11.1.2.x may use a slightly older version, but it would still be in the range of impacted versions.

The exploit allows the attacker to completely take over a server without needing credentials of any kind, other than finding their way into your network when your EPM system is behind a corporate firewall.  This "get inside the network" scenario is not as far-fetched as it sounds, unless your IT Security folks have implemented 2-factor security on the VPN.  We then get into a "social engineering" discussion, which I won't launch into here.

I can't view the code for Oracle's EPM Cloud, so I can't say for sure if the cloud is impacted.  My gut tells me the EPM Cloud is based upon on-premises EPM technology in some fashion - Apache 2.0 and Oracle WebLogic for sure (why re-invent the wheel?).  If the cloud URL is not protected (many EPM Cloud customers nowadays DO protect their URL so only folks inside their network may hit the URL, which is a GOOD thing), then there may be an issue here.

The BleepingComputer.com link which begins this article offers some suggestions.  One is to patch log4j to log4j 2.15.0.  Based upon my screenshot above, we'd have to do this in multiple places where the version is less than this.  (Which according to my scan, is all of them).  My gut tells me this effort would not be trivial, as we'd have to touch every EPM server having these directories.

A simpler fix may be to apply one of these changes suggested by BleepingComputer:
The flaw can also be mitigated in previous releases (2.10 and later) by setting system property "log4j2.formatMsgNoLookups" to "true" or removing the JndiLookup class from the classpath.
In the case of Windows-hosted EPM systems, we're talking about editing setdomainenv.bat in user_projects\domains\EPMSystem\bin so the WebLogic Admin Server has an additional -D argument for the Java options, and editing the Windows Registry on each Windows server for each Weblogic Managed Server underneath the "Hyperion Solutions" registry hive.  Fun times.  Actually not "simple", but you're not going out and patching multiple directories across multiple servers.

I suspect many of my readers won't have an appetite for this level of "surgery".  The next Oracle Critical Patch Update is due in mid-January 2022, but only for those on EPM 11.2.x.  Those of you still on EPM 11.1.2.x with no immediate plans to upgrade or move to the EPM Cloud may consider the surgery I mentioned above.

Those of us on EPM 11.2.x can hop out to Oracle's support site and search for the various Oracle Fusion Middleware patches.  The patches will likely come out before the next Critical Patch Update is formally announced in mid-January.

Disclaimer: This blog post is speculative and may become dated after the January 2022 Critical Patch Update.  I am not an employee of Oracle Corporation and thus do not speak for them.

Dec 12, 2021 Update:
The Apache Foundation has released a patched Java .jar file for log4j that claims to fix the issue.  I'll try it in my sandbox later today to verify it doesn't brick things.

Wednesday, October 27, 2021

EPM 11.2.7 is Available

 EPM 11.2.7 README

The EPM 11.2.x README (linked above) has been updated for 11.2.7.  The software is available on eDelivery for both 64-bit Windows and 64-bit Linux.

I'm in the process of standing up a fresh Windows Server 2019 VM and will share my findings after I get through the Install+Config of EPM 11.2.7.  I will be doing it fresh, rather than as an in-place upgrade from a prior release.  That exercise will come later.

One of the main things I'm interested in is to see how 11.2.7 handles the October 2021 Oracle Critical Patch Update ("CPU").  In particular... WebLogic.  When doing the patch search for the October 2021 CPU, I kept seeing "this patch has been superseded".  Grabbing the latest & greatest WebLogic patch actually bricked my 11.2.6 system and so I had to roll back the patch to get things working again.

Tuesday, September 14, 2021

Essbase Studio Server and Linux

So before I dive in, I need to clear the air a bit.  If you are currently using Essbase Studio, find an alternative.

Essbase Studio was removed in Hyperion / Oracle EPM 11.2.6.0.  You need to figure out a different way to build your Essbase cubes.  As the point of this blog post is not to give free application consulting, I'm going to pass over this particular topic and instead help those who are still using Essbase Studio 11.1.2.3 through 11.2.5.0 on Linux.

There's a little bug with how Essbase Studio Server was packaged for Linux, and I first detected it when doing an 11.1.2.3 upgrade back in the day.  The problem persists in 11.1.2.4, and I suspect the 11.2.0.0 through 11.2.5.0 releases have the exact same issue.

And here is the root cause and solution!

The symptom:  You run the pre-built start script for Essbase Studio Server on Linux, in your user_projects/epmsystem1/bin/ folder.  The script doesn't complain, but you never see the Java process for Essbase Studio Server start up.  What's going on here?

The root cause:  installtool.sh doesn't set the UNIX filesystem permissions correctly for the back-end Essbase Studio Server scripts.

The fix:

cd to your EPMSystem11R1 folder
Type (you will have to tweak this depending upon the variation of UNIX you are running)
find . -name *.sh -exec chmod ug+x {} \;

Done.  You've nuked it from orbit and changed all shell scripts to be executable by the owner and the group.

Sunday, August 8, 2021

11.2.6 Doesn't Fix My Bricked In-Place Upgrade

Those of you who've been following my blog for some time know that I love to experiment.

A while back, I had attempted an in-place upgrade within the Hyperion / Oracle EPM series after applying one of the quarterly Oracle Critical Patch Updates.  I have at least 2 posts here on that very topic: my original findings, and a 2nd post offering a solution shared by one of our frequent commenters here, Deezel.

The README for EPM 11.2.6.0, the newest EPM Release as of this writing, claims to automatically roll back Middleware patches are part of the in-place upgrade process.  I decided to roll the dice without first applying the solution linked above.  Mostly, I was just curious to see what would happen!

Well..... this happened:












(Trying to redeploy other web applications, such as Calculation Manager, yields a similar error)

Given this is happening in a throwaway sandbox that isn't customer-facing, I opted not to open a Service Request on this.

Other weird errors encountered during the in-place upgrade:

  • EPMSystem11R1\diagnostics\logs\install\ gobbled up 80GB of space due to a runaway 32-bit Oracle DB Client silent install.  It ate up all remaining disk space I had available, and I ended up having to forcibly terminate the javaw.exe process for installTool.  I also ended up renaming the dbclient32 folder and deleting its entry in the global Oracle Inventory.
  • The OHS upgrade failed at 50% completion status with a vague error.  I renamed the ohs folder and installTool completed during the 2nd attempt.
  • 11.2 still has the Known Issue with respect to cancelling out of configTool; it blew away both setEnv.bat and configTool.bat in the epmsystem1 instance folder.
I need to move on to other matters of investigation now, so I destroyed the VM to free up space for the next one.

It was an interesting exercise.  I'm encouraging all my peers to back up the Middleware folder (minus user_projects) prior to applying the quarterly Oracle CPU Middleware patches.  I think I still have some of my hair remaining....


Saturday, July 31, 2021

Hyperion Life Cycle Management Utility in EPM 11.2 - Logging Issue

The venerable user_projects\EPMINSTANCE\bin\utility.bat thankfully still exists in Hyperion / Oracle EPM 11.2.x.  We especially need to use this due to a bug in the user interface when we attempt to LCM export or import the Document Repository (formerly Reporting and Analysis) through the Shared Services Console.

Whether or not you hit the bug depends upon the EPM 11.2 dot release you're using, and also a roll of the dice.  The system either points the Document Repository at the correct underlying relational database, or it does not.  In the LCM graphical user interface, we get the vague EPMLCM "user not provisioned" or "not available" errors.  We'll all seen these before across multiple EPM products & releases.  So I'll skip the screenshots and dive right in.

You might not be able to export/import Document Repository through the HSS Console if the system is pointing it to the wrong database.  You can inspect this by launching the old "FRConfig" utility and examining the jdbcURL in the mBeans tab.

So when this happens, sometimes we need to use the LCM command-line utility.  The Workspace->Explore->File->Import/Export menus still work, but if you have 100s of reports you may be clicking "Overwrite" or "Skip" hundreds of times during the import process.

I noticed a problem in EPM 11.2.5.0 where the LCM export kept throwing hundreds of "user not provisioned" errors behind the scenes in the SharedServices_Security.log back-end log.  These errors are happening because the DocRep jdbcurl is pointing to the wrong database.  Coupled with this, errors relating to "cannot write to file" are also present....

Because the back-end logs are rotating faster than LCM can keep up!

Here's why, and the fix!

Examine this file on your Foundation server where you're running utility.bat from:

\Oracle\Middleware\user_projects\epmsystem1\config\FoundationServices\logging.xml

Inspect these sections:





Highlighted above is our first problem.  The log rotates once it hits 1MB in size, and only 4 backups are retained.  In my testing within a customer's environment, the SharedServices_Security_Client.log was rotating 100s of times, and LCM's back-end log started throwing fits.

Why is it writing to the log so many times?  Answer:





I've never been a fan of logging in TRACE by default, unless we're investigating an ongoing problem.

So the 2 screenshots above in combination explain what's going on.  Bump up both maxFileSize and maxLogSize for the epmcss-handler by a factor of 10, and then change the logger level for epmcss-handler from TRACE:32 to something like NOTIFICATION:1.  The latter will greatly cut down on the noise, and now LCM will be able to keep up with the logs.

Once I did these things, I stopped seeing complaints from the system about "unable to write to file SharedServices_Security.log.321" and such.  The underlying bug is still there about the Document Repository jdbcURL connection -- as far as I can tell, it is NOT getting it from the Shared Services Registry database; the HSS Registry database is correct according to the registry report.

Friday, July 30, 2021

EAS Console 11.1.2.4.044, EPM 11.12.4.x, and Java

One of my colleagues hit the following issue in his Hyperion / Oracle EPM 11.1.2.4 system earlier today, and reached out for help.  We got it fixed in just a few minutes, and here's the fix.

Root cause:  You've just patched your Essbase Suite to 11.1.2.4.044, uninstalled your EAS Console and installed the new one.  The EPM servers themselves are still running 11.1.2.4.x and an older version of Java.  The issue described herein doesn't apply to EPM 11.2.x.

Symptom:  On a server (we often test directly on the servers), we double-click the EAS Console thick client launcher, and see a message that looks like this:

You are using older java version 1.7.0_***
Install JDK8 and set JAVA_HOME variable to JDK8 installed location

Depending upon the status of your EPM 11.1.2.4 system, that first line will say either 1.6.0_updatelevel or 1.7.0_updatelevel.

I advise against following some of the blogs and tutorials online concerning this issue.  Here's why:

The blogs and tutorials are essentially telling you to use the Windows Control Panel to set a JAVA_HOME variable that points to the parent folder of a Java 8 "\java\bin" folder.

Don't do this on an 11.1.2.4 server!

11.1.2.4 isn't fully certified for Java 8.  Setting a JAVA_HOME variable that's pointing to a Java 8 could spell disaster the next time you start up Hyperion services or run Hyperion tools.  This entirely depends upon how your EPM system is setup.  Some EPM Infrastructure consultants can get... creative.  In some cases, the WebLogic scripts and other EPM software check if a JAVA_HOME variable is already set, and if so, it sets the variable for the process being started to whatever \Oracle\Middleware\EPMSystem11R1\common\config\11.1.2.0\setJavaRuntime.bat defines.

I'd need to spin up one of my mothballed 11.1.2.4 sandboxes to see if this situation breaks the system or not.  As I'm focusing exclusively on EPM 11.2.x now, I might not have time for this.  Perhaps others can test and comment below!

I'm also concerned about other software IT or anyone else may have installed on the server.  They might not like a JAVA_HOME pointing to a Java 8.

Quick Solution: Download a Java 8 from Oracle Support.  EAS Console doesn't require a JDK to function, so you can grab your choice of either JRE 8 or JDK 8.  Install it and let it go to the C: drive using the default suggested directory location.

Again, let it install to the C: drive.  Don't be tempted to overlay the jdk in your \Oracle\Middleware folder.

Now launch Windows Explorer and navigate to the Java folder you installed on C.  Copy the complete folder path to your clipboard, such as C:\Program Files\Java\jdk1.8.0_xxx

Now the final step.  Edit the EAS launcher script.  By default it would be 
\Oracle\Middleware\EPMSystem11R1\products\Essbase\eas\console\bin\admincon.bat

Look for these lines:

IF DEFINED JAVA_HOME (
    set PATH=%JAVA_HOME%\bin;%PATH:)=^)%
)

Immediately before this, add 1 new line like this:

SET JAVA_HOME="C:\Program Files\Java\jdk1.8.0_xxx"
You need those double quotes due to the space in your path.

Save the script and you're ready to rock & roll.

No messing around with global environment variables, potentially interfering with other apps.  Congratulations, you've just fixed your EAS Console!


Addendum: if on a client workstation, be very wary about installing Oracle Java there.  There's a chance one could run afoul of an Oracle Java license compliance audit, especially if the end-user starts using that Oracle Java for non-Oracle products.  As a workaround for this, Red Hat OpenJDK8 is an acceptable solution.  As OpenJDK is open source and legally free to distribute, you won't run into license compliance issues on non-Hyperion servers.  

I'm no lawyer and cannot speculate on how an end-user's workstation would be treated.... the user is accessing a properly-licensed Oracle product, EAS Console, so there ought not be an issue here.  But again, you don't know what'll happen on that workstation after you walk away!

Tuesday, July 20, 2021

EPM 11.2.6.0 is Here!

Hyperion / Oracle EPM 11.2.6.0 is out on Oracle eDelivery today (original date of this post: July 20, 2021... because The Internet Is Forever).

As with all EPM 11.2.x releases, it is very important to carefully crawl through the README before installing it.  Especially if you've never installed an EPM 11.2.x release before.  11.2.x is not your granddaddy's Hyperion!

If you're new to my blog, here are a few high-level points to be aware of:

Oracle has provided guidance within their latest EPM 11.2.x README's that, generally speaking, updates will be provided as new .dot releases rather than as PSUs for individual modules.  The exceptions to this rule are the Essbase 11.1.2.4 Suite that's used under the covers and also Oracle DRM.

RCU is not your friend.  Search the "RCU" tag on my blog.  RCU is not optional in 11.2.x and you will suffer if you overlook this crucial step.

Likewise search my blog for the "OHS" tag.  You don't get a Windows service for Oracle HTTP Server ("OHS") in 11.2.

There are a few one-off PSEs coming out, but first check the quarterly Critical Patch Update announcements, such as the one that came out today.  There are some very severe vulnerability scores addressed in the July 2021 update:

July 2021 Oracle Critical Patch Update

Other one-off patches may come out at Oracle's discretion, so check Oracle's on-premises EPM blog every month to stay aware of what's new:

Oracle on-premises EPM blog

Back to the topic at hand....

I tend to go through 2-3 iterations when new 11.2 dot releases come out:

  1. Fresh install on a fresh Windows VM.
  2. Attempt an in-place upgrade on an older Windows VM.
  3. Fresh install on a fresh Red Hat Linux VM.
Each iteration is actually fascinating and reveals new details about the product as I work through issues I've never encountered before.... and I've been at this since Arbor Essbase 3.2 in the 1990s.

As always, I'll attempt to write a summary post of my findings after I get through at least test case #1 above.

Microsoft Patch Tuesday for July 2021 just finished in my new sandbox, so its time to shutdown and back it up to a new Gold image.  Be well and patch up!

EPM 11.2.6 README is Available

Oracle EPM 11.2.6.0 README

As of this blog post's publication date & time, Hyperion / Oracle EPM 11.2.6.0 is not yet available on Oracle eDelivery for download.  Likewise, the Certification Matrix hasn't been updated yet for this new release.  I'd expect things to firm up over the coming days.

Update: 11.2.6.0 is now available for download on eDelivery.

The README confirms what the 11.2.5.0 one hinted at, plus we get a few new goodies.

  • Removal of Essbase Studio Client.  It says this was removed from the system installer (installTool), but I don't know if the underlying MSI for it is also removed entirely.
  • Planning REST API new features.
  • MS SQL Server 2019 is now certified for 11.2.6.  I don't know if they will retroactively go back and certify it for the older 11.2 releases... we'll have to wait and see.
  • Improved documentation for database password change procedures.  This information is sorely needed, especially where Oracle RCU is concerned!  I hope they include RCU in this documentation.
  • installTool apparently includes a new option concerning how Oracle Fusion Middleware patches are handled.  I'm very curious to see what it will look like.
The next release after this one, which will be 11.2.7.0 roughly 3 months from now, will remove FDMEE's support for Visual Basic Scripting.  Get that stuff converted over to Jython while there's time!

Sunday, July 18, 2021

PrintNightmare and Dinosaur Hyperion Versions

I hope by now you've learned about "PrintNightmare", one of the latest of a series of security vulnerabilities to hit Microsoft's Print Spooler service.  If you have not, please take the following actions immediately:

  1. Put "PrintNightmare" into your favorite Internet search engine to learn what it is.  This is a very serious exploitable bug that allows an attacker inside your network to elevate their permissions to that of a Domain Administrator.  The attacker now "owns" your network at this point and can conduct what I'll call... mischief.
  2. Back up your systems and apply the latest MS Windows Server patches right away.
  3. Disable the Print Spooler service on every server running Hyperion / Oracle EPM.
While patching is good, a security expert tells me the only 100% defense against this vulnerability is to stop and disable the Print Spooler on MS Windows servers.

Sadly, there's a catch if you are significantly behind on your Hyperion system upgrades.

If you're on Hyperion / Oracle EPM 11.1.2.1 or older, and you have the old Hyperion BI+ installed and running, chances are you have the 3rd party Ghostscript software installed and the old virtual printer drivers.

I've read that "Unsigned" printer drivers and the July 2021 Microsoft Windows Server patches don't play well together.  A follower on LinkedIn also told me he's been struggling with migrating PrintNightmare on 11.1.2.1.

In the very least, disable the Print Spooler service on the servers that don't run the Hyperion BI+ Print Server service.  Sadly, all it takes, however, is one vulnerable server to be discovered in order for your network to become exploited.

Unfortunately, "zero-day" exploit "proof of concept" (POC) code was accidentally published to GitHub the day PrintNightmare was disabled.  It was redacted not long after, but by then the Git repository was already cloned and the exploit code is out in the wild.  Hackers are furiously working on figuring out how to leverage the POC code.  A security expert tells me this type of exploit is something hackers absolutely love.

So for you remaining dinosaurs out there still on 11.1.2.1 or prior, here's yet another incentive to either upgrade or move to the EPM Cloud.  I can't offer advice more specific than this, as I don't have an 11.1.2.1 sandbox anymore.

Sunday, June 27, 2021

In-Place Upgrade Fix for 11.2.5 After Critical Patch Update

Credit for the following information goes to a commenter on this blog named Deezel.  And also, of course, to Oracle for figuring out the fix!

The issue:

In the Hyperion / Oracle EPM 11.2.x series of releases, Critical Patch Updates are available to fix significant security vulnerabilities in Oracle WebLogic Server 12c and Java 8.

Unfortunately, one of the WebLogic patches require an update to the OPatch utility, and the update cannot be rolled back successfully.  So if you're on EPM 11.2.2.0 through 11.2.4.0, you cannot redeploy your web applications after performing the in-place upgrade to 11.2.5.0.  You get a generic error in configTool and the back-end log error messages are unhelpful.

What follows are Oracle's instructions on how to work around the issue.  Again, a big thank you to commenter Deezel and I look forward to applying the fix in one of my bricked sandboxes.

I cannot emphasize enough the importance of following Oracle's advice by backing up the folders.  Best of luck!

ISSUE SUMMARY:
After Update from 11.2.4/11.2.3 to 11.2.5 deployment of products fail.
RECOVERY STEPS:
The steps listed here are to restore the Updated 11.2.5 instance with deployment failures to working state.
Back Up:
Take back up of EPMSystem11R1 and user_projects folder available under Middleware Home.
Steps:
1 Stop all services
2 Delete all the folders under MWH except EPMSystem11R1, dbclient32,dbclient64 and user_projects
3 Rename the .oracle.products available at EOH\.oracle.product as .oracle.products_11.2.5
4 Rename the .oracle.products0 as .oracle.products
5 Run Apply Update
6 Configure as per Update Guide
7 Start Services
8 Sanity test passes

What's Clobbering My Essbase Log?

Every now and then I need to educate a customer on how OPMN works.  Well, here's some knowledge I'm giving away for free.  (3 CEOs ago hated when I did this)

Here's the symptom of what's causing your on-premises Hyperion / Oracle EPM Essbase.log to keep growing 24x7, even when no end-users are logged in and no MaxL automation is running:

Essbase.log
What's going on here?  Nobody is logged in but the log keeps getting updated once every 20 seconds or so.

This, dear reader, is OPMN.

Oracle Process Management Notification Service.

OPMN is responsible for managing active-passive Essbase clustering.  These "hits" you're seeing in the log is OPMN pinging Essbase to see if it is alive.


"But wait a moment, Dave!  I have just one Essbase server per environment and am definitely not doing Active-Passive Essbase clustering!!!  How do I turn these pings off???"

I'm glad you asked.  Let's investigate the OPMN config file, located here by default:








Make a backup copy of opmn.xml if you've never performed this exercise before!  OPMN is very particular about the XML file, and Essbase will fail to boot up if you commit a typo error in this file.

Now click Search->Find in your favorite text editor (cough cough Notepad++ cough cough) for the word "death".  It is present in the opmn.xml file exactly once:




I have comments to offer on the restart-on-death setting.  We'll get to that in a moment.  But for now, you want to insert a new line immediately after <restart retry="2" timeout="600"/>

Paste exactly this:

<ping timeout="20" interval="0"/>

Bounce the Essbase service (Windows) or use your stopEssbase.sh / startEssbase.sh scripts in Linux.  Done.  Your modified file should look like this:








The line you pasted disables the 20-second ping.  Obviously, if you ARE indeed running Essbase in an Active-Passive cluster configuration, then you need to IGNORE my advice in this case.  That 20-second ping is what allows OPMN to fail over Essbase to the Passive node, if your environment is correctly setup.

Now let's briefly touch upon the restart-on-death setting.

The setting defaults to "true", which I tend not to like.  Why?  Because I prefer to use MaxL to gracefully shutdown the Essbase Agent rather than stopping the service or running stopEssbase.sh.  Stopping the service brings down Essbase unconditionallyAn ESSSVR process that's active - whether it is loading data, running a calc script, or restructuring - is now orphaned and cannot be restarted until the PID is killed.

How I bring down Essbase gracefully and send an alert to the system admins when it doesn't come down gracefully.... well, if you must know you may buy me off with a case of Guinness.

Happy Essbase'ing!

Sunday, June 13, 2021

Additional Thoughts: In-Place Upgrade to 11.2.5

So far I've performed three in-place upgrades to Hyperion / Oracle EPM 11.2.5.  Each scenario was different.  One went very smoothly, one was a trainwreck until I figured out how to fix the damage, and the third can only be repaired by copying files from a working 11.2.5 system.

Scenario One: 11.2.4 (Patched) to 11.2.5

This scenario is beyond repair unless you wipe everything under \Oracle\Middleware, except for the sole exception of \Oracle\Middleware\user_projects\epmsystem1.  Then bring over the Middleware folder from a fresh 11.2.5 install (from a throwaway VM sandbox for example).

As previously discussed on this blog and others on the Internet, when you apply the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware, the WebLogic Server patch requires an update to the OPatch utility that cannot be rolled back at this time.  This botches the entire in-place upgrade to 11.2.5 because you will be unable to redeploy any of the WebLogic Managed Servers.

Scenario Two: 11.2.4 (Not Patched) to 11.2.5

This in-place upgrade went flawlessly.

Scenario Three:  11.1.2.3.7xx to 11.2.5

Seriously, don't attempt this even though Oracle's 11.2.5 README says this upgrade path is certified.  It is possible to repair it if you know the back-end Middleware folder structure extremely well.  But you will spend waaaay more time repairing the system than if you used the swing server approach instead:  Build a fresh 11.1.2.3 system in a VM, with its own database server, patch it up to 11.1.2.3.7xx to exactly match your PROD system, then perform an in-place upgrade to 11.1.2.4, and finally LCM your stuff out and import it into a fresh 11.2.5 system.

Why the in-place upgrade from 11.1.2.3/11.1.2.4 to 11.2.5 is fraught with danger:

Your Oracle HTTP Server ("OHS") folder structure now contains a mixture of OHS 11.1.1.7 and OHS 12c binaries.  OHS refuses to start and you end up spending a lot of time troubleshooting until you discover the error messages (if you run launch.exe from the command line so you may see the error messages) really mean you're deep in what many IT professions call "DLL Hell".

Your domains\EPMSystem\bin folder scripts still refer to Java 6, JRockit 6, or Java 7 depending upon the state of your 11.1.2.x system.  The WebLogic Managed Servers won't start.

Not trusting the upgrade, I blew away everything except for user_projects\epmsystem1, and then re-ran installTool and had it install 11.2.5 fresh.  I then ran configTool and told it to re-use all of my databases, so that the application and Shared Services content was preserved.  I then redeployed everything.  This approach worked and all services, save one, are functional.

I'm still struggling on what to do with the old Reporting & Analysis Framework content.   It did not get converted over to Document Repository and I don't see any back-end utilities that handle this.  Perhaps I'm just overlooking something.

Because of this issue, Document Repository isn't visible in the Shared Services Console.  We also get that odd error message about missing the default document when logging in to EPM Workspace.  If you've encountered this issue have have overcome it, please let me know in the comments!  This one will need to become an Oracle SR.

I could of course reconfigure Financial Reporting and tell it to drop & recreate the database, but then I lose all of my old RA Framework content.  I can't "just LCM it over from the old system" because the customer's RAF repository is too huge to LCM.  I've asked the customer for their assistance in cleaning up old PDFs from years past that aren't needed anymore.  I'd then have to break up the LCM into smaller chunks and pray that 11.2.5 recognizes the content.... I suspect some Notepad++ trickery will be needed to do recursive find-replace operations.

Friday, June 4, 2021

In-Place Upgrade to EPM 11.2.5: OPMN

I have much to say about the in-place upgrade to Hyperion / Oracle EPM 11.2.5.  Anybody who knows me well... knows that I pretty much despise Hyperion in-place upgrades, and moving to 11.2.5 in-place is no exception.  In fact, it may be one of the harder ones.

We've already talked on this blog previously about how applying the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware - OPatch for Weblogic in particular - currently bricks your in-place upgrade from either 11.2.3 or 11.2.4 to 11.2.5.... unless you backed up your Middleware folder prior to patching it.

But this post is about other factors that'll make you want to tear your hair out.

Let's begin with OPMN.

No matter which version of EPM you're upgrading from, if you're upgrading in-place you will need to edit your opmn.xml.  The in-place upgrade doesn't require you to reconfigure Essbase, and that is messy anyway and trying to reconfigure Essbase makes the system think you're trying to add to the Essbase Cluster or create a new one, rather than updating the one that already exists.

One thing I do like about 11.2.5 is the name of the JDK folder is now generic; Oracle finally relented and removed the reference to the version and update numbers in the folder's name.  It is now simply \Oracle\Middleware\jdk instead of jdk1.8.0_181 or jdk160_35.  But because of this, you'll need to do some surgery.

Edit opmn.xml, find the reference to the jdk folder as shown in my screenshot above, and wipe out the version and update numbers so it is exactly DRIVELETTER:\Oracle\Middleware\jdk\jre\bin\server

The 11.2.5 web version of the README indicates an in-place upgrade on top of either 11.1.2.3 or 11.1.2.4 directly to 11.2.5.0 is "certified".  Um... no.  Caesar says "thumbs down".


I've been working my way through an in-place update to 11.2.5 for a customer who lagged behind and is still on 11.1.2.3.7xx.  I really cannot recommend this approach, despite the README's claim that it is certified.  I finally got OHS, Essbase, Planning RMI, and the WebLogic managed servers online.  This upgrade path is fraught with multiple problems and should only be attempted by EPM Infrastructure experts.  

Look forward to a more detailed post on this 11.1.2.3 -> 11.2.5.0 in-place upgrade -- problems & solutions, after the dust settles.  I have one last problem to solve, and this is: why the heck did the upgrade not convert the RA Framework content over to Document Repository?  Massaging LCM files isn't the answer as their RA Framework repository is too massive for LCM to handle all in one shot.  Ugh!


Tuesday, May 18, 2021

Oracle WebLogic Admin Server - as a Windows service made EASY


"BeaService", how did I beat you?


If you have never heard about or read about NSSM before, READ ON!

When I record installation sessions for Hyperion / Oracle EPM, I tend to pause the recording when it is time to deploy NSSM to get Oracle WebLogic Admin Server setup as a Windows service.

Because NSSM is an acronym that is not entirely nice.  I won't repeat it here.  But you NEED this tool if you don't have it!  Grab it here:  NSSM download

NSSM lets you create a Windows service for any executable .bat or .exe.  It does not work for OHS 12c and I have a separate post about that topic.  But it works wonderfully for the Oracle WebLogic Admin Server ("WLS") in either 10.x or 12.x.

Simply download NSSM and unzip it in a directory of your choice.  That directory should not be deleted later.  I save my copy in D:\Scripts\NSSM on my primary EPM Foundation Server where the master version of WLS resides.

The beauty of this utility is:

  • You can customize the name of the service, as in my first screenshot above.
  • You can make it rotate the log files.
  • You can force it to make the stdout and stderr logs reside in the same folder as your EPM service stdout/stderr logs!

I won't post the spoiler screenshots or else you won't hire me.  😈  But maybe I gave you a few helpful hints here?

Sunday, May 16, 2021

Patch 30695700 - Hyperion EPM Architect to DRM Conversion Utility

This post is going to be more painful than the last one I just wrote.

Suppose you are on Hyperion / Oracle EPM 11.1.2.4 or prior, and your metadata is under EPMA's control.  Further suppose you have many applications and converting them all to Classic Mode doesn't suit your business requirements.

You essentially have 2 alternatives to Classic Mode in EPM 11.2.x due to the elimination of EPMA.

1. Switch to the Oracle DRM Limited Use license.
2. Contact Abhi at EPMWare.com.

That's pretty much it.  If you, my dear reader, have a 3rd option I have not considered for on-premises EPM, please by all means share your solution in the comments!  With upgrades heating up in Fiscal 2021, this topic is going to come up over and over!

What the above out of the way, I'm going to focus specifically on option #1.

There is a patch numbered 30695700.  It is associated with DRM 11.1.2.4 (don't ask me why, as it really applies to an EPM 11.2.x implementation).

This is not a true "Opatch" opatch.  It is a standalone Java utility, and the Java source code is included.

Halt.  Stop right there.  Do not pass Go.  Do not collect $200 USD.

Depending upon which version of EPMA you're coming from, and also depending upon how your multi-line Essbase block storage member formulas are written, the utility crashes within seconds.

Sadly, the README delivered with patch 30695700 contains some rather.... odd... language for a non-certified and unsupported utility.

After the README states the utility is not certified by Oracle and is also unsupported, it includes a prohibition from sharing code modifications/enhancements.

So, I know exactly where in the code the crash issue happens, but sadly I am prohibited from sharing the solution.  I really believe the policy needs to be re-examined by Oracle Support so the EPM community may share knowledge with each other!

Suffice to say, a single-line member formula shouldn't have any problems during the conversion process.  The problem is when a multi-line formula contains empty lines (carriage returns for readability purposes) and the EPMA File Generator _ line continuation character doesn't appear where the conversion utility expects to find it.  That's about all I can say for now.

Quick Tip: Starting OHS 12 in a distributed Hyperion environment

This ought to be a simpler post than what I usually write.

The topic of how to start Oracle HTTP Server ("OHS") 12c in Hyperion / Oracle EPM 11.2.x comes up frequently.  I've previously posted about the little trick, buried within the EPM 11.2.x install/config guide, on how to save the WebLogic password for OHS so you aren't prompted every time you try to start up OHS.

"But hey! Dave, how do we start and stop OHS in a distributed MS Windows Server environment when OHS doesn't have its own Windows service anymore?  The good old NSSM trick we use for WebLogic Admin Server doesn't work for OHS!"

An excellent question.

Login to each OHS host server in your distributed environment as your EPM network service account.  Run the "storeuserconfig" trick to save epm_admin's password in your network service account's profile on the C drive.  Search the install/config guide for the text "storeuserconfig" and it'll take you directly to the needed command syntax.  Every member on your Hyperion support team who wants to start/stop services using their own distinct userID needs to perform this step.  This is a one-time-only step until such time as your security team forces you to change the WebLogic epm_admin password.

Now on your OHS server(s) foreign to where you run your master start/stop scripts from (you are running custom start/stop scripts, yes?), create a simple little DOS batch script like this:

sc start "Oracle Weblogic ohs NodeManager (D_Oracle_Middleware_ohs_wlserver)"  

start "StartOHS" /wait /i cmd /c %EPM_INSTANCE_HOME%\httpConfig\ohs\bin\startComponent.cmd ohs_component

Replace D_Oracle in that first command with the drive letter where you installed.  e.g. E:_Oracle, F:_Oracle or whatever is appropriate.  The drive letter is unfortunately embedded within the Oracle Node Manager 12c Windows service name.

Replace %EPM_INSTANCE_HOME% with the fully qualified path to your Oracle EPM Instance, which is DRIVE:\Oracle\Middleware\user_projects\epmsystem1 by default.

Note: the final line beginning with start "StartOHS" is all one line all the way through ohs_component.  Blogger is linewrapping it in an undesirable way.

Save this script as DRIVE:\scripts\startohs.bat or whatever you want.

Replace startComponnent.cmd with stopComponent.cmd and now you have your stopohs.bat script as well!

Finally, use this to invoke it remotely from your master start/stop scripts:

wmic /node:%FDNHOST2% process call create "DRIVE:\Scripts\startohs.bat"

wmic is a built-in Windows command.  Provided your userID is an administrator on the foreign server, you may use this syntax to invoke any batch file, command, or executable you wish.

Replace the variable %FDNHOST2% with the hostname of your 2nd OHS server that you're clustering for load balancing.  Repeat as needed if you have more than 2 OHS servers.

Simple & easy.  You're welcome!  👍

Wednesday, May 12, 2021

In-Place Upgrade to EPM 11.2.5.0

The comments section for my prior post about EPM 11.2.5.0 has really lit up with a common theme:

An in-place upgrade from either Hyperion / Oracle EPM 11.2.3.0 or 11.2.4.0 (the only certified paths to get to 11.2.5.0) fails if one applied the OPatch update necessary to subsequently apply the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware - most notably WebLogic and OHS.

The commenter community seems to be in agreement that the best approaches are:
  • If Oracle Middleware, Opatch in particular, has yet to be patched, back it up!  If disk space is available, back up \Oracle\Middleware in its entirety before applying the quarterly Cracle Critical Patch Updates.
  • If the system is unpatched, it is apparently safe to perform the in-place upgrade to 11.2.5.0.
  • If the system was patched, it will be necessary to restore \Oracle\Middleware\Opatch and perhaps \Oracle\Middleware\wlserver and \Oracle\Middlware\oracle_common.  The jury is out on exactly where the bug resides with respect to patching.  All that is known for certain is Opatch is a primary culprit and the Opatch update cannot be rolled back without performing a filesystem restore for the Opatch folder structure.
Additionally, I experienced issues with missing jar files where ODI 12.2.1.4 and FDMEE 11.2.5.0 is concerned.  I have yet to determine if this is due to the issue mentioned above....  an EPM 11.2 customer-facing upgrade that just got handed over from me to the UAT team has eaten up all of my "free time".

I've installed a fresh VM and a fresh/unpatched EPM 11.2.4.0 and will perform the in-place upgrade to 11.2.5.0 to vet if ODI/FDMEE is truly broken or not in this release.  The EPM Configurator tool does not reveal the problem when FDMEE is broken, but there's a way I can see it on the back-end.

Finally, time permitting, I'll take my unpatched 11.2.4.0 Middleware backup, restore it to my botched/patched 11.2.4.0->11.2.5.0 in-place upgrade VM and see if I can put Humpty Dumpty back together again.

And here we thought RCU was difficult....

Tuesday, April 27, 2021

EPM 11.2.5 - My Week is Full

We finally get Oracle Fusion Middleware (FMW) 12.2.1.4!

SQL Server 2017!

Essbase 21c certified!

The bad news:

If you're still using Essbase Studio, migrate away from it.  Essbase Studio is slated to be retired in EPM 11.2.6 as per this Oracle Statement Of Direction:  https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=81935405509356&id=2731169.1&_afrWindowMode=0&_adf.ctrl-state=rbcbyuw9y_4

Visual Basic scripts for FDMEE need to be migrated to Jython ASAP.  VB scripting is expected to be decertified in EPM 11.2.7.  We knew this some time ago, but now we now the release number where VB goes away for FDMEE.

Interesting news:

Oracle's readme for 11.2.5 confirms what I've suspected.  In 11.2.x we aren't getting Patch Set Updates (PSUs), but rather new "dot" releases.  Here's a copy/paste directly from the 11.2.5 readme:

EPM System updates are released on a quarterly basis, generally in January,
April, July, and October.

You can directly apply an update from the previous two updates.

I won't paste the entire thing.  When you download 11.2.5 from Oracle eDelivery, you get this folder:

EPM_11.2.5\EPM 11.2.5.0.000\EPM System Installation Documentation\EPM System Installation

Stop, open this folder.  Open the readme before doing anything.  Here's the Cliff Notes.  You can apply 11.2.5 as an in-place upgrade on top of either 11.2.3 or 11.2.4.  Do not attempt to apply it on top of 11.1.2.4 or any older release; you need to do a fresh install and LCM your apps from the older release.  Only 11.1.2.4 is certified for this migration path.

If you were an early adopter like me, you need to jump to 11.2.2 before you can get to 11.2.3 or higher.  You can apply 11.2.2 on top of either 11.2.0 or 11.2.1.  You can't get to 11.2.3, .4 or .5 unless you're on at least .2.  Job security!

Migration Path documentation:  11.1.2.4.x to 11.2.x

If you're on Financial Close Management 11.1.2.3 or older, prepare for pain.  I've done it and it is not fun.  :(

It is now time to back up my sandbox and attempt the upgrade.  Wish me luck!


Wednesday, March 31, 2021

EPM 11.2, Visual Studio, and SSIS Packages

This is a bit of a weird one, and it isn't Oracle's fault at all.  This is actually a Microsoft issue.

Suppose you are migrating from Hyperion / Oracle EPM 11.1.2.4 to EPM 11.2.x.  Further suppose you have custom data integrations in FDMEE/ODI or MaxL that rely upon SSIS (SQL Server Integration Services) automation.

Your EPM 11.2.x system would be using MS SQL Server 2016 SP2+ if you want to use SSIS.  I checked the certification matrix last week and Oracle still hasn't certified MSSQL 2019 for EPM 11.2.x.

(Disclaimer: the above paragraph may become be dated as Oracle might certified MSSQL 2019 in the future. The Internet is forever!)

Now here's the fun part.

To migrate your SSIS packages to MSSQL 2016, you will need to first import and upgrade them via MS Visual Studio.  The version available to download is Visual Studio 2019.

Once upgraded, you use MS SQL Management Console to import them into SSIS.  The version of this software as of this writing is 18.8.

This combination of software do not play well with each other, and you'll get an error message in MS SQL Console when you try to work with the upgraded packages.  This exact error and solution has been written about on many blogs.  I'll just jump straight to the solution.

You'll need to uninstall MS SQL Management Console 18.8, and then grab version 16.x from Microsoft's download page.  There's a link there that lets you download older versions, and 16.x is still available as of this writing.  Version 17.x does not solve our issue... we have to go all the way down to 16.x.

You may then complete your migration in MS SQL Console 16.x.

One final tip: if your custom integrations are using Microsoft's dtexec utility, you'll need to replace the reference in the script's path to dtexec from 110 (SQL Server 2012) to 130 (SQL Server 2016).

Saturday, March 13, 2021

Latest Windows Update Removes Abobe Flash

This is a follow-up about Adobe Flash Player EOL and On-Premises EPM 11.1.2.x.

As you may have heard, Adobe ended support for Flash Player on Jan 1, 2021.  This date has been known for some time, and as a result, Oracle released patches for EPM 11.1.2.4 in Summer 2020 that removes Flash dependency from Hyperion Planning, Hyperion Calculation Manager, and the Financial Close suite.

Some Oracle EPM / Hyperion customers may have deferred patching or upgrading.  Perhaps the thinking was "we can just leave it on end-user workstations as-is even though it is out of support."

Microsoft says "Not so fast!"


If you're on EPM 11.1.2.4 and are using any of the 3 modules I mentioned above, please apply the patches right away.  They're listed in my link above, although newer ones might be available by  now.

If you're on EPM 11.1.2.3.500+, you need to upgrade.

As mentioned in my earlier post, Adobe removed the Flash Player download from their website.  They further stated that downloading it from any 3rd party site is a license violation.

The good news is EPM 11.2.x has the fix built-in.  As does the EPM Cloud.

Thursday, March 11, 2021

FDMEE 11.2 - Java 6 reference can be buried within the database

Depending upon you migrate FDMEE from Hyperion / Oracle EPM 11.1.2.4 to EPM 11.2.x, you may have an issue when trying to run your load jobs.

In 11.2, Oracle included a handy migration package specifically to migrate the AIF* tables from 11.1.2.4 to 11.2.  You'll find it on your 11.2 FDMEE server in \Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\database\migrate.  There's a package each for MSSQL and Oracle - check the subfolders for the packages and instructions.

As per Oracle's documentation, you want to run this in 11.2 after you've executed "Configure Database" for FDMEE.  This because the migration package only migrates the AIF* (FDMEE) tables, and does not migrate the ODI-specific tables (SMP* etc.).

Now let's say you're in 11.2's Data Management and you're trying to run an import job.  It fails.  You check the log and see a reference to JAVA_HOME=%EPM_ORACLE_HOME%\..\jdk160_35.

This simply won't do!  Don't be tempted to just copy jdk160_35 from your 11.1.2.4 server to your 11.2.  Java SE 6 is out of support and won't be patched anymore.

You can run a simple SQL statement to fix the problem.  Here's how it looks in MSSQL, and Oracle's version of this would be the same:

Bounce your 11.2 FDMEE service(s) just to be on the safe side, and now when you go back in to Data Management, you won't see that reference to jdk160_35 in your process log anymore.


EPM 11.2 - Your environment has been set

All releases of Hyperion / Oracle EPM 11.2.0.0 through 11.2.4.0 hosted on MS Windows Server do the following:

While this doesn't hurt anything, if you want to clean things do, do the following after you've finished deploying all of your WebLogic services.

Open regedit and export the Hyperion Solutions registry hive to your desktop.

Open this file in notepad/Notepad++ and do a Search->Replace.

Change Your environment has been set.; to nothing.  Save the modified .reg file.

Double-click the .reg file to import it back into your registry.

Do this on every server.


Friday, February 19, 2021

OHS 12c Startup Script for EPM 11.2

The following assumes your OHS is hosted in MS Windows.  An equivalent bash shell script may be easily created to do the same in Linux/Exalytics.  The below assumes you have already executed the OHS storeUserConfig process previously documented on this blog.


SET CWD=%~dp0

REM This script sets up our environment variables.

CALL %CWD%ScriptEnv.bat

sc start "Oracle Weblogic ohs NodeManager (F_Oracle_Middleware_ohs_wlserver)"

start "StartOHS" /wait /i cmd /c %EPM_INSTANCE_HOME%\httpConfig\ohs\bin\startComponent.cmd ohs_component


You're welcome.

Change the "F" in red above to be your EPM system's drive letter.  The variable EPM_INSTANCE_HOME is the path to the instance, such as F:\Oracle\Middleware\user_projects\epmsystem1 and I define it in my custom ScriptEnv.bat.

Tuesday, February 9, 2021

Oracle EPM 11.2.x and Essbase Thick Client

Here's an intellectual exercise for my followers.

Suppose you've constructed a brand new Oracle EPM / Hyperion 11.2.x system (11.2.0.0 through 11.2.4.0 as of this writing).  Now also suppose you have a need to automate Essbase MaxL scripts... let's say for example an Essbase nightly level-0 backup, or an FDMEE nightly data load automation.

You run the Essbase Client installer, and MaxL works just fine.

Except the Essbase Client is 11.1.2.4 technology and you've just corrupted one small element of your 11.2.x system.

Trying launching the configTool graphical client.  The javaw.exe process crashes immediately and you never see the Oracle EPM Config logo pop up.

What to do?

Oracle does have a fix for this documented within their Knowledge Base.  You just need to know where to look.

Or, search for files dated in the year 2014 A.D.  Open the Windows Explorer and put in a date range of 1/1/2014 through 12/31/2014 as you see in my screenshot below.

 

Don't bother with the html files.  What you care about are the EXE and JAR file extensions.  The act of running EssbaseClient.exe causes the 2020-2021 versions of these files to be blown away by the 2014 versions.

The fix: stop everything.  Replace the 2014 files from a server where you did NOT run EssbaseClient.exe to install Essbase MaxL.  Then launch configTool and confirm it works again.  Customers who have security policies mandating database & WebLogic password changes every 90 days will need a working configTool utility.

Try this out in a DEV environment.  I like Ahi Tuna sashimi with a bit of soy sauce and wasabi, and thus you can thank me with that.  ;)