Hyperion Life Cycle Management Utility in EPM 11.2 - Logging Issue

The venerable user_projects\EPMINSTANCE\bin\utility.bat thankfully still exists in Hyperion / Oracle EPM 11.2.x.  We especially need to use this due to a bug in the user interface when we attempt to LCM export or import the Document Repository (formerly Reporting and Analysis) through the Shared Services Console.

Whether or not you hit the bug depends upon the EPM 11.2 dot release you're using, and also a roll of the dice.  The system either points the Document Repository at the correct underlying relational database, or it does not.  In the LCM graphical user interface, we get the vague EPMLCM "user not provisioned" or "not available" errors.  We'll all seen these before across multiple EPM products & releases.  So I'll skip the screenshots and dive right in.

You might not be able to export/import Document Repository through the HSS Console if the system is pointing it to the wrong database.  You can inspect this by launching the old "FRConfig" utility and examining the jdbcURL in the mBeans tab.

So when this happens, sometimes we need to use the LCM command-line utility.  The Workspace->Explore->File->Import/Export menus still work, but if you have 100s of reports you may be clicking "Overwrite" or "Skip" hundreds of times during the import process.

I noticed a problem in EPM 11.2.5.0 where the LCM export kept throwing hundreds of "user not provisioned" errors behind the scenes in the SharedServices_Security.log back-end log.  These errors are happening because the DocRep jdbcurl is pointing to the wrong database.  Coupled with this, errors relating to "cannot write to file" are also present....

Because the back-end logs are rotating faster than LCM can keep up!

Here's why, and the fix!

Examine this file on your Foundation server where you're running utility.bat from:

\Oracle\Middleware\user_projects\epmsystem1\config\FoundationServices\logging.xml

Inspect these sections:

Highlighted above is our first problem.  The log rotates once it hits 1MB in size, and only 4 backups are retained.  In my testing within a customer's environment, the SharedServices_Security_Client.log was rotating 100s of times, and LCM's back-end log started throwing fits.

Why is it writing to the log so many times?  Answer:

I've never been a fan of logging in TRACE by default, unless we're investigating an ongoing problem.

So the 2 screenshots above in combination explain what's going on.  Bump up both maxFileSize and maxLogSize for the epmcss-handler by a factor of 10, and then change the logger level for epmcss-handler from TRACE:32 to something like NOTIFICATION:1.  The latter will greatly cut down on the noise, and now LCM will be able to keep up with the logs.

Once I did these things, I stopped seeing complaints from the system about "unable to write to file SharedServices_Security.log.321" and such.  The underlying bug is still there about the Document Repository jdbcURL connection -- as far as I can tell, it is NOT getting it from the Shared Services Registry database; the HSS Registry database is correct according to the registry report.

EAS Console 11.1.2.4.044, EPM 11.12.4.x, and Java

One of my colleagues hit the following issue in his Hyperion / Oracle EPM 11.1.2.4 system earlier today, and reached out for help.  We got it fixed in just a few minutes, and here's the fix.

Root cause:  You've just patched your Essbase Suite to 11.1.2.4.044, uninstalled your EAS Console and installed the new one.  The EPM servers themselves are still running 11.1.2.4.x and an older version of Java.  The issue described herein doesn't apply to EPM 11.2.x.

Symptom:  On a server (we often test directly on the servers), we double-click the EAS Console thick client launcher, and see a message that looks like this:

You are using older java version 1.7.0_***

Install JDK8 and set JAVA_HOME variable to JDK8 installed location

Depending upon the status of your EPM 11.1.2.4 system, that first line will say either 1.6.0_updatelevel or 1.7.0_updatelevel.

I advise against following some of the blogs and tutorials online concerning this issue.  Here's why:

The blogs and tutorials are essentially telling you to use the Windows Control Panel to set a JAVA_HOME variable that points to the parent folder of a Java 8 "\java\bin" folder.

Don't do this on an 11.1.2.4 server!

11.1.2.4 isn't fully certified for Java 8.  Setting a JAVA_HOME variable that's pointing to a Java 8 could spell disaster the next time you start up Hyperion services or run Hyperion tools.  This entirely depends upon how your EPM system is setup.  Some EPM Infrastructure consultants can get... creative.  In some cases, the WebLogic scripts and other EPM software check if a JAVA_HOME variable is already set, and if so, it sets the variable for the process being started to whatever \Oracle\Middleware\EPMSystem11R1\common\config\11.1.2.0\setJavaRuntime.bat defines.

I'd need to spin up one of my mothballed 11.1.2.4 sandboxes to see if this situation breaks the system or not.  As I'm focusing exclusively on EPM 11.2.x now, I might not have time for this.  Perhaps others can test and comment below!

I'm also concerned about other software IT or anyone else may have installed on the server.  They might not like a JAVA_HOME pointing to a Java 8.

Quick Solution: Download a Java 8 from Oracle Support.  EAS Console doesn't require a JDK to function, so you can grab your choice of either JRE 8 or JDK 8.  Install it and let it go to the C: drive using the default suggested directory location.

Again, let it install to the C: drive.  Don't be tempted to overlay the jdk in your \Oracle\Middleware folder.

Now launch Windows Explorer and navigate to the Java folder you installed on C.  Copy the complete folder path to your clipboard, such as C:\Program Files\Java\jdk1.8.0_xxx

Now the final step.  Edit the EAS launcher script.  By default it would be 

\Oracle\Middleware\EPMSystem11R1\products\Essbase\eas\console\bin\admincon.bat

Look for these lines:

IF DEFINED JAVA_HOME (

    set PATH=%JAVA_HOME%\bin;%PATH:)=^)%

)

Immediately before this, add 1 new line like this:

SET JAVA_HOME="C:\Program Files\Java\jdk1.8.0_xxx"

You need those double quotes due to the space in your path.

Save the script and you're ready to rock & roll.

No messing around with global environment variables, potentially interfering with other apps.  Congratulations, you've just fixed your EAS Console!

Addendum: if on a client workstation, be very wary about installing Oracle Java there.  There's a chance one could run afoul of an Oracle Java license compliance audit, especially if the end-user starts using that Oracle Java for non-Oracle products.  As a workaround for this, Red Hat OpenJDK8 is an acceptable solution.  As OpenJDK is open source and legally free to distribute, you won't run into license compliance issues on non-Hyperion servers.  

I'm no lawyer and cannot speculate on how an end-user's workstation would be treated.... the user is accessing a properly-licensed Oracle product, EAS Console, so there ought not be an issue here.  But again, you don't know what'll happen on that workstation after you walk away!

EPM 11.2.6.0 is Here!

Hyperion / Oracle EPM 11.2.6.0 is out on Oracle eDelivery today (original date of this post: July 20, 2021... because The Internet Is Forever).

As with all EPM 11.2.x releases, it is very important to carefully crawl through the README before installing it.  Especially if you've never installed an EPM 11.2.x release before.  11.2.x is not your granddaddy's Hyperion!

If you're new to my blog, here are a few high-level points to be aware of:

Oracle has provided guidance within their latest EPM 11.2.x README's that, generally speaking, updates will be provided as new .dot releases rather than as PSUs for individual modules.  The exceptions to this rule are the Essbase 11.1.2.4 Suite that's used under the covers and also Oracle DRM.

RCU is not your friend.  Search the "RCU" tag on my blog.  RCU is not optional in 11.2.x and you will suffer if you overlook this crucial step.

Likewise search my blog for the "OHS" tag.  You don't get a Windows service for Oracle HTTP Server ("OHS") in 11.2.

There are a few one-off PSEs coming out, but first check the quarterly Critical Patch Update announcements, such as the one that came out today.  There are some very severe vulnerability scores addressed in the July 2021 update:

July 2021 Oracle Critical Patch Update

Other one-off patches may come out at Oracle's discretion, so check Oracle's on-premises EPM blog every month to stay aware of what's new:

Oracle on-premises EPM blog

Back to the topic at hand....

I tend to go through 2-3 iterations when new 11.2 dot releases come out:

1. Fresh install on a fresh Windows VM.
2. Attempt an in-place upgrade on an older Windows VM.
3. Fresh install on a fresh Red Hat Linux VM.

Each iteration is actually fascinating and reveals new details about the product as I work through issues I've never encountered before.... and I've been at this since Arbor Essbase 3.2 in the 1990s.

As always, I'll attempt to write a summary post of my findings after I get through at least test case #1 above.

Microsoft Patch Tuesday for July 2021 just finished in my new sandbox, so its time to shutdown and back it up to a new Gold image.  Be well and patch up!

EPM 11.2.6 README is Available

Oracle EPM 11.2.6.0 README

As of this blog post's publication date & time, Hyperion / Oracle EPM 11.2.6.0 is not yet available on Oracle eDelivery for download.  Likewise, the Certification Matrix hasn't been updated yet for this new release.  I'd expect things to firm up over the coming days.

Update: 11.2.6.0 is now available for download on eDelivery.

The README confirms what the 11.2.5.0 one hinted at, plus we get a few new goodies.

• Removal of Essbase Studio Client.  It says this was removed from the system installer (installTool), but I don't know if the underlying MSI for it is also removed entirely.
• Planning REST API new features.
• MS SQL Server 2019 is now certified for 11.2.6.  I don't know if they will retroactively go back and certify it for the older 11.2 releases... we'll have to wait and see.
• Improved documentation for database password change procedures.  This information is sorely needed, especially where Oracle RCU is concerned!  I hope they include RCU in this documentation.
• installTool apparently includes a new option concerning how Oracle Fusion Middleware patches are handled.  I'm very curious to see what it will look like.

The next release after this one, which will be 11.2.7.0 roughly 3 months from now, will remove FDMEE's support for Visual Basic Scripting.  Get that stuff converted over to Jython while there's time!

PrintNightmare and Dinosaur Hyperion Versions

I hope by now you've learned about "PrintNightmare", one of the latest of a series of security vulnerabilities to hit Microsoft's Print Spooler service.  If you have not, please take the following actions immediately:

1. Put "PrintNightmare" into your favorite Internet search engine to learn what it is.  This is a very serious exploitable bug that allows an attacker inside your network to elevate their permissions to that of a Domain Administrator.  The attacker now "owns" your network at this point and can conduct what I'll call... mischief.
2. Back up your systems and apply the latest MS Windows Server patches right away.
3. Disable the Print Spooler service on every server running Hyperion / Oracle EPM.

While patching is good, a security expert tells me the only 100% defense against this vulnerability is to stop and disable the Print Spooler on MS Windows servers.

Sadly, there's a catch if you are significantly behind on your Hyperion system upgrades.

If you're on Hyperion / Oracle EPM 11.1.2.1 or older, and you have the old Hyperion BI+ installed and running, chances are you have the 3rd party Ghostscript software installed and the old virtual printer drivers.

I've read that "Unsigned" printer drivers and the July 2021 Microsoft Windows Server patches don't play well together.  A follower on LinkedIn also told me he's been struggling with migrating PrintNightmare on 11.1.2.1.

In the very least, disable the Print Spooler service on the servers that don't run the Hyperion BI+ Print Server service.  Sadly, all it takes, however, is one vulnerable server to be discovered in order for your network to become exploited.

Unfortunately, "zero-day" exploit "proof of concept" (POC) code was accidentally published to GitHub the day PrintNightmare was disabled.  It was redacted not long after, but by then the Git repository was already cloned and the exploit code is out in the wild.  Hackers are furiously working on figuring out how to leverage the POC code.  A security expert tells me this type of exploit is something hackers absolutely love.

So for you remaining dinosaurs out there still on 11.1.2.1 or prior, here's yet another incentive to either upgrade or move to the EPM Cloud.  I can't offer advice more specific than this, as I don't have an 11.1.2.1 sandbox anymore.