Wednesday, October 27, 2021

EPM 11.2.7 is Available

 EPM 11.2.7 README

The EPM 11.2.x README (linked above) has been updated for 11.2.7.  The software is available on eDelivery for both 64-bit Windows and 64-bit Linux.

I'm in the process of standing up a fresh Windows Server 2019 VM and will share my findings after I get through the Install+Config of EPM 11.2.7.  I will be doing it fresh, rather than as an in-place upgrade from a prior release.  That exercise will come later.

One of the main things I'm interested in is to see how 11.2.7 handles the October 2021 Oracle Critical Patch Update ("CPU").  In particular... WebLogic.  When doing the patch search for the October 2021 CPU, I kept seeing "this patch has been superseded".  Grabbing the latest & greatest WebLogic patch actually bricked my 11.2.6 system and so I had to roll back the patch to get things working again.

Tuesday, September 14, 2021

Essbase Studio Server and Linux

So before I dive in, I need to clear the air a bit.  If you are currently using Essbase Studio, find an alternative.

Essbase Studio was removed in Hyperion / Oracle EPM  You need to figure out a different way to build your Essbase cubes.  As the point of this blog post is not to give free application consulting, I'm going to pass over this particular topic and instead help those who are still using Essbase Studio through on Linux.

There's a little bug with how Essbase Studio Server was packaged for Linux, and I first detected it when doing an upgrade back in the day.  The problem persists in, and I suspect the through releases have the exact same issue.

And here is the root cause and solution!

The symptom:  You run the pre-built start script for Essbase Studio Server on Linux, in your user_projects/epmsystem1/bin/ folder.  The script doesn't complain, but you never see the Java process for Essbase Studio Server start up.  What's going on here?

The root cause: doesn't set the UNIX filesystem permissions correctly for the back-end Essbase Studio Server scripts.

The fix:

cd to your EPMSystem11R1 folder
Type (you will have to tweak this depending upon the variation of UNIX you are running)
find . -name *.sh -exec chmod ug+x {} \;

Done.  You've nuked it from orbit and changed all shell scripts to be executable by the owner and the group.

Sunday, August 8, 2021

11.2.6 Doesn't Fix My Bricked In-Place Upgrade

Those of you who've been following my blog for some time know that I love to experiment.

A while back, I had attempted an in-place upgrade within the Hyperion / Oracle EPM series after applying one of the quarterly Oracle Critical Patch Updates.  I have at least 2 posts here on that very topic: my original findings, and a 2nd post offering a solution shared by one of our frequent commenters here, Deezel.

The README for EPM, the newest EPM Release as of this writing, claims to automatically roll back Middleware patches are part of the in-place upgrade process.  I decided to roll the dice without first applying the solution linked above.  Mostly, I was just curious to see what would happen!

Well..... this happened:

(Trying to redeploy other web applications, such as Calculation Manager, yields a similar error)

Given this is happening in a throwaway sandbox that isn't customer-facing, I opted not to open a Service Request on this.

Other weird errors encountered during the in-place upgrade:

  • EPMSystem11R1\diagnostics\logs\install\ gobbled up 80GB of space due to a runaway 32-bit Oracle DB Client silent install.  It ate up all remaining disk space I had available, and I ended up having to forcibly terminate the javaw.exe process for installTool.  I also ended up renaming the dbclient32 folder and deleting its entry in the global Oracle Inventory.
  • The OHS upgrade failed at 50% completion status with a vague error.  I renamed the ohs folder and installTool completed during the 2nd attempt.
  • 11.2 still has the Known Issue with respect to cancelling out of configTool; it blew away both setEnv.bat and configTool.bat in the epmsystem1 instance folder.
I need to move on to other matters of investigation now, so I destroyed the VM to free up space for the next one.

It was an interesting exercise.  I'm encouraging all my peers to back up the Middleware folder (minus user_projects) prior to applying the quarterly Oracle CPU Middleware patches.  I think I still have some of my hair remaining....

Saturday, July 31, 2021

Hyperion Life Cycle Management Utility in EPM 11.2 - Logging Issue

The venerable user_projects\EPMINSTANCE\bin\utility.bat thankfully still exists in Hyperion / Oracle EPM 11.2.x.  We especially need to use this due to a bug in the user interface when we attempt to LCM export or import the Document Repository (formerly Reporting and Analysis) through the Shared Services Console.

Whether or not you hit the bug depends upon the EPM 11.2 dot release you're using, and also a roll of the dice.  The system either points the Document Repository at the correct underlying relational database, or it does not.  In the LCM graphical user interface, we get the vague EPMLCM "user not provisioned" or "not available" errors.  We'll all seen these before across multiple EPM products & releases.  So I'll skip the screenshots and dive right in.

You might not be able to export/import Document Repository through the HSS Console if the system is pointing it to the wrong database.  You can inspect this by launching the old "FRConfig" utility and examining the jdbcURL in the mBeans tab.

So when this happens, sometimes we need to use the LCM command-line utility.  The Workspace->Explore->File->Import/Export menus still work, but if you have 100s of reports you may be clicking "Overwrite" or "Skip" hundreds of times during the import process.

I noticed a problem in EPM where the LCM export kept throwing hundreds of "user not provisioned" errors behind the scenes in the SharedServices_Security.log back-end log.  These errors are happening because the DocRep jdbcurl is pointing to the wrong database.  Coupled with this, errors relating to "cannot write to file" are also present....

Because the back-end logs are rotating faster than LCM can keep up!

Here's why, and the fix!

Examine this file on your Foundation server where you're running utility.bat from:


Inspect these sections:

Highlighted above is our first problem.  The log rotates once it hits 1MB in size, and only 4 backups are retained.  In my testing within a customer's environment, the SharedServices_Security_Client.log was rotating 100s of times, and LCM's back-end log started throwing fits.

Why is it writing to the log so many times?  Answer:

I've never been a fan of logging in TRACE by default, unless we're investigating an ongoing problem.

So the 2 screenshots above in combination explain what's going on.  Bump up both maxFileSize and maxLogSize for the epmcss-handler by a factor of 10, and then change the logger level for epmcss-handler from TRACE:32 to something like NOTIFICATION:1.  The latter will greatly cut down on the noise, and now LCM will be able to keep up with the logs.

Once I did these things, I stopped seeing complaints from the system about "unable to write to file SharedServices_Security.log.321" and such.  The underlying bug is still there about the Document Repository jdbcURL connection -- as far as I can tell, it is NOT getting it from the Shared Services Registry database; the HSS Registry database is correct according to the registry report.

Friday, July 30, 2021

EAS Console, EPM 11.12.4.x, and Java

One of my colleagues hit the following issue in his Hyperion / Oracle EPM system earlier today, and reached out for help.  We got it fixed in just a few minutes, and here's the fix.

Root cause:  You've just patched your Essbase Suite to, uninstalled your EAS Console and installed the new one.  The EPM servers themselves are still running and an older version of Java.  The issue described herein doesn't apply to EPM 11.2.x.

Symptom:  On a server (we often test directly on the servers), we double-click the EAS Console thick client launcher, and see a message that looks like this:

You are using older java version 1.7.0_***
Install JDK8 and set JAVA_HOME variable to JDK8 installed location

Depending upon the status of your EPM system, that first line will say either 1.6.0_updatelevel or 1.7.0_updatelevel.

I advise against following some of the blogs and tutorials online concerning this issue.  Here's why:

The blogs and tutorials are essentially telling you to use the Windows Control Panel to set a JAVA_HOME variable that points to the parent folder of a Java 8 "\java\bin" folder.

Don't do this on an server! isn't fully certified for Java 8.  Setting a JAVA_HOME variable that's pointing to a Java 8 could spell disaster the next time you start up Hyperion services or run Hyperion tools.  This entirely depends upon how your EPM system is setup.  Some EPM Infrastructure consultants can get... creative.  In some cases, the WebLogic scripts and other EPM software check if a JAVA_HOME variable is already set, and if so, it sets the variable for the process being started to whatever \Oracle\Middleware\EPMSystem11R1\common\config\\setJavaRuntime.bat defines.

I'd need to spin up one of my mothballed sandboxes to see if this situation breaks the system or not.  As I'm focusing exclusively on EPM 11.2.x now, I might not have time for this.  Perhaps others can test and comment below!

I'm also concerned about other software IT or anyone else may have installed on the server.  They might not like a JAVA_HOME pointing to a Java 8.

Quick Solution: Download a Java 8 from Oracle Support.  EAS Console doesn't require a JDK to function, so you can grab your choice of either JRE 8 or JDK 8.  Install it and let it go to the C: drive using the default suggested directory location.

Again, let it install to the C: drive.  Don't be tempted to overlay the jdk in your \Oracle\Middleware folder.

Now launch Windows Explorer and navigate to the Java folder you installed on C.  Copy the complete folder path to your clipboard, such as C:\Program Files\Java\jdk1.8.0_xxx

Now the final step.  Edit the EAS launcher script.  By default it would be 

Look for these lines:

    set PATH=%JAVA_HOME%\bin;%PATH:)=^)%

Immediately before this, add 1 new line like this:

SET JAVA_HOME="C:\Program Files\Java\jdk1.8.0_xxx"
You need those double quotes due to the space in your path.

Save the script and you're ready to rock & roll.

No messing around with global environment variables, potentially interfering with other apps.  Congratulations, you've just fixed your EAS Console!

Addendum: if on a client workstation, be very wary about installing Oracle Java there.  There's a chance one could run afoul of an Oracle Java license compliance audit, especially if the end-user starts using that Oracle Java for non-Oracle products.  As a workaround for this, Red Hat OpenJDK8 is an acceptable solution.  As OpenJDK is open source and legally free to distribute, you won't run into license compliance issues on non-Hyperion servers.  

I'm no lawyer and cannot speculate on how an end-user's workstation would be treated.... the user is accessing a properly-licensed Oracle product, EAS Console, so there ought not be an issue here.  But again, you don't know what'll happen on that workstation after you walk away!

Tuesday, July 20, 2021

EPM is Here!

Hyperion / Oracle EPM is out on Oracle eDelivery today (original date of this post: July 20, 2021... because The Internet Is Forever).

As with all EPM 11.2.x releases, it is very important to carefully crawl through the README before installing it.  Especially if you've never installed an EPM 11.2.x release before.  11.2.x is not your granddaddy's Hyperion!

If you're new to my blog, here are a few high-level points to be aware of:

Oracle has provided guidance within their latest EPM 11.2.x README's that, generally speaking, updates will be provided as new .dot releases rather than as PSUs for individual modules.  The exceptions to this rule are the Essbase Suite that's used under the covers and also Oracle DRM.

RCU is not your friend.  Search the "RCU" tag on my blog.  RCU is not optional in 11.2.x and you will suffer if you overlook this crucial step.

Likewise search my blog for the "OHS" tag.  You don't get a Windows service for Oracle HTTP Server ("OHS") in 11.2.

There are a few one-off PSEs coming out, but first check the quarterly Critical Patch Update announcements, such as the one that came out today.  There are some very severe vulnerability scores addressed in the July 2021 update:

July 2021 Oracle Critical Patch Update

Other one-off patches may come out at Oracle's discretion, so check Oracle's on-premises EPM blog every month to stay aware of what's new:

Oracle on-premises EPM blog

Back to the topic at hand....

I tend to go through 2-3 iterations when new 11.2 dot releases come out:

  1. Fresh install on a fresh Windows VM.
  2. Attempt an in-place upgrade on an older Windows VM.
  3. Fresh install on a fresh Red Hat Linux VM.
Each iteration is actually fascinating and reveals new details about the product as I work through issues I've never encountered before.... and I've been at this since Arbor Essbase 3.2 in the 1990s.

As always, I'll attempt to write a summary post of my findings after I get through at least test case #1 above.

Microsoft Patch Tuesday for July 2021 just finished in my new sandbox, so its time to shutdown and back it up to a new Gold image.  Be well and patch up!

EPM 11.2.6 README is Available


As of this blog post's publication date & time, Hyperion / Oracle EPM is not yet available on Oracle eDelivery for download.  Likewise, the Certification Matrix hasn't been updated yet for this new release.  I'd expect things to firm up over the coming days.

Update: is now available for download on eDelivery.

The README confirms what the one hinted at, plus we get a few new goodies.

  • Removal of Essbase Studio Client.  It says this was removed from the system installer (installTool), but I don't know if the underlying MSI for it is also removed entirely.
  • Planning REST API new features.
  • MS SQL Server 2019 is now certified for 11.2.6.  I don't know if they will retroactively go back and certify it for the older 11.2 releases... we'll have to wait and see.
  • Improved documentation for database password change procedures.  This information is sorely needed, especially where Oracle RCU is concerned!  I hope they include RCU in this documentation.
  • installTool apparently includes a new option concerning how Oracle Fusion Middleware patches are handled.  I'm very curious to see what it will look like.
The next release after this one, which will be roughly 3 months from now, will remove FDMEE's support for Visual Basic Scripting.  Get that stuff converted over to Jython while there's time!