EPM 11.2.x - Why Did My Web Service Crash During Startup?

So you're working on your shiny new upgrade: Oracle EPM / Hyperion 11.2.2.0 through 11.2.4.0.  (You want to be on 11.2.2.0 at a minimum, as that's the only path to subsequently jump to higher releases as of this writing).

You attempt a first-time start up of web services, and one or more web services shuts itself down.  The problem I'm about to describe existed in 11.1.2.4 and prior, but it still rears its ugly head in 11.2.x from time to time.

Here is the symptom.  For ease of discussion, I'm talking about an MS Windows server, but the problem can happen in Linux as well.  Flip the slashes and change .cmd to .sh.

Inspect Oracle\Middleware\user_projects\EPM_ORACLE_INSTANCE\diagnostics\logs\services\*-sysout.log for the failed process in question, and discover this text:

(Note: This "services" folder does not exist in Linux and you'll need to crawl through the WebLogic domain's log folders)

What to do here?

Answer: the fix is very simple.

1. Start up your WebLogic Admin Server.  There's no service for it (yet! A future post I'll write will discuss the easy way to do it.  Don't use the Knowledge Base article to create "beasvc"!)

For now, double-click Oracle\Middleware\user_projects\domains\EPMSystem\bin\startWeblogic.cmd.  Leave the DOS command window that pops up alone, and wait until you see the text RUNNING in upper-case.  This takes 2-1/2 to 5 minutes, depending upon your server.

This process encrypts the master boot.properties file for your WebLogic Admin Server.

2. Copy the file Oracle\Middleware\user_projects\domains\EPMSystem\servers\AdminServer\security\boot.properties to the Oracle\Middleware\user_projects\domains\EPMSystem\servers\FailedServerName\security\ folder, overwriting the boot.properties file that might exist in that folder.  Do this while your AdminServer process is still running in that DOS command window.

3. Restart the failed service in question.  Now it sees the encrypted boot.properties file and is able to communicate with AdminServer.

4. Once you've been able to successfully start all web services (Planning Web, HFM Web, EAS Web, etc.) you may hit Ctrl-C in that AdminServer DOS command window to end the process, if you wish to.

Appendix:  I've stood up every 11.2.x release released thus far:  11.2.0.0, 11.2.1.0, 11.2.2.0, 11.2.3.0, and now 11.2.4.0 as of this writing.  I've hit the issue described herein multiple times, and most recently in 11.2.4.0.  If you hit this issue yourself - you did nothing wrong during your config.  It just happens sometimes.

Once you fix it, the solution tends to be permanent and I've yet to have to do it a 2nd time in the same WebLogic domain.

EPM 11.2.4 - Online and Here's Some Findings!

One day after my 11.2.4 Initial Thoughts post, I have a near-full-stack online consisting of everything except for Tax and Cost/Profitability.  (The customers I've worked with who used those modules were a small slice of my customer population, so elected to go for the "biggest bang for the buck" where my  "spare time" is concerned.

(Disclaimer - as of Jan 29, 2021 this post is a rough draft and may contain typos.  I'll fix them over the weekend if any are found)

For the most part, standing up Oracle EPM / Hyperion 11.2.4.0 is exactly the same as Releases 11.2.0.0 through 11.2.3.0.  Meaning:

• Microsoft SQL Server 2016 SP2 only if you're going the Microsoft route, or Oracle RDBMS through 19c.
• Oracle WebLogic 12c is still behind one "dot" release.  Oracle Fusion Middleware ("FMW") 11.2.1.3 versus 11.2.1.4.  The Certification Matrix suggests EPM 11.2.4 could work on FMW 11.2.1.4, as it says FMW "11.2.1.3+".  This may be a permutation I attempt later after all Unit Testing is complete.
• Windows Server 2016 and 2019 and Linux 7 (RHEL and Oracle's only).  Essbase 21c is still limited to Linux 7-8 as of this writing.  
• While Essbase 21c is certified for Linux 7 and 8, Essbase 11.2.4 (really 11.1.2.4 behind the scenes) is certified for Windows 2016-2019 and Linux 7 only.  Pro tip: if you're going to upgrade, don't use Windows 2012.... Essbase 11.2.4 (11.1.2.4) works on it, but 2012 is coming out of support before you will see a budget approved for another upgrade.
• DRM and HFM still restricted to MS Windows servers only.
• RCU hasn't gone away in 11.2.4 and we still need to deal with it.  Click the link to see my posts which mention RCU.
• RCU does not support MS SQL Server when MSSQL is configured for SSL.
• Certain Financial Reporting and EAS Server configurations require manual edits of WebLogic and/or Oracle HTTP Server ("OHS") files.  Instructions are contained within the 11.2.4 Release Notes.
• OHS does not get setup as a Windows service.  The corresponding Linux pre-delivered startup scripts don't start OHS, either.  See my OHS StoreUserConfig post for a reminder on how OHS in 11.2.x (it is OHS 12c in this release) is different from all prior EPM releases going back to over a decade ago.
• Installing OHS under the Foundation node in installTool is deselected by default.  You need to manually select it and then ignore the warning message in the bottom pane about a missing "null" folder.  It installs just fine.
• Planning RMI still does not bind to its port and you have to apply my RMI Java solution.
• The Java JDK that is bundled with EPM 11.2.4.0 is still over one year old: Java 8 update 181.  You need to grab a newer Java 8 from the latest quarterly Oracle Critical Patch Update before your IT department runs a scanner and starts hectoring you about running vulnerable software.

Here are a few ways EPM 11.2.4 is different from 11.2.0.0 through 11.2.1.3:

• "Configure Database" step in configTool moved the checkbox that makes all DB entries the same.  Instead of being at the top left corner, it is now above the DB Name column.  In my testing, clicking that checkbox in an attempt to deselect all rows en-masse did not work.  I had to click each row by hand, so that I could make the database/schema names different.
• "Deploy to Single Managed Server" is still the default.  But now if you deselect it, deploy Foundation only, and then go back to deploy more things, the option is no longer grayed out.  I have not explored this further.  I avoid the "Single Managed Server" architecture and someday I'll write a lengthy blog post explaining my rationale.  I understand why some folks like it.  I don't.  To each their own!
• The pre-delivered validate batch file in EPM_INSTANCE/bin produces an HTML report that begins with "EPM Deployment Topology Report (Release 11.2.3.0.0)".  This is a typo.
• The EPM Workspace home page is back!!!!

A final note before I dive back in and resume my Unit Test.

I said above that the JDK is at least 1 year behind.  A lot of other things are behind as well.  Oracle Middleware (WebLogic, ADF and OHS) and the Essbase Suite.  Here's a quick reference of the patches you get "out of the box" after you finish running installTool.

Oracle/Middleware/EPMSystem11R1 OPatch #s

29260139

30717472

31206874

31206851

31206864

(Look them up and you'll see they're mostly the Essbase Suite 11.1.2.4 and 6 months to a year behind - EAS, Essbase RunTime Client, Essbase Server, etc).  Grab the Essbase Suite patches 11.1.2.4.041 or a higher version you trust, and you may apply them to your 11.2.4.0 system.  Try it and you will see OPatch lets you do it.

Oracle/Middleware/ohs OPatch #s

30687404 

26550599 

26355633

26287183  

26261906 

26051289 

25549931

25360603

25115951

Oracle/Middleware OPatch #s

26626168

30447461

29338031 

27747932

27430060

26355633

26287183

26261906

26051289

"Now wait a minute, Dave?  No oracle_common patches?  No util\bsu WebLogic patches?  You slacker?!?"  Welcome to Oracle Middleware 12c, my friend!  Those things are gone and what I listed in bold above are the -oh parameters you pass to the Opatch utility (add your drive letter of course).

/Oracle/Middleware is the combination of WebLogic and ADF/JDeveloper for Oracle 12c.  OHS and EPM still get their own OPatch folders as before.

I'm not looking up the English descriptions of those patch #s for you, as I've already given away too much for free.  That lookup is very easy to do with an Oracle Support subscription and access to https://support.oracle.com.

Snowed in this week, so I figured I'd make myself useful to the community somehow!

Be well, upgrade to 11.2, and patch up!

January 31, 2021 Update

My earlier remark about being able to change the "Deploy to Single Managed Server" option may have been premature.  I went back into configTool tonight, and the option is now grayed out.  The earlier behavior I experienced, where it was not grayed out, may have been a one-time fluke?

OHS 12c for EPM 11.2.x - StoreUserConfig

Because I know I'll be asked this again many times, here's a reminder on how to avoid being prompted for the WebLogic password when starting Oracle HTTP Server ("OHS") for Oracle EPM / Hyperion 11.2.x.  This applies in EPM 11.2.0.0 through 11.2.4.0 and likely beyond.

One time only, open a command prompt, CD to your EPM Oracle Instance ("epmsystem1" by default), and then navigate to the folder where startcomponent.cmd / startcomponent.sh resides.  This is pictured below for my testing sandbox.

Type:

startcomponent.[cmd|sh] ohs_component StoreUserConfig

This will ask for the WebLogic password (the same password you setup for the WebLogic admin user "epm_admin"), but now it will be saved behind the scenes so you'll never be asked again under the user account you're logged in as.  The "StoreUserConfig" option at the end of the command is what does this for you.

Notice the command tells you the filename where the password was stored.  If you have multiple people managing the system, copy that \.wlst\nm-key-ohs.props file from your home directory structure to theirs.

Having accomplished this, you may now add the startcomponent ohs_component command to your custom start-up script (you do have one, right?) to include OHS in your EPM start-up sequence.

EPM 11.2.4 - Initial Thoughts

It is time yet again for another "Initial Thoughts" post about on-premises Oracle EPM / Hyperion.  This time, the topic is EPM 11.2.4.0, which just came out.

I've completed the download and am about to fire off a clean install & config, but while I'm waiting for my VM to finish patching, I thought I'd share some thoughts from reading through the EPM 11.2.4 Release Notes.

My comments are by no means exhaustive.  I'm just mentioning the things that jump out at me as an EPM Infrastructure guy.  I encourage anyone interested in 11.2.4 to download the Installation document zip, which includes the Release Notes and Install+Config Guide.  The eDelivery product number for this is V1006299-01.

At a high-level view, this release fixes many bugs relating to HFM and FDMEE from 11.2.0.0 through 11.2.3.0.  Aside from bugfixes, I spotted these important features, improvements, and gotchas:

• LCM for Public Sector Planning is now supported to migrate from 11.1.2.4 to 11.2.4.  The Release Notes state this caveat: "Note: You can’t upgrade Public Sector Planning and Budgeting applications with Decision Packages or Budget Requests enabled."
• The Certification Matrix has been updated for 11.2.4.  It explicitly states that the "Core Server" option for MS Windows Server 2019 is not certified, and this makes complete sense to me.  The "Core Server" option means there's no graphical desktop for the server, which would make getting through installTool and configTool problematic.  I used the console version for UNIX back in the day, but the console version is painful to use.
• FDMEE install issues for Linux observed in 11.2.2 and 11.2.3 have been fixed in 11.2.4, provided all required products are installed at the same time on the Linux server.
• Numerous SSL fixes.  If you are using SSL within EPM itself, rather than doing it through an SSL-offloader such as F5, read the Release Notes carefully.
• Numerous FDMEE and HFM bug fixes.  28 issues fixed for FDMEE, and 14 for HFM.  I suggest reading carefully through the Release Notes where these products are concerned.
• SSL for MSSQL is still not supported where RCU is concerned.  No workaround has been published to my knowledge.
• EPM's validate.bat does not recognize all modules in an Apply Update situation. No workaround listed in the Release Notes.
• Tax Governance no longer depends upon Adobe Flash Player, which came out of Adobe support on Dec 31, 2020.  This is a very important update if you use Tax Governance.
• A few fixes for Shared Services and Financial Reporting / Workspace were included.
• HFML preview for Financial Reporting has been fixed, and "Related Content" links now work in Chrome & Edge.
• Linux.  Still limited to RHEL 7 and Oracle Linux 7.  Essbase 21c, on the other hand, is certified for both Releases 7 and 8 for Linux.
• There's a new command-line utility for Hyperion Planning: SetDBNEssbasePassword.  This utility allows you to update the Essbase and relational database passwords for multiple applications all in one shot.  
• I think the database side will work well for MSSQL shops, but most Oracle RDMBS shops I've worked with frequently have DBAs who prefer a unique password for each schema.  So your mileage will vary where this utility is concerned.  You're prompted to enter 1 password for Essbase and 1 password for relational.
• You can run the utility multiple times, specifying the application(s) for each run.

End-users will appreciate that the EPM Workspace home page has returned.  Favorites, recently opened documents, etc. should be visible again, just as we had in 11.1.2.4 and prior.

OK, and with all of the above, my VM's server patching is done and it is time to get to work on 11.2.4!

I intend to start with a pure Windows implementation first.  After this is working, I want to see if I can tack on a Linux instance running Essbase 21c and integrate it with the 11.2.4 system.

Essbase 21 - Initial Thoughts

Oracle Hyperion Essbase 21 came out in December 2020.  I didn't notice it as my Oracle eDelivery searches are always for "EPM".  With the arrival of Essbase 21, we need to search for "Essbase" in eDelivery.

Before attempting to install it, it is very important to read the Essbase 21 Installation Guide

I reviewed the guide over the weekend and wanted to offer my initial thoughts.

• Red Hat 7 and 8, and Oracle Linux 7 and 8 are the only platforms supported.
• Oracle VirtualBox is the only virtualization technique certified.  The usual caveat applies here;  many Hyperion shops use VMWare for virtualization because it is so ubiquitous across IT organizations.  When opening an SR with Oracle, don't tell them you're using VMWare or there's a small chance the SR can be kicked back.
• We're required to install a GNU c/c++ compiler.
• Essbase 21 is supposed to be able to co-habitate with EPM 11.1.2.4 and 11.2.2.x.  I see a potential problem here and I'll get to it in a minute.
• The documentation does not indicate an upgrade path.  Do we start fresh, or can we do an in-place upgrade -- say from 11.2.2.x?
• The installation guide contains a chapter named "Differences Between Essbase 11g and Essbase 21c".  Read this very carefully before getting started.  With respect to features, Oracle giveth and Oracle taketh away.
• We have to run RCU for Essbase 21.  I believe it is because Essbase 21 comes with a new web-based cube editor that is separate from EAS.
• If you use HFM, you will still need a Windows server in the mix.  I would not put FDMEE on the Linux server; I'd instead put it on the Windows server.
• There's some weird stuff in the documentation about Financial Reporting not being supported for HFM if you've got FR installed on the Essbase 21 machine.  So I'd put it on the HFM server.
  

Now let's talk about co-habitating Essbase 21 with EPM 11.2.2 or 11.1.2.4.

Essbase 21 requires that Oracle Fusion Middleware ("FMW") 12.2.1.4 be installed.  EPM 11.2.2 is bundled with FMW 12.2.1.3, so this implies an in-place FMW upgrade if you've already deployed EPM 11.2.2.

EPM 11.1.2.4 is coming out of support at the end of December 2021 (Safe Harbor applies), so I would not consider combining Essbase 21 with EPM 11.1.2.4.  I'm also not a big fan of mixed-release configurations.  I tried it in the past, wrote a white paper for Oracle about it, and Oracle stopped recommending it as far as I can tell.  (9.3.1 mixed with 11.1.1.0 - what a nightmare!!!).

Essbase 21 and FMW 12.2.1.4 want a version of Java 8 that is newer than what comes with EPM 11.2.2.  Java 8 is not certified for EPM 11.1.2.4.  For EPM 11.2.2 shops, you will want to grab the Java 8 patch 18143322.  The patch contains multiple files.  The easiest one to work with is jdk-8u271-linux-i586.tar.gz, because getting it installed doesn't require Linux root access.  Just drop that file where you want, gzip -d it, tar -xf it, and you're done.  The "bin" folder within the jdk will need to be added to your installation user's PATH.

The installation guide for FMW 12.2.1.4 lists additional patches & prerequisites.

At first brush, my reaction is:

1. Lay down the new Java 8 and add it to your PATH.
2. Do a fresh install of FMW 12.2.1.4 in a separate Middleware Home.
3. Install Essbase 21 after meeting the prerequisites (install GNU compiler, etc)
4. Install EPM 11.2.2, minus the Essbase components and anything else Essbase 21 laid down, and pray they all play together.

I've backed up my Linux 7 sandbox and will try to play with this later this week.  Stay tuned!

(1/13/2021 Edit: Fixed "Essbase 12" references so they read "Essbase 21")