Sunday, January 31, 2021

EPM 11.2.x - Why Did My Web Service Crash During Startup?

So you're working on your shiny new upgrade: Oracle EPM / Hyperion through  (You want to be on at a minimum, as that's the only path to subsequently jump to higher releases as of this writing).

You attempt a first-time start up of web services, and one or more web services shuts itself down.  The problem I'm about to describe existed in and prior, but it still rears its ugly head in 11.2.x from time to time.

Here is the symptom.  For ease of discussion, I'm talking about an MS Windows server, but the problem can happen in Linux as well.  Flip the slashes and change .cmd to .sh.

Inspect Oracle\Middleware\user_projects\EPM_ORACLE_INSTANCE\diagnostics\logs\services\*-sysout.log for the failed process in question, and discover this text:

(Note: This "services" folder does not exist in Linux and you'll need to crawl through the WebLogic domain's log folders)

What to do here?

Answer: the fix is very simple.

1. Start up your WebLogic Admin Server.  There's no service for it (yet! A future post I'll write will discuss the easy way to do it.  Don't use the Knowledge Base article to create "beasvc"!)

For now, double-click Oracle\Middleware\user_projects\domains\EPMSystem\bin\startWeblogic.cmd.  Leave the DOS command window that pops up alone, and wait until you see the text RUNNING in upper-case.  This takes 2-1/2 to 5 minutes, depending upon your server.

This process encrypts the master file for your WebLogic Admin Server.

2. Copy the file Oracle\Middleware\user_projects\domains\EPMSystem\servers\AdminServer\security\ to the Oracle\Middleware\user_projects\domains\EPMSystem\servers\FailedServerName\security\ folder, overwriting the file that might exist in that folder.  Do this while your AdminServer process is still running in that DOS command window.

3. Restart the failed service in question.  Now it sees the encrypted file and is able to communicate with AdminServer.

4. Once you've been able to successfully start all web services (Planning Web, HFM Web, EAS Web, etc.) you may hit Ctrl-C in that AdminServer DOS command window to end the process, if you wish to.

Appendix:  I've stood up every 11.2.x release released thus far:,,,, and now as of this writing.  I've hit the issue described herein multiple times, and most recently in  If you hit this issue yourself - you did nothing wrong during your config.  It just happens sometimes.

Once you fix it, the solution tends to be permanent and I've yet to have to do it a 2nd time in the same WebLogic domain.


  1. Hello Dave,

    As always, thank you so much for updates on latest version 11.2.4.
    We are on 11.2.3(Windows Server 2016) and applied latest Essbase patches (Essbase Server/APS/EAS) Now my question is if I have to upgrade to 11.2.4 on top of 11.2.3 do I have to rollback Essbase patches or can I apply on top of 11.2.3 with Essbase Please help.


    1. Hi Veera,

      installTool in 11.2.4 wants to apply pre-delivered patches for the Essbase suite. I'm doing maintenance on my system right now, but if memory serves it wants to apply Essbase or .033, depending upon the module.

      Run opatch lsinventory for the EPMSystem11R1 Oracle home and check after your upgrade is complete to see what happens. The automatic Essbase Suite patches might get rejected as "not applicable" due to the Oracle Inventory showing a higher patch level exists. It might or might not attempt an automatic rollback.

      Be mindful that 11.2.4 wants to do the same for WebLogic and OHS. Check this if you applied any of the Oracle "Critical Patch Update" quarterly patches.


Thank you very much for your interest in this blog! I hope you're finding it helpful.

Please keep comments relevant to the topic in the post, as this blog is not a free-for-all substitute for Oracle Support or traditional consulting. If you have many questions unrelated to the specific topic at hand, consider contacting me on LinkedIn ( so we may discuss the possibility of consulting.

Commenting on posts older than 90 days unfortunately goes into moderation, thanks to spammers who've been hitting this blog. Please have patience, and thanks for your understanding!

Comments including URLs linking back to gambling or other things unrelated to Oracle EPM will be deleted on sight. If you're an EPM consultant and are offering me constructive criticism or a tip, go ahead and DO link back to your blog or firm's website if you so desire.

Thanks again for reading!