So you're working on your shiny new upgrade: Oracle EPM / Hyperion 18.104.22.168 through 22.214.171.124. (You want to be on 126.96.36.199 at a minimum, as that's the only path to subsequently jump to higher releases as of this writing).
You attempt a first-time start up of web services, and one or more web services shuts itself down. The problem I'm about to describe existed in 188.8.131.52 and prior, but it still rears its ugly head in 11.2.x from time to time.
Here is the symptom. For ease of discussion, I'm talking about an MS Windows server, but the problem can happen in Linux as well. Flip the slashes and change .cmd to .sh.
Inspect Oracle\Middleware\user_projects\EPM_ORACLE_INSTANCE\diagnostics\logs\services\*-sysout.log for the failed process in question, and discover this text:
(Note: This "services" folder does not exist in Linux and you'll need to crawl through the WebLogic domain's log folders)
What to do here?
Answer: the fix is very simple.
1. Start up your WebLogic Admin Server. There's no service for it (yet! A future post I'll write will discuss the easy way to do it. Don't use the Knowledge Base article to create "beasvc"!)
For now, double-click Oracle\Middleware\user_projects\domains\EPMSystem\bin\startWeblogic.cmd. Leave the DOS command window that pops up alone, and wait until you see the text RUNNING in upper-case. This takes 2-1/2 to 5 minutes, depending upon your server.
This process encrypts the master boot.properties file for your WebLogic Admin Server.
2. Copy the file Oracle\Middleware\user_projects\domains\EPMSystem\servers\AdminServer\security\boot.properties to the Oracle\Middleware\user_projects\domains\EPMSystem\servers\FailedServerName\security\ folder, overwriting the boot.properties file that might exist in that folder. Do this while your AdminServer process is still running in that DOS command window.
3. Restart the failed service in question. Now it sees the encrypted boot.properties file and is able to communicate with AdminServer.
4. Once you've been able to successfully start all web services (Planning Web, HFM Web, EAS Web, etc.) you may hit Ctrl-C in that AdminServer DOS command window to end the process, if you wish to.
Appendix: I've stood up every 11.2.x release released thus far: 184.108.40.206, 220.127.116.11, 18.104.22.168, 22.214.171.124, and now 126.96.36.199 as of this writing. I've hit the issue described herein multiple times, and most recently in 188.8.131.52. If you hit this issue yourself - you did nothing wrong during your config. It just happens sometimes.
Once you fix it, the solution tends to be permanent and I've yet to have to do it a 2nd time in the same WebLogic domain.
As always, thank you so much for updates on latest version 11.2.4.
We are on 11.2.3(Windows Server 2016) and applied latest Essbase patches (Essbase Server/APS/EAS) 184.108.40.206.042. Now my question is if I have to upgrade to 11.2.4 on top of 11.2.3 do I have to rollback Essbase patches or can I apply on top of 11.2.3 with Essbase 220.127.116.11.042. Please help.
installTool in 11.2.4 wants to apply pre-delivered patches for the 18.104.22.168 Essbase suite. I'm doing maintenance on my system right now, but if memory serves it wants to apply Essbase 22.214.171.124.031 or .033, depending upon the module.
Run opatch lsinventory for the EPMSystem11R1 Oracle home and check after your upgrade is complete to see what happens. The automatic Essbase Suite patches might get rejected as "not applicable" due to the Oracle Inventory showing a higher patch level exists. It might or might not attempt an automatic rollback.
Be mindful that 11.2.4 wants to do the same for WebLogic and OHS. Check this if you applied any of the Oracle "Critical Patch Update" quarterly patches.