Tuesday, September 14, 2021
Sunday, August 8, 2021
Those of you who've been following my blog for some time know that I love to experiment.
A while back, I had attempted an in-place upgrade within the Hyperion / Oracle EPM series after applying one of the quarterly Oracle Critical Patch Updates. I have at least 2 posts here on that very topic: my original findings, and a 2nd post offering a solution shared by one of our frequent commenters here, Deezel.
The README for EPM 126.96.36.199, the newest EPM Release as of this writing, claims to automatically roll back Middleware patches are part of the in-place upgrade process. I decided to roll the dice without first applying the solution linked above. Mostly, I was just curious to see what would happen!
Well..... this happened:
(Trying to redeploy other web applications, such as Calculation Manager, yields a similar error)
Given this is happening in a throwaway sandbox that isn't customer-facing, I opted not to open a Service Request on this.
Other weird errors encountered during the in-place upgrade:
- EPMSystem11R1\diagnostics\logs\install\ gobbled up 80GB of space due to a runaway 32-bit Oracle DB Client silent install. It ate up all remaining disk space I had available, and I ended up having to forcibly terminate the javaw.exe process for installTool. I also ended up renaming the dbclient32 folder and deleting its entry in the global Oracle Inventory.
- The OHS upgrade failed at 50% completion status with a vague error. I renamed the ohs folder and installTool completed during the 2nd attempt.
- 11.2 still has the Known Issue with respect to cancelling out of configTool; it blew away both setEnv.bat and configTool.bat in the epmsystem1 instance folder.
Saturday, July 31, 2021
The venerable user_projects\EPMINSTANCE\bin\utility.bat thankfully still exists in Hyperion / Oracle EPM 11.2.x. We especially need to use this due to a bug in the user interface when we attempt to LCM export or import the Document Repository (formerly Reporting and Analysis) through the Shared Services Console.
Whether or not you hit the bug depends upon the EPM 11.2 dot release you're using, and also a roll of the dice. The system either points the Document Repository at the correct underlying relational database, or it does not. In the LCM graphical user interface, we get the vague EPMLCM "user not provisioned" or "not available" errors. We'll all seen these before across multiple EPM products & releases. So I'll skip the screenshots and dive right in.
You might not be able to export/import Document Repository through the HSS Console if the system is pointing it to the wrong database. You can inspect this by launching the old "FRConfig" utility and examining the jdbcURL in the mBeans tab.
So when this happens, sometimes we need to use the LCM command-line utility. The Workspace->Explore->File->Import/Export menus still work, but if you have 100s of reports you may be clicking "Overwrite" or "Skip" hundreds of times during the import process.
I noticed a problem in EPM 188.8.131.52 where the LCM export kept throwing hundreds of "user not provisioned" errors behind the scenes in the SharedServices_Security.log back-end log. These errors are happening because the DocRep jdbcurl is pointing to the wrong database. Coupled with this, errors relating to "cannot write to file" are also present....
Because the back-end logs are rotating faster than LCM can keep up!
Here's why, and the fix!
Examine this file on your Foundation server where you're running utility.bat from:
Inspect these sections:
Highlighted above is our first problem. The log rotates once it hits 1MB in size, and only 4 backups are retained. In my testing within a customer's environment, the SharedServices_Security_Client.log was rotating 100s of times, and LCM's back-end log started throwing fits.
Why is it writing to the log so many times? Answer:
I've never been a fan of logging in TRACE by default, unless we're investigating an ongoing problem.
So the 2 screenshots above in combination explain what's going on. Bump up both maxFileSize and maxLogSize for the epmcss-handler by a factor of 10, and then change the logger level for epmcss-handler from TRACE:32 to something like NOTIFICATION:1. The latter will greatly cut down on the noise, and now LCM will be able to keep up with the logs.
Once I did these things, I stopped seeing complaints from the system about "unable to write to file SharedServices_Security.log.321" and such. The underlying bug is still there about the Document Repository jdbcURL connection -- as far as I can tell, it is NOT getting it from the Shared Services Registry database; the HSS Registry database is correct according to the registry report.
Friday, July 30, 2021
Tuesday, July 20, 2021
Hyperion / Oracle EPM 184.108.40.206 is out on Oracle eDelivery today (original date of this post: July 20, 2021... because The Internet Is Forever).
As with all EPM 11.2.x releases, it is very important to carefully crawl through the README before installing it. Especially if you've never installed an EPM 11.2.x release before. 11.2.x is not your granddaddy's Hyperion!
If you're new to my blog, here are a few high-level points to be aware of:
Oracle has provided guidance within their latest EPM 11.2.x README's that, generally speaking, updates will be provided as new .dot releases rather than as PSUs for individual modules. The exceptions to this rule are the Essbase 220.127.116.11 Suite that's used under the covers and also Oracle DRM.
RCU is not your friend. Search the "RCU" tag on my blog. RCU is not optional in 11.2.x and you will suffer if you overlook this crucial step.
Likewise search my blog for the "OHS" tag. You don't get a Windows service for Oracle HTTP Server ("OHS") in 11.2.
There are a few one-off PSEs coming out, but first check the quarterly Critical Patch Update announcements, such as the one that came out today. There are some very severe vulnerability scores addressed in the July 2021 update:
Other one-off patches may come out at Oracle's discretion, so check Oracle's on-premises EPM blog every month to stay aware of what's new:
Back to the topic at hand....
I tend to go through 2-3 iterations when new 11.2 dot releases come out:
- Fresh install on a fresh Windows VM.
- Attempt an in-place upgrade on an older Windows VM.
- Fresh install on a fresh Red Hat Linux VM.
As of this blog post's publication date & time, Hyperion / Oracle EPM 18.104.22.168 is not yet available on Oracle eDelivery for download. Likewise, the Certification Matrix hasn't been updated yet for this new release. I'd expect things to firm up over the coming days.
Update: 22.214.171.124 is now available for download on eDelivery.
The README confirms what the 126.96.36.199 one hinted at, plus we get a few new goodies.
- Removal of Essbase Studio Client. It says this was removed from the system installer (installTool), but I don't know if the underlying MSI for it is also removed entirely.
- Planning REST API new features.
- MS SQL Server 2019 is now certified for 11.2.6. I don't know if they will retroactively go back and certify it for the older 11.2 releases... we'll have to wait and see.
- Improved documentation for database password change procedures. This information is sorely needed, especially where Oracle RCU is concerned! I hope they include RCU in this documentation.
- installTool apparently includes a new option concerning how Oracle Fusion Middleware patches are handled. I'm very curious to see what it will look like.
Sunday, July 18, 2021
I hope by now you've learned about "PrintNightmare", one of the latest of a series of security vulnerabilities to hit Microsoft's Print Spooler service. If you have not, please take the following actions immediately:
- Put "PrintNightmare" into your favorite Internet search engine to learn what it is. This is a very serious exploitable bug that allows an attacker inside your network to elevate their permissions to that of a Domain Administrator. The attacker now "owns" your network at this point and can conduct what I'll call... mischief.
- Back up your systems and apply the latest MS Windows Server patches right away.
- Disable the Print Spooler service on every server running Hyperion / Oracle EPM.