Tuesday, September 14, 2021
Sunday, August 8, 2021
Those of you who've been following my blog for some time know that I love to experiment.
A while back, I had attempted an in-place upgrade within the Hyperion / Oracle EPM series after applying one of the quarterly Oracle Critical Patch Updates. I have at least 2 posts here on that very topic: my original findings, and a 2nd post offering a solution shared by one of our frequent commenters here, Deezel.
The README for EPM 22.214.171.124, the newest EPM Release as of this writing, claims to automatically roll back Middleware patches are part of the in-place upgrade process. I decided to roll the dice without first applying the solution linked above. Mostly, I was just curious to see what would happen!
Well..... this happened:
(Trying to redeploy other web applications, such as Calculation Manager, yields a similar error)
Given this is happening in a throwaway sandbox that isn't customer-facing, I opted not to open a Service Request on this.
Other weird errors encountered during the in-place upgrade:
- EPMSystem11R1\diagnostics\logs\install\ gobbled up 80GB of space due to a runaway 32-bit Oracle DB Client silent install. It ate up all remaining disk space I had available, and I ended up having to forcibly terminate the javaw.exe process for installTool. I also ended up renaming the dbclient32 folder and deleting its entry in the global Oracle Inventory.
- The OHS upgrade failed at 50% completion status with a vague error. I renamed the ohs folder and installTool completed during the 2nd attempt.
- 11.2 still has the Known Issue with respect to cancelling out of configTool; it blew away both setEnv.bat and configTool.bat in the epmsystem1 instance folder.
Saturday, July 31, 2021
The venerable user_projects\EPMINSTANCE\bin\utility.bat thankfully still exists in Hyperion / Oracle EPM 11.2.x. We especially need to use this due to a bug in the user interface when we attempt to LCM export or import the Document Repository (formerly Reporting and Analysis) through the Shared Services Console.
Whether or not you hit the bug depends upon the EPM 11.2 dot release you're using, and also a roll of the dice. The system either points the Document Repository at the correct underlying relational database, or it does not. In the LCM graphical user interface, we get the vague EPMLCM "user not provisioned" or "not available" errors. We'll all seen these before across multiple EPM products & releases. So I'll skip the screenshots and dive right in.
You might not be able to export/import Document Repository through the HSS Console if the system is pointing it to the wrong database. You can inspect this by launching the old "FRConfig" utility and examining the jdbcURL in the mBeans tab.
So when this happens, sometimes we need to use the LCM command-line utility. The Workspace->Explore->File->Import/Export menus still work, but if you have 100s of reports you may be clicking "Overwrite" or "Skip" hundreds of times during the import process.
I noticed a problem in EPM 126.96.36.199 where the LCM export kept throwing hundreds of "user not provisioned" errors behind the scenes in the SharedServices_Security.log back-end log. These errors are happening because the DocRep jdbcurl is pointing to the wrong database. Coupled with this, errors relating to "cannot write to file" are also present....
Because the back-end logs are rotating faster than LCM can keep up!
Here's why, and the fix!
Examine this file on your Foundation server where you're running utility.bat from:
Inspect these sections:
Highlighted above is our first problem. The log rotates once it hits 1MB in size, and only 4 backups are retained. In my testing within a customer's environment, the SharedServices_Security_Client.log was rotating 100s of times, and LCM's back-end log started throwing fits.
Why is it writing to the log so many times? Answer:
I've never been a fan of logging in TRACE by default, unless we're investigating an ongoing problem.
So the 2 screenshots above in combination explain what's going on. Bump up both maxFileSize and maxLogSize for the epmcss-handler by a factor of 10, and then change the logger level for epmcss-handler from TRACE:32 to something like NOTIFICATION:1. The latter will greatly cut down on the noise, and now LCM will be able to keep up with the logs.
Once I did these things, I stopped seeing complaints from the system about "unable to write to file SharedServices_Security.log.321" and such. The underlying bug is still there about the Document Repository jdbcURL connection -- as far as I can tell, it is NOT getting it from the Shared Services Registry database; the HSS Registry database is correct according to the registry report.
Friday, July 30, 2021
Tuesday, July 20, 2021
Hyperion / Oracle EPM 188.8.131.52 is out on Oracle eDelivery today (original date of this post: July 20, 2021... because The Internet Is Forever).
As with all EPM 11.2.x releases, it is very important to carefully crawl through the README before installing it. Especially if you've never installed an EPM 11.2.x release before. 11.2.x is not your granddaddy's Hyperion!
If you're new to my blog, here are a few high-level points to be aware of:
Oracle has provided guidance within their latest EPM 11.2.x README's that, generally speaking, updates will be provided as new .dot releases rather than as PSUs for individual modules. The exceptions to this rule are the Essbase 184.108.40.206 Suite that's used under the covers and also Oracle DRM.
RCU is not your friend. Search the "RCU" tag on my blog. RCU is not optional in 11.2.x and you will suffer if you overlook this crucial step.
Likewise search my blog for the "OHS" tag. You don't get a Windows service for Oracle HTTP Server ("OHS") in 11.2.
There are a few one-off PSEs coming out, but first check the quarterly Critical Patch Update announcements, such as the one that came out today. There are some very severe vulnerability scores addressed in the July 2021 update:
Other one-off patches may come out at Oracle's discretion, so check Oracle's on-premises EPM blog every month to stay aware of what's new:
Back to the topic at hand....
I tend to go through 2-3 iterations when new 11.2 dot releases come out:
- Fresh install on a fresh Windows VM.
- Attempt an in-place upgrade on an older Windows VM.
- Fresh install on a fresh Red Hat Linux VM.
As of this blog post's publication date & time, Hyperion / Oracle EPM 220.127.116.11 is not yet available on Oracle eDelivery for download. Likewise, the Certification Matrix hasn't been updated yet for this new release. I'd expect things to firm up over the coming days.
Update: 18.104.22.168 is now available for download on eDelivery.
The README confirms what the 22.214.171.124 one hinted at, plus we get a few new goodies.
- Removal of Essbase Studio Client. It says this was removed from the system installer (installTool), but I don't know if the underlying MSI for it is also removed entirely.
- Planning REST API new features.
- MS SQL Server 2019 is now certified for 11.2.6. I don't know if they will retroactively go back and certify it for the older 11.2 releases... we'll have to wait and see.
- Improved documentation for database password change procedures. This information is sorely needed, especially where Oracle RCU is concerned! I hope they include RCU in this documentation.
- installTool apparently includes a new option concerning how Oracle Fusion Middleware patches are handled. I'm very curious to see what it will look like.
Sunday, July 18, 2021
I hope by now you've learned about "PrintNightmare", one of the latest of a series of security vulnerabilities to hit Microsoft's Print Spooler service. If you have not, please take the following actions immediately:
- Put "PrintNightmare" into your favorite Internet search engine to learn what it is. This is a very serious exploitable bug that allows an attacker inside your network to elevate their permissions to that of a Domain Administrator. The attacker now "owns" your network at this point and can conduct what I'll call... mischief.
- Back up your systems and apply the latest MS Windows Server patches right away.
- Disable the Print Spooler service on every server running Hyperion / Oracle EPM.
Sunday, June 27, 2021
Credit for the following information goes to a commenter on this blog named Deezel. And also, of course, to Oracle for figuring out the fix!
In the Hyperion / Oracle EPM 11.2.x series of releases, Critical Patch Updates are available to fix significant security vulnerabilities in Oracle WebLogic Server 12c and Java 8.
Unfortunately, one of the WebLogic patches require an update to the OPatch utility, and the update cannot be rolled back successfully. So if you're on EPM 126.96.36.199 through 188.8.131.52, you cannot redeploy your web applications after performing the in-place upgrade to 184.108.40.206. You get a generic error in configTool and the back-end log error messages are unhelpful.
What follows are Oracle's instructions on how to work around the issue. Again, a big thank you to commenter Deezel and I look forward to applying the fix in one of my bricked sandboxes.
I cannot emphasize enough the importance of following Oracle's advice by backing up the folders. Best of luck!
After Update from 11.2.4/11.2.3 to 11.2.5 deployment of products fail.
The steps listed here are to restore the Updated 11.2.5 instance with deployment failures to working state.
Take back up of EPMSystem11R1 and user_projects folder available under Middleware Home.
1 Stop all services
2 Delete all the folders under MWH except EPMSystem11R1, dbclient32,dbclient64 and user_projects
3 Rename the .oracle.products available at EOH\.oracle.product as .oracle.products_11.2.5
4 Rename the .oracle.products0 as .oracle.products
5 Run Apply Update
6 Configure as per Update Guide
7 Start Services
8 Sanity test passes
Every now and then I need to educate a customer on how OPMN works. Well, here's some knowledge I'm giving away for free. (3 CEOs ago hated when I did this)
Here's the symptom of what's causing your on-premises Hyperion / Oracle EPM Essbase.log to keep growing 24x7, even when no end-users are logged in and no MaxL automation is running:
Make a backup copy of opmn.xml if you've never performed this exercise before! OPMN is very particular about the XML file, and Essbase will fail to boot up if you commit a typo error in this file.
Now click Search->Find in your favorite text editor (cough cough Notepad++ cough cough) for the word "death". It is present in the opmn.xml file exactly once:
I have comments to offer on the restart-on-death setting. We'll get to that in a moment. But for now, you want to insert a new line immediately after <restart retry="2" timeout="600"/>
Paste exactly this:
<ping timeout="20" interval="0"/>
Bounce the Essbase service (Windows) or use your stopEssbase.sh / startEssbase.sh scripts in Linux. Done. Your modified file should look like this:
Sunday, June 13, 2021
Friday, June 4, 2021
I have much to say about the in-place upgrade to Hyperion / Oracle EPM 11.2.5. Anybody who knows me well... knows that I pretty much despise Hyperion in-place upgrades, and moving to 11.2.5 in-place is no exception. In fact, it may be one of the harder ones.
We've already talked on this blog previously about how applying the quarterly Oracle Critical Patch Update ("CPU") for Oracle Fusion Middleware - OPatch for Weblogic in particular - currently bricks your in-place upgrade from either 11.2.3 or 11.2.4 to 11.2.5.... unless you backed up your Middleware folder prior to patching it.
But this post is about other factors that'll make you want to tear your hair out.
Let's begin with OPMN.
Tuesday, May 18, 2021
"BeaService", how did I beat you?
If you have never heard about or read about NSSM before, READ ON!
When I record installation sessions for Hyperion / Oracle EPM, I tend to pause the recording when it is time to deploy NSSM to get Oracle WebLogic Admin Server setup as a Windows service.
Because NSSM is an acronym that is not entirely nice. I won't repeat it here. But you NEED this tool if you don't have it! Grab it here: NSSM download
NSSM lets you create a Windows service for any executable .bat or .exe. It does not work for OHS 12c and I have a separate post about that topic. But it works wonderfully for the Oracle WebLogic Admin Server ("WLS") in either 10.x or 12.x.
Simply download NSSM and unzip it in a directory of your choice. That directory should not be deleted later. I save my copy in D:\Scripts\NSSM on my primary EPM Foundation Server where the master version of WLS resides.
The beauty of this utility is:
- You can customize the name of the service, as in my first screenshot above.
- You can make it rotate the log files.
- You can force it to make the stdout and stderr logs reside in the same folder as your EPM service stdout/stderr logs!
I won't post the spoiler screenshots or else you won't hire me. 😈 But maybe I gave you a few helpful hints here?
Sunday, May 16, 2021
This ought to be a simpler post than what I usually write.
The topic of how to start Oracle HTTP Server ("OHS") 12c in Hyperion / Oracle EPM 11.2.x comes up frequently. I've previously posted about the little trick, buried within the EPM 11.2.x install/config guide, on how to save the WebLogic password for OHS so you aren't prompted every time you try to start up OHS.
"But hey! Dave, how do we start and stop OHS in a distributed MS Windows Server environment when OHS doesn't have its own Windows service anymore? The good old NSSM trick we use for WebLogic Admin Server doesn't work for OHS!"
An excellent question.
Login to each OHS host server in your distributed environment as your EPM network service account. Run the "storeuserconfig" trick to save epm_admin's password in your network service account's profile on the C drive. Search the install/config guide for the text "storeuserconfig" and it'll take you directly to the needed command syntax. Every member on your Hyperion support team who wants to start/stop services using their own distinct userID needs to perform this step. This is a one-time-only step until such time as your security team forces you to change the WebLogic epm_admin password.
Now on your OHS server(s) foreign to where you run your master start/stop scripts from (you are running custom start/stop scripts, yes?), create a simple little DOS batch script like this:
sc start "Oracle Weblogic ohs NodeManager (D_Oracle_Middleware_ohs_wlserver)"
start "StartOHS" /wait /i cmd /c %EPM_INSTANCE_HOME%\httpConfig\ohs\bin\startComponent.cmd ohs_component
Replace D_Oracle in that first command with the drive letter where you installed. e.g. E:_Oracle, F:_Oracle or whatever is appropriate. The drive letter is unfortunately embedded within the Oracle Node Manager 12c Windows service name.
Replace %EPM_INSTANCE_HOME% with the fully qualified path to your Oracle EPM Instance, which is DRIVE:\Oracle\Middleware\user_projects\epmsystem1 by default.
Note: the final line beginning with start "StartOHS" is all one line all the way through ohs_component. Blogger is linewrapping it in an undesirable way.
Save this script as DRIVE:\scripts\startohs.bat or whatever you want.
Replace startComponnent.cmd with stopComponent.cmd and now you have your stopohs.bat script as well!
Finally, use this to invoke it remotely from your master start/stop scripts:
wmic /node:%FDNHOST2% process call create "DRIVE:\Scripts\startohs.bat"
wmic is a built-in Windows command. Provided your userID is an administrator on the foreign server, you may use this syntax to invoke any batch file, command, or executable you wish.
Wednesday, May 12, 2021
- If Oracle Middleware, Opatch in particular, has yet to be patched, back it up! If disk space is available, back up \Oracle\Middleware in its entirety before applying the quarterly Cracle Critical Patch Updates.
- If the system is unpatched, it is apparently safe to perform the in-place upgrade to 220.127.116.11.
- If the system was patched, it will be necessary to restore \Oracle\Middleware\Opatch and perhaps \Oracle\Middleware\wlserver and \Oracle\Middlware\oracle_common. The jury is out on exactly where the bug resides with respect to patching. All that is known for certain is Opatch is a primary culprit and the Opatch update cannot be rolled back without performing a filesystem restore for the Opatch folder structure.
Tuesday, April 27, 2021
We finally get Oracle Fusion Middleware (FMW) 18.104.22.168!
SQL Server 2017!
Essbase 21c certified!
The bad news:
If you're still using Essbase Studio, migrate away from it. Essbase Studio is slated to be retired in EPM 11.2.6 as per this Oracle Statement Of Direction: https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=81935405509356&id=2731169.1&_afrWindowMode=0&_adf.ctrl-state=rbcbyuw9y_4
Visual Basic scripts for FDMEE need to be migrated to Jython ASAP. VB scripting is expected to be decertified in EPM 11.2.7. We knew this some time ago, but now we now the release number where VB goes away for FDMEE.
Oracle's readme for 11.2.5 confirms what I've suspected. In 11.2.x we aren't getting Patch Set Updates (PSUs), but rather new "dot" releases. Here's a copy/paste directly from the 11.2.5 readme:
EPM System updates are released on a quarterly basis, generally in January,
April, July, and October.
You can directly apply an update from the previous two updates.
I won't paste the entire thing. When you download 11.2.5 from Oracle eDelivery, you get this folder:
EPM_11.2.5\EPM 22.214.171.124.000\EPM System Installation Documentation\EPM System Installation
Stop, open this folder. Open the readme before doing anything. Here's the Cliff Notes. You can apply 11.2.5 as an in-place upgrade on top of either 11.2.3 or 11.2.4. Do not attempt to apply it on top of 126.96.36.199 or any older release; you need to do a fresh install and LCM your apps from the older release. Only 188.8.131.52 is certified for this migration path.
If you were an early adopter like me, you need to jump to 11.2.2 before you can get to 11.2.3 or higher. You can apply 11.2.2 on top of either 11.2.0 or 11.2.1. You can't get to 11.2.3, .4 or .5 unless you're on at least .2. Job security!
Migration Path documentation: 184.108.40.206.x to 11.2.x
If you're on Financial Close Management 220.127.116.11 or older, prepare for pain. I've done it and it is not fun. :(
It is now time to back up my sandbox and attempt the upgrade. Wish me luck!
Wednesday, March 31, 2021
This is a bit of a weird one, and it isn't Oracle's fault at all. This is actually a Microsoft issue.
Suppose you are migrating from Hyperion / Oracle EPM 18.104.22.168 to EPM 11.2.x. Further suppose you have custom data integrations in FDMEE/ODI or MaxL that rely upon SSIS (SQL Server Integration Services) automation.
Your EPM 11.2.x system would be using MS SQL Server 2016 SP2+ if you want to use SSIS. I checked the certification matrix last week and Oracle still hasn't certified MSSQL 2019 for EPM 11.2.x.
(Disclaimer: the above paragraph may become be dated as Oracle might certified MSSQL 2019 in the future. The Internet is forever!)
Now here's the fun part.
To migrate your SSIS packages to MSSQL 2016, you will need to first import and upgrade them via MS Visual Studio. The version available to download is Visual Studio 2019.
Once upgraded, you use MS SQL Management Console to import them into SSIS. The version of this software as of this writing is 18.8.
This combination of software do not play well with each other, and you'll get an error message in MS SQL Console when you try to work with the upgraded packages. This exact error and solution has been written about on many blogs. I'll just jump straight to the solution.
You'll need to uninstall MS SQL Management Console 18.8, and then grab version 16.x from Microsoft's download page. There's a link there that lets you download older versions, and 16.x is still available as of this writing. Version 17.x does not solve our issue... we have to go all the way down to 16.x.
You may then complete your migration in MS SQL Console 16.x.
One final tip: if your custom integrations are using Microsoft's dtexec utility, you'll need to replace the reference in the script's path to dtexec from 110 (SQL Server 2012) to 130 (SQL Server 2016).
Saturday, March 13, 2021
This is a follow-up about Adobe Flash Player EOL and On-Premises EPM 11.1.2.x.
As you may have heard, Adobe ended support for Flash Player on Jan 1, 2021. This date has been known for some time, and as a result, Oracle released patches for EPM 22.214.171.124 in Summer 2020 that removes Flash dependency from Hyperion Planning, Hyperion Calculation Manager, and the Financial Close suite.
Some Oracle EPM / Hyperion customers may have deferred patching or upgrading. Perhaps the thinking was "we can just leave it on end-user workstations as-is even though it is out of support."
Microsoft says "Not so fast!"
If you're on EPM 126.96.36.199 and are using any of the 3 modules I mentioned above, please apply the patches right away. They're listed in my link above, although newer ones might be available by now.
If you're on EPM 188.8.131.52.500+, you need to upgrade.
As mentioned in my earlier post, Adobe removed the Flash Player download from their website. They further stated that downloading it from any 3rd party site is a license violation.
The good news is EPM 11.2.x has the fix built-in. As does the EPM Cloud.
Thursday, March 11, 2021
Depending upon you migrate FDMEE from Hyperion / Oracle EPM 184.108.40.206 to EPM 11.2.x, you may have an issue when trying to run your load jobs.
In 11.2, Oracle included a handy migration package specifically to migrate the AIF* tables from 220.127.116.11 to 11.2. You'll find it on your 11.2 FDMEE server in \Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\database\migrate. There's a package each for MSSQL and Oracle - check the subfolders for the packages and instructions.
As per Oracle's documentation, you want to run this in 11.2 after you've executed "Configure Database" for FDMEE. This because the migration package only migrates the AIF* (FDMEE) tables, and does not migrate the ODI-specific tables (SMP* etc.).
Now let's say you're in 11.2's Data Management and you're trying to run an import job. It fails. You check the log and see a reference to JAVA_HOME=%EPM_ORACLE_HOME%\..\jdk160_35.
This simply won't do! Don't be tempted to just copy jdk160_35 from your 18.104.22.168 server to your 11.2. Java SE 6 is out of support and won't be patched anymore.
You can run a simple SQL statement to fix the problem. Here's how it looks in MSSQL, and Oracle's version of this would be the same:
Bounce your 11.2 FDMEE service(s) just to be on the safe side, and now when you go back in to Data Management, you won't see that reference to jdk160_35 in your process log anymore.
All releases of Hyperion / Oracle EPM 22.214.171.124 through 126.96.36.199 hosted on MS Windows Server do the following:
While this doesn't hurt anything, if you want to clean things do, do the following after you've finished deploying all of your WebLogic services.
Open regedit and export the Hyperion Solutions registry hive to your desktop.
Open this file in notepad/Notepad++ and do a Search->Replace.
Change Your environment has been set.; to nothing. Save the modified .reg file.
Double-click the .reg file to import it back into your registry.
Do this on every server.
Friday, February 19, 2021
The following assumes your OHS is hosted in MS Windows. An equivalent bash shell script may be easily created to do the same in Linux/Exalytics. The below assumes you have already executed the OHS storeUserConfig process previously documented on this blog.
REM This script sets up our environment variables.
sc start "Oracle Weblogic ohs NodeManager (F_Oracle_Middleware_ohs_wlserver)"
start "StartOHS" /wait /i cmd /c %EPM_INSTANCE_HOME%\httpConfig\ohs\bin\startComponent.cmd ohs_component
Change the "F" in red above to be your EPM system's drive letter. The variable EPM_INSTANCE_HOME is the path to the instance, such as F:\Oracle\Middleware\user_projects\epmsystem1 and I define it in my custom ScriptEnv.bat.
Tuesday, February 9, 2021
Here's an intellectual exercise for my followers.
Suppose you've constructed a brand new Oracle EPM / Hyperion 11.2.x system (188.8.131.52 through 184.108.40.206 as of this writing). Now also suppose you have a need to automate Essbase MaxL scripts... let's say for example an Essbase nightly level-0 backup, or an FDMEE nightly data load automation.
You run the Essbase Client installer, and MaxL works just fine.
Except the Essbase Client is 220.127.116.11 technology and you've just corrupted one small element of your 11.2.x system.
Trying launching the configTool graphical client. The javaw.exe process crashes immediately and you never see the Oracle EPM Config logo pop up.
What to do?
Oracle does have a fix for this documented within their Knowledge Base. You just need to know where to look.
Or, search for files dated in the year 2014 A.D. Open the Windows Explorer and put in a date range of 1/1/2014 through 12/31/2014 as you see in my screenshot below.
Don't bother with the html files. What you care about are the EXE and JAR file extensions. The act of running EssbaseClient.exe causes the 2020-2021 versions of these files to be blown away by the 2014 versions.
The fix: stop everything. Replace the 2014 files from a server where you did NOT run EssbaseClient.exe to install Essbase MaxL. Then launch configTool and confirm it works again. Customers who have security policies mandating database & WebLogic password changes every 90 days will need a working configTool utility.
Try this out in a DEV environment. I like Ahi Tuna sashimi with a bit of soy sauce and wasabi, and thus you can thank me with that. ;)