Saturday, June 27, 2020

Release the Kraken! EPM 11.2.2 is Out and Includes Linux

EPM is out and includes installers for Linux as well as MS Windows.  This is the one many organizations have been waiting for.

Time to spin up a new sandbox and take it for a test drive!

Edit: As of this writing, the 11.2.x documentation portal was last updated in May 2020.  While it is still 11.2.1-centric, there is a comment within the Installation Readme stating that Microsoft IE11 will not be supported in 11.2.2.  So we'll want to carefully review the 11.2.2 certification matrix once it is published.  The path forward looks to be MS Edge, Google Chrome and Firefox.  I'll publish a separate post on this topic once Oracle has updated their documentation.

News Flash: Oracle Extends Essbase Extended Support Through 2024

No supporting detail to digest yet, but here it is:

In short, the post linked above says Extended Support for Essbase 11.1.2.x is extended through DEC 2024.  (Safe Harbor: the Internet is forever, so if you found my blog through a Google search, please check Oracle's site for the latest information)

We also see there is again a break-out between Premiere Support and Extended Support.  Until yesterday, June 26, 2020, Premiere Support and Extended Support were on the same DEC 2021 expiration timeline.  Premiere Support remains on the DEC 2021 timeline, but Extended Support now gets the new DEC 2024 timeline.

Keep a few things in mind.

Although the post says "Essbase 11.1.2.x", in reality we're talking just about  The ship has sailed on new patches coming out for Essbase and prior a long time ago.  The terminal Oracle Essbase patch is and it was issued Dec 10, 2015.

Also note the post linked above is careful to talk about Essbase alone - not the entire Oracle EPM / Hyperion stack for 11.1.2.x.

What does this mean for on-premises EPM shops?

It suggests to me that Oracle intends to continue maintaining and patching Essbase past the prior DEC 2021 Safe Harbor date. 

Perhaps COVID-19 delayed efforts to gather the Dev team together to get Essbase 19c ready to roll out to EPM 11.2.1.x.  Certainly, I can say COVID-19 has impacted many organizations and they delayed their upgrade from 11.1.2.x to by 1-2 years.  It is possible Oracle received push-back from their customers and this could have contributed to extending the date.

Organizations still need to take Microsoft end-of-life dates into account.

Windows 7 is already out of support and the end-of-life date for Internet Explorer 11 has been aligned with end-of-life for Windows 10.  Customers who have moved to the Oracle EPM Cloud, but still use Windows 7, have 2 months remaining (as of the date of this post) to upgrade to Windows 10 so Microsoft Edge may be used; EPM Cloud will stop supporting Internet Explorer 11 in Fall 2020 (see the July 2020 EPM Cloud Update notes for details on this).

I am glad to see Essbase has received a breath of life.  This particular component has been the most heavily patched within the EPM on-premises suite, with a new cumulative patch coming out with significant bugfixes at a pace of one every 2 months.  For the sake of comparison, Hyperion Planning has 10 cumulative patches as of this writing, whereas Essbase has 40.

Those of us who rely upon on-premises Oracle EPM will need to re-check Oracle's site periodically to learn if the DEC 2024 Extended Support date might be applied to the rest of the EPM stack, or if it remains applicable to Essbase only.  The fact that EPM and use Essbase under the covers is a reason why I believe the date may have been extended specifically for Essbase.

Thursday, June 18, 2020 EOS White Paper

I authored a white paper on the topic of Oracle EPM / Hyperion coming out of Premiere and Extended Support.  You may have attended one of my ODTUG webinars or read some of my blog posts on this subject.  But here it is all in one place!

Download it and share if you need to give higher-ups a "push" to get a budget approved to either upgrade to 11.2 or move to the EPM Cloud.

Safe Harbor applies to my statement below:

The patching sunset date for is currently set to DEC 2021. then moves to Lifetime/Sustaining support starting JAN 2022.  Oracle could potentially change this due to so many of their customers having project delays due to COVID-19, but that is anyone's guess.  As always, look at Oracle's official site for the latest direction on this subject.

Regardless of what Oracle decides, you will want to either move to EPM or EPM Cloud before Microsoft ends support for Internet Explorer 11.  I do not expect any sort of patch that would pack-port's support for Microsoft Edge down to

Saturday, May 30, 2020

FCM/ARM 11.2 - Why Can't I Find The Target Libraries?

The Hyperion / Oracle EPM 11.2 Install/Config Guide says to do the following if you want to setup Financial Close / Account Rec:

This is a familiar step for anyone who has configured FCM/ARM in EPM

But in 11.2, I couldn't see either epm-misc-libraries or epm-aif-odi-libraries in the expected screen.  What's going on here?

The answer is WebLogic Admin Console 12c behaves a little different from your grandfather's WebLogic 10.3.6.  Click "Customize this table", REMOVE this checkbox, and then click Apply!

I can understand why Oracle made this design decision to make this the default setting - when libraries are included, the Deployments screen contains over 100 entries if you're doing a full-stack EPM implementation.

So take away "Exclude libraries when displaying deployments", set your Targets as needed, and then go back into "Customize this table" and turn the checkbox back on when you're satisfied.

Sunday, April 26, 2020

Stop Editing Workspace EAR Files - EASY Way to Change the Title!

I mentioned in my last ODTUG webinar that Oracle EPM 11.2.x's workspace looks exactly the same as's.  Everything looks identical right down to the color schema and title of the tab in your browser.

In the old days, we had to manipulate the WAR file inside of the EAR file to change the Oracle logo, manipulate the color codes, or whatnot so as to discriminate between Production, non-Production, or even different versions of EPM.  Replacing the logo image requires the know-how to make an image of the exact same height & width -- or being a fairly good Internet meme creator.

Worse, I sometimes find evidence that some consultants made edits directly within WebLogic's domains/EPMSystem/servers/FoundationServices0/tmp folder, rather than within the EAR+WAR file itself.

Stop doing that!!!

Do you know that a 1-line SQL statement makes your EPM Workspace title and browser tab look similar to my screenshot above?

Execute the SQL statement and bounce the EPM Foundation service, and you'll never have to worry about what happens if somebody subsequently patches Workspace or blows away WebLogic's /tmp folder.

Which SQL statement????   Here you go.  As my dear mother used to tell me in her dry humor, "never say I didn't do anything for you..."

UPDATE hss_component_property_values
SET property_value = 'EPM Workspace - Datavail Sandbox'
WHERE property_name = 'ApplicationName'

property_name = 'ApplicationName' is case-sensitive for both Oracle and MSSQL.  What you specify for property_value may be anything you want.  My suggestion is to indicate the version (e.g., or as appropriate) and the environment designation (PROD, UAT, DEV, etc...)

The above SQL statement works for all 11.1.2.x and 11.2.x versions as of this writing.  If you're still on 11.1.1.x or 9.3.x..... good luck!

Wednesday, April 22, 2020

EPM to In-Place Upgrade

Here's a quick follow-up to my earlier post regarding EPM Initial Thoughts.

If you installed Oracle EPM shortly after it was released in December 2019, you are likely considering moving up to EPM so you may enjoy the new certifications for Microsoft Edge and Google Chrome.

But do you need to start all over?  No, you don't!

As mentioned in the installation guide, provides a Maintenance Upgrade option, whereby you may perform an in-place upgrade on and bring it up to

I ran this process over the weekend, and here are my observations:

In short, the process is easy and and is identical to how the in-place upgrade worked within the 11.1.1.x and 11.1.2.x series of releases.  The installTool.cmd utility detects the previous installation and forces you to upgrade the existing components first before allowing you to go back and install additional components.

When you perform an in-place upgrade from to, the binaries for Oracle WebLogic Server and Oracle HTTP Server initially remain untouched.  All of the .ear files underneath \Oracle\Middleware\EPMSystem11R1\products, however, will be replaced by the upgrade.

One thing to note is when installTool.cmd reaches 97% progress and seems stuck at "updating Oracle Inventory", there are some additional things happening behind the scenes; the installer applies Oracle Middleware and Essbase patches during this phase.  Patches previously applied by's installTool are re-applied by's installTool.  These patches include Oracle WebLogic, Oracle HTTP Server, and Essbase.

If you use SmartView for Essbase, you will need to reapply the Java fix for Essbase SmartView ad-hoc.  This is because installTool applies the Essbase patches in a slightly incorrect order, leaving two Java artifacts in Oracle\Middleware\EPMSystem11R1\common\EssbaseJavaAPI\\lib\ with an incorrect version.  Click the link within this paragraph to see the exact fix.

After you finish running installTool, configTool indicates the web applications need to be redeployed.  In truth, this doesn't provide much benefit as WebLogic automatically picks up the newer .ear files the next time you restart services.

It isn't a bad idea to run the redeploy step, though, because the act of redeploying makes a few minor updates to the Shared Services Registry database.  Specifically, a few components will have their version numbers updated from to  This makes the EPM System Registry Report and also Help->About EPM System->Show Details slightly more accurate (although not exactly accurate in all cases).  Doing this will help reduce some confusion should you need to work with Oracle Support in the future.

April 23, 2020 update:  Michael Fredericks of FinWeb Solutions noted he received a Communication Error when trying to login to EPM Workspace after performing the upgrade.  He resolved the issue by running the redeploy step.  He also reminds us that when redeploying, we need to go back and re-apply any Java Heap customizations.  Thanks for your comment on LinkedIn, Michael!

One final point about this in-place upgrade: You don't need to touch or re-do RCU.

Monday, April 20, 2020

EPM Initial Thoughts

Let's begin with a comparison chart I whipped up...

Here are my observations after standing up Oracle EPM
  • The install and config steps in are identical to in all respects.  You still need to worry about RCU, OHS still isn't its own Windows service, etc.
  • The Oracle Middleware layer is also identical between and, with the exception of 2 patches that automatically installs: 27747932 and 27430060. Simply applying these 2 patches to an system doesn't give it the Edge & Chrome certifications; you need to be on as Oracle recompiled a lot of their Java classes within the EPM .ear files for Workspace, Planning, etc.  We can see evidence of this when we crawl through the .war files within the .ear files in a tool such as 7-zip.
  • As with, you'll find is one year behind (as of this writing) in terms of the quarterly Oracle Critical Patch Updates for WebLogic, Java, and OHS.
  • There is no need to install first and then in-place upgrade to  You may install fresh and then LCM your application content from
The chief benefits to over are the certifications added for Microsoft Edge and Google Chrome.

IE11 will continue to receive Microsoft security updates for as long as Windows 10 remains supported, but as they say, "shots have been fired".  As Microsoft has indicated Edge is the browser of the future, so has Oracle and thus (I suspect) the push to get out the door.  (People have been clamoring for Chrome certification as well)

Further reading on Microsoft's site: Microsoft Lifecycle FAQ

Thursday, April 9, 2020

Time to Spin up EPM

Oracle Announcement

More analysis will be forthcoming.  I'm inclined to pick the fresh download & install option vs. upgrading from

Of interest: support for Google Chrome is back with this new release, and we finally get Microsoft Edge support as well (bringing EPM on-premises more into parity with EPM Cloud).

Many people have asked me about MS Windows Server 2016 certification, and this new release includes that as well.

On this point, Oracle indicates EPM is on the development roadmap, and Internet Explorer will be de-certified at that time.

Stay tuned!  I haven't seen a release come out this quickly since the Essbase 5.0 fiasco.

Monday, March 30, 2020

Planning RMI 11.2: "Could not find the service start class" SOLUTION

I spent more time troubleshooting this than I care to admit.  Rather than walking you through the things I tried, let's cut to the chase!

Symptom Observed:
You start the "Oracle Hyperion RMI Registry" service in the Control Panel or via a script.  The service shows it is running and no logfiles reveal an up/down status for it.

Meanwhile, your classic Planning batch scripts don't work like they used to...

Digging deeper with traditional port checking tools, we see the RMI port 11333 is not listening.

Finally, we check the Windows Event Viewer.

Oh boy, this is going to be a long day...

Quick Tip:
Search engines and the Oracle Knowledge Base don't have current information on this "could not find the service start class" error yet, at least not where Oracle EPM 11.2 is concerned.  There's some older articles out there about ports, service names, etc. that don't lead you to a solution.  Hopefully this blog entry will help!

First, export this portion of your Windows Registry in case you want to roll back to the original configuration:


And here it is!

Planning RMI is still using the old Java 6.

I haven't dug enough to find the exact culprit, but there's likely a mix of Java class files in RMI where some were compiled with Java 6, and others with Java 8.  Java 6 can't run Java 8 classes, but Java 8 can often run both Java 6 and Java 8 classes.

So the trick is to locate a "jvm.dll" file that is Java 8 and 32-bit. I tried using a 64-bit one and it immediately complained I was mixing 32-bit and 64-bit.

In a vanilla EPM 11.2 system, you will have a 32-bit Java 8 if you allowed it to install the 32-bit Oracle Database Client.  The file is:


Replace the "JVMLibrary" Data value in my screenshot above so your system looks like this:

Bounce RMI, and suddenly port 11333 is working and you may continue with your day.

Here's hoping a more official fix will be forthcoming!

Sunday, March 29, 2020

Q&A From March 12 2020 Webinar

We ran out of time in our ODTUG EPM 11.2 Webinar to get to everyone's questions, so we had to rush through it.  Without further ado, here's the questions submitted in writing by attendees.  I'll discuss them in no particular order and exactly as they were submitted.

Q: Does 11.2 still have rapid deployment option?
A: Yes.

Q: So no more CopyApp Utilities to migrate HFM Apps between environments?
A: No, the venerable HFMCopyApplication utility was deprecated in  In theory, the latest patch of the utility from could work, followed by running the HFMUpgradeApplication utility.  In practice, I believe Oracle Support would advise migrating via the HFM LCM Snapshot instead (if, and only if, you are on  Or, use the Import Application link in Navigate->Administer->Consolidation Applications.

Q: Is RCU schema/database needed per application server or per environment?
A: One per application server.  A pure Essbase server with no WebLogic web apps wouldn't need it.

Q: Is Windows Server 2019 only supported or okay to install Windows Server 2012 or 2016?
A: In theory, yes you can put 11.2 on either Server 2012 R2 or 2016.  I've done it myself.  In practice, I'd advise against it using 2012 and 2016.  You want to use server operating systems specifically vetted and certified by Oracle Corporation.  Oracle has only certified Server 2019 at this time for 11.2.

Q: Any word on Oracle 18c or 19c support?
A: Terrific question and I'm getting asked this more and more.  I'm told by DBAs that 19c is essentially 12c technology with bugfixes sitting on top of it.  I know of at least one customer who is using 19c successfully for their EPM system.  I would love to see certification from Oracle on 19c.

Q: Understand Financial Reports with Essbase connection cannot be migrated from to 11.2? Is that right?
A: The Essbase connection type is visible as a drop-down in Database Connection Manager. I'll need one of my report developers to kick the tires to see if it works.  Sorry I don't have a definitive answer yet!

Q: Hello, how can I  save the nodemanager password so when I start OHS I don't have to type the password in windows?
A: Yes!  The syntax is all on one line:
\Oracle\Middleware\user_projects\epmsystem1\httpConfig\ohs\bin\startComponent.cmd ohs_component storeUserConfig
You only need to do this one time per OHS server.  Be mindful that if a different userID needs to perform restart maintenance, that person either needs to be trained on this process, or you put it in a script from them to double-click.  You will be prompted to type in epm_admin's password, and then you won't be asked again.

Q: will you be supported by oracle if you install 11.2 on ANYTHING other than Windows 2019 server.
A: I can't answer for Oracle on this point.  Depending upon who is assigned to work on your SR, someone might kick it back at you.  My personal recommendation is to use certified operating systems only - especially where Production is concerned.

Q: Any word on when Rhel7 server version for EPM 11.2 will come out?  Other then soon?
A: "Soon".  :)  I have not received any specific guidance beyond what is available for public consumption.

Q: Users will probably miss the Home page in 11.2.  Any workaround, or plan for bringng that back?
A: As a workaround, users can set their preferences so a specific app launches upon login.  The loss of the Home page is a byproduct of the architectural change that removed Reporting & Analysis Framework.

Q: what is the replacement for the HFR studio thick client?
A: The Reporting Web Studio is available within EPM Workspace and was first introduced by Financial Reporting patch set update  In there's a manual OHS configuration step before it appears to users.  It works out-of-the-box in 11.2 without needing to manually update OHS.  Several PSUs were issued since then which improved parity with the thick client.  Customers who are on or higher are urged to become familiar with the web studio before making the jump to 11.2.  Alternatives would be saved SmartView templates or other technology.

Thursday, March 26, 2020

March 12 2020 webinar - because you REALLY want to hear my voice?

EPM 11.2 Upgrade - Lessons Learned

I have a face for radio, but not the voice!  If you can withstand my voice, click the link above and listen to findings about the Oracle EPM 11.2 upgrade.

A big THANK YOU to the nice people at ODTUG for hosting this session and allowing me to present!

Wednesday, March 4, 2020

Upcoming EPM 11.2 Webinar March 12 2020

Thursday, March 12, 2020 12:00 PM - 1:00 PM EDT
EPM 11.2: Lessons Learned and 2021 Preparedness
Dave Shay, Datavail

Webinar abstract:

As we all know, Oracle EPM 11.2 is here! But…it was released too late in 2019 for most organizations to budget an on-premises EPM upgrade for fiscal 2020. However, the end of support for is also looming in 2021. If you’re staying on premises, an upgrade to 11.2 should go live no later than December 2021 (earlier if subject to SOX controls).

Rather than waiting for the next budget cycle to roll around, this webinar will show attendees how to prepare for an upgrade this year without spending significant time and capital. We’ll also share what we’ve learned while upgrading to 11.2 and what you can expect post install.

Visit this link to reserve your free spot!

Avid readers of this blog will be familiar with a good chunk of the content I'll present.  In this session I'll do my best to condense observations collected since the Dec 18 2019 Oracle EPM 11.2 Release Date into a 1-hour session.

The session will include Q&A at the end.

Thanks to ODTUG for sponsoring this session!

Saturday, February 22, 2020

EPM 11.2 - More Than a Cupful of Java

If your IT department runs deep-scans of your Oracle EPM server filesystems looking for older/vulnerable versions of Java, this post is certainly for you.

This blog has previously written about how Oracle Jan 2020 Critical Patch Update Affects EPM 11.2  In the post linked here, you'll find information about a newer version of Java SE 8 that's applicable to EPM  (You can expect quarterly Java 8 updates until it reaches end-of-life... so get used to this if you aren't moving to the Cloud)

So let's say you've downloaded the newer Java and want to replace the content of your pre-existing Java folders with the newer one.  Just how many Java locations do you have on your EPM 11.2 server, anyway???

Being a UNIX nerd from the beginning of my IT career, I was just dying to take the Cygwin utilities for MS Windows for a spin on my EPM 11.2 sandbox server.  Here's what I found!

CD \Oracle\Middleware
C:\cygwin\bin\find . -type f -name java.exe -print -exec {} -version ;

java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) Client VM (build 25.171-b11, mixed mode)
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) Client VM (build 25.171-b11, mixed mode)
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b52)
Java HotSpot(TM) Client VM (build 20.10-b01, mixed mode)

Ewww!!!  Java 6?  See Java 6 Not Eradicated in EPM 11.2

java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

That's 10 separate java instances.  Now that you finished scrolling down and witnessed The Matrix fly past your eyes, here's some thoughts.

The "dbclient32" and "dbclient64" folders only exist if you selected them when you used EPM's installTool.cmd wizard.  (Picking either FDMEE or HFM will cause the Oracle DB clients to be automatically selected... you may deselect them if you're using MS SQL Server exclusively).

The "dbclient32" folder is the only 32-bit Java you'll have.  Keep this if you intend to use the SQL Developer tool, which is embedded within the dbclient32 folder structure.  SQL Developer in this release still expects a 32-bit Java.  All of the other folders listed above are 64-bit.

As of this writing, Java 1.8.0_241-b26 (Java SE 8 Update 241) is the version Oracle published in their latest Critical Patch Update.  This information will become dated starting in mid-April 2020, but the concepts described here will remain the same.

Do not be tempted to rename the \Oracle\Middleware\jdk1.8.0_181 folder or your EPM 11.2 system breaks!  This folder name is hard-coded throughout the system (Windows Registry, batch scripts, and other files).  This may be a topic for another blog entry.  For now, to get security-compliant you might back up the 181 folder and then replace its contents with the contents of 241.

I'm out of Java coffee.  Time to stop blogging.

Tuesday, February 11, 2020

SmartView Essbase Ad-Hoc and EPM 11.2

Here's a short(er) blog entry about Hyperion / Oracle EPM 11.2, I promise!

Are you getting a pop-up error in SmartView when trying a top-level SmartView ad-hoc retrieve from Essbase?  In my case, I got a weird Java message complaining about "essggridoption.setAncestorTop()".

Our culprit here is a combination of the order by which Essbase patches were automatically applied when EPM 11.2 was installed, and perhaps some funny business involving how files revert to read-only status in Windows Server 2019.

The EPM 11.2 installer automatically applies these Essbase patches where appropriate:

Your problem here is this:

The Essbase Java API files used by Analytic Provider Services (APS) doesn't match what the Essbase server expects.  There's an Essbase API protocol mis-match here!

The patches automatically applied by the EPM 11.2 installtool are stored here:
Sure enough, there's a subfolder here for 29749671.

Stop your EPM services on the APS host machine(s).

Attempt to copy \Oracle\Middleware\oracle_common\OPatch\Patches\patchFiles\29749671\files\common\EssbaseJavaAPI\\lib\
choosing to overwrite the existing files.

One of three things will happen:
  1. The copy happens OK and you may restart services.
  2. It complains that a Java process is using one of the files you're trying to overwrite.  You'll have to use your Windows Task Manager to hunt down and terminate those processes (or reboot).  Then try again.
  3. It complains the target file is read-only.  If this is the case, right-click the target file, click Properties, and remove the Read-Only flag.  Then try again.
#2 above happened to me.  After I killed the offending process, replaced the files and restarted services, SmartView worked as expected!

Saturday, February 8, 2020

EPM 11.2 Distributed Environment - Yup, Another RCU Post!

My blog entry EPM 11.2 RCU and you apparently struck a nerve, so consider this a follow-up!

Before we get to the "do this, do that" part, let's examine how we can determine if our attempt to deploy Hyperion / Oracle EPM 11.2 web services across multiple machines is... well, botched.

Part 1 - Is my distributed deployment botched?

Let's assume you've followed the instructions within the post linked above, and you have successfully created your WebLogic domain and have deployed a few things on machine #1 (EPM Foundation, Oracle HTTP Server, and perhaps a few supporting modules such as APS, EAS, CalcMgr, etc).  Now you're in the EPM System Configurator on machine #2 and are getting the dreaded red X when you try the Deploy To Application Server task.

user_projects\EPMINSTANCE\diagnostics\logs\configtool.log on machine #2 offers nothing beyond the vague and dreaded InvocationTargetException Java stack trace error.

By the way, ERROR messages prior to this point may typically be ignored.  You will first see "ERROR: Jars manifest check failed with message "Some referenced jars do not exist" and then a little further into the log you'll see "env.isWindows is not a function".  Don't waste your time troubleshooting those, as they're unrelated to your issue.

Once you see InvocationTargetException, close out of the log and now open wlst_debug.log in the same directory.  This points us to the underlying issue!

Do you see "WLSTException: No element JDBCSystemResource was found" at the top of a lengthy Java stack trace error?  If so, look just a few lines above.  The message doesn't look like an error at first glance:

2020-02-08 13:38:53,392 INFO  [1] - find JDBCSystemResource!JdbcResource!JDBCDriverParams "opss-audit-DBDS!opss-audit-DBDS!NO_NAME_0" as obj5
How could this be bad?  It is an [INFO] message after all, not an error.

It turns out there's an OPSS schema that's created by the RCU utility described within my earlier post.  The WebLogic Scripting Tool (WLST) can't find the OPSS jdbc definition files on machine #2 because they don't exist.  More than likely, this folder on your machine #2 is empty.

If you find yourself nodding your head at this point, yes.... your distributed deployment is botched.  But it can quickly be rescued!

(Note: Don't be tempted to copy those files over from machine #1.  They contain references to the RCU prefix and I haven't tested what might happen if you do that -- I suspect it would be something bad?)

Part 2 - How to fix machine #2 so we can deploy web services there?

Follow these steps:
  1. Login to your WebLogic Admin Console on machine #1 (http://machine1:7001/console) as epm_admin and click "Release Configuration" if you see that button is available.
  2. On machine #2, blow away user_projects\domains\EPMSystem -- all files & subfolders.
  3. On machine #2, launch RCU (Middleware\oracle_common\bin\rcu.bat)
  4. Use the "create" option and choose a new prefix.  Pick something sensible like the EPMINSTANCE name (assuming you aren't deploying as "epmsystem1" everywhere) or machine #2's hostname.
  5. Pick all checkboxes except for the Oracle Data Integrator ones.
  6. Pick the option to use the same password for all schemas.  This way is much easier.
  7. Let 'er rip and close the RCU utility when done.

Last steps...

I've read in the comments on the RCU blog entry that Oracle Support is telling some people to use different "schemas" for each machine's RCU database. 

When in doubt, go with Oracle's suggested approach of 1 Oracle Schema or 1 MS SQL Database per RCU Prefix (machine), just in case you run into trouble later and need to open an SR.  This means more things for your DBA to back up on a regular basis, but it may cause less confusion for Oracle Support as you work through the SR together.

Let's take a look at the back-end database.  In this example I'll use MS SQL Server 2016.

I have more machines deployed than are visible within this screenshot, but hopefully the concept comes across!

Each EPMINSTANCE ("machine" if you are not scaling vertically) has its own RCU "prefix" assigned.  In this example, "DEV" and "LANDO".  These schema users were created by the RCU utility.

I'm using same SQL database for these and so far things are working OK.

Should you want to cluster out even further for High Availability etc., you might consider either using Oracle's suggested approach (1 database/schema for each EPMINSTANCE), or at least one per each cluster....

... e.g, EPMRCU0 has the RCU prefixes for Foundation0, CalcMgr0, EAS0, etc.  Then EPMRCU1 could have the *1 servers, EPMRCU2 could have the *2 servers, and so on.

Getting back to the task at hand, copy from machine #1's EPMSystem11R1\common\config\ folder to the same location on #2.  Edit it on machine #2 and change the "schemaPrefix=" line to the prefix you created for machine #2.  You shouldn't need to edit the two SysDBA*= lines.  If you aren't using the same SQL database / Oracle schema, you can wipe out the RcuSchemaPassword= value and add the plain text password (the password will be automatically encrypted if machine #2 becomes un-botched) and also DbURL= as appropriate.

Finally, go back into the configtool and re-attempt your deployment.  So that you don't have to re-read through old log entries, you might rename EPMINSTANCE\diagnostics\logs\config to some other name before you do this.

In Closing

Yes, I did botch one of my machines on purpose... I wanted to see what would happen if I skipped the RCU step for a machine where I did NOT intend to deploy EPM Foundation.  Why did I do this?  (Aside from enjoying pain)

Hop out to the EPM 11.2 documentation landing page on Oracle's site, and review:

Deployment Options Guide
    Section 2.1 Clustering Java Web Applications
        "When scaling Oracle Hyperion Foundation Services (underline emphasis mine), 
        you must create a new schema using RCU and edit on each
        machine in the deployment."

This specific underlined sentence fragment led me to believe RCU was only need on machines where Foundation would be deployed.  Silly me for taking the documentation seriously!

Another related point.  See:

Installation and Configuration Guide
    Section 6.2 Creating Infrastructure Schemas Using Repository Creation Utility
        "You must run RCU on each machine in your environment."

I'd add one caveat here; RCU is only needed on machines where you want to deploy WebLogic Managed Servers (Planning Web, HFM Web, CalcMgr, etc).  Essbase, Essbase Studio, DRM, HFM Java Server, and other non-WebLogic processes don't need it because they are independent from WebLogic.  So the general rule of thumb here is: if EPM System Configurator creates a user_projects\domains\EPMSystem folder on the machine, that machine needs RCU.

Time to enjoy the rest of my weekend.  Be safe out there, and don't ask me about Oracle SOA!

Monday, February 3, 2020

January 2020 Oracle Critical Patch Update affects EPM 11.2

So you've spun up your shiny new Hyperion / Oracle EPM  And now... it is time to patch!

I haven't come across a comprehensive patching guide for EPM 11.2 yet, so here's my attempt to create one.  This guide assumes you've patched EPM 11.1.x.x in the past and are familiar with Oracle's "OPatch" utility.

As usual, assume you will take an outage and shut down the EPM stack across all servers before patching (ONE environment at a time, please!!  "Gee Dave, what could possibly go wrong?")

Oracle's quarterly "Critical Patch Update" (CPU for short) was published in mid-January 2020 right on schedule.  As expected, there are impacts to EPM 11.2, even though EPM isn't mentioned by name.

For your reference, there's a link to Oracle's JAN2020CPU (as they call it) announcement:
Oracle JAN2020CPU Security Alert

In a nutshell, here's what is believed to apply to EPM

Java SE 8

EPM ships with Java SE 1.8 Update 181.  This is "old" as it apparently was published in the APR2019CPU, and Java has been updated every Quarter since then.  The version we should use as of January 2020 is Java SE 1.8 Update 241.

The patch # for Java SE 8 is 18143322 and you should hop into and bookmark that patch.  Oracle will re-use this patch # every time a new Java 8 patch is issued.  Make your patch search screen look like this so you can find it quickly:

This is not a "patch"; it is a full install.  You can install it on just one server's C: drive, deselect the "public JRE" option within the install wizard, and accept the other default prompts.  When finished, copy it to your D/E/F drive as appropriate to your various Hyperion servers and save it as \Oracle\Middleware\jdk8

Once you've finished this task, you can uninstall it via the Windows Control Panel from the C: drive on the one server you installed it to.

You will then face a decision that is worthy of a separate blog post:  Do you replace the content of \Oracle\Middleware\jdk1.8.0_181 or do you update the Windows Registry and various bat/cmd files to replace all references of jdk1.8.0_181 to jdk8?  Replacing the content of the jdk1.8.0_181 folder is the fastest and easiest solution by far... until somebody in IT notices and asks why a vulnerable version of Java is installed.

Before you decide right away, remember Java 8 will likely be updated by Oracle every 3 months.

Oracle WebLogic / Fusion Middleware

(For my friends on the functional non-infrastructure side, here's why you care):  Hyperion does not function without WebLogic / FMW running behind the scenes.  When you start up web services like EPM Foundation, Planning Web, HFM Web, EAS Server, etc., you are starting an "Oracle WebLogic Managed Server".  When WEbLogic / FMW has a security vulnerability, then you should consider your EPM system to be equally vulnerable.

The patch # for this is 30675853 and it is "WebLogic PSU 191217.1425".  This is a cumulative patch.

The very good news is we no longer need to use the BEA Smart Update (bsu.bat) utility!  In fact, that entire directory structure doesn't exist anymore.  We now use good ol' Oracle OPatch.

The "-oh" parameter you pass to \Oracle\Middleware\OPatch\opatch.bat is \Oracle\Middleware

In addition to updating Oracle WebLogic, it also updates some files within the \Oracle\Middleware\oracle_common directory hierarchy (Oracle ADF / JDeveloper and the like).

Oracle HTTP Server

(For my friends on the functional non-infrastructure side, here's why you care): OHS is the front-end proxy for EPM Workspace.  This is what listens to port 19000 / 19443.  As stated in the WebLogic section above, if OHS can be exploited then so too can your EPM system.

The patch # for this is 30687404 and it is "Oracle HTTP Server PSU 191219.2319" .  This is also a cumulative patch and you also use OPatch to apply it.
(Update: Credit to "RJ" in the comments below who pointed out I had pasted an incorrect patch ID #.  30687404 is the correct patch ID # to use as of this writing)

The "-oh" parameter you pass to \Oracle\Middleware\ohs\OPatch\opatch.bat is \Oracle\Middleware\ohs

Oracle EPM / Hyperion

No references to EPM 11.2 in JAN2020CPU as mentioned earlier.

Essbase Suite

No references to Oracle Essbase/APS/EAS in JAN2020CPU.  There is a separate Oracle Inventory on your EPM 11.2 server(s) for Essbase Suite, if installed, so you can still patch it if you want some of the newer Essbase-related bugfixes.

Here's what you have out of the box in EPM where Essbase Suite is concerned:

29260139 Essbase Studio Server
29260160 Essbase Admin Server 
29749671 Analytic Provider Services
29749652 Essbase RTC
29749662 Essbase Server

Oracle Data Integrator

ODI is referenced within JAN2020CPU, but I'd avoid it for now until more information is forthcoming.  If you want to take a look, the patch # is 29778645 for "ODI BUNDLE PATCH (CPU)".  This patch updates the ODI Studio thick client, which expects some minor tweaks to ODI's Master repository.

You can apply this patch if your ODI is standalone and is NOT the one bundled with FDMEE  This is because the patch wants you to run the Oracle Upgrade Assistant utility, which expects that the ODI Master and Work repositories were built through the Repository Creation Utility ("RCU") rather than by the EPM System Configuration tool when you ran the "Configure Database" step for FDMEE.

We will likely need to wait for an FDMEE patch, which I'd expect to include a new *.drv file that instructs FDMEE to update the ODI repository when FDMEE is restarted.

Clear As Mud???

One question I'm asked by customer IT departments from time to time:  "Do we really need to do this?"

Here's one way to answer IT's question.  Forward Oracle's JAN2020CPU advisory article, linked at the top of this blog entry.  Here's just one specific example from the article:

I highlighted in red some, but not all, of the important nuggets of information:
  • CVE#. IT can paste into their favorite Internet search engine to learn more.
  • Product.  Oracle WebLogic Server is the technology behind pretty much all of Hyperion's web services in 11.2, except for DRM.
  • Remote Exploit without Auth.  A "yes" means an attacker doesn't need a userID/password for the server or application in order to cause mischief.
  • Base Score.  This is on a scale of 1 to 10, with 10 being the most serious.
  • Supported Versions Affected.  I highlighted 10.3.6 for our friends who are still on EPM or -- those use WebLogic 10.3.6.  EPM uses WebLogic
Give this example to your IT counterpart, along with the link to Oracle's advisory article, and ask if IT is willing to live with the scores listed throughout the article.  (Again, the above is just 1 example, but there are many ranging from the low 4s to the high 8s and 9s).

Be mindful the JAN2020CPU advisory mentions everything Oracle; Oracle Database, MySQL, Java, Hyperion, Oracle Financials, etc.  This makes reading through the entire advisory a daunting task.  I advise doing so only after ingesting my caffeine than it took you to get all the way to this paragraph!

You can mitigate any panic by explaining the on-premises Oracle EPM system typically sits behind a corporate firewall and is not exposed to the public Internet. By applying these various patches, you are mitigating risk of a disgruntled employee wanting to cause mischief or steal data before their last day.

Wednesday, January 29, 2020

Having FDMEE 11.2 Adapter Troubles?

Having stood up EPM with HFM, Essbase, Planning, FDMEE and all the trimmings, I'm starting to go crawl through the back-end logs for the various modules and am.... finding things.

Here's one that struck me:


[timestamp] Starting import of the adapters.
[timestamp] [erpi] [ERROR] [] [oracle.erpi.localization] [tid: 1] [ecid: 07669c97-8ef1-4a3c-aa80-8e1c25fcd267-00000002,0] Error incountered while importing adapters.[[
java.lang.RuntimeException: Either directory F:\Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\odi\\adapters does not exist or is not a directory
    at com.hyperion.aif.install.AdapterImport.main(
This entry happens to be the very first thing we see within the log.  The [timestamp] just so happens to correspond with the first-time DB Config and WebLogic deployment of FDMEE.

Bear in mind the configtool's graphical interface did not report any errors here.  Config/Deploy status for FDMEE was given a green checkmark.

Now, does the directory \Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\odi\\adapters actually exist?  Of course not!  We're installing EPM

And guess what?  Directory \Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\odi\\adapters does exist and I see adapter XML files in there for JDE and Open Interface.

So I rolled the dice, created an empty directory \Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\odi\ and copied\adapters to\adapters.

Just to be on the safe side, I used epmsys_registry.bat command-line options to blow away the FDMEE stuff from the EPM System Registry, so I could return to configtool and run a fresh DB config and deploy for FDMEE.

This time around, no errors in aif-localization.log and the adapter imports were reported as a success:

[new timestamp] [erpi] [NOTIFICATION] [] [oracle.erpi.localization] [tid: 1] [ecid: 29cae779-08ae-4e92-a5c8-df87557c5c15-00000002,0] Starting import of the adapters.
[new timestamp] [erpi] [NOTIFICATION] [] [oracle.erpi.localization] [tid: 1] [ecid: 29cae779-08ae-4e92-a5c8-df87557c5c15-00000002,0] Importing adapter F:\Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\odi\\adapters/JDE_Adapter.xml
[new timestamp] [erpi] [NOTIFICATION] [] [oracle.erpi.localization] [tid: 1] [ecid: 29cae779-08ae-4e92-a5c8-df87557c5c15-00000002,0] Finished Importing adapter F:\Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\odi\\adapters/JDE_Adapter.xml
[new timestamp] [erpi] [NOTIFICATION] [] [oracle.erpi.localization] [tid: 1] [ecid: 29cae779-08ae-4e92-a5c8-df87557c5c15-00000002,0] Importing adapter F:\Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\odi\\adapters/Open_Interface_Adapter.xml
[new timestamp] [erpi] [NOTIFICATION] [] [oracle.erpi.localization] [tid: 1] [ecid: 29cae779-08ae-4e92-a5c8-df87557c5c15-00000002,0] Finished Importing adapter F:\Oracle\Middleware\EPMSystem11R1\products\FinancialDataQuality\odi\\adapters/Open_Interface_Adapter.xml
[new timestamp] [erpi] [NOTIFICATION] [] [oracle.erpi.localization] [tid: 1] [ecid: 29cae779-08ae-4e92-a5c8-df87557c5c15-00000002,0] Finished importing all adapters.
I'm curious to know if anyone out there has tried to use FDMEE yet and ran into trouble using either the JDE or Open Interface adapters.  If so, please post your observations, then try the above and advise our readers if this info helped!

Friday, January 17, 2020

Cumulative Feature Overview - Updated for EPM 11.2

Oracle's Cumulative Feature Overview web page now allows you to compare prior releases against 11.2.

Sadly, when I picked the core modules HFM, Planning and Essbase, I didn't see anything about features removed (Simplified Interface, Workforce Planning, CapEx Planning, etc).

It isn't surprising no changes are listed for Essbase.  Essbase 11.2 is actually Essbase under the covers.  (Looking at the Essbase client DLLs tells you exactly this)

I do see a 2 tuning confirmation options mentioned for HFM.  Here's a direct copy & paste:

"Two new configuration settings have been added for calculations.  MaxNumConcurrentCalculations - The maximum number of concurrent calculations allowed per server per application. The default value is 8.
ConcurrentCalculationWaitTimeInSecond - If the maximum number of concurrent calculations are running and a user launches a new calculation, the system will wait until the timeout and log a message in the system log about the timeout with the POV details. The default value is 30 seconds. If the maximum number of concurrent calculations are running and ConcurrentCalculationWaitTimeInSecond = -1, and a user launches a new calculation, then the system will wait infinitely until one of the calculations finishes."
An HFM Task Option feature is also mentioned. Quote:
"In Task Audit, the system now generates entries for Journal Period Opened, Journal Period Closed, Data Locked, and Data Unlocked. Each lock and unlock audit event will have a description indicating which cell or cells were locked or unlocked, including the Scenario, Year, Entity (and descendants if applicable), and Period."
Everything else I noticed pertains to the back-up technology uplift to Ora

Tuesday, January 14, 2020

Windows 7 End of Life is Here

One of my colleagues shared his post about Windows 7 End of Life so I thought I'd mention it here.  I work with some customers who still have Windows 7 in their environment, whether it be physical desktops/laptops or virtual desktops (e.g. VDI).

Windows 7 is due to hit Microsoft End of Life today... January 14, 2020.

No pressure, right?

Fortunately, both Hyperion / Oracle EPM and 11.2 work with Windows 10.  EPM doesn't care which desktop operating system you use if you're only using Oracle SmartView for Office and EPM Workspace.

If you're on EPM and want to migrate to Windows 10, you may want to check out this Oracle Knowledge Base article:

"Enterprise Performance Management Certification for Windows 10 with IE11 (Doc ID 2085672.1)"

I'm not aware of any Windows 10 caveats where EPM 11.2 is concerned.

Feb 22, 2020 update - Comments on this blog entry are now locked, thanks to spammers.