Thursday, December 26, 2019

EPM 11.2, RCU and You!

Feb 8, 2020 Update:  There's a follow up blog entry concerning EPM 11.2 Distributed Environments but please first read through the blog entry below, as it contains per-requisite knowledge.  Thank you!


There's a venerable utility for Oracle Fusion Middleware that has been available for quite some time.  Those of us who configured Oracle Data Integrator in the pre-FDMEE days, Hyperion Financial Close Management Suite through, or Oracle Business Intelligence know what I'm about to discuss.

Oracle Repository Creation Utility, or "RCU" for short.

In EPM and prior, you didn't need to worry about running RCU if you weren't using the modules I mentioned within my first paragraph.

Those days are behind us now.  EPM 11.2 requires that you run through the RCU process before you attempt to create your Oracle WebLogic domain and deploy your EPM Foundation web service.

R2D2 says: "Technically, EPM Foundation is an Oracle WebLogic Managed Server."
C3P0 responds: "Don't get technical with me [further insults added here]."

So let's talk about what has changed in EPM 11.2.  Your actions will depend upon if you're using Microsoft SQL Server or an Oracle database.

If you're using Microsoft SQL Server, the steps are a little more complex because your DBA needs to create an empty database for you.  Furthermore, the DBA needs to run these queries against that database before you may continue and run RCU.  Let's assume the DBA has named the RCU database "epmrcu":


Note the COLLATE option is slightly different for RCU's database vs. everything else EPM wants to use.

Source:  SQL Server Database Creation Requirements

The above steps aren't needed if you're using an Oracle Database; RCU creates the schemas and associated users for you.

Now for both MS SQL Server or Oracle Database, you need to fire up Oracle/Middleware/oracle_common/bin/rcu.bat and follow the instructions here:
Using RCU 

Unless you're a DBA, you will need a DBA's assistance.  The RCU utility requires someone with elevated rights to type in their credentials for a DBA superuser account ("sa" for MS SQL or "sys as sysdba" for Oracle).  There's the option to save the commands to a SQL query so your DBA can run it on their own, but I've experienced mixed results with this in the past.

So you now have your RCU database initialized.  There's just one more manual step to do.

On each server where any Oracle WebLogic components run, you need to manually edit this file:


Here's the fun part:  passwords are exposed in plain text until you run through configTool on the server, at which time the passwords become encrypted.
Source: Updating File

(Edit: I blew away my MS Server 2016 VM and spun up MS Server 2019, after being advised by Oracle that Server 2016 hasn't been certified yet for EPM 11.2.  After running through RCU, I inspected the file and noticed encrypted passwords.  I still had to update the JDBC URL and some other information)

Once you complete all of these tasks, you may finally launch configTool and deploy your WebLogic Admin Server and your various WebLogic Managed Services.  If you overlooked these instructions from the config guide, unfortunately you will need to blow away the entire domains/EPMSystem folder structure and start over.

Eying the Christmas Dinner leftovers in my fridge.... I may be a little delayed on writing my next post!  👀

Saturday, December 21, 2019

Installing EPM 11.2: Things Waiting to Trip You Up

Christmas 2019 came early this year, and just about every EPM gear-head I know is trying to install EPM so it may be taken for a spin.

Not so fast!  There's a few things to take care of first!

Christmas Eve 2019 update: Stay tuned!  New information has come in from Oracle and I will update this post after I finish a few plates of Christmas Day ham.  First and foremost, Windows Server 2016  is NOT certified by Oracle for EPM 11.2 at this time.  Please upgrade your sandboxes to Windows Server 2019.  Use Oracle 12 or SQL Server 2016 as your RDBMS until further guidance.

January 2020 update: Oracle's feedback is posted here: EPM 11.2 Certification Matrix post
Look for the text in red font.

EPM only worked up through MS Windows Server 2012 R2.  EPM 11.2 works on 2016 and 2019.  (My previous post about the 11.2 Certification Matrix mentions 2016 as a potential omission)

MS Windows Server 2012 R2 imposed some new changes that weren't a problem for us in 2008 R2.  Server 2016 and 2019 present the same challenges, plus newer ones.

Before doing anything, change this Windows Registry key from 1 to 0 and reboot.


You will thank me later.  If you find a few days later that the key's value reverted from 0 back to 1, contact your IT department and get an exception to Group Policy Update so your EPM servers are excluded from the change to this key.

Here's the classic symptoms this key is wrong:
  • Pop-up message asking for permission to change something within a directory.
  • You try to pick a folder for the Oracle eDelivery download helper, and it says your access is denied.
  • You manage to try a deployment and notice a folder named "Oracle EPM System" in the root of your Hyperion/EPM drive.
 Now for the fun ones:  Windows Defender and Windows Firewall

Windows Firewall.  Don't stop the service.  Your Remote Desktop session will immediately be terminated and you'll need someone with console access to turn the service back on.  Instead, go into the Windows Firewall settings and turn off notifications and blocking for Domain and Local network.  Leave it turned on for Public network.  Otherwise you will have to deal with a mess of individual port exceptions.

Windows Defender.  Go into the Windows Defender panel and add on-access scanning exceptions for where you're downloading the software to, and where you're installing it to.  Otherwise, look forward to a 20-30% performance degradation.  Unlike 2012 and prior, you can't disable or stop this Windows service.  There are also firewall settings in this service and you'll need to inspect and modify them.

Finally, was the install/config so bad that you need to blow away either user_projects or EPMSystem11R1?  You may hit the "path too long" error Microsoft helpfully introduced.  The poor man's solution is to install freely available 7-zip.  Launch 7-zip, do File->Open and pick the parent folder.  Then left-click the child folder to be deleted, and click Shift-Delete.  This will blow it away without encountering the Microsoft error.

Thoughts On EPM 11.2 Certification Matrix

I updated my earlier post Important Oracle EPM 11.2 Links with a link to the EPM 11.2 Certification Matrix

I'd like to share a few thoughts about what's currently in this matrix.

Christmas Eve 2019 update: Stay tuned!  New information has come in from Oracle and I will update this post after I finish a few plates of Christmas Day ham.  First and foremost, Windows Server 2016  is NOT certified by Oracle for EPM 11.2 at this time.  Please upgrade your sandboxes to Windows Server 2019.  Use Oracle 12 or SQL Server 2016 as your RDBMS until further guidance.

New Years Day Update January 1 2020: Answers to my questions below have been addressed by a senior person in Oracle.  I posted the answers in red text immediately after the relevant question.  If you are color blind, fear not as I prefixed the text with "ORACLE'S ANSWER".  I offered to provide accreditation to the Oracle person who was kind enough to read through this post thoroughly and provided a thoughtful response, but the person asked not to be named.  Again, a big THANK YOU to the person at Oracle who provided these answers!  We now have more clarity, as you will see in the red text below!  As the Oracle person asked not to be identified, I paraphrased the answers in my own words.

Questions posed:

  • Why isn't Windows Server 2016 listed?  ORACLE'S ANSWER: Windows Server 2019 was the focus of Q&A testing because it has the "longest runway" from a Microsoft Extended Support perspective.  It might work on 2016 and 2012, but your mileage will vary and is not guaranteed.
  • Yes, Windows Server 2012 R2 is coming out of Microsoft Extended Support in a few years, but will it work there? ORACLE'S ANSWER: As above, not worth the R&D / Quality Assurance time as Server 2012 has a limited shelf-life from a Microsoft perspective.  DATAVAIL ANSWER:  Ping me in LinkedIn if you couldn't attend my ODTUG webinar on this topic and I can quickly debrief you.
  • The only version of MS SQL Server listed is 2016.  (One person told me they tried it on 2019 and it didn't work, so he's downgrading to 2016).  ORACLE'S ANSWER: From a Microsoft perspective, SQL Server 2016 is the version certified for Oracle Fusion Middleware 12c.
  • The Firefox ESR version numbers are not listed.  In EPM, certain EPM screens stopped working once you surpassed a certain Firefox version.  Has this been fixed in 11.2?  PERSONAL ANSWER: Once the 2020 New Year champagne wears off I need to go back and check if this was answered.
  • Windows 10 is the only supported Windows OS listed.  What about 8.1?  Not everyone has finished their migration to 10 yet.  ORACLE'S ANSWER: 8.1 is coming out of Microsoft Extended Support soon, so testing/QA emphasis was spent on Windows 10 from a desktop perspective.  DATAVAIL'S ANSWER: Windows 7 is close to end-of-life and a desktop upgrade to Windows 10 is recommended.
  • Why is Windows Server 2012 listed as a certified client OS, but not as a certified server OS?  ORACLE'S ANSWER: As above.  Server testing was focused on MS Server 2019 due to the predicted shelf life.  PERSONAL ANSWER: Server 2012 is essentially the same technology as Desktop 10, so this kind of makes sense.
  • JRE 8 is the only client Java plug-in supported?  Is the EPM system not backward-compatible for organizations who haven't completed their client desktop updates yet?  ORACLE AND PERSONAL JOINT ANSWER: Java 6 came out of Oracle Extended Support on Jan 1 2019.  Java 7 has a limited shelf life and the security updates for it are only available to paying customers; the general public doesn't have access to those updates.  Java 8 has the longest shelf life until a subsequent patch is issued for EPM 11.2 that certifies newer versions.
  • Certification Matrices for and prior listed Microsoft Office versions.  Microsoft Office is conspicuously missing here. We know 365 Cloud won't be certified until a newer version of SmartView is released, but could we get some details in the meantime.  ORACLE'S ANSWER: Oracle SmartView for Office is the only desktop client remaining that cares about the version of MS Office.  A conscious decision was made to remove Office from the matrix so as to reduce confusion.
No mention of Microsoft Edge.  We know this is on the roadmap, but it would be nice to see a statement in the matrix to the effect of "not supported yet; to be certified in a subsequent patch" or "it is certified now".  ORACLE'S ANSWER: IE11 for now. Stay tuned.

Speaking for myself, I've got a secret underground bunker / mad scientist lab with a 6-month Trial Edition of MS Windows Server 2016 and MS SQL Server 2016 Developer Edition.  I've gotten through the EPM install and config, and everything works fine except for Oracle HTTP Server 12.  I'll write a separate post on that once I complete the troubleshooting process and can actually login.

The certification matrix is widely considered to be a "one stop shop"on these types of questions, rather than having to wade through a variety of readmes, installation and admin guides.  I'd love to see the matrix tightened up and improved.

Did I miss any of your questions?  Post them in the comments below!

Wednesday, December 18, 2019

EPM 11.2: No More Financial Reporting Essbase Connections?

I'm starting to crawl through the various Oracle Hyperion / EPM 11.2 readmes, and spotted this nugget:

Hyperion BI Plus. You can’t migrate Release reports that used Essbase as a data source to Release 11.2. In Release 11.2, Oracle Hyperion Financial Reporting, which is now a component of Hyperion Financial Applications, continues to support connections to a Planning data source using an Essbase connection, as well as Profitability and Cost Management.
Across my customer portfolio, there are a sizable number of reports that use the Essbase data source.  This will be a migration impact.

I'm worried how this will impact customers who DON'T have Hyperion Planning, and just have Essbase.  There's quite a few of these environments out there.  If a customer doesn't have Planning, what happens when they have HFR reports that hit Essbase?

In all prior releases, when you setup data connections for HFR, you picked the data source type.  If you picked Planning, it would automatically "know" about the Planning apps because that information was picked up from Shared Services.  In this brave new world of EPM 11.2, what happens if you pick a Planning data source type for a pure Essbase app?  Would you even see it?

I hope to know the answer later tonight.  If I learn anything material, I will update this post.

EPM 11.2 Important Oracle Links

I will update this post after I can find the updated Certification Matrix for EPM 11.2.

Here's Oracle's EPM 11.2 Release Acccouncement

EPM 11.2 Documentation Landing Page.  This is your one-stop-shop for all of the administration and installation guides.

You can download the software by signing in with your Oracle Support credentials to Oracle eDelivery
Once you login, type "EPM" into the search box and then click the Search button.
You will see multiple "" download packages.  You don't need to grab them all, as there is overlap across many of them.

Bookmark this page and come back to it soon.  As of this writing, it doesn't have the Supported Platform / Certification Matrix for EPM 11.2, but I suspect it will be updated very soon:
Hyperion Supported Platforms

Dec 20, 2019 Update: The EPM 11.2 Certification Matrix is now available.

Generally speaking, you will want to download:
  • Oracle HTTP Server 12
  • ODI Studio 12 if you are doing anything with FDMEE
  • EPM zips  Grab the full set and then you can pick and choose which modules to install when you run through the installation tool.
  • Windows Client Installers
  • Installation Readmes
Some of the packages include optional packages you might not need, such as Oracle SOA or Oracle Identity Manager.

Analysis to follow after I've had a chance to finish downloading and can spin it up in a sandbox.

EPM 11.2 Is Here

My shortest blog post ever.

Analysis to follow separately.

Sunday, December 8, 2019

ODI Tables Missing in FDMEE Here's why!

So you did a fresh install of FDMEE, applied the latest patch, and then ran the config tool to "Create Database" and "Deploy To Application Server".  This is how I do it, as I hate repeating the same activity twice (before and after patching).

But lo' and behold, the ODI tables (SNP* etc) are missing within the FDMEE repository.  What's going on here?


On your FDMEE server, inspect this file.  If FDMEE is clustered across multiple servers, you need to do this on just one server only.  Let's assume your 1st FDMEE server is hosted on Windows and resides on the D drive:


Click the image if it doesn't render completely.  I'm an EPM Infrastructure guy and am thus graphically-challenged.  But the key nugget of information is this:  Do you see "" or ""?  Change that 3 to a 4!

The "FinancialDataQuality\odi\" folder doesn't exist in this patch, so your ODI Master and Work repository tables never get created!  The config wizard doesn't bark at you that you hit this problem.

Oh, there's one other thing.  In the later FDMEE patches, you get only one shot at creating the ODI tables within FDMEE.  Look within the same createODIRepos.bat file mentioned above. You'll see a line remarked out that was NOT remarked out in the unpatched version of the software.  Do you see it?  Post in the comments below and I'll give you a hint.  This remarked line is what makes the configurator wizard create the ODI Master and Work tables!

Fix the .bat file, re-run FDMEE's "Configure Database" task from the EPM Configurator on just one server.  No need to redeploy to WebLogic.  Restart FDMEE and you're good.

I can understand why Oracle decided to remark out that line; if you need to re-run "Configure Database" for FDMEE for any reason, maybe you don't want to lose your ODI customizations.  But what if the ODI tables never got created to start with?  Best to backup the original file, make your edits, run the configurator, and then when things are working OK you can restore your backup copy of the file to its original name.

Will this remain a bug in 11.2, or will it be fixed?  Time will tell!

Friday, December 6, 2019

Hot Off the Press: EPM 11.2 is Actually Coming in Dec 2019?

Oracle updated their EPM 11.2 Release Date blog post today, Dec 6, 2019, indicating Oracle is still projecting a release of EPM 11.2 sometime in December 2019 (*).  The precise day of the month isn't known yet.

(*) - Oracle Safe Harbor applies.  Date can be subject to change.

The blog post I've linked above contains a wealth of information, so I suggest you read it if you haven't checked it out lately.

Monday, November 25, 2019

Link to Nov 21 ODTUG Webinar recording

If you are a full ODTUG member, you may download a recording of last week's webinar I presented.

Here's the link: Support Expiration Dates


Date: 2019-11-22
Subtopic: EPM

Dave Shay, Datavail
Extended Support deadlines are drawing near for the technology undergirding on-premises Oracle EPM/Hyperion systems. Learn about vendor Extended Support deadlines for Java, Oracle JRockit, Microsoft Windows Server, Microsoft SQL Server, Linux, and Oracle EPM and prior, and learn how they will affect your EPM/Hyperion applications. While some of these dates are a few years away, others are not, which may surprise you.

Also, learn about implications of an upgrade vs. moving to the cloud if your system is subject to Sarbanes-Oxley or similar change audit controls. If your Oracle EPM system is subject to these controls, take note of ways to avoid being red-flagged in a future year’s SOX audit.

Monday, November 11, 2019

OCT2019CPU WebLogic "3L3H" Breaks Jython?

In a recent post, I listed the patches included in Oracle's October 2019 "Critical Patch Update".  Among them is a patch for Oracle WebLogic 10.3.6, which is part of the Oracle Middleware infrastructure that supports Hyperion

Customers who use Jython scripting within FDMEE may want to do extra regression testing before promoting this patch into Production.

A little background:

My team uses custom scripts to maintain 7 rolling days of Hyperion LCM backups.  One stumbling block with this is: when rolling off the 7th folder so it may be replaced, traditional MS-DOS "rmdir" doesn't work for certain subfolders in Windows Server 2012 R2 (usually Reporting Analysis Framework and Financial Close / ARM).  This is due to the famous "directory path too deep" error.

To work around this, we created a very simple Jython script.  Jython bypasses the Microsoft API, allowing us to delete folders containing paths which exceed the Microsoft limitation.

You do this:

SET CLASSPATH=%CLASSPATH%;%HYPDRIVE%\Oracle\Middleware\oracle_common\modules\oracle.jrf_11.1.1\jrf-wlstman.jar
%JAVA_HOME%\bin\java weblogic.WLST %SCRIPTDIR%/

And the Jython script looks like this:

# This Python/Jython script removes the oldest LCM backup folder.
# We use this technique to work around the Windows Server 2012
# limitation concerning directories containing pathnames exceeding
# 260 characters.
#  Written on 11/02/2016 by Dave Shay (Datavail)
# Modified on MM/DD/YYYY by Your Name - Briefly list changes

import shutil


I like simple solutions like this.  It uses technology that is already baked into the EPM infrastructure.  Now, a new challenger appears: WebLogic OCT2019CPU patch ID "3L3H".  This is what happens when trying to run our Jython script after applying the patch.

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Problem invoking WLST - Traceback (innermost last):
  File "D:\Scripts\", line 13, in ?
  File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/shutil$py.class", line 117, in rmtree
  File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/shutil$py.class", line 132, in _build_cmdtuple
  File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/javaos$py.class", line 135, in listdir

The Rotation7 folder remains and now we essentially have 6 rolling days of backups instead of 7.

When I look at the "jython-modules.jar" file, I can see the timestamp changed to when it was patched.  It would appear Oracle accidentally broke something?

My team will open an SR with Oracle and try to get to the bottom of this.  In the meantime, I would tend to shy away from this specific patch.

Thursday, November 7, 2019

ODTUG Webinar Registration Link

Here's where to sign up for my upcoming ODTUG Webinar:

Registration Link

"Hyperion/EPM, Java, SQL Server…Upcoming Extended Support Deadlines and What They Mean for You
Dave Shay, Datavail"

This encapsulates information I've posted here across multiple threads, but I'll present it as a single discussion.

In the very least, some of you will finally get to hear my voice.  Please do not heckle... too much.  😆

Thursday, October 17, 2019

October 2019 Quarterly Critical Patch Updates announced for Hyperion

Hi EPM/Hyperion on-premises Infrastructure friends,

The following information is what I was able to dig up concerning OCT2019CPU, published just a few days ago.

Here are the specific patch ID #s which apply to Hyperion.

Oracle WebLogic Server, versions
(Available for and only)
November 11 update: Please consult WebLogic Breaks Jython? before promoting this patch to Production.

Oracle Java SE, version 7
30384776: Oracle JDK 7 Update 241 b31
Requires our upgrade procedure from Java 6 & JRockit 6 to Java 7 is performed first:
(Available for only)

Hyperion Data Relationship Management, version
(This is a stale patch – the last DRM patch is from Feb 2019)
Hyperion Enterprise Performance Management Architect, version
(Extremely stale patch from 2015 - only applies to customers who integrate EPMA with DRM)

Hyperion Financial Reporting, version
(patch is 4 months old)

Hyperion Planning
(patch is 3 months old)
Check the patch readme and Oracle forum for Essbase certification issues.  Prior Essbase patches were de-certified by Oracle for Planning  I don’t know if this has been fixed for .009.

Hyperion Shared Services
28314691: Patch Set Update: HSS
(patch is 1 year old. Depends upon prior PSEs being applied first and in a specific order.  Consult the readme)

EPM Workspace / RA Framework
Extreme care must be taken when downloading Workspace / RA Framework patches!
We must look for this text within the patch readme:

Unless things have changed, co-mingling RA Framework / Workspace with an system could corrupt Workspace and make it non-functional.

Source of the above patch info from
“Critical Patch Update (CPU) Program Oct 2019 Patch Availability Document (PAD) (Doc ID 2568292.1)”