Sunday, December 8, 2019

ODI Tables Missing in FDMEE Here's why!

So you did a fresh install of FDMEE, applied the latest patch, and then ran the config tool to "Create Database" and "Deploy To Application Server".  This is how I do it, as I hate repeating the same activity twice (before and after patching).

But lo' and behold, the ODI tables (SNP* etc) are missing within the FDMEE repository.  What's going on here?


On your FDMEE server, inspect this file.  If FDMEE is clustered across multiple servers, you need to do this on just one server only.  Let's assume your 1st FDMEE server is hosted on Windows and resides on the D drive:


Click the image if it doesn't render completely.  I'm an EPM Infrastructure guy and am thus graphically-challenged.  But the key nugget of information is this:  Do you see "" or ""?  Change that 3 to a 4!

The "FinancialDataQuality\odi\" folder doesn't exist in this patch, so your ODI Master and Work repository tables never get created!  The config wizard doesn't bark at you that you hit this problem.

Oh, there's one other thing.  In the later FDMEE patches, you get only one shot at creating the ODI tables within FDMEE.  Look within the same createODIRepos.bat file mentioned above. You'll see a line remarked out that was NOT remarked out in the unpatched version of the software.  Do you see it?  Post in the comments below and I'll give you a hint.  This remarked line is what makes the configurator wizard create the ODI Master and Work tables!

Fix the .bat file, re-run FDMEE's "Configure Database" task from the EPM Configurator on just one server.  No need to redeploy to WebLogic.  Restart FDMEE and you're good.

I can understand why Oracle decided to remark out that line; if you need to re-run "Configure Database" for FDMEE for any reason, maybe you don't want to lose your ODI customizations.  But what if the ODI tables never got created to start with?  Best to backup the original file, make your edits, run the configurator, and then when things are working OK you can restore your backup copy of the file to its original name.

Will this remain a bug in 11.2, or will it be fixed?  Time will tell!

Friday, December 6, 2019

Hot Off the Press: EPM 11.2 is Actually Coming in Dec 2019?

Oracle updated their EPM 11.2 Release Date blog post today, Dec 6, 2019, indicating Oracle is still projecting a release of EPM 11.2 sometime in December 2019 (*).  The precise day of the month isn't known yet.

(*) - Oracle Safe Harbor applies.  Date can be subject to change.

The blog post I've linked above contains a wealth of information, so I suggest you read it if you haven't checked it out lately.

Monday, November 25, 2019

Link to Nov 21 ODTUG Webinar recording

If you are a full ODTUG member, you may download a recording of last week's webinar I presented.

Here's the link: Support Expiration Dates


Date: 2019-11-22
Subtopic: EPM

Dave Shay, Datavail
Extended Support deadlines are drawing near for the technology undergirding on-premises Oracle EPM/Hyperion systems. Learn about vendor Extended Support deadlines for Java, Oracle JRockit, Microsoft Windows Server, Microsoft SQL Server, Linux, and Oracle EPM and prior, and learn how they will affect your EPM/Hyperion applications. While some of these dates are a few years away, others are not, which may surprise you.

Also, learn about implications of an upgrade vs. moving to the cloud if your system is subject to Sarbanes-Oxley or similar change audit controls. If your Oracle EPM system is subject to these controls, take note of ways to avoid being red-flagged in a future year’s SOX audit.

Monday, November 11, 2019

OCT2019CPU WebLogic "3L3H" Breaks Jython?

In a recent post, I listed the patches included in Oracle's October 2019 "Critical Patch Update".  Among them is a patch for Oracle WebLogic 10.3.6, which is part of the Oracle Middleware infrastructure that supports Hyperion

Customers who use Jython scripting within FDMEE may want to do extra regression testing before promoting this patch into Production.

A little background:

My team uses custom scripts to maintain 7 rolling days of Hyperion LCM backups.  One stumbling block with this is: when rolling off the 7th folder so it may be replaced, traditional MS-DOS "rmdir" doesn't work for certain subfolders in Windows Server 2012 R2 (usually Reporting Analysis Framework and Financial Close / ARM).  This is due to the famous "directory path too deep" error.

To work around this, we created a very simple Jython script.  Jython bypasses the Microsoft API, allowing us to delete folders containing paths which exceed the Microsoft limitation.

You do this:

SET CLASSPATH=%CLASSPATH%;%HYPDRIVE%\Oracle\Middleware\oracle_common\modules\oracle.jrf_11.1.1\jrf-wlstman.jar
%JAVA_HOME%\bin\java weblogic.WLST %SCRIPTDIR%/

And the Jython script looks like this:

# This Python/Jython script removes the oldest LCM backup folder.
# We use this technique to work around the Windows Server 2012
# limitation concerning directories containing pathnames exceeding
# 260 characters.
#  Written on 11/02/2016 by Dave Shay (Datavail)
# Modified on MM/DD/YYYY by Your Name - Briefly list changes

import shutil


I like simple solutions like this.  It uses technology that is already baked into the EPM infrastructure.  Now, a new challenger appears: WebLogic OCT2019CPU patch ID "3L3H".  This is what happens when trying to run our Jython script after applying the patch.

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

Problem invoking WLST - Traceback (innermost last):
  File "D:\Scripts\", line 13, in ?
  File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/shutil$py.class", line 117, in rmtree
  File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/shutil$py.class", line 132, in _build_cmdtuple
  File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/javaos$py.class", line 135, in listdir

The Rotation7 folder remains and now we essentially have 6 rolling days of backups instead of 7.

When I look at the "jython-modules.jar" file, I can see the timestamp changed to when it was patched.  It would appear Oracle accidentally broke something?

My team will open an SR with Oracle and try to get to the bottom of this.  In the meantime, I would tend to shy away from this specific patch.

Thursday, November 7, 2019

ODTUG Webinar Registration Link

Here's where to sign up for my upcoming ODTUG Webinar:

Registration Link

"Hyperion/EPM, Java, SQL Server…Upcoming Extended Support Deadlines and What They Mean for You
Dave Shay, Datavail"

This encapsulates information I've posted here across multiple threads, but I'll present it as a single discussion.

In the very least, some of you will finally get to hear my voice.  Please do not heckle... too much.  😆

Thursday, October 17, 2019

October 2019 Quarterly Critical Patch Updates announced for Hyperion

Hi EPM/Hyperion on-premises Infrastructure friends,

The following information is what I was able to dig up concerning OCT2019CPU, published just a few days ago.

Here are the specific patch ID #s which apply to Hyperion.

Oracle WebLogic Server, versions
(Available for and only)
November 11 update: Please consult WebLogic Breaks Jython? before promoting this patch to Production.

Oracle Java SE, version 7
30384776: Oracle JDK 7 Update 241 b31
Requires our upgrade procedure from Java 6 & JRockit 6 to Java 7 is performed first:
(Available for only)

Hyperion Data Relationship Management, version
(This is a stale patch – the last DRM patch is from Feb 2019)
Hyperion Enterprise Performance Management Architect, version
(Extremely stale patch from 2015 - only applies to customers who integrate EPMA with DRM)

Hyperion Financial Reporting, version
(patch is 4 months old)

Hyperion Planning
(patch is 3 months old)
Check the patch readme and Oracle forum for Essbase certification issues.  Prior Essbase patches were de-certified by Oracle for Planning  I don’t know if this has been fixed for .009.

Hyperion Shared Services
28314691: Patch Set Update: HSS
(patch is 1 year old. Depends upon prior PSEs being applied first and in a specific order.  Consult the readme)

EPM Workspace / RA Framework
Extreme care must be taken when downloading Workspace / RA Framework patches!
We must look for this text within the patch readme:

Unless things have changed, co-mingling RA Framework / Workspace with an system could corrupt Workspace and make it non-functional.

Source of the above patch info from
“Critical Patch Update (CPU) Program Oct 2019 Patch Availability Document (PAD) (Doc ID 2568292.1)”

Friday, October 4, 2019

I might host a webinar soon?!?

Stay tuned.  Submitted application and we'll see if it is accepted.

But I can already hear it in my head:   "I award you no points."

the "I Award You No Points" guy

Sunday, September 29, 2019

SOX and On-Premises EPM and prior

Let's talk about one of our favorite subjects: Sarbanes-Oxley compliance combined with on-premises Oracle EPM / Hyperion.

Auditors and IT Risk Management departments tend to frown on running SOX-relevant financial applications on systems where a vendor's Extended Support has expired.  Plain English: no ongoing defect remediation via patches, and no new security vulnerability patches.

As I wrote in a prior post, this ship has already sailed for:
  • Oracle EPM and prior versions
  • Microsoft Windows Server 2008 R2
  • Microsoft SQL Server 2008 (all Service Packs)
  • Java 6 and prior versions
  • JRockit 6
The next Extended Support expiration dates looming are:
  • Oracle EPM
  • Microsoft Windows Server 2012 R2
  • Microsoft SQL Server 2012 SP3
  • Java 7
Of the above, the first deadline we'll hit is December 2021, or December 2020, and that's for Oracle EPM / Hyperion  ("Safe Harbor" - I do not speak on behalf of Oracle Corporation, and their dates may change).  Standard Support is set to expire Dec 2020, and Extended Support is set to expire Dec 2021.  Check your Oracle Support contract to see which option you're on.

What readers need to consider is their timeline to either upgrade to EPM 11.2 (once released), or migrate to the Oracle EPM Cloud.

December 2021 seems like a long time away, but let's again re-visit SOX.

Let's say your fiscal year aligns with the calendar year: Jan:Dec.  In this scenario, SOX-relevant applications only get 2 windows per year to complete upgrades and do a go-live cutover to a new system:  May and September.  Shoot for May, and use September as your fall-back position.  Going live during either your fiscal 1st Quarter or 4th Quarter will trigger a red flag in your SOX audit.

So keep these dates in mind and then start counting backward.  Don't wait until late in 2021 to either upgrade or move to the cloud.  By then most EPM consulting partners, such as the firm I work for, will likely be slammed trying to hit that Sept 2021 SOX deadline.  I'm reminded of when Microsoft revoked support for browsers older than IE11... we were insanely busy because many customers were still on EPM or older, and IT Risk Management departments forced Finance to upgrade to remain compliant.

One final thought: I've recently been contacted by a competitor promising cheaper support rates than Oracle's.  I want to discourage people from considering this, unless you intend to completely retire Hyperion and switch to a different platform on or before Q3 2021.  A 3rd party partner/consultant will face legal problems if they are discovered installing patches or upgrades a former Oracle customer is no longer entitled to receive.

Friday, September 6, 2019

EPM 11.2: What Will the Upgrade Project Cost?

I'm getting asked these questions more and more as the EPM 11.2 release looms nearer:

"What will it cost me?"
"How long will it take?"

My answer tends to be:

"I won't know for sure until I see the Release Notes and install it in a lab."

But, I do have some directional answers.

Before I begin, however, there is a very important point:  You must remain current on your Oracle Support maintenance agreement.  This is the only way you can legally obtain the EPM 11.2 installation media.  A partner who uploads their own copy of the media will lose their partner status with Oracle and face legal action.

Cost and Time are related

Whether you use a partner or perform the upgrade in-house, cost is unavoidable.

If you do it in-house, you are essentially paying opportunity cost;  the people working on the upgrade are not working on other things to support the pre-existing system, do development work, etc.  If you use a partner, of course you are either paying Time & Materials or are paying Fixed Price.

Either way, there is cost involved from a labor perspective.

IT Infrastructure

You will want to do what Oracle calls an Out of Place upgrade (or what I call Lift & Shift).  See my post Why Upgrade to EPM 11.2 or Move to the Cloud on why you don't want to do an in-place upgrade.  (Speaking from a personal note, I hate in-place upgrades.  They're almost always messy)

Translated to English:
  1. Stand up new servers using a newer operating system, faster CPUs, etc.  Your existing servers are likely 4+ years old.  Size the CPU core count, RAM, and disk to be no less than what you're utilizing in current Production.  Remember that on an Essbase server, the disk needs to be double in size so you don't run out of space during an Essbase dense restructure.
  2. Stand up a new database server running newer database software, such as SQL Server 2016 instead of SQL Server 2012.
  3. Size the database schemas (Oracle 12.2), or databases (MS SQL Server 2016) to be no smaller than what you're using in current Production.  Think forward and increase the size to accommodate future growth for the next few years.  This is especially the case for folks who use HFM and/or FDMEE.
  4. Plan ahead and ask IT to add an Anti-Virus "On Access" scanning exception for D:\Oracle
For many companies, a cost is associated with this. IT allocates their data center costs back to the business units consuming those resources. For a period of time, you will be paying allocated expenses for both your old environments and the new.

EPM Architect

Somewhere between 40-50% of the EPM customer population I work with use EPMA.  EPMA is gone in 11.2, and replaced by DRM.

Oracle has stated a utility will be made available that converts EPMA content into DRM.  That being said, account for some extra testing time by your Subject Matter Experts (and your partner if you will use one).

Financial Reporting

The architecture has changed and "Reporting & Analysis Framework" has been deprecated.  I'm a big fan of this; RA Framework is always messy to troubleshoot when it won't start back up.  The way reports are stored within Hyperion LCM has changed as a result.  Assume there will be additional time spent on:

  1. Migrating reports from the old system into the new (hopefully the migration utility handles this OK).
  2. Migrating objects that aren't reports, but are stored within Workspace.  Such as PDFs, videos, MS Word documents, MS Excel documents, etc.
  3. Updating LCM backup scripts, if you have nightly LCM backups enabled.  (You should!!!)
The Great Unknown

From prior experience, I can say the first 5-6 months of a new EPM release can be.... interesting.
Especially where HFM and Essbase are concerned.

Have you counted the number of patches released for Essbase alone in  I've been using Essbase since the 3.2 days in the mid-90s, and have never seen so many patches for it since  Expect perhaps some bumps in the road, especially if you use ASO Essbase.

Critical Security Patches During Your Project

Congratulations, you are now back on fully-supported software, from an Operating System, Database, and Oracle EPM standpoint.  (EPM has about 1-2 years to go, depending upon your support contract).

Most EPM shops haven't moved their servers from Java 6 to Java 7.  Now you will be on Java 8.  Oracle issues security patches (actually full installs) for both Java 7 and Java 8 every three months.  You will also be on a newer version of Oracle WebLogic:  You're going to move from WebLogic 10 to 12.  This also gets critical security patches issued every Quarter.

This means, you can anticipate at least 1 major outage during your upgrade project, multiplied by the number of EPM 11.2 environments you expect to have (e.g. DEV, UAT & PROD).  You will want to account for this in your Project Plan.

OK, But How Long Does an Upgrade Normally Take?

My rule of thumb is 5-7 business days per environment to:
  1. Install
  2. Patch
  3. Config/Deploy
  4. Migrate apps
This rule gets thrown out the window when we're talking about a brand new release.  Take my guestimate and double it.

Add more time if you use SSL and/or SSO. Where SSL is concerned, you may need new SSL certificates. The new system is expected to comply with TLS 1.2, whereas EPM and prior could only use TLS 1.0. For SSO, I expect no changes to Shared Services / EPM Foundation, but we'll have to wait and see!

Workforce, CapEx, Project Modules for Planning

OK, here's the fun one.  These modules don't exist in Planning 11.2.  If you currently use them, put some extra hours+cost in your Project Plan for them.  At best, we could LCM them into 11.2 and they will work OK.  At worst, they will have to be re-implemented from scratch.  Yuck!

Hyperion Tax - Gone!

If you're one of the few who use the Tax module, add some hours+cost to consider how this case would be handled.

If memory serves, the same will be true for Profitability, Essbase Studio and Strategic Finance.  Oracle has the final say and we won't know for sure until the Release Notes are issued.

Essbase Studio - If it is Gone and You Use Drill-Through, What Next?

Not everyone uses Essbase Studio for drill-through.  But if you do, plan on spending time in Design, Build and Test for whatever alternate solution you choose to use.

SOX Auditors

Who doesn't love them?

If your system is SOX-relevant, you might face extra cost if you utilize external auditors. The auditors, and your own people who interact with them, may need to perform their activities twice during the fiscal year when your EPM 11.2 Production cut-over goes live; one audit for the old system, and one audit for the new system.

This extra cost can be avoided if you cut over so the first month of the new fiscal year is operated within the new system. Many customers I've interacted with have difficulty achieving this, due to a variety of factors.

In Conclusion

Don't plan to go "all in" once the software is released.  Check out my post Action Plan for Early EPM 11.2 Adopters and plan on running a test lab / sandbox to "kick the tires" before you spin up the servers for the "real" EPM 11.2 environment that you plan on using.

Now it is time for me to go back and keep refreshing the Oracle eDelivery page....

Thursday, September 5, 2019

EPM 11.2 Release Date - Bookmark Oracle's Post

Oracle's EPM 11.2 On-Premises Release Date Blog Post

As of this writing, both the blog post linked above and the Oracle Knowledge Base article which links to the above (last dated Aug 1, 2019) list an "Oracle Safe Harbor" projected release date of September 2019.

Late October 2019 Update: "Safe Harbor" has been bumped from December 2019.  As always, "wait and see"!

Be sure to bookmark the above and check back.

The primary Oracle Knowledge Base article to check would be "When will EPM 11.2 become Available? (Doc ID 2553915.1)"

October 2, 2019 Update: the article 2553915.1 mentioned above isn't available anymore!

For those of you unfamiliar with term "Oracle Safe Harbor", it essentially means the posted information is directional in nature, and could change.  The full text of Oracle Safe Harbor can be read within the Knowledge Base.  Just put "EPM 11.2" into the KB's search box and several articles on this topic will appear.