If You're Enjoying My Posts Here....

With heavy heart I announce my employer laid me off today.  Anyone who knows the sales dynamic between Oracle EPM Cloud vs. Oracle EPM On-Premises requires no further explanation.

It was a good run.

This means my access to Oracle Support (patches and Knowledge Base) and eDelivery ought to be cut off.  I'll cut myself off, of course.  (Nobody expects the Spanish Inquisition)

I believe on-premises Hyperion / Oracle EPM has good years ahead.  The vast majority of mid-market companies I've spoken with over the past few years aren't ready to move to the cloud, and want to stay on-premises for at least the next 3 years.  Perhaps longer.  On-premises upgrades are not going away.

The dates are fluid, but we are being told EPM on-premises will be fully supported through "at least" 2030 and I heard 2031 tossed out there as well.  Safe Harbor applies; Oracle Corporation reserves the right to change their support policy.  This is just the best information I have in writing as of today.

Time to catch up on overdue yardwork and prepare the homestead for an Iowa winter.

Then it'll be time to dust off my resume and pound the virtual pavement.

Anyway, the rate of my posts on this blog will slow down until I find a new sponsor for an Oracle Support/eDelivery account.

So if 11.2.4+ or when Essbase 21 come out and you want me to test and write about them, click the link on the right-hand side of this page for my LinkedIn profile and you may contact me there to discuss a sponsorship.  Sorry, existing customers of my ex-employer may not contact me for this due to non-compete.  I write a separate post when my non-compete eventually expires.

Be well,

Dave

 

EPM 11.2.3.0 In-Place Upgrade: Why I Did It, and Issues Encountered

As mentioned in my previous post about EPM 11.2.3.0, I decided to attempt an in-place upgrade from Hyperion / Oracle EPM 11.2.2.0 to 11.2.3.0.

Before I launch into this discussion, I want to be very up-front:  I do not like EPM in-place upgrades.  They are fraught with risk as you never know what issues you'll encounter.  Once you encounter them, you have to work non-stop to fix them because the system you've attempted to upgrade is now non-functional.

This spells disaster for a Production system where the customer only provides a weekend to get it all done; everything must be online by 7AM Monday morning.

So why did I even bother???

Let's look at the recent Oracle EPM 11.2.x release history:

1. EPM 11.2.0.0 released December 2019
2. EPM 11.2.1.0 released April 2020; Chrome and Edge fully certified, IE11 de-certified
   
3. EPM 11.2.2.0 released June 2020 and includes Linux
4. EPM 11.2.3.0 released sometime between September 15 - early October 2020

Now lets examine the list of patches for EPM 11.2.x:

1. Ziltch
2. None
3. Nada

Oracle is patching Essbase 11.1.2.4, and I'd have to check but I believe patches for DRM 11.2 are also on a separate track.  The rest of the EPM suite, however, only has bugfixes as provided in EPM Releases 11.2.1.0 through 11.2.3.0 as listed above.

The question lingers in my mind: will we see PSEs or PSUs start to trickle out for the non-Essbase components, or will we continue to see new "dot" releases every 3 months or so?

Doubting the former, I considered the latter.

If Oracle were to continue this trend of issuing dot releases instead of PSEs/PSUs, then us remaining on-premises Infrastructure people ought to get comfortable with performing in-place upgrades within the EPM 11.2.x.x releases.  Thus, my adventure began.

The above out of the way, here's my observations after performing the in-place upgrade and troubleshooting it.  In no particular order:

• The documentation is absolutely correct that we need to edit \Oracle\Middleware\EPMSystem11R1\products\Essbase\eas\console\bin\admincon.bat and change the reference of Java from Java 6 to Java 8.  Java 6 has been out of Extended Support for some time.
  
• Essbase Sample Applications not visible after installing them via installtool.  In 11.2.2.0, I didn't install the sample applications.  In 11.2.3.0 I wanted to test them.  Installtool said the install was a success.  After digging around, I found them in \Oracle\Middleware\EPMSystem11R1\products\Essbase\EssbaseServer\app
• This was NOT my ARBORPATH
• I checked epmsys_registry and \Oracle\Middleware\user_projects\epmsystem1\EssbaseServer\essbaseserver1\bin\setEssbaseEnv.bat -- both of them had the ARBORPATH I wanted
• I'm not sure if this is the correct method, but I edited \Oracle\Middleware\user_projects\epmsystem1\EssbaseServer\essbaseserver1\bin\server.properties and followed the comments within it.  I added an entry for ARBORPATH, bounced Essbase, and then I could migrate or create apps and they appeared where I wanted.
  
• installtool re-applies Middleware and Essbase patches every time it runs. This behavior is not new and dates back to at least 11.1.2.4.  I don't recall if 11.1.2.3 and prior did this.  When you see installtool "stuck" at 97-98%, that's what it is doing.  The lesson here is don't start applying Essbase patches or the quarterly critical Oracle Middleware patches until you're satisfied you're completely done using installtool.
• Analytic Provider Services does not work.  This behavior started in 11.2.0.0 and continues through 11.2.3.0.
• The error message in the WebLogic log is "weblogic.application.ModuleException: java.lang.RuntimeException: Failed to deploy/initialize the application as given archive is missing required standard webservice deployment decriptor."  ("descriptor" typo is Oracle's, not mine).
• The fix is to patch the Essbase Suite up to a higher patch level, such as 11.1.2.4.040 or .041 as of this writing.  Consult the APS patch README to see which other patches correlate to the Analytic Provider Services patch you want to apply.
  
• Oracle database clients, WebLogic Server, and OHS would not upgrade and installtool shows errors during the upgrade process.  Don't worry about this as you don't need to upgrade them.  11.2.0.0 through 11.2.3.0 use the same versions of this software.  Just ignore and move on!
• On a positive note, the Google Chrome "white screen" rendering bug in LCM has been fixed in 11.2.3.0!
• Consolidation Administration->Settings->System still has the same defaults as 11.1.2.4.
• MinDataCacheSizeInMB: 2.25GB -- This is likely too high for sandboxes and DEV environments.  It might be OK for UAT and PROD.  You decide!  Tuning is an art, not science.
  
• SessionManagerTimeoutInMS 1,200,000 (20 minutes) -- Many HFM admins asked me to triple it. This affects EPM Workspace AND SmartView.
  
• Another positive note. Oracle has done a great job cleaning up the many references within scripts to Java 6 that I mentioned in my more than a cupful of Java post.  EAS Console thick client launcher and MetadataMerge.bat are the last remaining offenders now.
• I completely understand why EAS Console is still referring to Java 6.  EAS Console is 11.1.2.4 software at this time, so it needs to work in an 11.1.2.4 system that uses Java 6.
  
• Another positive note. My Windows Registry fix for the Planning RMI issue was preserved.  Bookmark the post I linked if you need to use the classic Planning command-line utilities, as they need RMI running properly.
  

That's about it!  (But isn't that enough?)

EPM 11.2.3.0 Is Available on Oracle eDelivery

The EPM Certification Matrix has been updated for on-premises Hyperion / OracleEPM 11.2.3.0.

Update: Oracle posted the following on Oct 16, 2020: Oracle 11.2.3 Release Announcement

The online Feature Comparison Tool hasn't been updated as of when I checked earlier this morning.  The README, however, lists a boatload of bugfixes.  HFM shops should inspect the README carefully and allow sufficient time for rigorous regression testing.

There's about 2GB remaining in my download queue, and then it will be time to take it for a spin.

Initial observations:

• Still no certified migration path from 11.1.2.3 and older.
• The certification matrix says HFM isn't available for Linux 7 yet.  Fortunately, I don't have any customers screaming for this (yet).  I do have customers who've expressed interest in running Essbase on Linux 7 instead of Linux 6.  Linux 6 is a dead product and Linux 7 has faster disk I/O drivers, among other improvements.
  
• Installation media is only available for 64-bit Microsoft Windows and Linux 7.  Solaris and AIX ("AIX ain't UNIX") aren't listed.
• In-place "Apply Updates" upgrade available if you're on on 11.2.0.0 through 11.2.2.0.
• The LCM bug for Google Chrome has apparently been fixed.
• The README provides the workaround for Planning RMI not binding to its port.
• No mention if the FDMEE Linux bugs have been fixed or not.
• The README explains why I can't login to EAS; an Oracle WebLogic policy update is needed.  The problem may have been introduced in 11.2.2.0, as my 11.2.1.0 sandbox works just fine.
• The README has a very important note about LDAP hosted through MSAD; every Microsoft shop will need to get in front of this before Micosoft forces our hand in 2021.
  

FDMEE 11.2.2.0, Linux, and numerous bugs - UPDATE

As mentioned in my previous post, I was able to spend this weekend doing some R&D in Hyperion / Oracle EPM 11.2.2.0 in Linux instead of doing heads-down customer-related work.

One of the items re-visited were the numerous bugs encountered with FDM Enterprise Edition (FDMEE).  These bugs are specific to the Linux edition of EPM 11.2.2.0; the Windows edition of the same release works just fine.  To see all of the gory details, expand August 2020 on the right-hand side of this page to see my 3-part series about the FDMEE 11.2.2.0 bugs in Linux.  Part 1 of the 3-part series starts here.

If you're impatient and wish to avoid going through all of the error evidence, I can summarize FDMEE's problems in 2 major categories:

1. When running "Configure Database" for FDMEE, many dozens of ODI tables are missing.  The ODI Agent is thus unusable when FDMEE attempts to invoke it.
2. When running "Deploy to Application Server" for FDMEE, WebLogic's configuration is incomplete.  The FDMEE process starts and WebLogic says it enters RUNNING mode, but is essentially useless.

Again, for the sake of clarification, the above 2 problems are specific to FDMEE 11.2.2.0 hosted in Linux.

Here is the quick & dirty solution I tested this weekend, and it worked perfectly for me.  I hope this helps you, too!

Solution

Create a throwaway virtual machine running Microsoft Windows Server 2016 or 2019.  Download HFM Plus 11.2.2.0 and install it.  You do not need to create a database or run configtool; completing the install for FDMEE is enough.

On your Linux FDMEE server(s), stop FDMEE and rename these 2 things:

1. aif.ear within the Middleware/EPMSystem11R1/products/FinancialDataQuality directory structure.  It is easy to find if you're not familiar with the back-end filesystem.  Just cd to the folder I just mentioned and type:  find . -type f -name aif.ear
2. Middleware/odi

Point of clarification: I can't say for sure that aif.ear is contributing to the problem.  I can say with high confidence that Middleware/odi has problems in EPM 11.2.2.0's Linux edition!

Now grab the same 2 things from your Windows filesystem.  In the case of Middleware\odi, you'll want to zip it up in Windows and unzip it in Linux.  If FDMEE is clustered across multiple servers, do this on each.

Once the above is done, launch configtool.sh in Linux and re-run both Configure Database and Deploy to Application Server for just FDMEE.  Pick all options again even though they have the green checkmark.  When Configure Database asks if you want to Drop & Create or Re-Use, pick the Drop & Recreate option.  If you have FDMEE clustered across multiple servers, do Configure Database on just one of them, and Deploy to Application Server on all FDMEE servers.

Once configtool is complete, check your FDMEE database/schema.  All of the SNP* tables that were missing before ought to be present now.

rm -rf ErpIntegrator*/tmp and ErpIntegrator*/cache

Restart FDMEE and regression test.  Middleware/user_projects/yourinstance/bin/validate.sh ought to come back green now where FDMEE is concerned.

If this blog entry does not help, I implore you to open a Service Request with Oracle.  As of this writing, there are still no patches out there for either Hyperion Infrastructure (installtool+configtool) or FDM Enterprise Edition for 11.2.x.  With 11.2.2.0 for Linux being so new, and many companies not having an EPM upgrade budget approved for Fiscal 2020, Oracle may be unaware of this issue.  There will likely be a deluge of bug reports once Fiscal 2021 rolls around!

The 11.2.x upgrades my team & I performed thus far either haven't included FDMEE, or had FDMEE exclusively on MS Windows.  So I could open an SR now about the issue, but the SR wouldn't have the same standing as if the problem were holding up a customer's upgrade project.

Provider Services 11.2.2.0, Linux, and the WebLogic bug - UPDATE

I posted previously about my adventures working with Hyperion / Oracle EPM 11.2.2.0 in a Linux environment.  One thing that stood out, which I've reproduced and have also received confirmation about from some of my commenters here, is Analytic Provider Services (APS) produces a WebLogic error message during APS's startup sequence.

The error message is hard to decyper, but it essentially complains that something is misconfigured within the aps.ear file.

This error does not happen in the Windows counterpart of EPM 11.2.2.0.

I discovered the following 2 solutions during my regression testing, and wanted to share.  So here goes!

Solution 1

Stand up a little throwaway Windows Server 2016 or 2019 virtual machine.  Install Essbase Plus 11.2.2.0.  In installtool.bat, just pick Foundation and APS.  You don't need to configure/deploy.  Just the install is sufficient!  Once APS is installed, stop APS on Linux and then copy aps.ear from the Windows VM to your Linux server.  Clear AnalyticProviderServices0's \tmp and \cache subfolders.  Then restart APS in Linux and you should be good to go.

Or....

Solution 2

APS is actually a patch within the 11.1.2.4 stack.  The patch is over one year old (11.1.2.4.033 or 11.1.2.4.031).  The Linux version of this specific patch is apparently problematic.  In Linux, download and OPatch a higher series of patches for the Essbase stack, such as 11.1.2.4.040 or 11.1.2.4.041 as of this writing.  You will need to patch the entire Essbase stack: Essbase Agent, Essbase Runtime, Essbase Client, APS, and EAS Server.  The APS patch notes tell you which patches are certified for the specific patch version you decide to grab.

Stop the stack, run OPatch for the patches where needed, clear /tmp and /cache for each web service you patched, and restart.  No need to reconfig/redeploy.

My results

I've done both Solution 1 and Solution 2 within different test sandboxes.  Both worked for me.

Take care with respect to Solution 2 if you're running Essbase ASO cubes.  If you read through the patch notes from 11.1.2.4.033 through 11.1.2.4.041, you'll see lots of issues relating to ASO.  I suspect the Essbase developers at Oracle aren't done tweaking ASO yet.