Weekend LinkedIn Commentary

Cross-posting what I wrote on my LinkedIn.

"Glad to see so many questions quietly coming in via private chat about Oracle EPM 11.2. I encourage anyone considering an on-premises upgrade in 2021 to spin up a VM now and make sure you can successfully get through RCU and the EPM Configurator. Don't wait until your project is scheduled to start in 2021.... act now while things are quiet and you still have time to troubleshoot. This will go a long way to have a successful project in 2021!

Please do NOT send any diagrams or logfiles to me unless we have non-disclosure and SOW paperwork in place."

One additional point I made in private chat concerning FDMEE problems.

If you get errors about missing SNP_* tables when starting up FDMEE, the installation media is the likely culprit. Open a SEV-2 SR with Oracle so they know about the issue. 

 

While you wait from a response from Oracle, you could try this approach:

Create a new VM that is specifically for FDMEE. Download EPM 11.2.2.0 and do a fresh FDMEE install on that VM.  Download FDMEE 11.2.2.0 as soon as you read this post, as I question the 11.2.3.0 base installation media for FDMEE/ODI and it will eventually get removed from eDelivery.

 

You do not need to configure or RCU because this VM will be thrown away once you get through this exercise. Then throw the latest EPM 11.2.x (11.2.3.0 as of this writing) on top of it and do a Maintenance in-place update.

Finally, zip up Oracle\Middleware\odi\ on that VM and copy it over to the troublesome FDMEE machines. Re-run the configurator and pick "Configure Database" for FDMEE. Then check your database and verify the many dozen ODI tables are now present.

Your results may vary. Back up the odi filesystem on the FDMEE machine(s) before you replace it. Keep in mind the above is a suggestion and might not have Oracle's blessing at this time.

2021: EPM 11.2 Upgrade AND Data Center Change? Things to Consider

Many of my readers are looking at a situation where something needs to be done to their EPM system in Fiscal 2021:  Either upgrade, move to Oracle EPM Cloud, or move to a 3rd party solution such as OneStream.

The deciding factors are End Of Life for Hyperion / Oracle EPM 11.1.2.4 in DEC 2021, Microsoft IE11, Microsoft Server 2012 and prior, Microsoft SQL Server 2012 and prior, Linux 6, and so on.  I've written about these topics before and you can find details if you search through this blog's history.

One thing I would like to caution my readers about is conflating multiple significant changes in a single shot.  Such as:  Upgrading from EPM 11.1.2.4 or prior to EPM 11.2.x, AND doing a shift at the same time from an on-premises data center to a hosted cloud such as OCI, Azure, Amazon AWS, CenturyLink, etc.

Yes, many organizations combine these activities into a single project, often with success.  But please do consider one factor that goes beyond technology: your friends in Finance.

Finance MUST be involved in a significant project such as this, as they will need to validate the new system has apples-to-apples data results and performance (or better).

I mention this for the case where things don't quite match up right: data is wrong, or performance worsened.  Combining a significant version upgrade AND a shift in data center hosting leaves doubt as to where the problem stems from.

This being said, I'd like to offer general answers to questions I've received over the past few years along these lines.  Bear in mind these are general answers and every organization is different.  As I pose general questions & answers, if anything I say below triggers a further question, I highly suggest finding a Partner and locking them in now.  Don't wait until Spring/Summer 2021 as by then many Partners will already be booked for the year for their 2021 EPM 11.2 upgrade projects.

My favorite analogy along these lines relates back to my experience living in the US MidWest: try finding a competent roofer after a major storm passes through town.  The local experts are all booked, and you could be stuck with somebody from out of town - dubious credentials - who won't be around when you find problems later.

So without further ado, here we go....

Q: Any certification issues between Amazon AWS, Microsoft Azure, and Oracle Cloud Infrastructure (OCI) that I need to know about?

A:  No.  Certification revolves around the guest virtual operating system and relational database, not the physical host.  If you speak with Oracle Sales and if you are using Oracle RDBMS for Hyperion, you will be "guided" toward OCI because of database licensing.  This topic can get pretty detailed and is beyond the scope of this blog article.  Suffice to say, if you're using Oracle RDMBS and are looking at shifting from on-premises to hosted, make sure BOTH an IT Director AND your IT Procurement folks are involved.

Q: May I use my existing hardware to host EPM 11.2?

A: You can... but consider this upgrade project as an opportunity to leverage a hardware refresh for faster performance.  If your EPM platform is virtualized, the virtual guests can be moved to faster hardware as IT finds capacity.  If your EPM platform is physical and not virtual, you should either consider virtualizing (a topic for another blog post I could spend some time writing) or a hardware refresh.  Your physical EPM 11.1.2.4 servers are at least 5 years old and it is time for a speed upgrade.

Q: I want to change my database platform from X to Y.  Any issues?

A: If you're just running Essbase, no issues and go for it!  If you're running HFM or Financial Close, be careful.  By "change my database platform" I'm not talking about a version change -- say from Oracle 10 to 19, or MSSQL 2012 to 2016.  I'm talking about switching between Oracle & MSSQL in either direction.

The Hyperion Financial Close / Account Reconciliation suite for on-premises is a special animal.  You really should look at involving a Partner if you're on 11.1.2.3 or older.

HFM is very touchy when it comes to its database artifacts.  If you are switching between Oracle & MSSQ, just expect your project will see a 15-20% overhead due to working Service Requests (SRs) with Oracle.  Switching your database type for HFM isn't supported out-of-the-box and you will hit issues.

Q: How is EPM 11.2 different in terms of requirements for CPU cores, memory and disk?

A: The EPM Foundation 11.2 service takes a minute or two longer to start up, unless you have the WebLogic Admin Server (WLS) process running.  WLS is a bit hungerier for memory in this release.

Generally speaking, the various web processes (CalcMgr, EAS, Foundation, APS, etc) have about the same requirements in EPM 11.2 as in EPM 11.1.2.3.500 through EPM 11.1.2.4.x.  That is, you want to reserve 1 CPU core and 1.5GB-2GB of RAM for each web process.  Hyperion Planning and Hyperion Financial Reporting are exceptions due to a Java memory leak: set aside 8GB for each.

The above is just general advice.  Think of it as a rough rule of thumb.  When in a formal upgrade project, I'd want about an hour to speak with the IT Director/architect and DBA to dive into the details and develop an environment diagram appropriate to the situation at hand.

Q: Which is better?  Amazon AWS, Microsoft Azure, Oracle Cloud Infrastructure (OCI), or 3rd party host XYZ?

A: How much money do you have?  All kidding aside....

Spinning up EPM 11.2 in a hosted environment looks and feels exactly the same as on-premises.  Your users' experience ultimate depends upon several things:

1. Network connectivity between the users' location(s) and the hosted data center.
2. In the specific case of Essbase, is the Essbase server able to use disk equivalent to solid state or not?
3. In the specific cases of HFM, FCM and/or ARM, how fast is the database?
4. Backups and disaster recovery.  Can you control the backup retention, and is the backup / DR plan reliable?
5. Bad guys.  Assume 2021 will be worse than 2020 in terms of malware, data leaks, extortion, etc.  Couple this thought with what I just mentioned about backups and DR.  If you stay on-premises, you have your IT Security folks to yell at.  If you shift to a 3rd party host, now it is their problem to deal with, there must also be trust between you and them.
   

Q: IT wants to enact Secure Socket Layer (SSL) for everything.  What do I need to know about EPM 11.2?

A: This is a growing trend I've observed.  Be careful if you're using MSSQL Server and your DBA wants to turn on SSL.  EPM 11.2 requires Oracle's Repository Creation Utility (RCU), and it doesn't like MSSQL with SSL enabled.  Last I checked, no workaround has been published by Oracle for this specific configuration.  Be warned and be careful.

The good news is on the Oracle WebLogic and Oracle HTTP Server (OHS) side of the fence, EPM 11.2 allows more secure SSL/TLS protocols than EPM 11.1.2.4 and prior.  It is still messy as you have to deal with SSL certificates that eventually expire, but EPM 11.1.2.4 and prior use insecure SSL protocols that are essentially worthless.  EPM 11.2 leaps forward multiple generations on the back-end, and as a result TLS in particular is much better.

------

I've just scratched the surface here, but I hope you have a few things to consider now.  Again, as I said previously this blog article is general in nature, and may not be suited for your specific circumstance.  Safe Harbor, yada yada yada.

Hey Why Does My New SmartView Keep Timing Out?

Oracle SmartView for Office still needs the Windows Registry keys it has been wanting since... the very beginning really.  It does not matter if we're talking Hyperion / Oracle EPM on-premises or Oracle EPM Cloud.

If your user has upgraded to a new Windows 10 machine and is complaining about SmartView timeout issues, the culprit is almost always the Windows Registry.  This is what a non-tuned user's workstation looks like, from my point of view:

The registry key names you see in this example screenshot are shorthand for the full registry key name.

These registry keys and suggested values were taken directly out of a series of Oracle Knowledge Base articles that have been available for a very long time.

Here we're looking at two areas of focus: network and MS Office animations.

Network of course is the true silver bullet here.  Get the keys applied, reboot the user's workstation, and the problem is nearly always solved.

The animation keys help with weird flickering issues occasionally reported by SmartView users.  I haven't heard any of these complaints recently, as the flickering problem is very Office version-specific.  I've written about this issue here before.  If your user has the on-premises version of Office 365, they are likely OK and wouldn't have seen the issue.  It still wouldn't hurt to have the keys, though.

So here's the script that generates the above screenshot.  Within this very simple script -- nothing proprietary here! -- you can see the full key names and values expected.

 

echo ------------------------
echo KeepAliveTimeout should be set to dword:2bf20
echo Here is the current value:
reg query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v KeepAliveTimeout

echo ------------------------
echo ServerInfoTimeout should be set to dword:2bf20
echo Here is the current value:
reg query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ServerInfoTimeout

echo ------------------------
echo ReceiveTimeout should be set to dword:75300
echo Here is the current value:
reg query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ReceiveTimeout

echo ------------------------
echo DisableWindowTransitionsOnAddinTaskPanes should be set to dword:1
echo Here is the current value:
reg query HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Common\Toolbars /v DisableWindowTransitionsOnAddinTaskPanes

echo ------------------------
echo DisableAnimations should be set to dword:1
echo Here is the current value:
reg query HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Common\Graphics /v DisableAnimations

echo All done checking!
pause


One final point...  #MISSING suppression is handled on the end-user's workstation, not on the Essbase server (on-premises or Cloud - it does not matter).  If user requests a massive grid in either ad-hoc mode or by opening a Planning form in SmartView, the Essbase server is going to process it.  Essbase ships the complete grid back to SmartView and lets the chips fall where they may.  (The communication flow is more complex than this - I'm just trying to keep this post simple)

I mention this because I've seen some users attempt to "data mine" Essbase.  The mindset here is "give me everything and I'll sift through it in a linked sheet or in MS Access".  In my view, Essbase & Planning were never intended to be used this way.

When a user requests a grid containing a half million or more possible cells, it does not matter what you do to either the user's Windows Registry, the essbase.cfg file (only if you're on-premises), or tweaks to the cube's dense/sparse settings.  SmartView will remain locked up until the entire grid is fetched and processed.  I've noticed grid retrieval performance starts to get ugly when we're looking at 300,000 possible cells and higher.

Where on-premises is concerned, massive grid retrievals can lead to "OutOfMemoryException" errors in either Planning or Analytic Provider Services, depending upon the type of SmartView connection used.   You can bet these errors also happen in Oracle EPM Cloud, since the underlying technology is the same, but the Oracle WebLogic logs aren't exposed to us in the Cloud.  Bumping up the on-premises "-Xmx" registry keys for Planning & APS will provide some relief (e.g. changing from the 4GB default to 8GB), but all we're really doing here is throwing more RAM at the problem; it won't speed anything up.

The underlying issue to address, unfortunately, is end-user behavior.  Usually it is only 1-3 people within the organization who are trying to datamine the entire cube, and you'll typically find them in FP&A.  If you're in Oracle EPM Cloud, use EPMAutomate to grab the daily audit logs and that should help you identify who is pulling down large grids.  On-premises has different ways EPM Detectives such as yourself can probe and find who is bringing the system to its knees. 

So apply the keys recommended above, reboot the workstation, and keep retrieval slice sizes within reason.

If You're Enjoying My Posts Here....

With heavy heart I announce my employer laid me off today.  Anyone who knows the sales dynamic between Oracle EPM Cloud vs. Oracle EPM On-Premises requires no further explanation.

It was a good run.

This means my access to Oracle Support (patches and Knowledge Base) and eDelivery ought to be cut off.  I'll cut myself off, of course.  (Nobody expects the Spanish Inquisition)

I believe on-premises Hyperion / Oracle EPM has good years ahead.  The vast majority of mid-market companies I've spoken with over the past few years aren't ready to move to the cloud, and want to stay on-premises for at least the next 3 years.  Perhaps longer.  On-premises upgrades are not going away.

The dates are fluid, but we are being told EPM on-premises will be fully supported through "at least" 2030 and I heard 2031 tossed out there as well.  Safe Harbor applies; Oracle Corporation reserves the right to change their support policy.  This is just the best information I have in writing as of today.

Time to catch up on overdue yardwork and prepare the homestead for an Iowa winter.

Then it'll be time to dust off my resume and pound the virtual pavement.

Anyway, the rate of my posts on this blog will slow down until I find a new sponsor for an Oracle Support/eDelivery account.

So if 11.2.4+ or when Essbase 21 come out and you want me to test and write about them, click the link on the right-hand side of this page for my LinkedIn profile and you may contact me there to discuss a sponsorship.  Sorry, existing customers of my ex-employer may not contact me for this due to non-compete.  I write a separate post when my non-compete eventually expires.

Be well,

Dave

 

EPM 11.2.3.0 In-Place Upgrade: Why I Did It, and Issues Encountered

As mentioned in my previous post about EPM 11.2.3.0, I decided to attempt an in-place upgrade from Hyperion / Oracle EPM 11.2.2.0 to 11.2.3.0.

Before I launch into this discussion, I want to be very up-front:  I do not like EPM in-place upgrades.  They are fraught with risk as you never know what issues you'll encounter.  Once you encounter them, you have to work non-stop to fix them because the system you've attempted to upgrade is now non-functional.

This spells disaster for a Production system where the customer only provides a weekend to get it all done; everything must be online by 7AM Monday morning.

So why did I even bother???

Let's look at the recent Oracle EPM 11.2.x release history:

1. EPM 11.2.0.0 released December 2019
2. EPM 11.2.1.0 released April 2020; Chrome and Edge fully certified, IE11 de-certified
   
3. EPM 11.2.2.0 released June 2020 and includes Linux
4. EPM 11.2.3.0 released sometime between September 15 - early October 2020

Now lets examine the list of patches for EPM 11.2.x:

1. Ziltch
2. None
3. Nada

Oracle is patching Essbase 11.1.2.4, and I'd have to check but I believe patches for DRM 11.2 are also on a separate track.  The rest of the EPM suite, however, only has bugfixes as provided in EPM Releases 11.2.1.0 through 11.2.3.0 as listed above.

The question lingers in my mind: will we see PSEs or PSUs start to trickle out for the non-Essbase components, or will we continue to see new "dot" releases every 3 months or so?

Doubting the former, I considered the latter.

If Oracle were to continue this trend of issuing dot releases instead of PSEs/PSUs, then us remaining on-premises Infrastructure people ought to get comfortable with performing in-place upgrades within the EPM 11.2.x.x releases.  Thus, my adventure began.

The above out of the way, here's my observations after performing the in-place upgrade and troubleshooting it.  In no particular order:

• The documentation is absolutely correct that we need to edit \Oracle\Middleware\EPMSystem11R1\products\Essbase\eas\console\bin\admincon.bat and change the reference of Java from Java 6 to Java 8.  Java 6 has been out of Extended Support for some time.
  
• Essbase Sample Applications not visible after installing them via installtool.  In 11.2.2.0, I didn't install the sample applications.  In 11.2.3.0 I wanted to test them.  Installtool said the install was a success.  After digging around, I found them in \Oracle\Middleware\EPMSystem11R1\products\Essbase\EssbaseServer\app
• This was NOT my ARBORPATH
• I checked epmsys_registry and \Oracle\Middleware\user_projects\epmsystem1\EssbaseServer\essbaseserver1\bin\setEssbaseEnv.bat -- both of them had the ARBORPATH I wanted
• I'm not sure if this is the correct method, but I edited \Oracle\Middleware\user_projects\epmsystem1\EssbaseServer\essbaseserver1\bin\server.properties and followed the comments within it.  I added an entry for ARBORPATH, bounced Essbase, and then I could migrate or create apps and they appeared where I wanted.
  
• installtool re-applies Middleware and Essbase patches every time it runs. This behavior is not new and dates back to at least 11.1.2.4.  I don't recall if 11.1.2.3 and prior did this.  When you see installtool "stuck" at 97-98%, that's what it is doing.  The lesson here is don't start applying Essbase patches or the quarterly critical Oracle Middleware patches until you're satisfied you're completely done using installtool.
• Analytic Provider Services does not work.  This behavior started in 11.2.0.0 and continues through 11.2.3.0.
• The error message in the WebLogic log is "weblogic.application.ModuleException: java.lang.RuntimeException: Failed to deploy/initialize the application as given archive is missing required standard webservice deployment decriptor."  ("descriptor" typo is Oracle's, not mine).
• The fix is to patch the Essbase Suite up to a higher patch level, such as 11.1.2.4.040 or .041 as of this writing.  Consult the APS patch README to see which other patches correlate to the Analytic Provider Services patch you want to apply.
  
• Oracle database clients, WebLogic Server, and OHS would not upgrade and installtool shows errors during the upgrade process.  Don't worry about this as you don't need to upgrade them.  11.2.0.0 through 11.2.3.0 use the same versions of this software.  Just ignore and move on!
• On a positive note, the Google Chrome "white screen" rendering bug in LCM has been fixed in 11.2.3.0!
• Consolidation Administration->Settings->System still has the same defaults as 11.1.2.4.
• MinDataCacheSizeInMB: 2.25GB -- This is likely too high for sandboxes and DEV environments.  It might be OK for UAT and PROD.  You decide!  Tuning is an art, not science.
  
• SessionManagerTimeoutInMS 1,200,000 (20 minutes) -- Many HFM admins asked me to triple it. This affects EPM Workspace AND SmartView.
  
• Another positive note. Oracle has done a great job cleaning up the many references within scripts to Java 6 that I mentioned in my more than a cupful of Java post.  EAS Console thick client launcher and MetadataMerge.bat are the last remaining offenders now.
• I completely understand why EAS Console is still referring to Java 6.  EAS Console is 11.1.2.4 software at this time, so it needs to work in an 11.1.2.4 system that uses Java 6.
  
• Another positive note. My Windows Registry fix for the Planning RMI issue was preserved.  Bookmark the post I linked if you need to use the classic Planning command-line utilities, as they need RMI running properly.
  

That's about it!  (But isn't that enough?)

EPM 11.2.3.0 Is Available on Oracle eDelivery

The EPM Certification Matrix has been updated for on-premises Hyperion / OracleEPM 11.2.3.0.

Update: Oracle posted the following on Oct 16, 2020: Oracle 11.2.3 Release Announcement

The online Feature Comparison Tool hasn't been updated as of when I checked earlier this morning.  The README, however, lists a boatload of bugfixes.  HFM shops should inspect the README carefully and allow sufficient time for rigorous regression testing.

There's about 2GB remaining in my download queue, and then it will be time to take it for a spin.

Initial observations:

• Still no certified migration path from 11.1.2.3 and older.
• The certification matrix says HFM isn't available for Linux 7 yet.  Fortunately, I don't have any customers screaming for this (yet).  I do have customers who've expressed interest in running Essbase on Linux 7 instead of Linux 6.  Linux 6 is a dead product and Linux 7 has faster disk I/O drivers, among other improvements.
  
• Installation media is only available for 64-bit Microsoft Windows and Linux 7.  Solaris and AIX ("AIX ain't UNIX") aren't listed.
• In-place "Apply Updates" upgrade available if you're on on 11.2.0.0 through 11.2.2.0.
• The LCM bug for Google Chrome has apparently been fixed.
• The README provides the workaround for Planning RMI not binding to its port.
• No mention if the FDMEE Linux bugs have been fixed or not.
• The README explains why I can't login to EAS; an Oracle WebLogic policy update is needed.  The problem may have been introduced in 11.2.2.0, as my 11.2.1.0 sandbox works just fine.
• The README has a very important note about LDAP hosted through MSAD; every Microsoft shop will need to get in front of this before Micosoft forces our hand in 2021.
  

FDMEE 11.2.2.0, Linux, and numerous bugs - UPDATE

As mentioned in my previous post, I was able to spend this weekend doing some R&D in Hyperion / Oracle EPM 11.2.2.0 in Linux instead of doing heads-down customer-related work.

One of the items re-visited were the numerous bugs encountered with FDM Enterprise Edition (FDMEE).  These bugs are specific to the Linux edition of EPM 11.2.2.0; the Windows edition of the same release works just fine.  To see all of the gory details, expand August 2020 on the right-hand side of this page to see my 3-part series about the FDMEE 11.2.2.0 bugs in Linux.  Part 1 of the 3-part series starts here.

If you're impatient and wish to avoid going through all of the error evidence, I can summarize FDMEE's problems in 2 major categories:

1. When running "Configure Database" for FDMEE, many dozens of ODI tables are missing.  The ODI Agent is thus unusable when FDMEE attempts to invoke it.
2. When running "Deploy to Application Server" for FDMEE, WebLogic's configuration is incomplete.  The FDMEE process starts and WebLogic says it enters RUNNING mode, but is essentially useless.

Again, for the sake of clarification, the above 2 problems are specific to FDMEE 11.2.2.0 hosted in Linux.

Here is the quick & dirty solution I tested this weekend, and it worked perfectly for me.  I hope this helps you, too!

Solution

Create a throwaway virtual machine running Microsoft Windows Server 2016 or 2019.  Download HFM Plus 11.2.2.0 and install it.  You do not need to create a database or run configtool; completing the install for FDMEE is enough.

On your Linux FDMEE server(s), stop FDMEE and rename these 2 things:

1. aif.ear within the Middleware/EPMSystem11R1/products/FinancialDataQuality directory structure.  It is easy to find if you're not familiar with the back-end filesystem.  Just cd to the folder I just mentioned and type:  find . -type f -name aif.ear
2. Middleware/odi

Point of clarification: I can't say for sure that aif.ear is contributing to the problem.  I can say with high confidence that Middleware/odi has problems in EPM 11.2.2.0's Linux edition!

Now grab the same 2 things from your Windows filesystem.  In the case of Middleware\odi, you'll want to zip it up in Windows and unzip it in Linux.  If FDMEE is clustered across multiple servers, do this on each.

Once the above is done, launch configtool.sh in Linux and re-run both Configure Database and Deploy to Application Server for just FDMEE.  Pick all options again even though they have the green checkmark.  When Configure Database asks if you want to Drop & Create or Re-Use, pick the Drop & Recreate option.  If you have FDMEE clustered across multiple servers, do Configure Database on just one of them, and Deploy to Application Server on all FDMEE servers.

Once configtool is complete, check your FDMEE database/schema.  All of the SNP* tables that were missing before ought to be present now.

rm -rf ErpIntegrator*/tmp and ErpIntegrator*/cache

Restart FDMEE and regression test.  Middleware/user_projects/yourinstance/bin/validate.sh ought to come back green now where FDMEE is concerned.

If this blog entry does not help, I implore you to open a Service Request with Oracle.  As of this writing, there are still no patches out there for either Hyperion Infrastructure (installtool+configtool) or FDM Enterprise Edition for 11.2.x.  With 11.2.2.0 for Linux being so new, and many companies not having an EPM upgrade budget approved for Fiscal 2020, Oracle may be unaware of this issue.  There will likely be a deluge of bug reports once Fiscal 2021 rolls around!

The 11.2.x upgrades my team & I performed thus far either haven't included FDMEE, or had FDMEE exclusively on MS Windows.  So I could open an SR now about the issue, but the SR wouldn't have the same standing as if the problem were holding up a customer's upgrade project.

Provider Services 11.2.2.0, Linux, and the WebLogic bug - UPDATE

I posted previously about my adventures working with Hyperion / Oracle EPM 11.2.2.0 in a Linux environment.  One thing that stood out, which I've reproduced and have also received confirmation about from some of my commenters here, is Analytic Provider Services (APS) produces a WebLogic error message during APS's startup sequence.

The error message is hard to decyper, but it essentially complains that something is misconfigured within the aps.ear file.

This error does not happen in the Windows counterpart of EPM 11.2.2.0.

I discovered the following 2 solutions during my regression testing, and wanted to share.  So here goes!

Solution 1

Stand up a little throwaway Windows Server 2016 or 2019 virtual machine.  Install Essbase Plus 11.2.2.0.  In installtool.bat, just pick Foundation and APS.  You don't need to configure/deploy.  Just the install is sufficient!  Once APS is installed, stop APS on Linux and then copy aps.ear from the Windows VM to your Linux server.  Clear AnalyticProviderServices0's \tmp and \cache subfolders.  Then restart APS in Linux and you should be good to go.

Or....

Solution 2

APS is actually a patch within the 11.1.2.4 stack.  The patch is over one year old (11.1.2.4.033 or 11.1.2.4.031).  The Linux version of this specific patch is apparently problematic.  In Linux, download and OPatch a higher series of patches for the Essbase stack, such as 11.1.2.4.040 or 11.1.2.4.041 as of this writing.  You will need to patch the entire Essbase stack: Essbase Agent, Essbase Runtime, Essbase Client, APS, and EAS Server.  The APS patch notes tell you which patches are certified for the specific patch version you decide to grab.

Stop the stack, run OPatch for the patches where needed, clear /tmp and /cache for each web service you patched, and restart.  No need to reconfig/redeploy.

My results

I've done both Solution 1 and Solution 2 within different test sandboxes.  Both worked for me.

Take care with respect to Solution 2 if you're running Essbase ASO cubes.  If you read through the patch notes from 11.1.2.4.033 through 11.1.2.4.041, you'll see lots of issues relating to ASO.  I suspect the Essbase developers at Oracle aren't done tweaking ASO yet.

If you could be Dave Shay for a day...

No, I don't wish this on anyone.

But in seriousness, suppose you could set my speaking agenda. What would you want me to talk about?

ODTUG is looking for abstracts. Rather than throwing pasta at the wall, how about stuff you care about?

I only ask for no RCU.

ZeroLogon Vulnerability and EPM On-Premises AND Cloud

I don't normally write about Microsoft vulnerabilities and related patches, but this one is important for all Oracle EPM / Hyperion instances... whether on-premises or in Oracle's EPM SaaS Cloud.

A little background.  Vulnerabilities are ranked on a score from 0.1 to 10.0.  What I'm about to discuss here is a 10.0, which is the most dangerous score.

The official designation of this particular critter is "CVE-2020-1472".  Independent security research firms, such as Secura, refer to it as ZeroLogon.

Microsoft issued a patch for it in August 2020's "Patch Tuesday", but the extent of the problem wasn't fully known at the time.

If you want to read the gory details, you can check out Secura's white paper on the subject.  I'll summarize in brief:

The vulnerability allows anyone having access to the network to become a Windows Domain Administrator.  You don't even need network credentials if you stroll into the office and plug a device into an Ethernet port.  Remote workers, of course, often have the access required. The point being that once the attacker runs the exploit and elevates himself to a domain admin, or creates a new domain admin account with a known password, he can cause all sorts of mischief with far-reaching consequences throughout the organization.

Now let's talk about EPM, starting with on-premises and then moving on to Oracle's EPM SaaS Cloud (PBC, FCC, etc.).

Microsoft Active Directory ("MSAD") is ubiquitous within the on-premises EPM space.  The vast majority of EPM implementations I've supported, installed, or health checked use MSAD for end-user authentication.  Hyperion Shared Services and the various EPM components connect to a Windows Domain Controller in order to authenticate end-user login attempts.

Disclaimer: the following paragraph contains theoretical conjecture.  We won't know the effects for sure until an non-patched system is attacked. 

Our fictional attacker who exploits ZeroLogon can completely break this.  Worse, the attacker could kick the EPM servers out of the domain, making it hard to hop on the EPM servers and troubleshoot why nobody can login.

I have worked with a few customers who use alternatives to Microsoft for end-user authentication, such as Novell eDirectory or other LDAP solutions.  By and large, though, there can be a Microsoft Windows Domain lurking somewhere within the network.

They key takeaway here is EPM system stakeholders should inquire with the IT department and confirm the Domain Controllers have had the August 2020 Microsoft patches applied.  I've noticed it is a mixed bag "out in the wild"; some organizations patch immediately, while others lag behind... especially during financial Quarter-End or Year-End change freezes.

Now let's talk Cloud briefly.

Oracle's EPM SaaS Cloud products for Consolidation, Planning, Account Rec, etc. all share one thing in common: EPMAutomate.

EPMAutomate is the Cloud's command-line utility used for a variety of tasks: upload data to the Cloud, run it through Data Management, fire off Calculation Rules, download reports and audit logs, and more.  EPMAutomate resides on a server under the customer's control, either on-premises or in a hosted cloud such as AWS, Azure, OCI, etc.  The vast majority of EPMAutomate implementations I've seen happen to sit on MS Windows servers.  (It can be hosted on Linux, and sometimes I witness that variation)

If EPMAutomate is hosted on MS Windows, and that machine happens to be joined to the MS Windows Domain... well, there's a possibility your EPM Cloud automation might stop working someday if an intruder bricks your network account or kicks the EPMAutomate host server out of the domain.  (Again, I use the word possibility until we see the fallout when it eventually happens)

2020 has been an awful year thus far, so please do your part not to make it... awful-er.  Insist your network domain controllers get patched for "CVE-2020-1472", included in August 2020 Microsoft Patch Tuesday.

Hat tip: Penetration Tester Dustin Heywood

Adobe Flash Player EOL and On-Premises EPM 11.1.2.x.

If you are already live on Oracle EPM / Hyperion 11.2.x (you brave soul!) or in the Oracle EPM SaaS cloud, this post isn't for you.

EPM 11.1.2.3.500 through 11.1.2.4.x both have dependencies upon Adobe Flash Player on the end-user side for Hyperion Calculation Manager and Hyperion Planning.  Hyperion Financial Close / Account Rec in 11.1.2.3 also uses Flash.

Adobe announced earlier that End Of Life for Flash Player is December 31, 2020.  Not far away now!

So let's review the scenarios for on-premises EPM:

• For Hyperion Planning 11.1.2.4, patch 31365862 – 11.1.2.4.010 takes care of this.
• For Hyperion Calculation Manager 11.1.2.4, patch 28557058 – 11.1.2.4.014 takes care of this.
• For 11.1.2.3 and older, there is no solution other than upgrading or moving to the cloud.
• For 11.2.0.0 and higher, the solution is already baked into the base release and there is no need to patch.

What does it mean if you don't patch before the deadline, or you're on an older release and can't patch at all?

Adobe has stated very clearly that the ability to download Flash Player will be removed once the support deadline of Dec 31, 2020 has passed.  Neither the latest version nor older versions will be available to download.  Furthermore, no new security patches will be issued.

I haven't seen this in writing, but expect Firefox to quickly flag the Flash Player extension as vulnerable in January 2021.  I wouldn't be surprised at all if the extension gets disabled without the option to re-enable it.

Adobe has further stated that a Flash Player installer downloaded from any 3rd party site will be considered "Unauthorized".

If you're on 11.1.2.4 and haven't applied the patches I mentioned above, my recommendation would be to patch and regression test now before you enter a fiscal 2020 4th Quarter change freeze.

There's an added benefit if you're working directly on a server for testing purposes. Flash Player often isn't installed on MS Windows Server 2012 and is hard to get.  The latest download page on Adobe's website flags your browser as coming from Windows 8 and shows a Knowledge Base article instead of letting you download the installer.  If you need to get into the Calculation Manager Rules or Variable designers, the browser wants to invoke Flash Player and the page hangs.  This exact issue hit me yesterday, which led me down the road of investigating these patches.

(A quick reminder to carefully read the patch READMEs for the 2 patches I listed.  Planning 11.1.2.4.010 contains a new optional application property, some files need to be copied over to the Financial Reporting server, and the CalcMgr patch needs to be installed on multiple machines in a typical distributed environment)

EPM 11.2.2.0 - Notable Differences Between Windows & Linux

Weekends sometimes offer time for some leisurely R&D (especially with the current lock-down / mask rules).  I posted previously about some issues I noticed in the Linux edition of Oracle EPM / Hyperion 11.2.2.0.  Well, this weekend I finished standing up the Windows edition.  Here are some key configuration errors/issues I spotted.

FDM Enterprise Edition (FDMEE)

You'll find a 3-part post concerning 11.2.2.0 FDMEE errors in Linux.  In the Windows edition, I encountered none of those issues.  The ODI tables built successfully, and FDMEE in Workspace, ODI Web Studio and ODI Agent were all functional.

Suggestion for Linux users:

Install the 11.2.2.0 Windows edition (you don't need to configure) and zip up \Oracle\Middleware\odi, ftp it over to your Linux server, and unzip it with the Replace option in /Oracle/Middleware/odi

Also copy aif.ear from Windows to Linux.

Then re-try your Configure Database and Deploy for FDMEE.

Planning RMI

Edit: An earlier version of this post had an incorrect link to the Planning RMI fix.  This is now corrected.

This starts up just fine in Linux and successfully binds to port 11333.  In Windows, we still have the issue I wrote about for 11.2.0.0 and 11.2.1.0.  You'll find the solution to fixing the Windows Registry for Planning RMI here on my blog.

Analytic Provider Services

In Windows, it deploys just fine.  In Linux, it gets a deployment error.

Suggested solution for Linux users:

Install the Windows edition of EPM 11.2.2.0 (you don't need to deploy).  Copy aps.ear from Windows to Linux, and then redeploy in Linux.

Hyperion Financial Reporting 11.2.x LCM Issue

I don't have a solution for this yet, but the problem I noticed in Oracle EPM / Hyperion Financial Reporting 11.2.0.0 carries forward into 11.2.1.0 and 11.2.2.0.

The Hyperion LCM command-line utility works OK for Financial Reporting (now "Document Repository"), but the user interface in the Shared Services Console does not.  This issue is specific to "Document Repository" only.

The observed symptom happens when trying to import or export Document Repository within the Shared Services Console.  When attempting an export, you'll see the infamous EPMLCM-13000 error, which could mean just about anything (usually it means the module is non-functional, offline, or you lack permission to the module in question).  When attempting an import, it says you don't have permission to Document Repository.

The culprit, however, is the EPM System Configurator tool.

Here are "before" and "after" screenshots of the legacy FRConfig tool (\Oracle\Middleware\EPMSystem11R1\products\financialreporting\bin\frconfig.bat|sh).

Before.  Here is after we only deployed EPM Foundation, Financial Reporting, and nothing else:

 

The correct dbName is shown above.  This is the dbName I use for EPM Foundation.  In EPM 11.2.x, the Reporting Analysis Framework repository is no more, and instead Financial Reporting's tables ought to (in my opinion) co-located with EPM Foundation.  Look at the table names and you'll see why.

And here is after we deployed a few more modules.

Here you can see the configuration was switched to use FDMEE's database.

Note that I did NOT touch Financial Reporting when deploying FDMEE.

It is blind luck whether Financial Reporting's configuration stays as-is, or is switched behind the scenes to use Planning, FDMEE, or some other product that has its own database.

The dbName and DbURL properties are read-only, despite the very old Oracle Knowledge Base articles concerning the FRConfig tool.

Your workarounds until Oracle issues a patch are:

1. Perform Exports and Imports through Workspace Explore ("the old way"), or
2. Use the LCM command-line utility.  Here pasted below is an LCM XML file for Document Repository that works for me in 11.2.x:

<?xml version="1.0" encoding="UTF-8"?>
<Package>
    <LOCALE>en_US</LOCALE>
    <User name="admin" password="admin's password here"/>
    <Task>
        <Source type="Application" product="DOCREP" project="Default Application Group" application="Document Repository"/>
        <Target type="FileSystem" filePath="/DOCREP-Document Repository"/>
        <Artifact recursive="true" parentPath="/Repository Objects" pattern="*"/>
    </Task>
</Package>

Syntax I use to invoke the utility is:

CD F:\Oracle\Middleware\user_projects\epmsystem1\bin
SET XMLDIR=folder where you've saved ExportReports.xml as pasted above
SET LCMDIR=folder where you want the exports to be saved
start "ExportReports" /wait /i cmd /c Utility.bat %XMLDIR%\ExportReports.xml -b %LCMDIR%\Reports ^> %LCMDIR%\ExportReports.log 2^> %LCMDIR%\ExportReports.err

Oracle's "utility.bat" only works on the EPM Foundation server, and only when you're sitting in your Instance /bin folder.  A workaround is available to run it on other machines, and Oracle has a Knowledge Base article about this.

The start /wait /i cmd /c utility.bat syntax is helpful if your script is going to do other operations after this one.  Oracle's utility.bat invokes a Java program that will cause your script to immediately bomb and exit without warning should the Java program abnormally exits for any reason.

Utility.bat|sh works both ways (it can import as well as export) and the LifeCycle Management User's Guide contains the syntax for this.

Check back with support.oracle.com from time to time to see if a patch has come addressing this.  In addition to checking for Financial Reporting, also check "Hyperion Infrastructure" as fixes to configtool may appear within this section.