Tuesday, January 12, 2021

Essbase 21 - Initial Thoughts

Oracle Hyperion Essbase 21 came out in December 2020.  I didn't notice it as my Oracle eDelivery searches are always for "EPM".  With the arrival of Essbase 21, we need to search for "Essbase" in eDelivery.

Before attempting to install it, it is very important to read the Essbase 21 Installation Guide

I reviewed the guide over the weekend and wanted to offer my initial thoughts.

  • Red Hat 7 and 8, and Oracle Linux 7 and 8 are the only platforms supported.
  • Oracle VirtualBox is the only virtualization technique certified.  The usual caveat applies here;  many Hyperion shops use VMWare for virtualization because it is so ubiquitous across IT organizations.  When opening an SR with Oracle, don't tell them you're using VMWare or there's a small chance the SR can be kicked back.
  • We're required to install a GNU c/c++ compiler.
  • Essbase 21 is supposed to be able to co-habitate with EPM 11.1.2.4 and 11.2.2.x.  I see a potential problem here and I'll get to it in a minute.
  • The documentation does not indicate an upgrade path.  Do we start fresh, or can we do an in-place upgrade -- say from 11.2.2.x?
  • The installation guide contains a chapter named "Differences Between Essbase 11g and Essbase 21c".  Read this very carefully before getting started.  With respect to features, Oracle giveth and Oracle taketh away.
  • We have to run RCU for Essbase 21.  I believe it is because Essbase 21 comes with a new web-based cube editor that is separate from EAS.
  • If you use HFM, you will still need a Windows server in the mix.  I would not put FDMEE on the Linux server; I'd instead put it on the Windows server.
  • There's some weird stuff in the documentation about Financial Reporting not being supported for HFM if you've got FR installed on the Essbase 21 machine.  So I'd put it on the HFM server.

Now let's talk about co-habitating Essbase 21 with EPM 11.2.2 or 11.1.2.4.

Essbase 21 requires that Oracle Fusion Middleware ("FMW") 12.2.1.4 be installed.  EPM 11.2.2 is bundled with FMW 12.2.1.3, so this implies an in-place FMW upgrade if you've already deployed EPM 11.2.2.

EPM 11.1.2.4 is coming out of support at the end of December 2021 (Safe Harbor applies), so I would not consider combining Essbase 21 with EPM 11.1.2.4.  I'm also not a big fan of mixed-release configurations.  I tried it in the past, wrote a white paper for Oracle about it, and Oracle stopped recommending it as far as I can tell.  (9.3.1 mixed with 11.1.1.0 - what a nightmare!!!).

Essbase 21 and FMW 12.2.1.4 want a version of Java 8 that is newer than what comes with EPM 11.2.2.  Java 8 is not certified for EPM 11.1.2.4.  For EPM 11.2.2 shops, you will want to grab the Java 8 patch 18143322.  The patch contains multiple files.  The easiest one to work with is jdk-8u271-linux-i586.tar.gz, because getting it installed doesn't require Linux root access.  Just drop that file where you want, gzip -d it, tar -xf it, and you're done.  The "bin" folder within the jdk will need to be added to your installation user's PATH.

The installation guide for FMW 12.2.1.4 lists additional patches & prerequisites.

At first brush, my reaction is:

  1. Lay down the new Java 8 and add it to your PATH.
  2. Do a fresh install of FMW 12.2.1.4 in a separate Middleware Home.
  3. Install Essbase 21 after meeting the prerequisites (install GNU compiler, etc)
  4. Install EPM 11.2.2, minus the Essbase components and anything else Essbase 21 laid down, and pray they all play together.

I've backed up my Linux 7 sandbox and will try to play with this later this week.  Stay tuned!

(1/13/2021 Edit: Fixed "Essbase 12" references so they read "Essbase 21")

Saturday, December 5, 2020

Weekend LinkedIn Commentary

Cross-posting what I wrote on my LinkedIn.

"Glad to see so many questions quietly coming in via private chat about Oracle EPM 11.2. I encourage anyone considering an on-premises upgrade in 2021 to spin up a VM now and make sure you can successfully get through RCU and the EPM Configurator. Don't wait until your project is scheduled to start in 2021.... act now while things are quiet and you still have time to troubleshoot. This will go a long way to have a successful project in 2021!


Please do NOT send any diagrams or logfiles to me unless we have non-disclosure and SOW paperwork in place."

One additional point I made in private chat concerning FDMEE problems.


If you get errors about missing SNP_* tables when starting up FDMEE, the installation media is the likely culprit. Open a SEV-2 SR with Oracle so they know about the issue. 

 

While you wait from a response from Oracle, you could try this approach:


Create a new VM that is specifically for FDMEE. Download EPM 11.2.2.0 and do a fresh FDMEE install on that VM.  Download FDMEE 11.2.2.0 as soon as you read this post, as I question the 11.2.3.0 base installation media for FDMEE/ODI and it will eventually get removed from eDelivery.

 

You do not need to configure or RCU because this VM will be thrown away once you get through this exercise. Then throw the latest EPM 11.2.x (11.2.3.0 as of this writing) on top of it and do a Maintenance in-place update.


Finally, zip up Oracle\Middleware\odi\ on that VM and copy it over to the troublesome FDMEE machines. Re-run the configurator and pick "Configure Database" for FDMEE. Then check your database and verify the many dozen ODI tables are now present.


Your results may vary. Back up the odi filesystem on the FDMEE machine(s) before you replace it. Keep in mind the above is a suggestion and might not have Oracle's blessing at this time.

Monday, November 30, 2020

2021: EPM 11.2 Upgrade AND Data Center Change? Things to Consider

Many of my readers are looking at a situation where something needs to be done to their EPM system in Fiscal 2021:  Either upgrade, move to Oracle EPM Cloud, or move to a 3rd party solution such as OneStream.

The deciding factors are End Of Life for Hyperion / Oracle EPM 11.1.2.4 in DEC 2021, Microsoft IE11, Microsoft Server 2012 and prior, Microsoft SQL Server 2012 and prior, Linux 6, and so on.  I've written about these topics before and you can find details if you search through this blog's history.

One thing I would like to caution my readers about is conflating multiple significant changes in a single shot.  Such as:  Upgrading from EPM 11.1.2.4 or prior to EPM 11.2.x, AND doing a shift at the same time from an on-premises data center to a hosted cloud such as OCI, Azure, Amazon AWS, CenturyLink, etc.

Yes, many organizations combine these activities into a single project, often with success.  But please do consider one factor that goes beyond technology: your friends in Finance.

Finance MUST be involved in a significant project such as this, as they will need to validate the new system has apples-to-apples data results and performance (or better).

I mention this for the case where things don't quite match up right: data is wrong, or performance worsened.  Combining a significant version upgrade AND a shift in data center hosting leaves doubt as to where the problem stems from.

This being said, I'd like to offer general answers to questions I've received over the past few years along these lines.  Bear in mind these are general answers and every organization is different.  As I pose general questions & answers, if anything I say below triggers a further question, I highly suggest finding a Partner and locking them in now.  Don't wait until Spring/Summer 2021 as by then many Partners will already be booked for the year for their 2021 EPM 11.2 upgrade projects.

My favorite analogy along these lines relates back to my experience living in the US MidWest: try finding a competent roofer after a major storm passes through town.  The local experts are all booked, and you could be stuck with somebody from out of town - dubious credentials - who won't be around when you find problems later.

So without further ado, here we go....

Q: Any certification issues between Amazon AWS, Microsoft Azure, and Oracle Cloud Infrastructure (OCI) that I need to know about?

A:  No.  Certification revolves around the guest virtual operating system and relational database, not the physical host.  If you speak with Oracle Sales and if you are using Oracle RDBMS for Hyperion, you will be "guided" toward OCI because of database licensing.  This topic can get pretty detailed and is beyond the scope of this blog article.  Suffice to say, if you're using Oracle RDMBS and are looking at shifting from on-premises to hosted, make sure BOTH an IT Director AND your IT Procurement folks are involved.

Q: May I use my existing hardware to host EPM 11.2?

A: You can... but consider this upgrade project as an opportunity to leverage a hardware refresh for faster performance.  If your EPM platform is virtualized, the virtual guests can be moved to faster hardware as IT finds capacity.  If your EPM platform is physical and not virtual, you should either consider virtualizing (a topic for another blog post I could spend some time writing) or a hardware refresh.  Your physical EPM 11.1.2.4 servers are at least 5 years old and it is time for a speed upgrade.

Q: I want to change my database platform from X to Y.  Any issues?

A: If you're just running Essbase, no issues and go for it!  If you're running HFM or Financial Close, be careful.  By "change my database platform" I'm not talking about a version change -- say from Oracle 10 to 19, or MSSQL 2012 to 2016.  I'm talking about switching between Oracle & MSSQL in either direction.

The Hyperion Financial Close / Account Reconciliation suite for on-premises is a special animal.  You really should look at involving a Partner if you're on 11.1.2.3 or older.

HFM is very touchy when it comes to its database artifacts.  If you are switching between Oracle & MSSQ, just expect your project will see a 15-20% overhead due to working Service Requests (SRs) with Oracle.  Switching your database type for HFM isn't supported out-of-the-box and you will hit issues.

Q: How is EPM 11.2 different in terms of requirements for CPU cores, memory and disk?

A: The EPM Foundation 11.2 service takes a minute or two longer to start up, unless you have the WebLogic Admin Server (WLS) process running.  WLS is a bit hungerier for memory in this release.

Generally speaking, the various web processes (CalcMgr, EAS, Foundation, APS, etc) have about the same requirements in EPM 11.2 as in EPM 11.1.2.3.500 through EPM 11.1.2.4.x.  That is, you want to reserve 1 CPU core and 1.5GB-2GB of RAM for each web process.  Hyperion Planning and Hyperion Financial Reporting are exceptions due to a Java memory leak: set aside 8GB for each.

The above is just general advice.  Think of it as a rough rule of thumb.  When in a formal upgrade project, I'd want about an hour to speak with the IT Director/architect and DBA to dive into the details and develop an environment diagram appropriate to the situation at hand.

Q: Which is better?  Amazon AWS, Microsoft Azure, Oracle Cloud Infrastructure (OCI), or 3rd party host XYZ?

A: How much money do you have?  All kidding aside....

Spinning up EPM 11.2 in a hosted environment looks and feels exactly the same as on-premises.  Your users' experience ultimate depends upon several things:

  1. Network connectivity between the users' location(s) and the hosted data center.
  2. In the specific case of Essbase, is the Essbase server able to use disk equivalent to solid state or not?
  3. In the specific cases of HFM, FCM and/or ARM, how fast is the database?
  4. Backups and disaster recovery.  Can you control the backup retention, and is the backup / DR plan reliable?
  5. Bad guys.  Assume 2021 will be worse than 2020 in terms of malware, data leaks, extortion, etc.  Couple this thought with what I just mentioned about backups and DR.  If you stay on-premises, you have your IT Security folks to yell at.  If you shift to a 3rd party host, now it is their problem to deal with, there must also be trust between you and them.

Q: IT wants to enact Secure Socket Layer (SSL) for everything.  What do I need to know about EPM 11.2?

A: This is a growing trend I've observed.  Be careful if you're using MSSQL Server and your DBA wants to turn on SSL.  EPM 11.2 requires Oracle's Repository Creation Utility (RCU), and it doesn't like MSSQL with SSL enabled.  Last I checked, no workaround has been published by Oracle for this specific configuration.  Be warned and be careful.

The good news is on the Oracle WebLogic and Oracle HTTP Server (OHS) side of the fence, EPM 11.2 allows more secure SSL/TLS protocols than EPM 11.1.2.4 and prior.  It is still messy as you have to deal with SSL certificates that eventually expire, but EPM 11.1.2.4 and prior use insecure SSL protocols that are essentially worthless.  EPM 11.2 leaps forward multiple generations on the back-end, and as a result TLS in particular is much better.

------

I've just scratched the surface here, but I hope you have a few things to consider now.  Again, as I said previously this blog article is general in nature, and may not be suited for your specific circumstance.  Safe Harbor, yada yada yada.

Friday, November 20, 2020

Hey Why Does My New SmartView Keep Timing Out?

Oracle SmartView for Office still needs the Windows Registry keys it has been wanting since... the very beginning really.  It does not matter if we're talking Hyperion / Oracle EPM on-premises or Oracle EPM Cloud.

If your user has upgraded to a new Windows 10 machine and is complaining about SmartView timeout issues, the culprit is almost always the Windows Registry.  This is what a non-tuned user's workstation looks like, from my point of view:

The registry key names you see in this example screenshot are shorthand for the full registry key name.

These registry keys and suggested values were taken directly out of a series of Oracle Knowledge Base articles that have been available for a very long time.

Here we're looking at two areas of focus: network and MS Office animations.

Network of course is the true silver bullet here.  Get the keys applied, reboot the user's workstation, and the problem is nearly always solved.

The animation keys help with weird flickering issues occasionally reported by SmartView users.  I haven't heard any of these complaints recently, as the flickering problem is very Office version-specific.  I've written about this issue here before.  If your user has the on-premises version of Office 365, they are likely OK and wouldn't have seen the issue.  It still wouldn't hurt to have the keys, though.

So here's the script that generates the above screenshot.  Within this very simple script -- nothing proprietary here! -- you can see the full key names and values expected.

 

echo ------------------------
echo KeepAliveTimeout should be set to dword:2bf20
echo Here is the current value:
reg query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v KeepAliveTimeout

echo ------------------------
echo ServerInfoTimeout should be set to dword:2bf20
echo Here is the current value:
reg query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ServerInfoTimeout

echo ------------------------
echo ReceiveTimeout should be set to dword:75300
echo Here is the current value:
reg query "HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings" /v ReceiveTimeout

echo ------------------------
echo DisableWindowTransitionsOnAddinTaskPanes should be set to dword:1
echo Here is the current value:
reg query HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Common\Toolbars /v DisableWindowTransitionsOnAddinTaskPanes

echo ------------------------
echo DisableAnimations should be set to dword:1
echo Here is the current value:
reg query HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Common\Graphics /v DisableAnimations

echo All done checking!
pause

One final point...  #MISSING suppression is handled on the end-user's workstation, not on the Essbase server (on-premises or Cloud - it does not matter).  If user requests a massive grid in either ad-hoc mode or by opening a Planning form in SmartView, the Essbase server is going to process it.  Essbase ships the complete grid back to SmartView and lets the chips fall where they may.  (The communication flow is more complex than this - I'm just trying to keep this post simple)

I mention this because I've seen some users attempt to "data mine" Essbase.  The mindset here is "give me everything and I'll sift through it in a linked sheet or in MS Access".  In my view, Essbase & Planning were never intended to be used this way.

When a user requests a grid containing a half million or more possible cells, it does not matter what you do to either the user's Windows Registry, the essbase.cfg file (only if you're on-premises), or tweaks to the cube's dense/sparse settings.  SmartView will remain locked up until the entire grid is fetched and processed.  I've noticed grid retrieval performance starts to get ugly when we're looking at 300,000 possible cells and higher.

Where on-premises is concerned, massive grid retrievals can lead to "OutOfMemoryException" errors in either Planning or Analytic Provider Services, depending upon the type of SmartView connection used.   You can bet these errors also happen in Oracle EPM Cloud, since the underlying technology is the same, but the Oracle WebLogic logs aren't exposed to us in the Cloud.  Bumping up the on-premises "-Xmx" registry keys for Planning & APS will provide some relief (e.g. changing from the 4GB default to 8GB), but all we're really doing here is throwing more RAM at the problem; it won't speed anything up.

The underlying issue to address, unfortunately, is end-user behavior.  Usually it is only 1-3 people within the organization who are trying to datamine the entire cube, and you'll typically find them in FP&A.  If you're in Oracle EPM Cloud, use EPMAutomate to grab the daily audit logs and that should help you identify who is pulling down large grids.  On-premises has different ways EPM Detectives such as yourself can probe and find who is bringing the system to its knees. 

So apply the keys recommended above, reboot the workstation, and keep retrieval slice sizes within reason.

Wednesday, October 21, 2020

If You're Enjoying My Posts Here....

With heavy heart I announce my employer laid me off today.  Anyone who knows the sales dynamic between Oracle EPM Cloud vs. Oracle EPM On-Premises requires no further explanation.

It was a good run.

This means my access to Oracle Support (patches and Knowledge Base) and eDelivery ought to be cut off.  I'll cut myself off, of course.  (Nobody expects the Spanish Inquisition)

I believe on-premises Hyperion / Oracle EPM has good years ahead.  The vast majority of mid-market companies I've spoken with over the past few years aren't ready to move to the cloud, and want to stay on-premises for at least the next 3 years.  Perhaps longer.  On-premises upgrades are not going away.

The dates are fluid, but we are being told EPM on-premises will be fully supported through "at least" 2030 and I heard 2031 tossed out there as well.  Safe Harbor applies; Oracle Corporation reserves the right to change their support policy.  This is just the best information I have in writing as of today.

Time to catch up on overdue yardwork and prepare the homestead for an Iowa winter.

Then it'll be time to dust off my resume and pound the virtual pavement.

Anyway, the rate of my posts on this blog will slow down until I find a new sponsor for an Oracle Support/eDelivery account.

So if 11.2.4+ or when Essbase 21 come out and you want me to test and write about them, click the link on the right-hand side of this page for my LinkedIn profile and you may contact me there to discuss a sponsorship.  Sorry, existing customers of my ex-employer may not contact me for this due to non-compete.  I write a separate post when my non-compete eventually expires.

Be well,

Dave

 


EPM 11.2.3.0 In-Place Upgrade: Why I Did It, and Issues Encountered

As mentioned in my previous post about EPM 11.2.3.0, I decided to attempt an in-place upgrade from Hyperion / Oracle EPM 11.2.2.0 to 11.2.3.0.

Before I launch into this discussion, I want to be very up-front:  I do not like EPM in-place upgrades.  They are fraught with risk as you never know what issues you'll encounter.  Once you encounter them, you have to work non-stop to fix them because the system you've attempted to upgrade is now non-functional.

This spells disaster for a Production system where the customer only provides a weekend to get it all done; everything must be online by 7AM Monday morning.

So why did I even bother???

Let's look at the recent Oracle EPM 11.2.x release history:

  1. EPM 11.2.0.0 released December 2019
  2. EPM 11.2.1.0 released April 2020; Chrome and Edge fully certified, IE11 de-certified
  3. EPM 11.2.2.0 released June 2020 and includes Linux
  4. EPM 11.2.3.0 released sometime between September 15 - early October 2020

Now lets examine the list of patches for EPM 11.2.x:

  1. Ziltch
  2. None
  3. Nada

Oracle is patching Essbase 11.1.2.4, and I'd have to check but I believe patches for DRM 11.2 are also on a separate track.  The rest of the EPM suite, however, only has bugfixes as provided in EPM Releases 11.2.1.0 through 11.2.3.0 as listed above.

The question lingers in my mind: will we see PSEs or PSUs start to trickle out for the non-Essbase components, or will we continue to see new "dot" releases every 3 months or so?

Doubting the former, I considered the latter.

If Oracle were to continue this trend of issuing dot releases instead of PSEs/PSUs, then us remaining on-premises Infrastructure people ought to get comfortable with performing in-place upgrades within the EPM 11.2.x.x releases.  Thus, my adventure began.

The above out of the way, here's my observations after performing the in-place upgrade and troubleshooting it.  In no particular order:

  • The documentation is absolutely correct that we need to edit \Oracle\Middleware\EPMSystem11R1\products\Essbase\eas\console\bin\admincon.bat and change the reference of Java from Java 6 to Java 8.  Java 6 has been out of Extended Support for some time.
  • Essbase Sample Applications not visible after installing them via installtool.  In 11.2.2.0, I didn't install the sample applications.  In 11.2.3.0 I wanted to test them.  Installtool said the install was a success.  After digging around, I found them in \Oracle\Middleware\EPMSystem11R1\products\Essbase\EssbaseServer\app
    • This was NOT my ARBORPATH
    • I checked epmsys_registry and \Oracle\Middleware\user_projects\epmsystem1\EssbaseServer\essbaseserver1\bin\setEssbaseEnv.bat -- both of them had the ARBORPATH I wanted
    • I'm not sure if this is the correct method, but I edited \Oracle\Middleware\user_projects\epmsystem1\EssbaseServer\essbaseserver1\bin\server.properties and followed the comments within it.  I added an entry for ARBORPATH, bounced Essbase, and then I could migrate or create apps and they appeared where I wanted.
  • installtool re-applies Middleware and Essbase patches every time it runs. This behavior is not new and dates back to at least 11.1.2.4.  I don't recall if 11.1.2.3 and prior did this.  When you see installtool "stuck" at 97-98%, that's what it is doing.  The lesson here is don't start applying Essbase patches or the quarterly critical Oracle Middleware patches until you're satisfied you're completely done using installtool.
  • Analytic Provider Services does not work.  This behavior started in 11.2.0.0 and continues through 11.2.3.0.
    • The error message in the WebLogic log is "weblogic.application.ModuleException: java.lang.RuntimeException: Failed to deploy/initialize the application as given archive is missing required standard webservice deployment decriptor."  ("descriptor" typo is Oracle's, not mine).
    • The fix is to patch the Essbase Suite up to a higher patch level, such as 11.1.2.4.040 or .041 as of this writing.  Consult the APS patch README to see which other patches correlate to the Analytic Provider Services patch you want to apply.
  • Oracle database clients, WebLogic Server, and OHS would not upgrade and installtool shows errors during the upgrade process.  Don't worry about this as you don't need to upgrade them.  11.2.0.0 through 11.2.3.0 use the same versions of this software.  Just ignore and move on!
  • On a positive note, the Google Chrome "white screen" rendering bug in LCM has been fixed in 11.2.3.0!
  • Consolidation Administration->Settings->System still has the same defaults as 11.1.2.4.
    • MinDataCacheSizeInMB: 2.25GB -- This is likely too high for sandboxes and DEV environments.  It might be OK for UAT and PROD.  You decide!  Tuning is an art, not science.
    • SessionManagerTimeoutInMS 1,200,000 (20 minutes) -- Many HFM admins asked me to triple it. This affects EPM Workspace AND SmartView.
  • Another positive note. Oracle has done a great job cleaning up the many references within scripts to Java 6 that I mentioned in my more than a cupful of Java post.  EAS Console thick client launcher and MetadataMerge.bat are the last remaining offenders now.
    • I completely understand why EAS Console is still referring to Java 6.  EAS Console is 11.1.2.4 software at this time, so it needs to work in an 11.1.2.4 system that uses Java 6.
  • Another positive note. My Windows Registry fix for the Planning RMI issue was preserved.  Bookmark the post I linked if you need to use the classic Planning command-line utilities, as they need RMI running properly.

That's about it!  (But isn't that enough?)

Saturday, October 17, 2020

EPM 11.2.3.0 Is Available on Oracle eDelivery

The EPM Certification Matrix has been updated for on-premises Hyperion / OracleEPM 11.2.3.0.

Update: Oracle posted the following on Oct 16, 2020: Oracle 11.2.3 Release Announcement

The online Feature Comparison Tool hasn't been updated as of when I checked earlier this morning.  The README, however, lists a boatload of bugfixes.  HFM shops should inspect the README carefully and allow sufficient time for rigorous regression testing.

There's about 2GB remaining in my download queue, and then it will be time to take it for a spin.

Initial observations:

  • Still no certified migration path from 11.1.2.3 and older.
  • The certification matrix says HFM isn't available for Linux 7 yet.  Fortunately, I don't have any customers screaming for this (yet).  I do have customers who've expressed interest in running Essbase on Linux 7 instead of Linux 6.  Linux 6 is a dead product and Linux 7 has faster disk I/O drivers, among other improvements.
  • Installation media is only available for 64-bit Microsoft Windows and Linux 7.  Solaris and AIX ("AIX ain't UNIX") aren't listed.
  • In-place "Apply Updates" upgrade available if you're on on 11.2.0.0 through 11.2.2.0.
  • The LCM bug for Google Chrome has apparently been fixed.
  • The README provides the workaround for Planning RMI not binding to its port.
  • No mention if the FDMEE Linux bugs have been fixed or not.
  • The README explains why I can't login to EAS; an Oracle WebLogic policy update is needed.  The problem may have been introduced in 11.2.2.0, as my 11.2.1.0 sandbox works just fine.
  • The README has a very important note about LDAP hosted through MSAD; every Microsoft shop will need to get in front of this before Micosoft forces our hand in 2021.