Wednesday, October 4, 2017

HFM 11.1.2.4.205 and SmartView

Oracle has acknowledged a bug in HFM 11.1.2.4.205 PSU and is working on a fix.

The observed behavior is you have an HFM data grid open in SmartView and you get the session expired / disconnected message well in advance of the defined session timeout setting.

I've noticed this seems to happen more frequently if your user has a bunch of VB SmartView retrieval functions on their retrieval template instead of defining a traditional ad-hoc grid.  This is pure speculation on my part as I don't have too many systems I'm supporting that are on this particular patch.  It seems that each VB retrieval cell maintains its own connection, and apparently once one cell gets disconnected, all of them do and your back-end XFM ODL logs are flooded with messages.

Oracle Development has indicated the bug will be fixed in the upcoming patch HFM 11.1.2.4.206 PSU.

For now, the only known workaround available for the issue is to downgrade to HFM 11.1.2.4.204 PSU.

Monday, September 25, 2017

Apply CVE-2017-9805 today - updated top of post

Update:  Please scroll through the comments and review the comment posted by user cjratliff on October 4, 2017 at 3:02 PM.  He has posted a tip about Oracle Knowledge Base article # 2255054.1 and a response from Oracle Support that I have yet to receive on my SR.  Ugh!!!  This stuff is a nightmare to navigate. Thank you, cjratliff .
 
There was a bit of buzz on the Internet a few weeks ago concerning a zero-day exploit discovered within Apache Struts 2.  I did some sleuthing around in my EPM 11.1.2.4 on-premises lab and found evidence of Struts within several WebLogic processes (Calculation Manager, EPM Foundation, and more).

This past Friday, September 22, Oracle issued a Security Alert they've named CVE-2017-9805.  This includes a fix for Apache Struts 2 within WebLogic 10.3.6, which every EPM 11.1.2.3 and 11.1.2.4 system on the face of the planet uses behind the scenes.

The patch number to download for WebLogic 10.3.6 is 26835212.

The procedure to apply this patch is different from how we normally apply EPM patches.  On each server in the environment where you have a folder named \Oracle\Middleware\wlserver_10.3, you will want to edit this file:

Windows:
\Oracle\Middleware\utils\bsu\bsu.cmd

UNIX:
/Oracle/Middleware/utils/bsu/bsu.sh

Modify the MEM_ARGS line to be as so:

set MEM_ARGS=-Xms2048m -Xmx2048m

If you don't do this, the patch utility runs for a long time and then fails with an OutOfMemory exception error.

Next, copy the unzipped contents of the patch into this folder:

\Oracle\Middleware\utils\bsu\cache_dir

You would then shutdown EPM web services and execute the bsu script you edited earlier, and examine the output.

I  would take things 1 step further and blow away the /cache and /tmp folders for each WebLogic Managed Server underneath \Oracle\Middleware\user_projects\domains\EPMSystem\servers

Stay safe out there!


Update: I recommend using the command-line patch interface rather than the graphical interface. On a 4.0GHz machine with solid state disk, it takes 12 minutes just for the GUI to render. Another 12 minutes to process the patch after you are given the opportunity to click. With the command-line interface, you can eliminate the first 12 minutes.

Also, this patch has a conflict with the April 2017 critical patch "RVBS". RVBS needs to be rolled back first. This adds another 12 minutes.

Wednesday, August 30, 2017

Financial Reporting 11.1.2.4.706 web designer 403-Forbidden error

The latest patch for Hyperion Financial Reporting, which is Patch Set Update 11.1.2.4.706, contains a flaw that causes the web designer to be unusable.  We get a 403-Forbidden error when trying to connect to the web designer URL http://OHShost:19000/frdesigner

This worked in PSUs 11.1.2.4.700 through 11.1.2.4.705.

I've opened an SR with Oracle and will share the solution once it is discovered.

Tuesday, August 29, 2017

ASO, ESSCMDQ, and Essbase 11.1.2.4.015+

Some of my clients automate EPMA's command line batch utility to pump metadata updates into ASO Essbase cubes.  Anybody who has done this for any length of time knows about the ASO design flaw involving an internal dimension restructure counter that is capped at 255.

Oracle published a workaround for this a while ago, in the form of the ESSCMDQ utility.  Once installed and we automate the suggested ASO restructure script, this was a reliable way to avoid the 255 limitation.... until very recently!

A few patches ago, ESSCMDQ started behaving... erratically.  Sometimes the restructure script would complete, and other times ESSCMDQ would crash.  The crash message would complain about a DLL within the Microsoft Windows version of the 11.1.2.4 Essbase Client, such as ESSOPGN.dll.

It just so happens that Oracle discovered this and published an update, but you have to dig for it within the Knowledge Base!

The Knowledge Base article # on support.oracle.com is "2273191.1".  This article acknowledges the bug and includes a download link to a version of ESSCMDQ.exe recompiled for Essbase 11.1.2.4.015 and higher.  Unfortunately, this version isn't included on the main download page for ESSCMDQ.

Oracle is issuing patches for on-premises Essbase 11.1.2.4 at the pace of about once every 2 months.  Since 11.1.2.4.0 first came out, as of this writing 19 cumulative patches have been issued for Essbase. This means if you are using ASO and are on 11.1.2.4.015 through 11.1.2.4.019, you will want to hit the "2273191.1" article and grab the newer version of ESSCMDQ.

Friday, August 11, 2017

Blank Member Selection window in Planning 11.1.2.4 & IE11???

I recently worked with a client who upgraded from Oracle Hyperion Planning 11.1.2.2.300 to 11.1.2.4. One of their Planning administrators tried to create a new form and complained the Member Selection pop-up window is blank in IE11.

The frustrating thing is the window renders correctly in IE8.  What's going on here?

This:











When we migrate a Planning application via LCM from one environment/version to another, the Application Properties are typically included within the migrated artifacts.  ORACLE_ADF_UI = false is the culprit here!

"Sherman, set the wayback machine to Hyperion Planning patch set 11.1.2.2.300!"

When Oracle rolled out PSU 11.1.2.2.300 for Planning, they introduced the ADF user interface.  This interface is what we know and love today in 11.1.2.4.  Back in the 11.1.2.2.300 days, however, some people were still using browsers older than IE9, which couldn't fully utilize ADF.  So as a workaround, Oracle documented a method to deliberately disable ADF.  By manually adding the ORACLE_ADF_UI property and setting it to false, one could force Planning to behave as it did in releases prior to 11.1.2.2.300.

The problem here is IE11 doesn't know how to render the Member Selection pop-up window when ORACLE_ADF_UI is present and set to false.

A few final notes:

When either deleting this property or changing it from false to true, it is necessary to stop and restart the Planning web service before the change takes effect.

Oracle also advises that once we are running in ADF mode and new Planning web forms are designed, it is not advisable to revert back to non-ADF mode.  The forms you built in ADF mode might not render the way you intend when you revert to non-ADF mode.

Saturday, August 5, 2017

LCM Backups, Windows 2012, and You

If you have an on-premises Oracle EPM / Hyperion system 11.1.2.0 or later (and you really should be on 11.1.2.4 by now!), this post is for you.

Traditional relational database backups and routine disk backups / VM snapshots are great things to have automated, but are not sufficient in and of themselves to fully protect your EPM system from disaster.

"Disaster", in the context of this post, is an Administrator or PowerUser doing any of the following:
  • Edit the design of a Planning web form, Calculation Manager rule, or Financial Report in such a way that it is no longer usable.  Adding insult to injury, the developer doesn't remember which exact changes were made, and there's no Edit->Undo after clicking Save.
  • A click-and-drag operation in the EPMA Dimension Library gone horribly wrong.
  • Delete a folder hierarchy within Reporting Framework in EPM Workspace.  (Yes, it gives an "Are you sure?" prompt, and yes, I've had to do a restore because someone clicked Yes in Production by mistake).
The 3 examples cited above, all of which I have experienced in "real life", cannot be fixed by a simple VM snapshot restore.  The types of objects mentioned above reside as records within various relational repositories, so you would need to bring the EPM system offline and restore to your last good relational backup.

In the case where Reporting Framework is concerned, we additionally need to restore the ReportingAnalysis\data folder on the RAF Agent server, and that needs to be synchronized with the RAF relational restore.

But I digress....

Best practice is to automate nightly exports via Oracle's LCM command-line utility for EPM.  If you're unfamiliar with this utility, read LCM user guide Chapter 7.

My personal preference is to maintain multiple rolling rotations of LCM backups.  This is because sometimes a problem isn't reported for a few days.

And now we get to the nugget of why I'm writing this post today....

Certain LCM artifacts have extremely deep directory paths.  Reporting Framework and Financial Close Management immediately jump to mind.  In Windows Server 2008 R2, rotating and pruning the LCM export folders wasn't a problem.  But with 11.1.2.4's support for Windows Server 2012, we hit a new issue we didn't have to deal with in the past:  Microsoft's deep directory path character limitation.

In Windows Server 2012, try running RMDIR /S /Q on your oldest LCM folder for RAF or FCM.  You will likely see a failure message stating the directory path is too deep.

So off we go to our favorite web search engine to find a solution.  The 2 most frequently posted solutions are too "clunky" to use, in my opinion:
  1. Mount a temporary drive to a point in the path before the # of characters reach 255-260.  CD to it and delete from there.  Then delete the temporary drive.
  2. Use a tool like 7-zip, which uses a different API and doesn't have the character limitation.  You can navigate to the parent folder and shift-delete it, and it is gone.
I really don't like either of the above for my automated LCM backups.  When I want to roll off the oldest backup, here's what I do.

Create this Jython script:

# rmRotation7.py
#
# This Jython script removes the oldest LCM backup folder.
# We use this technique to work around the Windows Server 2012
# limitation concerning directories containing deep pathnames.
#
#  Written on 11/02/2016 by Dave Shay (Datavail)
# Modified on MM/DD/YYYY by Your Name - Briefly list changes

import shutil

shutil.rmtree('E:/Backup/LCM/Rotation7')

Why Jython? Because that shutil.rmtree function does everything we need with just 2 lines of code, and all modern Hyperion systems have access to Jython!

OK, so how do we invoke it?

Paste these 2 lines of code into your LCM automation wrapper script:

SET CLASSPATH=%CLASSPATH%;E:\Oracle\Middleware\oracle_common\modules\oracle.jrf_11.1.1\jrf-wlstman.jar
%JAVA_HOME%\bin\java weblogic.WLST E:/Scripts/rmRotation7.py

The first line is what prepares your DOS shell so that it can run Jython scripts. It does run a little slow and will delay your script for a few seconds. The second line invokes your Jython script and prunes the folder named in rmRotation7.py.

There are other ways to tackle this problem, such as installing 3rd party tools like Cygwin. My preference when working with customers' Hyperion systems is to utilize the framework already available. In my opinion, it makes knowledge transfer and ongoing maintenance a little easier. When we instead install a 3rd party tool, now we've introduced yet another thing we need to potentially patch and maintain.

Friday, August 4, 2017

FDMEE Batch Wrapper

I recently helped a customer troubleshoot why their FDMEE data load automation stopped working.  The solution was actually quite simple, and so I thought I'd share it here!

The Observed Symptom

An upstream system automatically provides ASCII text files for FDMEE to load.  The timing of the delivery varies from day to day, so a batch process is kicked off by the Windows Task Scheduler to continually check for the files' presence and process them when discovered.

On an intermittent basis, the FDMEE load doesn't properly complete.  The files are detected, the FDMEE load is attempted and fails, and then the files are moved into an archive folder by the consultant's DOS wrapper script.

Root Cause Analysis

The consultant setup the FDMEE automation via a Windows Task Scheduler job, which triggered to run "On Startup".  The script loops with a 60 second pause.  Once files are detected in the expected folder location, the pre-delivered FDMEE load utility is invoked.  The files are moved into an archive folder after this runs.

Unfortunately, the FDMEE server was rebooted via Windows Update.  Some time after the reboot, the files were delivered and picked up by the automation, but FDMEE has not yet finished its Oracle WebLogic startup sequence.

Plain English:  The FDMEE's load utility couldn't process the file, since FDMEE wasn't online.

The Fix

FDMEE 11.1.2.4 is one of the services that takes the longest to complete its WebLogic start-up sequence.  Depending upon the computing resources available, this can require 3-5 minutes or longer.

When FDMEE is fully initialized and ready to accept connections, the system is listening to TCP port 6550 (this is the default FDMEE port unless someone changed it).

We add 1 line to the top of our FDMEE automation wrapper script:

powershell D:\Scripts\WaitForFDMEE.ps1

Next, we create the WaitForFDMEE.ps1 script.  Here is the script in full:

# WaitForFDMEE.ps1
#
# This script loops until FDMEE's port is online.
#
# If you receive a security policy error about “unsigned” Powershell scripts when 
# running this process, open a command prompt and type:
# powershell.exe Set-ExecutionPolicy Unrestricted
#
#  Written on 08/04/2017 by Dave Shay (Datavail)
# Modified on MM/DD/YYYY by Your Name - Briefly list changes

$ErrorActionPreference = "SilentlyContinue"

# Loop forever until FDMEE is online
do
{
 $socket = new-object System.Net.Sockets.TcpClient("localhost", 6550)
 
 if ($socket -eq $null)
 {
  write-host "FDMEE isn't fully initialized yet.  Sleeping 20 seconds..."
  powershell.exe Start-Sleep -s 20
 }
} until ($socket -ne $null)

write-host "FDMEE is ready to accept connections."

Finally, we copy & paste this line to the command prompt. This prevents a Powershell security error. We only need to issue this command one time.

powershell.exe Set-ExecutionPolicy Unrestricted

And that's it! The FDMEE automation wrapper script now sleeps until it detects that FDMEE is online.