Monday, November 11, 2019
Customers who use Jython scripting within FDMEE may want to do extra regression testing before promoting this patch into Production.
A little background:
My team uses custom scripts to maintain 7 rolling days of Hyperion LCM backups. One stumbling block with this is: when rolling off the 7th folder so it may be replaced, traditional MS-DOS "rmdir" doesn't work for certain subfolders in Windows Server 2012 R2 (usually Reporting Analysis Framework and Financial Close / ARM). This is due to the famous "directory path too deep" error.
To work around this, we created a very simple Jython script. Jython bypasses the Microsoft API, allowing us to delete folders containing paths which exceed the Microsoft limitation.
You do this:
%JAVA_HOME%\bin\java weblogic.WLST %SCRIPTDIR%/rmRotation7.py
And the Jython script looks like this:
# This Python/Jython script removes the oldest LCM backup folder.
# We use this technique to work around the Windows Server 2012
# limitation concerning directories containing pathnames exceeding
# 260 characters.
# Written on 11/02/2016 by Dave Shay (Datavail)
# Modified on MM/DD/YYYY by Your Name - Briefly list changes
I like simple solutions like this. It uses technology that is already baked into the EPM infrastructure. Now, a new challenger appears: WebLogic OCT2019CPU patch ID "3L3H". This is what happens when trying to run our Jython script after applying the patch.
Initializing WebLogic Scripting Tool (WLST) ...
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
Problem invoking WLST - Traceback (innermost last):
File "D:\Scripts\rmRotation7.py", line 13, in ?
File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/shutil$py.class", line 117, in rmtree
File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/shutil$py.class", line 132, in _build_cmdtuple
File "D:\Oracle\Middleware\wlserver_10.3\common\wlst\modules\jython-modules.jar\Lib/javaos$py.class", line 135, in listdir
The Rotation7 folder remains and now we essentially have 6 rolling days of backups instead of 7.
When I look at the "jython-modules.jar" file, I can see the timestamp changed to when it was patched. It would appear Oracle accidentally broke something?
My team will open an SR with Oracle and try to get to the bottom of this. In the meantime, I would tend to shy away from this specific patch.
Thursday, November 7, 2019
"Hyperion/EPM, Java, SQL Server…Upcoming Extended Support Deadlines and What They Mean for You
Dave Shay, Datavail"
This encapsulates information I've posted here across multiple threads, but I'll present it as a single discussion.
In the very least, some of you will finally get to hear my voice. Please do not heckle... too much. 😆
Thursday, October 17, 2019
The following information is what I was able to dig up concerning OCT2019CPU, published just a few days ago.
“Critical Patch Update (CPU) Program Oct 2019 Patch Availability Document (PAD) (Doc ID 2568292.1)”
Friday, October 4, 2019
Sunday, September 29, 2019
Let's talk about one of our favorite subjects: Sarbanes-Oxley compliance combined with on-premises Oracle EPM / Hyperion.
Auditors and IT Risk Management departments tend to frown on running SOX-relevant financial applications on systems where a vendor's Extended Support has expired. Plain English: no ongoing defect remediation via patches, and no new security vulnerability patches.
As I wrote in a prior post, this ship has already sailed for:
- Oracle EPM 188.8.131.52 and prior versions
- Microsoft Windows Server 2008 R2
- Microsoft SQL Server 2008 (all Service Packs)
- Java 6 and prior versions
- JRockit 6
- Oracle EPM 184.108.40.206
- Microsoft Windows Server 2012 R2
- Microsoft SQL Server 2012 SP3
- Java 7
What readers need to consider is their timeline to either upgrade to EPM 11.2 (once released), or migrate to the Oracle EPM Cloud.
December 2021 seems like a long time away, but let's again re-visit SOX.
Let's say your fiscal year aligns with the calendar year: Jan:Dec. In this scenario, SOX-relevant applications only get 2 windows per year to complete upgrades and do a go-live cutover to a new system: May and September. Shoot for May, and use September as your fall-back position. Going live during either your fiscal 1st Quarter or 4th Quarter will trigger a red flag in your SOX audit.
So keep these dates in mind and then start counting backward. Don't wait until late in 2021 to either upgrade or move to the cloud. By then most EPM consulting partners, such as the firm I work for, will likely be slammed trying to hit that Sept 2021 SOX deadline. I'm reminded of when Microsoft revoked support for browsers older than IE11... we were insanely busy because many customers were still on EPM 220.127.116.11 or older, and IT Risk Management departments forced Finance to upgrade to remain compliant.
One final thought: I've recently been contacted by a competitor promising cheaper support rates than Oracle's. I want to discourage people from considering this, unless you intend to completely retire Hyperion and switch to a different platform on or before Q3 2021. A 3rd party partner/consultant will face legal problems if they are discovered installing patches or upgrades a former Oracle customer is no longer entitled to receive.
Friday, September 6, 2019
"What will it cost me?"
"How long will it take?"
My answer tends to be:
"I won't know for sure until I see the Release Notes and install it in a lab."
But, I do have some directional answers.
Before I begin, however, there is a very important point: You must remain current on your Oracle Support maintenance agreement. This is the only way you can legally obtain the EPM 11.2 installation media. A partner who uploads their own copy of the media will lose their partner status with Oracle and face legal action.
Cost and Time are related
Whether you use a partner or perform the upgrade in-house, cost is unavoidable.
If you do it in-house, you are essentially paying opportunity cost; the people working on the upgrade are not working on other things to support the pre-existing system, do development work, etc. If you use a partner, of course you are either paying Time & Materials or are paying Fixed Price.
Either way, there is cost involved from a labor perspective.
You will want to do what Oracle calls an Out of Place upgrade (or what I call Lift & Shift). See my post Why Upgrade to EPM 11.2 or Move to the Cloud on why you don't want to do an in-place upgrade. (Speaking from a personal note, I hate in-place upgrades. They're almost always messy)
Translated to English:
- Stand up new servers using a newer operating system, faster CPUs, etc. Your existing servers are likely 4+ years old. Size the CPU core count, RAM, and disk to be no less than what you're utilizing in current Production. Remember that on an Essbase server, the disk needs to be double in size so you don't run out of space during an Essbase dense restructure.
- Stand up a new database server running newer database software, such as SQL Server 2016 instead of SQL Server 2012.
- Size the database schemas (Oracle 12.2), or databases (MS SQL Server 2016) to be no smaller than what you're using in current Production. Think forward and increase the size to accommodate future growth for the next few years. This is especially the case for folks who use HFM and/or FDMEE.
- Plan ahead and ask IT to add an Anti-Virus "On Access" scanning exception for D:\Oracle
Somewhere between 40-50% of the EPM customer population I work with use EPMA. EPMA is gone in 11.2, and replaced by DRM.
Oracle has stated a utility will be made available that converts EPMA content into DRM. That being said, account for some extra testing time by your Subject Matter Experts (and your partner if you will use one).
The architecture has changed and "Reporting & Analysis Framework" has been deprecated. I'm a big fan of this; RA Framework is always messy to troubleshoot when it won't start back up. The way reports are stored within Hyperion LCM has changed as a result. Assume there will be additional time spent on:
- Migrating reports from the old system into the new (hopefully the migration utility handles this OK).
- Migrating objects that aren't reports, but are stored within Workspace. Such as PDFs, videos, MS Word documents, MS Excel documents, etc.
- Updating LCM backup scripts, if you have nightly LCM backups enabled. (You should!!!)
From prior experience, I can say the first 5-6 months of a new EPM release can be.... interesting.
Especially where HFM and Essbase are concerned.
Have you counted the number of patches released for Essbase alone in 18.104.22.168?? I've been using Essbase since the 3.2 days in the mid-90s, and have never seen so many patches for it since 22.214.171.124. Expect perhaps some bumps in the road, especially if you use ASO Essbase.
Critical Security Patches During Your Project
Congratulations, you are now back on fully-supported software, from an Operating System, Database, and Oracle EPM standpoint. (EPM 126.96.36.199 has about 1-2 years to go, depending upon your support contract).
Most EPM shops haven't moved their servers from Java 6 to Java 7. Now you will be on Java 8. Oracle issues security patches (actually full installs) for both Java 7 and Java 8 every three months. You will also be on a newer version of Oracle WebLogic: You're going to move from WebLogic 10 to 12. This also gets critical security patches issued every Quarter.
This means, you can anticipate at least 1 major outage during your upgrade project, multiplied by the number of EPM 11.2 environments you expect to have (e.g. DEV, UAT & PROD). You will want to account for this in your Project Plan.
OK, But How Long Does an Upgrade Normally Take?
My rule of thumb is 5-7 business days per environment to:
- Migrate apps
Add more time if you use SSL and/or SSO. Where SSL is concerned, you may need new SSL certificates. The new system is expected to comply with TLS 1.2, whereas EPM 188.8.131.52 and prior could only use TLS 1.0. For SSO, I expect no changes to Shared Services / EPM Foundation, but we'll have to wait and see!
Workforce, CapEx, Project Modules for Planning
OK, here's the fun one. These modules don't exist in Planning 11.2. If you currently use them, put some extra hours+cost in your Project Plan for them. At best, we could LCM them into 11.2 and they will work OK. At worst, they will have to be re-implemented from scratch. Yuck!
Hyperion Tax - Gone!
If you're one of the few who use the Tax module, add some hours+cost to consider how this case would be handled.
If memory serves, the same will be true for Profitability, Essbase Studio and Strategic Finance. Oracle has the final say and we won't know for sure until the Release Notes are issued.
Essbase Studio - If it is Gone and You Use Drill-Through, What Next?
Not everyone uses Essbase Studio for drill-through. But if you do, plan on spending time in Design, Build and Test for whatever alternate solution you choose to use.
Who doesn't love them?
If your system is SOX-relevant, you might face extra cost if you utilize external auditors. The auditors, and your own people who interact with them, may need to perform their activities twice during the fiscal year when your EPM 11.2 Production cut-over goes live; one audit for the old system, and one audit for the new system.
This extra cost can be avoided if you cut over so the first month of the new fiscal year is operated within the new system. Many customers I've interacted with have difficulty achieving this, due to a variety of factors.
Don't plan to go "all in" once the software is released. Check out my post Action Plan for Early EPM 11.2 Adopters and plan on running a test lab / sandbox to "kick the tires" before you spin up the servers for the "real" EPM 11.2 environment that you plan on using.
Now it is time for me to go back and keep refreshing the Oracle eDelivery page....
Thursday, September 5, 2019
As of this writing, both the blog post linked above and the Oracle Knowledge Base article which links to the above (last dated Aug 1, 2019) list an "Oracle Safe Harbor" projected release date of September 2019.
Late October 2019 Update: "Safe Harbor" has been bumped from December 2019. As always, "wait and see"!
Be sure to bookmark the above and check back.
The primary Oracle Knowledge Base article to check would be "When will EPM 11.2 become Available? (Doc ID 2553915.1)"
October 2, 2019 Update: the article 2553915.1 mentioned above isn't available anymore!
For those of you unfamiliar with term "Oracle Safe Harbor", it essentially means the posted information is directional in nature, and could change. The full text of Oracle Safe Harbor can be read within the Knowledge Base. Just put "EPM 11.2" into the KB's search box and several articles on this topic will appear.