Monday, March 30, 2020

Planning RMI 11.2: "Could not find the service start class" SOLUTION

I spent more time troubleshooting this than I care to admit.  Rather than walking you through the things I tried, let's cut to the chase!

Symptom Observed:
You start the "Oracle Hyperion RMI Registry" service in the Control Panel or via a script.  The service shows it is running and no logfiles reveal an up/down status for it.

Meanwhile, your classic Planning batch scripts don't work like they used to...


Digging deeper with traditional port checking tools, we see the RMI port 11333 is not listening.

Finally, we check the Windows Event Viewer.


Oh boy, this is going to be a long day...

Quick Tip:
Search engines and the Oracle Knowledge Base don't have current information on this "could not find the service start class" error yet, at least not where Oracle EPM 11.2 is concerned.  There's some older articles out there about ports, service names, etc. that don't lead you to a solution.  Hopefully this blog entry will help!

Solution:
First, export this portion of your Windows Registry in case you want to roll back to the original configuration:

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\HyS9RMIRegistry_epmsystem1\Parameters

And here it is!

Planning RMI is still using the old Java 6.

I haven't dug enough to find the exact culprit, but there's likely a mix of Java class files in RMI where some were compiled with Java 6, and others with Java 8.  Java 6 can't run Java 8 classes, but Java 8 can often run both Java 6 and Java 8 classes.

So the trick is to locate a "jvm.dll" file that is Java 8 and 32-bit. I tried using a 64-bit one and it immediately complained I was mixing 32-bit and 64-bit.

In a vanilla EPM 11.2 system, you will have a 32-bit Java 8 if you allowed it to install the 32-bit Oracle Database Client.  The file is:

\Oracle\Middleware\dbclient32\jdk\jre\bin\client\jvm.dll

Replace the "JVMLibrary" Data value in my screenshot above so your system looks like this:


Bounce RMI, and suddenly port 11333 is working and you may continue with your day.

Here's hoping a more official fix will be forthcoming!

Sunday, March 29, 2020

Q&A From March 12 2020 Webinar

We ran out of time in our ODTUG EPM 11.2 Webinar to get to everyone's questions, so we had to rush through it.  Without further ado, here's the questions submitted in writing by attendees.  I'll discuss them in no particular order and exactly as they were submitted.

Q: Does 11.2 still have rapid deployment option?
A: Yes.

Q: So no more CopyApp Utilities to migrate HFM Apps between environments?
A: No, the venerable HFMCopyApplication utility was deprecated in 11.1.2.4.  In theory, the latest patch of the utility from 11.2.3.7xx could work, followed by running the HFMUpgradeApplication utility.  In practice, I believe Oracle Support would advise migrating via the HFM LCM Snapshot instead (if, and only if, you are on 11.1.2.4).  Or, use the Import Application link in Navigate->Administer->Consolidation Applications.

Q: Is RCU schema/database needed per application server or per environment?
A: One per application server.  A pure Essbase server with no WebLogic web apps wouldn't need it.

Q: Is Windows Server 2019 only supported or okay to install Windows Server 2012 or 2016?
A: In theory, yes you can put 11.2 on either Server 2012 R2 or 2016.  I've done it myself.  In practice, I'd advise against it using 2012 and 2016.  You want to use server operating systems specifically vetted and certified by Oracle Corporation.  Oracle has only certified Server 2019 at this time for 11.2.

Q: Any word on Oracle 18c or 19c support?
A: Terrific question and I'm getting asked this more and more.  I'm told by DBAs that 19c is essentially 12c technology with bugfixes sitting on top of it.  I know of at least one customer who is using 19c successfully for their EPM 11.1.2.4 system.  I would love to see certification from Oracle on 19c.

Q: Understand Financial Reports with Essbase connection cannot be migrated from 11.1.2.4 to 11.2? Is that right?
A: The Essbase connection type is visible as a drop-down in Database Connection Manager. I'll need one of my report developers to kick the tires to see if it works.  Sorry I don't have a definitive answer yet!

Q: Hello, how can I  save the nodemanager password so when I start OHS I don't have to type the password in windows?
A: Yes!  The syntax is all on one line:
\Oracle\Middleware\user_projects\epmsystem1\httpConfig\ohs\bin\startComponent.cmd ohs_component storeUserConfig
You only need to do this one time per OHS server.  Be mindful that if a different userID needs to perform restart maintenance, that person either needs to be trained on this process, or you put it in a script from them to double-click.  You will be prompted to type in epm_admin's password, and then you won't be asked again.

Q: will you be supported by oracle if you install 11.2 on ANYTHING other than Windows 2019 server.
A: I can't answer for Oracle on this point.  Depending upon who is assigned to work on your SR, someone might kick it back at you.  My personal recommendation is to use certified operating systems only - especially where Production is concerned.

Q: Any word on when Rhel7 server version for EPM 11.2 will come out?  Other then soon?
A: "Soon".  :)  I have not received any specific guidance beyond what is available for public consumption.

Q: Users will probably miss the Home page in 11.2.  Any workaround, or plan for bringng that back?
A: As a workaround, users can set their preferences so a specific app launches upon login.  The loss of the Home page is a byproduct of the architectural change that removed Reporting & Analysis Framework.

Q: what is the replacement for the HFR studio thick client?
A: The Reporting Web Studio is available within EPM Workspace and was first introduced by Financial Reporting patch set update 11.1.2.4.700.  In 11.1.2.4.7xx there's a manual OHS configuration step before it appears to users.  It works out-of-the-box in 11.2 without needing to manually update OHS.  Several PSUs were issued since then which improved parity with the thick client.  Customers who are on 11.1.2.4.700 or higher are urged to become familiar with the web studio before making the jump to 11.2.  Alternatives would be saved SmartView templates or other technology.


Thursday, March 26, 2020

March 12 2020 webinar - because you REALLY want to hear my voice?

EPM 11.2 Upgrade - Lessons Learned

I have a face for radio, but not the voice!  If you can withstand my voice, click the link above and listen to findings about the Oracle EPM 11.2 upgrade.

A big THANK YOU to the nice people at ODTUG for hosting this session and allowing me to present!

Wednesday, March 4, 2020

Upcoming EPM 11.2 Webinar March 12 2020


Thursday, March 12, 2020 12:00 PM - 1:00 PM EDT
EPM 11.2: Lessons Learned and 2021 Preparedness
Dave Shay, Datavail

Webinar abstract:


As we all know, Oracle EPM 11.2 is here! But…it was released too late in 2019 for most organizations to budget an on-premises EPM upgrade for fiscal 2020. However, the end of support for 11.1.2.4 is also looming in 2021. If you’re staying on premises, an upgrade to 11.2 should go live no later than December 2021 (earlier if subject to SOX controls).

Rather than waiting for the next budget cycle to roll around, this webinar will show attendees how to prepare for an upgrade this year without spending significant time and capital. We’ll also share what we’ve learned while upgrading to 11.2 and what you can expect post install.


Visit this link to reserve your free spot!

Avid readers of this blog will be familiar with a good chunk of the content I'll present.  In this session I'll do my best to condense observations collected since the Dec 18 2019 Oracle EPM 11.2 Release Date into a 1-hour session.

The session will include Q&A at the end.

Thanks to ODTUG for sponsoring this session!

Saturday, February 22, 2020

EPM 11.2 - More Than a Cupful of Java

If your IT department runs deep-scans of your Oracle EPM server filesystems looking for older/vulnerable versions of Java, this post is certainly for you.

This blog has previously written about how Oracle Jan 2020 Critical Patch Update Affects EPM 11.2  In the post linked here, you'll find information about a newer version of Java SE 8 that's applicable to EPM 11.2.0.0.  (You can expect quarterly Java 8 updates until it reaches end-of-life... so get used to this if you aren't moving to the Cloud)

So let's say you've downloaded the newer Java and want to replace the content of your pre-existing Java folders with the newer one.  Just how many Java locations do you have on your EPM 11.2 server, anyway???

Being a UNIX nerd from the beginning of my IT career, I was just dying to take the Cygwin utilities for MS Windows for a spin on my EPM 11.2 sandbox server.  Here's what I found!

F:
CD \Oracle\Middleware
C:\cygwin\bin\find . -type f -name java.exe -print -exec {} -version ;

./dbclient32/jdk/bin/java.exe
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) Client VM (build 25.171-b11, mixed mode)
./dbclient32/jdk/jre/bin/java.exe
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) Client VM (build 25.171-b11, mixed mode)
./dbclient64/jdk/bin/java.exe
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
./dbclient64/jdk/jre/bin/java.exe
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

./EPMSystem11R1/common/JRE/Sun/1.6.0/bin/java.exe
java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b52)
Java HotSpot(TM) Client VM (build 20.10-b01, mixed mode)

Ewww!!!  Java 6?  See Java 6 Not Eradicated in EPM 11.2

./EPMSystem11R1/common/JRE/Sun/1.8.0_181/bin/java.exe
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
./EPMSystem11R1/common/ODBC-64/Merant/8.0/jre/bin/java.exe
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)
./jdk1.8.0_181/bin/java.exe
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
./jdk1.8.0_181/jre/bin/java.exe
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
./ohs/oracle_common/jdk/jre/bin/java.exe
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

That's 10 separate java instances.  Now that you finished scrolling down and witnessed The Matrix fly past your eyes, here's some thoughts.

The "dbclient32" and "dbclient64" folders only exist if you selected them when you used EPM's installTool.cmd wizard.  (Picking either FDMEE or HFM will cause the Oracle DB clients to be automatically selected... you may deselect them if you're using MS SQL Server exclusively).

The "dbclient32" folder is the only 32-bit Java you'll have.  Keep this if you intend to use the SQL Developer tool, which is embedded within the dbclient32 folder structure.  SQL Developer in this release still expects a 32-bit Java.  All of the other folders listed above are 64-bit.

As of this writing, Java 1.8.0_241-b26 (Java SE 8 Update 241) is the version Oracle published in their latest Critical Patch Update.  This information will become dated starting in mid-April 2020, but the concepts described here will remain the same.

Do not be tempted to rename the \Oracle\Middleware\jdk1.8.0_181 folder or your EPM 11.2 system breaks!  This folder name is hard-coded throughout the system (Windows Registry, batch scripts, and other files).  This may be a topic for another blog entry.  For now, to get security-compliant you might back up the 181 folder and then replace its contents with the contents of 241.

I'm out of Java coffee.  Time to stop blogging.

Tuesday, February 11, 2020

SmartView Essbase Ad-Hoc and EPM 11.2

Here's a short(er) blog entry about Hyperion / Oracle EPM 11.2, I promise!

Are you getting a pop-up error in SmartView when trying a top-level SmartView ad-hoc retrieve from Essbase?  In my case, I got a weird Java message complaining about "essggridoption.setAncestorTop()".

Our culprit here is a combination of the order by which Essbase 11.1.2.4 patches were automatically applied when EPM 11.2 was installed, and perhaps some funny business involving how files revert to read-only status in Windows Server 2019.

The EPM 11.2 installer automatically applies these Essbase 11.1.2.4 patches where appropriate:

Your problem here is this:

The Essbase Java API files used by Analytic Provider Services (APS) doesn't match what the Essbase server expects.  There's an Essbase API protocol mis-match here!

The patches automatically applied by the EPM 11.2 installtool are stored here:
\Oracle\Middleware\oracle_common\OPatch\Patches\patchFiles
Sure enough, there's a subfolder here for 29749671.

Stop your EPM services on the APS host machine(s).

Attempt to copy \Oracle\Middleware\oracle_common\OPatch\Patches\patchFiles\29749671\files\common\EssbaseJavaAPI\11.1.2.0\lib\
to
Oracle\Middleware\EPMSystem11R1\common\EssbaseJavaAPI\11.1.2.0\lib\
choosing to overwrite the existing files.

One of three things will happen:
  1. The copy happens OK and you may restart services.
  2. It complains that a Java process is using one of the files you're trying to overwrite.  You'll have to use your Windows Task Manager to hunt down and terminate those processes (or reboot).  Then try again.
  3. It complains the target file is read-only.  If this is the case, right-click the target file, click Properties, and remove the Read-Only flag.  Then try again.
#2 above happened to me.  After I killed the offending process, replaced the files and restarted services, SmartView worked as expected!

Saturday, February 8, 2020

EPM 11.2 Distributed Environment - Yup, Another RCU Post!

My blog entry EPM 11.2 RCU and you apparently struck a nerve, so consider this a follow-up!

Before we get to the "do this, do that" part, let's examine how we can determine if our attempt to deploy Hyperion / Oracle EPM 11.2 web services across multiple machines is... well, botched.

Part 1 - Is my distributed deployment botched?

Let's assume you've followed the instructions within the post linked above, and you have successfully created your WebLogic domain and have deployed a few things on machine #1 (EPM Foundation, Oracle HTTP Server, and perhaps a few supporting modules such as APS, EAS, CalcMgr, etc).  Now you're in the EPM System Configurator on machine #2 and are getting the dreaded red X when you try the Deploy To Application Server task.

user_projects\EPMINSTANCE\diagnostics\logs\configtool.log on machine #2 offers nothing beyond the vague and dreaded InvocationTargetException Java stack trace error.

By the way, ERROR messages prior to this point may typically be ignored.  You will first see "ERROR: Jars manifest check failed with message "Some referenced jars do not exist" and then a little further into the log you'll see "env.isWindows is not a function".  Don't waste your time troubleshooting those, as they're unrelated to your issue.

Once you see InvocationTargetException, close out of the log and now open wlst_debug.log in the same directory.  This points us to the underlying issue!

Do you see "WLSTException: No element JDBCSystemResource was found" at the top of a lengthy Java stack trace error?  If so, look just a few lines above.  The message doesn't look like an error at first glance:

2020-02-08 13:38:53,392 INFO  [1] com.oracle.cie.domain.script.ScriptExecutor - find JDBCSystemResource!JdbcResource!JDBCDriverParams "opss-audit-DBDS!opss-audit-DBDS!NO_NAME_0" as obj5
How could this be bad?  It is an [INFO] message after all, not an error.

It turns out there's an OPSS schema that's created by the RCU utility described within my earlier post.  The WebLogic Scripting Tool (WLST) can't find the OPSS jdbc definition files on machine #2 because they don't exist.  More than likely, this folder on your machine #2 is empty.


If you find yourself nodding your head at this point, yes.... your distributed deployment is botched.  But it can quickly be rescued!

(Note: Don't be tempted to copy those files over from machine #1.  They contain references to the RCU prefix and I haven't tested what might happen if you do that -- I suspect it would be something bad?)

Part 2 - How to fix machine #2 so we can deploy web services there?

Follow these steps:
  1. Login to your WebLogic Admin Console on machine #1 (http://machine1:7001/console) as epm_admin and click "Release Configuration" if you see that button is available.
  2. On machine #2, blow away user_projects\domains\EPMSystem -- all files & subfolders.
  3. On machine #2, launch RCU (Middleware\oracle_common\bin\rcu.bat)
  4. Use the "create" option and choose a new prefix.  Pick something sensible like the EPMINSTANCE name (assuming you aren't deploying as "epmsystem1" everywhere) or machine #2's hostname.
  5. Pick all checkboxes except for the Oracle Data Integrator ones.
  6. Pick the option to use the same password for all schemas.  This way is much easier.
  7. Let 'er rip and close the RCU utility when done.

Last steps...

I've read in the comments on the RCU blog entry that Oracle Support is telling some people to use different "schemas" for each machine's RCU database. 

When in doubt, go with Oracle's suggested approach of 1 Oracle Schema or 1 MS SQL Database per RCU Prefix (machine), just in case you run into trouble later and need to open an SR.  This means more things for your DBA to back up on a regular basis, but it may cause less confusion for Oracle Support as you work through the SR together.

Let's take a look at the back-end database.  In this example I'll use MS SQL Server 2016.

I have more machines deployed than are visible within this screenshot, but hopefully the concept comes across!

Each EPMINSTANCE ("machine" if you are not scaling vertically) has its own RCU "prefix" assigned.  In this example, "DEV" and "LANDO".  These schema users were created by the RCU utility.

I'm using same SQL database for these and so far things are working OK.

Should you want to cluster out even further for High Availability etc., you might consider either using Oracle's suggested approach (1 database/schema for each EPMINSTANCE), or at least one per each cluster....

... e.g, EPMRCU0 has the RCU prefixes for Foundation0, CalcMgr0, EAS0, etc.  Then EPMRCU1 could have the *1 servers, EPMRCU2 could have the *2 servers, and so on.


Getting back to the task at hand, copy RCUSchema.properties from machine #1's EPMSystem11R1\common\config\11.1.2.0 folder to the same location on #2.  Edit it on machine #2 and change the "schemaPrefix=" line to the prefix you created for machine #2.  You shouldn't need to edit the two SysDBA*= lines.  If you aren't using the same SQL database / Oracle schema, you can wipe out the RcuSchemaPassword= value and add the plain text password (the password will be automatically encrypted if machine #2 becomes un-botched) and also DbURL= as appropriate.

Finally, go back into the configtool and re-attempt your deployment.  So that you don't have to re-read through old log entries, you might rename EPMINSTANCE\diagnostics\logs\config to some other name before you do this.

In Closing

Yes, I did botch one of my machines on purpose... I wanted to see what would happen if I skipped the RCU step for a machine where I did NOT intend to deploy EPM Foundation.  Why did I do this?  (Aside from enjoying pain)

Hop out to the EPM 11.2 documentation landing page on Oracle's site, and review:

Deployment Options Guide
    Section 2.1 Clustering Java Web Applications
        "When scaling Oracle Hyperion Foundation Services (underline emphasis mine), 
        you must create a new schema using RCU and edit RCUSchema.properties on each
        machine in the deployment."

This specific underlined sentence fragment led me to believe RCU was only need on machines where Foundation would be deployed.  Silly me for taking the documentation seriously!

Another related point.  See:

Installation and Configuration Guide
    Section 6.2 Creating Infrastructure Schemas Using Repository Creation Utility
        "You must run RCU on each machine in your environment."

I'd add one caveat here; RCU is only needed on machines where you want to deploy WebLogic Managed Servers (Planning Web, HFM Web, CalcMgr, etc).  Essbase, Essbase Studio, DRM, HFM Java Server, and other non-WebLogic processes don't need it because they are independent from WebLogic.  So the general rule of thumb here is: if EPM System Configurator creates a user_projects\domains\EPMSystem folder on the machine, that machine needs RCU.

Time to enjoy the rest of my weekend.  Be safe out there, and don't ask me about Oracle SOA!