Tuesday, April 27, 2021

EPM 11.2.5 - My Week is Full

We finally get Oracle Fusion Middleware (FMW) 12.2.1.4!

SQL Server 2017!

Essbase 21c certified!

The bad news:

If you're still using Essbase Studio, migrate away from it.  Essbase Studio is slated to be retired in EPM 11.2.6 as per this Oracle Statement Of Direction:  https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=81935405509356&id=2731169.1&_afrWindowMode=0&_adf.ctrl-state=rbcbyuw9y_4

Visual Basic scripts for FDMEE need to be migrated to Jython ASAP.  VB scripting is expected to be decertified in EPM 11.2.7.  We knew this some time ago, but now we now the release number where VB goes away for FDMEE.

Interesting news:

Oracle's readme for 11.2.5 confirms what I've suspected.  In 11.2.x we aren't getting Patch Set Updates (PSUs), but rather new "dot" releases.  Here's a copy/paste directly from the 11.2.5 readme:

EPM System updates are released on a quarterly basis, generally in January,
April, July, and October.

You can directly apply an update from the previous two updates.

I won't paste the entire thing.  When you download 11.2.5 from Oracle eDelivery, you get this folder:

EPM_11.2.5\EPM 11.2.5.0.000\EPM System Installation Documentation\EPM System Installation

Stop, open this folder.  Open the readme before doing anything.  Here's the Cliff Notes.  You can apply 11.2.5 as an in-place upgrade on top of either 11.2.3 or 11.2.4.  Do not attempt to apply it on top of 11.1.2.4 or any older release; you need to do a fresh install and LCM your apps from the older release.  Only 11.1.2.4 is certified for this migration path.

If you were an early adopter like me, you need to jump to 11.2.2 before you can get to 11.2.3 or higher.  You can apply 11.2.2 on top of either 11.2.0 or 11.2.1.  You can't get to 11.2.3, .4 or .5 unless you're on at least .2.  Job security!

Migration Path documentation:  11.1.2.4.x to 11.2.x

If you're on Financial Close Management 11.1.2.3 or older, prepare for pain.  I've done it and it is not fun.  :(

It is now time to back up my sandbox and attempt the upgrade.  Wish me luck!


35 comments:

  1. Thanks for the news. One question slightly different from the topic. Essbase provided with these versions are of restricted use. It cannot be used for stand alone essbase purpose. And stand-alone essbase (like 21c) do not support FR. What should be the upgrade path for someone uses essbase and FR together currently with 11.1.2

    ReplyDelete
  2. Thanks for the news. One question slightly different from the topic. Essbase provided with these versions are of restricted use. It cannot be used for stand alone essbase purpose. And stand-alone essbase (like 21c) do not support FR. What should be the upgrade path for someone uses essbase and FR together currently with 11.1.2

    ReplyDelete
  3. The upgrade from 11.2.4 to 11.2.5 will fail if you have previously patched OPatch to 13.9.4.2.4. (for example to apply the Weblogic CPU). It's not possible to roll-back the Opatch patch...

    ReplyDelete
    Replies
    1. I think I'm hitting this exactly issue, RJ.

      Delete
    2. If you find this in your install logs, you can be fairly certain you are hitting this issue.

      FDMEE (ODI):
      Validations are disabled for this session.
      Verifying data
      Copying Files
      Internal Error: File Copy failed. Aborting Install

      Weblogic:
      Validations are disabled for this session.
      Verifying data
      Copying Files
      Internal Error: File Copy failed. Aborting Install

      Delete
    3. Looks a similar issue. Anyone managed to get passed this?

      We have the latest CPU and Opatch.
      11.2.4 to 11.2.5- Some components does deploy but some does not. Also weblogic shows old version when we start workspace without starting updeployed servers. Looks a shaky release
      An ongoing oracle sr, but dim progress.

      Delete
    4. We tried uninstalling the opatch updates (both .4 and .5 was tried) , installation gets further, but eventually fails. I guess the only way to get past this would have been a full backup of Middleware *before* applying the opatch patch. I doubt anyone has done that however.

      Without opatch patched, the upgrade works. From the look of things, there are new issues, and issues that were supposed to be fixed does not seem to be =(

      I think we have now have four or five SRs regarding the upgrade another few on other issues.

      Delete
    5. Hi Deezel,
      Thanks for your comments. It is very upsetting to know, we just removed prior Opatch backup to make space for prior 11.2.5 backup­čśŤ

      Apparently for us the apply updates show green via the installer (opatch 13.9.4.2.5). But yes the logs says exactly like what you mentioned.

      Has this been officially confirmed by Oracle to you (no way to go to 11.2.5 with Optach)?

      Also I noticed, RCUSchema.properties being overwritten while applying update. Strangely some component still get delpoyed while accessing the blank rcuschema file, that is another mystery.

      For now, we manually put back the old rcuschema file which we backed up each server.

      Delete
    6. No update since Friday on any of the SRs. Mostly "checking internally" type of replies.

      No official word on this yet.

      Everything was green installing the upgrade, so was hopeful. After deployment however... not so much. After checking the install logs and checking the install logs in the %TMP% location.
      Ex weblogic show:

      The distribution Oracle Fusion Middleware 12c Infrastructure 12.2.1.4.0 contains incompatible features with the following: [[
      Oracle Fusion Middleware 12c Infrastructure 12.2.1.3.0

      and then a massive amount of libs. Same type of error for ODI, but different content.

      Never checked on the RCUSchema, readme only said to run RCU after "install". I probably would have checked it if I hadn't already checked the install logs ;)

      Delete
    7. Same story with Oracle SR. Even though it has the utmost priority as it is PROD, still saying awaiting internally since Friday.

      I guess we will need to go back to 11.2.4 restoration :(

      Its not about running rcu, but the rcuschema.properties file is getting updated with the oringial default(kind of blank).
      Not clear if it needs to be updated with the old one as sure the configuration is accessing it.
      But we tried that as well...and fdmee and planning still not getting deployed.

      Keep us updated if you have some workaround or response from Oracle


      Delete
    8. Well, latest word now is that

      "WE already have bug on this bug # 32822099 dev team is working on it will update you"

      Delete
    9. This whole apply update for this release is total junk. I am running into the same issues as the rest of you. It just blew up 2 of my servers. I am pretty sure it's this whole OPatch issue and not only if you tried to patch it to apply updates to the prior version. I didn't apply any updates to my servers and it still exhibited the same behavior. I think Oracle should pull down this update because it just doesn't update properly, as the prior releases did. It all seems tied to the FMW 12.2.1.4 update, as far as I can see. Really disappointing.

      Delete
    10. Well, colleague applied it to a clean 11.2.4 and it seemed to work. That was only a personal VM however. I haven't tried it yet as I am a bit busy and my VM happen to have the opatch and all the security patches on it. Already know it will fail basically.

      Delete
    11. I also confirm it work fine in a clean install-- 11.2.4 and then 11.2.5 updates.

      Delete
    12. and now they come back and ask if opatch was patched... which directly. Also they have all logs. Makes me wonder what they have been doing for two weeks.

      Delete
    13. and now they come back and ask if opatch was patched... which directly. Also they have all logs. Makes me wonder what they have been doing for two weeks.

      Delete
    14. Deezel, your answer: they are working on SRs where the customer is louder than you are.

      If you are still having an issue and it is delaying your upgrade project, bump up the severity and then call the toll-free number and ask to speak with a support manager.

      Delete
    15. Yeah, I had hoped it would tie in to another issue where we had Dev attention already. But customer decided to go live on 11.2.4 and got a PSE for the other issue. So 11.2.5 is not burning at this customer atm. Will see where it ends up.

      Delete
    16. I ran into this recently too on an 11.2.3 in-place update -- in our case, the jdk under middleware_home had been updated (opatch had also been updated, and several weblogic security patches had been applied). Luckily we had taken snapshots before running the 11.2.5 installer. We rolled the machines back to before the update, rolled back the JDK to the out of the box version, and the installation seems to have passed successfully. I manually verified that all the FMW infra components were at the correct version, and the weblogic / fdm / odi installation logs no longer recorded the 'Internal Error: File Copy failed. Aborting Install' error: just a normal installation and a success message.

      I did not roll back the weblogic security updates or the opatch updates, just the JDK under middleware_home. The weblogic BSU requires an updated JDK in some EPM versions, so I wonder if it's not actually opatch or any weblogic updates behind all this, but rather the JDK version behind them. YMMV of course, but this was my extremely recent experience with all this nonsense. good luck y'all!

      Delete
    17. Dave for you SR thing. We already did the max we can (severity and escalation), but took more than a week. Seems like oracle doesn't bother on-premise. Finally we got fed up and rolled back.

      Currently , we are discussing with the Oracle installation package guy, who identified Opatch issue but still need to provide a fix solution

      Delete
    18. I don't think JDK version matters as we have also tested without touching it. We have tried the following scenarios

      Install 11.2.4
      apply opatch patch 13.9.4.2.5
      apply 11.2.5
      Fail

      Install 11.2.4
      apply opatch patch 13.9.4.2.5
      uninstall opatch patch 13.9.4.2.5 (it still think it's 13.9.4.2.5 however)
      apply 11.2.5
      Fail

      Support said that the version of opatch after uninstall should be 13.9.4.2.0, I have never seen that be so after uninstall in any of the tests we have done. Same guy updated us today that Dev said the last scenario works for them. We are waiting for info on exactly how they accomplished that. Maybe there are some extra parameters you can set? Not holding my breath... We'll see I guess.

      Delete
    19. Latest update, they have a workaround that is currently in QA testing.

      Delete
    20. That's encouraging news, Deezel. Appreciate the update!

      Delete
    21. They have started posting the solution now.

      One this I will add is if this is NOT the first maintenance upgrade you do, you must edit the .oracle.products manually to get your upgrade to work (it will most likely miss products otherwise and the installation will go fine, but the configuration will fail).

      I have tested this on my 11.2.3 (which has been upgraded from 11.2.0 -> 11.2.2 -> 11.2.3 -> 11.2.5) and it seem to work fine, haven't done extensive testing yet however.

      Happy EPM-ing ;)

      ISSUE SUMMARY:
      After Update from 11.2.4/11.2.3 to 11.2.5 deployment of products fail.
      RECOVERY STEPS:
      The steps listed here are to restore the Updated 11.2.5 instance with deployment failures to working state.
      Back Up:
      Take back up of EPMSystem11R1 and user_projects folder available under Middleware Home.
      Steps:
      1 Stop all services
      2 Delete all the folders under MWH except EPMSystem11R1, dbclient32,dbclient64 and user_projects
      3 Rename the .oracle.products available at EOH\.oracle.product as .oracle.products_11.2.5
      4 Rename the .oracle.products0 as .oracle.products
      5 Run Apply Update
      6 Configure as per Update Guide
      7 Start Services
      8 Sanity test passes
      9 Delete the back up folders for EPMSystem11R1 and user_projects

      Delete
    22. Thank you for sharing, Deezel! Mind if I write this as a standalone blog post? I'll give you credit.

      Delete
    23. Be my guest Dave. I am sure Oracle will eventually release a document on it =)

      Delete
  4. Hello Dave,
    Did you figured out how to upgrade ODI for FDMEE ("Manual Configuration Tasks for FDMEE"). I get the error that there are no schemas for ODI (and it is clear because ODI schema is created by FDMEE configuration not by running RCU).
    P.S. I'm using Oracle DB and did RCU configuration by non-sysdba user.
    Thanks.

    ReplyDelete
    Replies
    1. Hi Vlad,

      I played around with this and FDMEE 11.2 doesn't like an ODI that was independently created through RCU. If we create ODI through the FDMEE database config process, we cannot run the Manual Configuration Tasks for FDMEE.

      The Manual Configuration Tasks as documented in the 11.2 install/config guide is telling us to run the Upgrade Assistant. UA only works against schemas created by RCU.

      Delete
    2. Works fine for us using ms SQL server.
      May be you are missing entry in the registry version table of FDMEE. ODI rcu picks schema detail from there

      Delete
  5. It's nice that they're beginning support of Essbase 21c but note that Planning is NOT yet supported with 21c, only with the packaged 11.1.2.4.040 Essbase. Maybe next quarter.

    ReplyDelete
  6. Hello,

    Thank you so much for all your comments. We are on 11.2.4 and not applied any updates to OPatch or any Critical patch updates. Do you think it is okay and safe to upgrade to 11.2.5?

    Thanks,
    Sreenivasa

    ReplyDelete
    Replies
    1. I tried this on my test VM server and it worked. As long as your OPatch is still the default 13.9.4.2.0 version, the upgrade should work.

      Delete
  7. I see the OPatch that is delivered with 11.2.4 is 11.1.0.9. Are you saying we need to update OPatch to apply Weblogic Critical Patches to 12.2.1.3 version?

    ReplyDelete
  8. I would completely backup or snapshot your server before you attempt it. The problem is with the FMW OPatch version, which is 13.9.4.2.2, not the one delivered with EPM itself. I don't recall applying any Oracle CPUs on this one environment, to update any Weblogic, FMW or Java, which would have required updating the FMW OPatch to at least 13.9.4.2.4 or 13.9.4.2.5, but, I could be mistaken. I decided to just wipe the whole install and install clean with 11.2.5 and that seems to work. The the UI seems a bit clunkier in this release, with a black box outline around the spot where your mouse is hovering, in Workspace.

    Until Oracle comes up with a fix for this potential OPatch issue, I personally wouldn't recommend trying to apply as an update from 11.2.3 or 11.2.4 to 11.2.5.

    ReplyDelete
  9. Thank you so much RJ and Larry for your responses. I am able to Upgrade from 11.2.4 to 11.2.5 successfully. I found couple of issues during testing 1) HFR Web Studio is not working with multiple FR instances, it works with one instance 2) Unable to add 2nd instance of APS server in EAS client. In both cases first instance working fine, 2nd horizontally scaled instances are having problem.

    ReplyDelete

Thank you very much for your interest in this blog! I hope you're finding it helpful.

Please keep comments relevant to the topic in the post, as this blog is not a free-for-all substitute for Oracle Support or traditional consulting. If you have many questions unrelated to the specific topic at hand, consider contacting me on LinkedIn (https://www.linkedin.com/in/daveshay) so we may discuss the possibility of consulting.

Commenting on posts older than 90 days unfortunately goes into moderation, thanks to spammers who've been hitting this blog. Please have patience, and thanks for your understanding!

Comments including URLs linking back to gambling or other things unrelated to Oracle EPM will be deleted on sight. If you're an EPM consultant and are offering me constructive criticism or a tip, go ahead and DO link back to your blog or firm's website if you so desire.

Thanks again for reading!