Thursday, December 26, 2019

EPM 11.2, RCU and You!

----------------
Feb 8, 2020 Update:  There's a follow up blog entry concerning EPM 11.2 Distributed Environments but please first read through the blog entry below, as it contains per-requisite knowledge.  Thank you!

----------------

There's a venerable utility for Oracle Fusion Middleware that has been available for quite some time.  Those of us who configured Oracle Data Integrator in the pre-FDMEE days, Hyperion Financial Close Management Suite 11.1.2.1 through 11.1.2.4.2xx, or Oracle Business Intelligence know what I'm about to discuss.

Oracle Repository Creation Utility, or "RCU" for short.

In EPM 11.1.2.4 and prior, you didn't need to worry about running RCU if you weren't using the modules I mentioned within my first paragraph.

Those days are behind us now.  EPM 11.2 requires that you run through the RCU process before you attempt to create your Oracle WebLogic domain and deploy your EPM Foundation web service.

R2D2 says: "Technically, EPM Foundation is an Oracle WebLogic Managed Server."
C3P0 responds: "Don't get technical with me [further insults added here]."

So let's talk about what has changed in EPM 11.2.  Your actions will depend upon if you're using Microsoft SQL Server or an Oracle database.

If you're using Microsoft SQL Server, the steps are a little more complex because your DBA needs to create an empty database for you.  Furthermore, the DBA needs to run these queries against that database before you may continue and run RCU.  Let's assume the DBA has named the RCU database "epmrcu":

ALTER DATABASE epmrcu Set READ_COMMITTED_SNAPSHOT ON
GO
ALTER DATABASE epmrcu Set ALLOW_SNAPSHOT_ISOLATION ON
GO
ALTER DATABASE epmrcu COLLATE LATIN1_GENERAL_CS_AS
GO


Note the COLLATE option is slightly different for RCU's database vs. everything else EPM wants to use.

Source:  SQL Server Database Creation Requirements

The above steps aren't needed if you're using an Oracle Database; RCU creates the schemas and associated users for you.

Now for both MS SQL Server or Oracle Database, you need to fire up Oracle/Middleware/oracle_common/bin/rcu.bat and follow the instructions here:
Using RCU 

Unless you're a DBA, you will need a DBA's assistance.  The RCU utility requires someone with elevated rights to type in their credentials for a DBA superuser account ("sa" for MS SQL or "sys as sysdba" for Oracle).  There's the option to save the commands to a SQL query so your DBA can run it on their own, but I've experienced mixed results with this in the past.

So you now have your RCU database initialized.  There's just one more manual step to do.

On each server where any Oracle WebLogic components run, you need to manually edit this file:

Oracle\Middleware\EPMSystem11R1\common\config\11.1.2.0\RCUSchema.properties

Here's the fun part:  passwords are exposed in plain text until you run through configTool on the server, at which time the passwords become encrypted.
Source: Updating RCU.properties File

(Edit: I blew away my MS Server 2016 VM and spun up MS Server 2019, after being advised by Oracle that Server 2016 hasn't been certified yet for EPM 11.2.  After running through RCU, I inspected the RCU.properties file and noticed encrypted passwords.  I still had to update the JDBC URL and some other information)

Once you complete all of these tasks, you may finally launch configTool and deploy your WebLogic Admin Server and your various WebLogic Managed Services.  If you overlooked these instructions from the config guide, unfortunately you will need to blow away the entire domains/EPMSystem folder structure and start over.

Eying the Christmas Dinner leftovers in my fridge.... I may be a little delayed on writing my next post!  👀


28 comments:

  1. even updating rcu& rcu properties file also getting issues and weblogic not deploeing and giving error as "
    com.hyperion.hit.tool.deploy.wlst.ScriptWlstConsole] trace: 5:Internal Exception: java.sql.SQLSyntaxErrorException: [FMWGEN][SQLServer JDBC Driver][SQLServer]Invalid object name 'JPS_DN'."

    ReplyDelete
    Replies
    1. Hi Raaju. You can't accept the default RCU options as presented. You need to pick all of the items underneath the "AS Common Schemas" node. If it picks Oracle Data Integrator for you, remove the checkbox for that one only.

      JPS_DN is part of OPSS (Oracle Platform Security Services). RCU might bark at you if you try to go back in and re-use the same schema prefix. If it does this, use the "drop" menu in RCU to blow it away, and then recreate.

      Delete
    2. Actually have tried it before -
      Selected all options under "AS Comman Schemas" and unchecked ODI.. while deploying weblogic got the error msgs like above ...

      later tried to create another as "DEV1" and removed "DEV" & completly new DB for EPMRCU and ran the required sql to update the db isolations etc....

      after installtion of hyp11.2 and did this rcu ..still weblogic keep on failing at 50% with the above reason..

      Delete
  2. Hi Dave. I'm also having a go at configuring EPM 11.2. Spotted this RCU requirement too, and gotten through a successful configuration now (after blowing away the whole lot and starting again!).

    How does your setup look though? For me, OHS doesn't startup automatically. I've found a way to get it started, but wondering if your OHS install worked straight away, or did you go for IIS (which they say isn't supported now - but the config option is there!) or Embedded?

    The other problem I have is two of my WebLogic deployments fail to start (I've configured everything on a single managed server) - those are calcmgr and AIF (which is FDMEE). How does yours look though if you configured them?

    I'll let you know if I figure out what's wrong with my deployments.

    ReplyDelete
    Replies
    1. Had the same problem doing a compact deployment, after many attempts to get that to work. I'm using SQL Server 2016 Express. Are you also using SQL Server?

      Delete
    2. OHS has been a problem to get working, and I am considering punting and going with embedded weblogic until Oracle announces a fix.

      Delete
    3. Thanks for your updates Larry and Dave.

      I'm actually testing on Oracle 12c rather than SQL Server, but yes there seems to be some issues with compact deployments:
      1. My calc mgr / FDMEE problem was solved by redeploying, but upon doing that it forces you to redeploy to a new managed server. I'm concerned that the deployment of these didn't work in a single managed server, but I saw no concerning error during the config.
      2. If this is the case, it's going to be a problem later when patching compact deployments as sometimes opatch needs a manual re-deployment step after the patch.

      For OHS, I fixed it by copying Middleware\user_projects\epmsystem1\httpConfig\ohs\config\fmwconfig\components\OHS\ohs_component over Middleware\user_projects\epmsystem1\httpConfig\ohs\config\fmwconfig\components\OHS\instances\ohs1

      This needs to be repeated on each re-configure.

      It still doesn't autostart, but it looks like it never will but you could write a script that calls:
      Middleware\user_projects\epmsystem1\httpConfig\ohs\bin\startComponent.cmd ohs1

      It prompts for a password, but to solve that run this once and it'll stop asking after that:
      Middleware\user_projects\epmsystem1\httpConfig\ohs\bin\startComponent.cmd ohs1 storeUserConfig

      Delete
    4. I think the compact deployment into the single managed server wasn't tested enough by Oracle. Seems to be a problem regardless of RDBMS. My FDMEE issues went away by clearing out the compiled temp files and allowing WLS to recompile them, however, Calc Mgr is still barking about not being able to find the CSS.jar file, for some reason. I'm going to continue to pick and prod at it to get it to behave. I've solved problems like it before. I'm sure it will get into line.
      Thanks for the tips on OHS. It's definitely more cumbersome in 12c with having the config file in multiple places and having to start it from a cmd line or using a script. I hope to find a way to get OHS to run as a service, as it had previously. There has to be a way :)

      Delete
  3. For the RCU DB, ALLOW_SNAPSHOT_ISOLATION ON is NOT listed as needed, if you are using SQL Server. It, and the other 2 settings listed, are listed as needed for all EPM DBs, except HFM. In 11.1.2.x, only EPMA had the requirement. Was hoping that those settings would go away, but, now they are become requirements basically across the board. Oh well.

    ReplyDelete
    Replies
    1. I have to say the SQL Server requirements are a bit confusing. One set of requirements for the main EPM stack, and another set of requirements specifically for RCU. RCU's requirements have always been different and I hit issues every time I tried implementing Financial Close / Account Rec until I had the "oops" moment and remembered RCU is special. For now it seems that stuff carries forward into EPM 11.2 and I'd love for the manual steps to be removed in a future Fusion Middleware update.

      Delete
  4. @Dan C: Appreciate your detailed feedback. I tend to avoid compact deployments but I found your remarks related to OHS very interesting. I'm running Windows Update on the lab server now and then will give your suggestion about copying the files and using a script a try. My team uses a custom script to start EPM services anyway, so adding one line to it to customize how OHS is started is pretty much immaterial.

    Here's hoping for a happy and prosperous 2020 to all reading!

    ReplyDelete
  5. Hi Dave and Dan,

    We had the same problem with the deployment of FDMEE and Calc, but we have managed to resolve it. In the logging we found the following “Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid.”
    We start Weblogic admin, and then reenter the user/pass for the deployed weblogic applications (below the steps).

    * Start the Weblogic Admin Server using \Oracle\Middleware\user_projects\domains\EPMSystem\bin\startWebLogic.cmd.
    * Log in to the Admin Console
    * Then go to Environment > Servers > Summary of Servers >[server that is not starting].
    * Now go to the "Server Start" tab all the way right.
    * Click the "Lock & Edit" button
    * find the User Name and Password section all the way down.
    * Enter the User Name and Password and click Save.
    * Now click "Activate Changes".
    * Now you can start the service(s).
    you have to do this for all deployments, the ones that is troubling you 😊

    ReplyDelete
    Replies
    1. the service which running normal dosen't have user name and password can you clear which user name and password i have to use

      thanks in advance
      Ahmed

      Delete
    2. From time to time, I get "boot identity not valid" and I use this simple method to remediate it.

      1. Start WebLogic Admin Server on its proper machine and wait for "server state changed to RUNNING".
      2. Copy user_projects\EPMSystem\servers\AdminServer\security\boot.properties to the machine and WebLogic Managed Server you're having difficulty starting.
      3. Start up the problematic WebLogic Managed Server.

      This approach works for me most of the time.

      Example: WebLogic Admin Server runs on machine DEV1, and you can't start Planning on machine DEV2 due to the "boot identity" error.

      1. Start WebLogic Admin Server on DEV1 and wait for the RUNNING state (in the startup console window, not the Windows Services panel).
      2. Copy AdminServer\security\boot.properties from DEV1 to Planning0\security\boot.properties on DEV2. Create the \security folder underneath Planning0 if it is missing. Back up and overlay Planning0\security\boot.properties if it exists.
      3. Keep Admin Server up and running on DEV1. Now start Planning on DEV2.

      Delete
  6. Hi,
    Has any one tried a distributed install. I've Foundation,FCM,FR installed on node 1 and when I'm trying to configure Essbase, Planning, HPCM, APS on the second node, the deployment is failing. The logs indicate the message - "The schema EPM_OPSS is already in use for security store". I had copied the RCUSchema.properties file from the first node to second node. But deployment fails with above error. I don't think I need to run RCU for each node, do I?

    ReplyDelete
    Replies
    1. I had the same issue. The method to resolve is this:

      1. On your second machine, attempt the "Deploy to Application Server" tasks for your chosen applications. Let it run, fail and complete.
      2.Then browse to "E:\Oracle\Middleware\user_projects\domains\EPMSystem\" directory and rename the "config" folder to "config_old".
      3. Next copy the "config" folder from the location above on your first server, to the same location on your second server.
      4. Finally re-run the configtool, selecting the same "Deploy to Application Server" tasks. This time it will succeed.

      I've got a blog post about exactly this issue: http://hyperipwn.blogspot.com/2020/01/epm-112-reusing-single-rcu-repository.html

      Delete
    2. Thank you so much for your response.
      As usual, the Oracle documentation is confusing as it mentioned "You must run RCU and modify RCU schema properties on each machine in your environment". I ended up running RCU on all the servers. Let me try your workaround. Thanks again!

      Delete
    3. We had issues deploy to application server task from second machine onwards .Followed EPM Bloke steps able to finish the deployment successfully but could not bring services from second server getting below error . Do you face similar issues when you bring services up?

      onwards .weblogic.management.provider.internal.RuntimeAccessImpl$ParseException: [Management:141266]Parsing failure in config.xml: java.lang.IllegalArgumentException: Unable to use KSS for Demo Key Stores, KSS is unavailable..

      Delete
    4. Hi Vanig,
      I'm low on resources and need to get another machine online so I can host more VMs. Once I do, I'm going to give the distributed install a try. Oracle's 11.2 install/config guide says we have to run the RCU process on each machine, but it isn't clear to me how instances 1+X might be handled differently than instance 1.

      Delete
    5. am also facing similar issue when running deploy to app server in second server is failing.
      any idea how to resolve this ?

      Server 1: FS,CM,Planning and Essbase
      Server 2:HFM and FDMEE

      I ran RCU in Server 2 with new prefix
      I got below update from Oracle Support regarding RCU



      In the case of a distributed environment, We need to create a new RCU schema on each box in the environment, by providing a new prefix name ( preferably provide the name of the box as Prefix, so it will be unique as well as easy to track ). The RCUSchema.properties file on each box should be updated with the respective Prefix ( created for that particular box ) and db details ( db sys credentials, db host name, etc. ). We should NOT use the same RCUSchema which was created on the first box ( parent box ) for subsequent boxes ( Child boxes ) in the environment. I will quickly list down the steps that I usually do :

      1. Install required EPM products as distributed on box 1 ( parent box – contains Foundation server ), box 2, box 3, etc.
      2. Create RCU Schema on box 1 by providing suitable Prefix ( host name of box 1 ) and update RCUSchema.properties file on box 1 with this information.
      3. Configure products selected for box 1 ( apart from OHS web server ).
      4. Start WL Admin server on box 1
      5. Create new RCU Schema on box 2 ( child box ) by providing a new prefix ( host name of box 2 ) and update RCUSchema.properties file on box 2 with this information.
      6. Configure products on box 2.
      7. Repeat steps 5 to 6 for subsequent boxes in the environment.
      8. Configure OHS Web Server on Box 1.

      Delete
    6. I've experimented with this and got Foundation successfully clustered to a 2nd EPM instance. I have a few more scenarios to test and then I'll write a separate blog entry about it. Otherwise I won't be able to keep up with the comments in this RCU blog entry, as the clustering topic is of interest and the Install/Config documentation could use some clarifications.

      Delete
    7. Thanks Nani & Dave,
      I also installed a distributed environment with 5 VM and also stuck in the same scenario.
      I followed Nani suggestions to create RCU for each VM and it works.
      Just two little tiny things to mention for detail:
      1.If you have already use EPM system configtool for the distributed VMs, it's better delete all configurations under user_projects; I tried just delete WebLogic domain(EPMSystem), or delete the instance(epmsystem1) I always get error message when I try to start the services.
      2.I still need to copy config folder under WebLogic domain(EPMSystem) in the first VM and redeploy to Application Server again in each VM.

      Delete
    8. did u install FDMEE ? do we need create new RCU schemas with ODI Master and Work Repo selected

      So when we configure FDMEE do we need to use the schema which we created using RCU or we need to give new schema when we configure for FDMEE ?

      Delete
    9. Hi Nani, no you do not use RCU to create the ODI Master and Work repositories. You let the FDMEE Configure step handle that. You -do- use the RCU utility on the FDMEE host server, though. See my follow-up post on this topic:
      http://www.epmonprem.pro/2020/02/epm-112-distributed-environment-yup.html

      Delete
  7. one more question does epm 11.2 support 19c Oracle DB ?

    ReplyDelete
  8. Hi Nani, as of the last update, it has not been tested so no it is not officially certified. See http://www.epmonprem.pro/2019/12/thoughts-on-epm-112-certification-matrix.html and bookmark the link to Oracle's URL where you may download the 11.2.0.0 certification matrix.

    ReplyDelete
  9. Update/correction. Oracle 19c -is- certified for EPM 11.1.2.4 and higher, all the way through EPM 11.2.1.3 as of this writing (Nov 2020).

    ReplyDelete

Thank you very much for your interest in this blog! I hope you're finding it helpful.

Please keep comments relevant to the topic in the post, as this blog is not a free-for-all substitute for Oracle Support or traditional consulting. If you have many questions unrelated to the specific topic at hand, consider contacting me on LinkedIn (https://www.linkedin.com/in/daveshay) so we may discuss the possibility of consulting.

Commenting on posts older than 90 days unfortunately goes into moderation, thanks to spammers who've been hitting this blog. Please have patience, and thanks for your understanding!

Comments including URLs linking back to gambling or other things unrelated to Oracle EPM will be deleted on sight. If you're an EPM consultant and are offering me constructive criticism or a tip, go ahead and DO link back to your blog or firm's website if you so desire.

Thanks again for reading!