Feb 8, 2020 Update: There's a follow up blog entry concerning EPM 11.2 Distributed Environments but please first read through the blog entry below, as it contains per-requisite knowledge. Thank you!
----------------
There's a venerable utility for Oracle Fusion Middleware that has been available for quite some time. Those of us who configured Oracle Data Integrator in the pre-FDMEE days, Hyperion Financial Close Management Suite 11.1.2.1 through 11.1.2.4.2xx, or Oracle Business Intelligence know what I'm about to discuss.
Oracle Repository Creation Utility, or "RCU" for short.
In EPM 11.1.2.4 and prior, you didn't need to worry about running RCU if you weren't using the modules I mentioned within my first paragraph.
Those days are behind us now. EPM 11.2 requires that you run through the RCU process before you attempt to create your Oracle WebLogic domain and deploy your EPM Foundation web service.
R2D2 says: "Technically, EPM Foundation is an Oracle WebLogic Managed Server."
C3P0 responds: "Don't get technical with me [further insults added here]."
So let's talk about what has changed in EPM 11.2. Your actions will depend upon if you're using Microsoft SQL Server or an Oracle database.
If you're using Microsoft SQL Server, the steps are a little more complex because your DBA needs to create an empty database for you. Furthermore, the DBA needs to run these queries against that database before you may continue and run RCU. Let's assume the DBA has named the RCU database "epmrcu":
ALTER DATABASE epmrcu Set READ_COMMITTED_SNAPSHOT ON
GO
ALTER DATABASE epmrcu Set ALLOW_SNAPSHOT_ISOLATION ON
GO
ALTER DATABASE epmrcu COLLATE LATIN1_GENERAL_CS_AS
GO
Note the COLLATE option is slightly different for RCU's database vs. everything else EPM wants to use.
Source: SQL Server Database Creation Requirements
The above steps aren't needed if you're using an Oracle Database; RCU creates the schemas and associated users for you.
Now for both MS SQL Server or Oracle Database, you need to fire up Oracle/Middleware/oracle_common/bin/rcu.bat and follow the instructions here:
Using RCU Unless you're a DBA, you will need a DBA's assistance. The RCU utility requires someone with elevated rights to type in their credentials for a DBA superuser account ("sa" for MS SQL or "sys as sysdba" for Oracle). There's the option to save the commands to a SQL query so your DBA can run it on their own, but I've experienced mixed results with this in the past.
Source: Updating RCU.properties File
(Edit: I blew away my MS Server 2016 VM and spun up MS Server 2019, after being advised by Oracle that Server 2016 hasn't been certified yet for EPM 11.2. After running through RCU, I inspected the RCU.properties file and noticed encrypted passwords. I still had to update the JDBC URL and some other information)
Once you complete all of these tasks, you may finally launch configTool and deploy your WebLogic Admin Server and your various WebLogic Managed Services. If you overlooked these instructions from the config guide, unfortunately you will need to blow away the entire domains/EPMSystem folder structure and start over.
even updating rcu& rcu properties file also getting issues and weblogic not deploeing and giving error as "
ReplyDeletecom.hyperion.hit.tool.deploy.wlst.ScriptWlstConsole] trace: 5:Internal Exception: java.sql.SQLSyntaxErrorException: [FMWGEN][SQLServer JDBC Driver][SQLServer]Invalid object name 'JPS_DN'."
Hi Raaju. You can't accept the default RCU options as presented. You need to pick all of the items underneath the "AS Common Schemas" node. If it picks Oracle Data Integrator for you, remove the checkbox for that one only.
DeleteJPS_DN is part of OPSS (Oracle Platform Security Services). RCU might bark at you if you try to go back in and re-use the same schema prefix. If it does this, use the "drop" menu in RCU to blow it away, and then recreate.
Actually have tried it before -
DeleteSelected all options under "AS Comman Schemas" and unchecked ODI.. while deploying weblogic got the error msgs like above ...
later tried to create another as "DEV1" and removed "DEV" & completly new DB for EPMRCU and ran the required sql to update the db isolations etc....
after installtion of hyp11.2 and did this rcu ..still weblogic keep on failing at 50% with the above reason..
Hi Dave. I'm also having a go at configuring EPM 11.2. Spotted this RCU requirement too, and gotten through a successful configuration now (after blowing away the whole lot and starting again!).
ReplyDeleteHow does your setup look though? For me, OHS doesn't startup automatically. I've found a way to get it started, but wondering if your OHS install worked straight away, or did you go for IIS (which they say isn't supported now - but the config option is there!) or Embedded?
The other problem I have is two of my WebLogic deployments fail to start (I've configured everything on a single managed server) - those are calcmgr and AIF (which is FDMEE). How does yours look though if you configured them?
I'll let you know if I figure out what's wrong with my deployments.
Had the same problem doing a compact deployment, after many attempts to get that to work. I'm using SQL Server 2016 Express. Are you also using SQL Server?
DeleteOHS has been a problem to get working, and I am considering punting and going with embedded weblogic until Oracle announces a fix.
DeleteThanks for your updates Larry and Dave.
DeleteI'm actually testing on Oracle 12c rather than SQL Server, but yes there seems to be some issues with compact deployments:
1. My calc mgr / FDMEE problem was solved by redeploying, but upon doing that it forces you to redeploy to a new managed server. I'm concerned that the deployment of these didn't work in a single managed server, but I saw no concerning error during the config.
2. If this is the case, it's going to be a problem later when patching compact deployments as sometimes opatch needs a manual re-deployment step after the patch.
For OHS, I fixed it by copying Middleware\user_projects\epmsystem1\httpConfig\ohs\config\fmwconfig\components\OHS\ohs_component over Middleware\user_projects\epmsystem1\httpConfig\ohs\config\fmwconfig\components\OHS\instances\ohs1
This needs to be repeated on each re-configure.
It still doesn't autostart, but it looks like it never will but you could write a script that calls:
Middleware\user_projects\epmsystem1\httpConfig\ohs\bin\startComponent.cmd ohs1
It prompts for a password, but to solve that run this once and it'll stop asking after that:
Middleware\user_projects\epmsystem1\httpConfig\ohs\bin\startComponent.cmd ohs1 storeUserConfig
I think the compact deployment into the single managed server wasn't tested enough by Oracle. Seems to be a problem regardless of RDBMS. My FDMEE issues went away by clearing out the compiled temp files and allowing WLS to recompile them, however, Calc Mgr is still barking about not being able to find the CSS.jar file, for some reason. I'm going to continue to pick and prod at it to get it to behave. I've solved problems like it before. I'm sure it will get into line.
DeleteThanks for the tips on OHS. It's definitely more cumbersome in 12c with having the config file in multiple places and having to start it from a cmd line or using a script. I hope to find a way to get OHS to run as a service, as it had previously. There has to be a way :)
For the RCU DB, ALLOW_SNAPSHOT_ISOLATION ON is NOT listed as needed, if you are using SQL Server. It, and the other 2 settings listed, are listed as needed for all EPM DBs, except HFM. In 11.1.2.x, only EPMA had the requirement. Was hoping that those settings would go away, but, now they are become requirements basically across the board. Oh well.
ReplyDeleteI have to say the SQL Server requirements are a bit confusing. One set of requirements for the main EPM stack, and another set of requirements specifically for RCU. RCU's requirements have always been different and I hit issues every time I tried implementing Financial Close / Account Rec until I had the "oops" moment and remembered RCU is special. For now it seems that stuff carries forward into EPM 11.2 and I'd love for the manual steps to be removed in a future Fusion Middleware update.
Delete@Dan C: Appreciate your detailed feedback. I tend to avoid compact deployments but I found your remarks related to OHS very interesting. I'm running Windows Update on the lab server now and then will give your suggestion about copying the files and using a script a try. My team uses a custom script to start EPM services anyway, so adding one line to it to customize how OHS is started is pretty much immaterial.
ReplyDeleteHere's hoping for a happy and prosperous 2020 to all reading!
Hi Dave and Dan,
ReplyDeleteWe had the same problem with the deployment of FDMEE and Calc, but we have managed to resolve it. In the logging we found the following “Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid.”
We start Weblogic admin, and then reenter the user/pass for the deployed weblogic applications (below the steps).
* Start the Weblogic Admin Server using \Oracle\Middleware\user_projects\domains\EPMSystem\bin\startWebLogic.cmd.
* Log in to the Admin Console
* Then go to Environment > Servers > Summary of Servers >[server that is not starting].
* Now go to the "Server Start" tab all the way right.
* Click the "Lock & Edit" button
* find the User Name and Password section all the way down.
* Enter the User Name and Password and click Save.
* Now click "Activate Changes".
* Now you can start the service(s).
you have to do this for all deployments, the ones that is troubling you 😊
the service which running normal dosen't have user name and password can you clear which user name and password i have to use
Deletethanks in advance
Ahmed
From time to time, I get "boot identity not valid" and I use this simple method to remediate it.
Delete1. Start WebLogic Admin Server on its proper machine and wait for "server state changed to RUNNING".
2. Copy user_projects\EPMSystem\servers\AdminServer\security\boot.properties to the machine and WebLogic Managed Server you're having difficulty starting.
3. Start up the problematic WebLogic Managed Server.
This approach works for me most of the time.
Example: WebLogic Admin Server runs on machine DEV1, and you can't start Planning on machine DEV2 due to the "boot identity" error.
1. Start WebLogic Admin Server on DEV1 and wait for the RUNNING state (in the startup console window, not the Windows Services panel).
2. Copy AdminServer\security\boot.properties from DEV1 to Planning0\security\boot.properties on DEV2. Create the \security folder underneath Planning0 if it is missing. Back up and overlay Planning0\security\boot.properties if it exists.
3. Keep Admin Server up and running on DEV1. Now start Planning on DEV2.
Hi,
ReplyDeleteHas any one tried a distributed install. I've Foundation,FCM,FR installed on node 1 and when I'm trying to configure Essbase, Planning, HPCM, APS on the second node, the deployment is failing. The logs indicate the message - "The schema EPM_OPSS is already in use for security store". I had copied the RCUSchema.properties file from the first node to second node. But deployment fails with above error. I don't think I need to run RCU for each node, do I?
I had the same issue. The method to resolve is this:
Delete1. On your second machine, attempt the "Deploy to Application Server" tasks for your chosen applications. Let it run, fail and complete.
2.Then browse to "E:\Oracle\Middleware\user_projects\domains\EPMSystem\" directory and rename the "config" folder to "config_old".
3. Next copy the "config" folder from the location above on your first server, to the same location on your second server.
4. Finally re-run the configtool, selecting the same "Deploy to Application Server" tasks. This time it will succeed.
I've got a blog post about exactly this issue: http://hyperipwn.blogspot.com/2020/01/epm-112-reusing-single-rcu-repository.html
Thank you so much for your response.
DeleteAs usual, the Oracle documentation is confusing as it mentioned "You must run RCU and modify RCU schema properties on each machine in your environment". I ended up running RCU on all the servers. Let me try your workaround. Thanks again!
We had issues deploy to application server task from second machine onwards .Followed EPM Bloke steps able to finish the deployment successfully but could not bring services from second server getting below error . Do you face similar issues when you bring services up?
Deleteonwards .weblogic.management.provider.internal.RuntimeAccessImpl$ParseException: [Management:141266]Parsing failure in config.xml: java.lang.IllegalArgumentException: Unable to use KSS for Demo Key Stores, KSS is unavailable..
Hi Vanig,
DeleteI'm low on resources and need to get another machine online so I can host more VMs. Once I do, I'm going to give the distributed install a try. Oracle's 11.2 install/config guide says we have to run the RCU process on each machine, but it isn't clear to me how instances 1+X might be handled differently than instance 1.
am also facing similar issue when running deploy to app server in second server is failing.
Deleteany idea how to resolve this ?
Server 1: FS,CM,Planning and Essbase
Server 2:HFM and FDMEE
I ran RCU in Server 2 with new prefix
I got below update from Oracle Support regarding RCU
In the case of a distributed environment, We need to create a new RCU schema on each box in the environment, by providing a new prefix name ( preferably provide the name of the box as Prefix, so it will be unique as well as easy to track ). The RCUSchema.properties file on each box should be updated with the respective Prefix ( created for that particular box ) and db details ( db sys credentials, db host name, etc. ). We should NOT use the same RCUSchema which was created on the first box ( parent box ) for subsequent boxes ( Child boxes ) in the environment. I will quickly list down the steps that I usually do :
1. Install required EPM products as distributed on box 1 ( parent box – contains Foundation server ), box 2, box 3, etc.
2. Create RCU Schema on box 1 by providing suitable Prefix ( host name of box 1 ) and update RCUSchema.properties file on box 1 with this information.
3. Configure products selected for box 1 ( apart from OHS web server ).
4. Start WL Admin server on box 1
5. Create new RCU Schema on box 2 ( child box ) by providing a new prefix ( host name of box 2 ) and update RCUSchema.properties file on box 2 with this information.
6. Configure products on box 2.
7. Repeat steps 5 to 6 for subsequent boxes in the environment.
8. Configure OHS Web Server on Box 1.
I've experimented with this and got Foundation successfully clustered to a 2nd EPM instance. I have a few more scenarios to test and then I'll write a separate blog entry about it. Otherwise I won't be able to keep up with the comments in this RCU blog entry, as the clustering topic is of interest and the Install/Config documentation could use some clarifications.
DeleteThanks Nani & Dave,
DeleteI also installed a distributed environment with 5 VM and also stuck in the same scenario.
I followed Nani suggestions to create RCU for each VM and it works.
Just two little tiny things to mention for detail:
1.If you have already use EPM system configtool for the distributed VMs, it's better delete all configurations under user_projects; I tried just delete WebLogic domain(EPMSystem), or delete the instance(epmsystem1) I always get error message when I try to start the services.
2.I still need to copy config folder under WebLogic domain(EPMSystem) in the first VM and redeploy to Application Server again in each VM.
did u install FDMEE ? do we need create new RCU schemas with ODI Master and Work Repo selected
DeleteSo when we configure FDMEE do we need to use the schema which we created using RCU or we need to give new schema when we configure for FDMEE ?
Hi Nani, no you do not use RCU to create the ODI Master and Work repositories. You let the FDMEE Configure step handle that. You -do- use the RCU utility on the FDMEE host server, though. See my follow-up post on this topic:
Deletehttp://www.epmonprem.pro/2020/02/epm-112-distributed-environment-yup.html
thank you
Deleteone more question does epm 11.2 support 19c Oracle DB ?
ReplyDeleteHi Nani, as of the last update, it has not been tested so no it is not officially certified. See http://www.epmonprem.pro/2019/12/thoughts-on-epm-112-certification-matrix.html and bookmark the link to Oracle's URL where you may download the 11.2.0.0 certification matrix.
ReplyDeleteUpdate/correction. Oracle 19c -is- certified for EPM 11.1.2.4 and higher, all the way through EPM 11.2.1.3 as of this writing (Nov 2020).
ReplyDeleteHi Dave/Nani,
ReplyDeleteQuestion to you guys. Thanks in advance.
I will be using a distributed environment to install 11.2.3.000 with the SQL server 2016 as RDBMS and I have four nodes.
1) Foundation and CM
2) FDMEE and FR ( though FR is in web but i think it's allowing to create schema for RA and can be used for configuration.
3) HFM 1
4) HFM 2
so the questions are as follows.
a) I will be creating 4 RCU schema for each and every box BUT do i need to create 4 Daatabase in SQL or ONLY ONE and can use that in four nodes while creating the 4 different schema?
b) I have 2 HFM box and did clustering in 11.2.4.2 so the process will be same while creating the cluster or any different approach?
c) I can use COPYAPP utility as well as IMPORT application option to migrate HFM application from 11,1,2,4 to 11.2.3 and then upgrade the application. Am i right here?
4) I can directly Install and Configure version 11.2.3.000 or I have to install 11.2.0 and then a inplace migration to 1.2.3?
5) Any link to migrate HFM and FDMEE artifacts from 11.1.2.4( HFM - 11.1.2.4.207 and FDMEE - 11.1.2.4.210) to 11.2.3.000 please
Thanks,
Sujoy
Hi Sujoy,
ReplyDeleteI think I may have anewered your primary question in LinkedIn chat already?
(a) For just 4 nodes you can have 1 SQL DB. Unique RCU "prefix" for each node.
(b) Same process as usual.
(c) Migrate the LCM HFM "profile" first, and then the snapshot. I haven't tested the copyapp utility yet.
(4) Until Oracle fixes the ODI bug, I would be inclined to go to 11.2.2.0 first and then do an in-place upgrade to 11.2.3.0.
(5) Now we're getting into "free consulting" territory. :)
Hi Dave,
ReplyDeleteI am trying to Configure RCU for Hyperion 11.2.2 for Oracle 19c on Windows , it is failing at Pre Req validation step with message Database is not supported , Any way to bypass the message and proceed with RCU creation.
Hi Ravi, regrets for not replying earlier as your comment was stuck in moderation.
DeleteOracle 19c is certified for 11.2.2.x, so I'm unsure why it is throwing this message at you. You'll want to verify you're using the RCU that came with 11.2.2 and not the one from 11.1.2. If you see a "Continue" button in that RCU screen, it means you may ignore the message and proceed.
At this time, the only database configuration I'm aware of that is not supported by RCU is hitting MS SQL Server that is configured for SSL.
If you don't see a "Continue" option in RCU, you'll likely need to open an SR with Oracle. All of my tests have been hitting SQL Server.
Best of luck!
Hi Dave, if mthe configuration step fails is it OK by renamming x:\Oracle\Middleware\user_projects\domains\EPMSystem\Config or is it necessary to do some deeper clenasing? thanks
ReplyDeleteI've had the best success following these steps when the initial configuration fails in EPM 11.2.x. This is drastic, but it works.
ReplyDelete1. Completely blow away domains\EPMSystem. The whole thing.
2. Delete and recreate empty databases/schemas for Foundation and RCU.
3. Delete the "Hyperion Solutions" registry hive if you're running MS Windows.
4. Re-run RCU.
5. Re-edit rcuschema.propeties.
6. Attempt your configtool steps again.
I might have forgotten a step... this issue is a royal pain.