The Observed Symptom
An upstream system automatically provides ASCII text files for FDMEE to load. The timing of the delivery varies from day to day, so a batch process is kicked off by the Windows Task Scheduler to continually check for the files' presence and process them when discovered.
On an intermittent basis, the FDMEE load doesn't properly complete. The files are detected, the FDMEE load is attempted and fails, and then the files are moved into an archive folder by the consultant's DOS wrapper script.
Root Cause Analysis
The consultant setup the FDMEE automation via a Windows Task Scheduler job, which triggered to run "On Startup". The script loops with a 60 second pause. Once files are detected in the expected folder location, the pre-delivered FDMEE load utility is invoked. The files are moved into an archive folder after this runs.
Unfortunately, the FDMEE server was rebooted via Windows Update. Some time after the reboot, the files were delivered and picked up by the automation, but FDMEE has not yet finished its Oracle WebLogic startup sequence.
Plain English: The FDMEE's load utility couldn't process the file, since FDMEE wasn't online.
FDMEE 220.127.116.11 is one of the services that takes the longest to complete its WebLogic start-up sequence. Depending upon the computing resources available, this can require 3-5 minutes or longer.
When FDMEE is fully initialized and ready to accept connections, the system is listening to TCP port 6550 (this is the default FDMEE port unless someone changed it).
We add 1 line to the top of our FDMEE automation wrapper script:
Next, we create the WaitForFDMEE.ps1 script. Here is the script in full:
# This script loops until FDMEE's port is online.
# If you receive a security policy error about “unsigned” Powershell scripts when
# running this process, open a command prompt and type:
# powershell.exe Set-ExecutionPolicy Unrestricted
# Written on 08/04/2017 by Dave Shay (Datavail)
# Modified on MM/DD/YYYY by Your Name - Briefly list changes
$ErrorActionPreference = "SilentlyContinue"
# Loop forever until FDMEE is online
$socket = new-object System.Net.Sockets.TcpClient("localhost", 6550)
if ($socket -eq $null)
write-host "FDMEE isn't fully initialized yet. Sleeping 20 seconds..."
powershell.exe Start-Sleep -s 20
} until ($socket -ne $null)
write-host "FDMEE is ready to accept connections."
Finally, we copy & paste this line to the command prompt. This prevents a Powershell security error. We only need to issue this command one time.
powershell.exe Set-ExecutionPolicy Unrestricted
And that's it! The FDMEE automation wrapper script now sleeps until it detects that FDMEE is online.