All of my blog posts are generally created out of need, this one is another example. I was helping out a client with a DR test of their JD Edwards and it was all looking great until the automatic start of the JD Edwards weblogic instances did not work. They went from starting to stopped in server manager. What!!! The start procedure has been tested 100 times and is solid as a rock, we use nmstart commands in a startup script. All nice and neat.
on further analysis of the individual JVM logs, I see the following (amongst the errors)
weblogic.home = D:\oracle\Middleware\Oracle_Home\wlserver\server
weblogic.management.server = http://10.38.144.11:7001
in the servername.out file in domainname\servers\servername\logs dir
It’s trying to use an IP address for the weblogic.management.server setting. Tidy darts, I’ll just use windows find to locate where the IP address is specified (surely just an xml file… no, maybe .config, no – maybe .cmd… no… .properties file… nothing say’s windows find). Needless to say windows find is the most useless tool that has every been put on the face of this planet. This is 2012R2 also…
Using the trusty
D:\oracle\Middleware>findstr /S /C:"10.38.144.11" *.* | findstr /V /C:".log" |findstr /V /C:".out"
… I find the following file (of course, not one of the extensions that I’d been expecting… proprietary binary format)
Oracle_Home\user_projects\domains\E1_Apps\servers\AdminServer\data\store\default
\_WLS_ADMINSERVER000000.DAT:loadFactorI thresholdxp?@ ♀ ► ♀t ♦sn
mpt ▬snmp://10.38.144.11:96t ☻t3t ¶t3://10.38.144.11:96t ♣httpst ¶t3://10.38.144
.11:96t ↑cluster-broadcast-securet ¶t3://10.38.144.11:96t ♦ldapt ▬ldap://10.38.1
44.11:96t ◄cluster-broadcastt ¶t3://10.38.144.11:96t ♦httpt ▬http://10.38.144.11
:96t ♣admint ¶t3://10.38.144.11:96t ♦iiopt ▬iiop://10.38.144.11:96t ♣ldapst ¶t3:
//10.38.144.11:96t ♥t3st ¶t3://10.38.144.11:96t ♣iiopst ¶t3://10.38.144.11:96xsq
…
What, this is a persistent filestore under the adminserver which is caching the IP address of the server before DR. This is painful! This is the ONLY location it’s specified anywhere.
Hmm, a bit of googling later leaves me quite sure that I cannot do anything with this in a rush, it looks complex. The one thing that I can do in a rush is to delete the file and restart the machine, see if my scripts work.
So, after taking down the adminServer (it locks the file hard) I backed up the file and removed it.
My server then started perfectly – but of course, the funny thing? Look at what it uses now in the persistent store:
weblogic.home = D:\oracle\Middleware\Oracle_Home\wlserver\server
weblogic.management.server = http://10.38.16.12:7001
weblogic.nmservice.RotationEnabled = true
The IP address of the DR machine!! haha. Monkey see, monkey do.
This must be some sort of bug in the adminserver which is caching the IP address not the hostname of the machine that the server is starting on. I’m considering raising an SR for this, we’ll see.
Remember, if you are using nmstart and changing IP addresses of your server, you probably want to delete the admin server persistent cache.