Production Down

Fridays are usually quiet around here, and this past Friday started out quiet too. However, at 10:12am my monitoring software informed me that my production FSCM PIA was down.
Immediate action required!

While trying to stay calm and before my phone started ringing furiously, I RDPed to my web server.

I went through these troubleshooting steps:

Troubleshooting

1/ Is the PIA service still running? – Check its still running according to Windows services

2/ What errors do I have in the logs?
Navigated to <PS_HOME>\webserv\PFSCM\servers\PIA\logs and opened the PFSCM.log file. Where I found the following entry:

####<Nov 4, 2016 10:12:13 AM MDT> <Error> <HTTP> <MY-SERVER> <PIA> <[ACTIVE] ExecuteThread: '18' for queue: 
'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1478275933163> <BEA-101017> 
<[ServletContext@1574297599[app:peoplesoft module:/ path:null spec-version:3.0]] Root cause of ServletException.
java.lang.OutOfMemoryError: Java heap space

Workaround

I quickly restarted the PIA service which resolved the immediate issue. The PIA came back up and our FSCM application was available again.

Solution

The real question is if I should increase the Java memory allocation for my PIA. This is done by editing the setEnv.cmd file and updating the following value:

SET JAVA_OPTIONS_WIN=-server -Xms1024m -Xmx1024m

Currently we have 1GB allocated to the Java VM for the PIA. I could increase this value, and then uninstall and reinstall the PIA service to make this take effect. On the other hand, the PIA has not crashed since it was installed, so at the moment I’ll not change it. Instead I think I’ll schedule a restart of the Windows server as it has not been rebooted for over 2 months.

If the out of memory condition happens again I’ll probably increase the memory allocation.