The other day I was trying to deploy a war file on Red Hat Enterprise Linux (RHEL) 5. It could just as easily have been CentOS 5. The system had mod_proxy_ajp installed as defaulted by Red Hat.
My problem: my test code, request.isSecure(), was always returning 'false'. This was very disconcerting, especially since the system was only accessible via https. In development, everything was fine.
I started digging in. What could cause this in production but not in development?
I started to compare httpd.conf, and to back out changes made by the previous sysadmin. No luck. I continued to undo changes until rpm -V httpd said all was well.
Still it did not work.
I then proceeded to look in the conf.d directory. I found the difference: dev was using mod_ssl but production was using mod_nss.
I changed dev to use mod_nss and retested. Well, dev still worked, and (of course) production still did not. I yanked all the files out of conf.d except the proxy_ajp.conf, welcome.conf and nss.conf. Production still was a no-go.
Next, I searched /etc/httpd/** and other httpd directories for files that had been modified.
Bingo! I found that mod_nss was not installed as an rpm via yum from Red Hat. But where was it installed from? The sysadmin had obviously got it on there somehow. Probably it was a source tar ball install. But I did not know which configure options, or even which version.
My first attempt was to backup and install the vendor rpm for mod_nss.
I re-stared apache httpd, but it errored out. It did not like the default OCSP responder directive (NSSOCSPDefaultResponder). We have to use a newer version than the vendor provided. This is needed because DoD root certificates do not have the OCSP extension. Therefore any chain will fail when validated by NSS, unless you default the OSCP queries to http://ocsp.disa.mil.
So I assumed that the previous sysadmin had compiled it on this box, and that he had left the work directory in place. I typed "locate mod_nss". I found several expanded tar balls - for each of which, 'make' had been run.
I then md5sum'ed each of the compiled artifacts I could find on the system. I got a match: he was using mod_nss-1.0.6. I checked the hash on the tar ball against the authority for mod_nss. It matched! This was my first bit of good luck.
Next I had to figure out why 1.0.6 did not work on RHEL5. I got the mod_nss-1.0.3-8.el5.src.rpm for mod_nss from Red Hat. It contained mod_nss-1.0.3 and 7 patches.
Red Hat only patches for two reasons: compatibility and security. Version 1.0.6 was obviously not compatible, since it was not working. I suspected that the patches could not be applied directly to 1.0.6, as they were designed for version 1.0.3.
I compared the 1.0.3 source from Red Hat and the source from the authority: they matched. I tested the patches and found they could not be applied directly to 1.0.6. Therefore I applied them to 1.0.4, then 1.0.5, then 1.0.6. It turns out that 2 of the patches (defer and eccinit) were back ports already included in 1.0.6.
Now I had an rpm for the working version, which includes the vendor's patches. I installed it. Success! Everything worked when I ran my tests.
You gain two benefits if you always use the package manager to install your custom patched version:
The maintenance fees that you already pay to your vendor are not wasted by taking unnecessary risks through bypassing the vendor's procedures; and your next sysadmin can follow the documented changes from the source rpm file.
As a precaution, I also created a faux version rpm to ensure that the next release from Red Hat does not clobber it during a yum update. This way you will be warned when the vendor releases an updated version.