Last night a blog post saved my life! (Or, Hey there’s a really annoying bug in Windows Server 2003 SMTP Service)
All comments aside about how Server 2003 is EOS in about 60 days, some of us still live in a world where those are in use. :-(
In the process of disabling RC4 ciphers on our Exchange servers, we had an IIS 6.0 SMTP service experience begin failing to deliver mail to Exchange. The IIS server was forwarding mail to Exchange and using TLS. The immediate presumption was that without RC4 there were no common ciphers with the Exchange server anymore, so we went about installing KB948693 to add AES cipher support to 2003, at least getting to the point where they could share a relatively modern set. This alone did not fix the issue and looking in Wireshark, there was some odd tailing data when the 2003 was trying to use a cipher other than the RC4 ones which were previously working. This shows up simply as “Ignored Unknown Record” when sniffing the attempted AES use, which by itself isn’t very telling, but appears in stark contrast where the conversation is much cleaner/understood when using RC4.
There’s an amazingly thorough write up of this on https://lbr.id.lv/#Windows_SMTP_bug_breaks_3DES_and_AES_CBC
Essentially the problem occurs when using block ciphers as opposed to stream ciphers. There’s an old update, KB957047, which mentions issues after setting FIPSAlgorithmPolicy =1, which we hadn’t done, but apparently has the same net effect as it forces a switch to 3DES (block cipher). Puzzling that the version of smtpsvc.dll that we had was newer than the one listed in the 957047, so at first glance, would have thought that we wouldn’t need it. Tried installing it anyway and got warning messages about failed missing dependencies in a package from KB976323, so after re-applying 976323 and then applying 957047, the Ignored Unknown Record disappears from the trace and mail flows normally . Neither 957047 or 976323 requires a reboot but do restart IIS. Seems like good practice to re-apply 976323 yet again as well.
So thanks lbr.id.lv as I’d unwittingly already been through MS Support for this issue before and not realized what side the issue was on.