Bug #46746 for bind9-public: HMAC fatal errors in native pkcs#11 mode on FIPS-mode enabled system

The following bugs was reported by Petr Mensik and Tomas Hozza from RedHat; I scored it 6.5 - CVSS:3.0/AV:N/AC:L/PR:N/UI:R/S:U/C:N/I:N/A:H/E:P/RL:O/RC:C But it affects only a very specific configuration - an FIPS-mode enabled system that fails to provide MD5 and SHA1 functions. ~~~ I am working now on a bug triggered in Red Hat Enterprise Linux, when system is running in FIPS mode. We in Red Hat enable FIPS mode only by configuration change, without rebuilding BIND or OpenSSL. In that cases, some openssl functions are disabled and are returning errors. That are MD5 and HMAC MD5 in bind. $ echo test | openssl md5 Error setting digest md5 140504757397408:error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips:digest.c:251: If PKCS#11 build is used (--with-pkcs11 --enable-pkcs11-native), any error reported from digest functions is fatal. Our bind-pkcs11 variant is built by default with softhsm2 module, linked dynamically to system openssl library. It fails if md5 algorithm is used anywhere from isc_md5_* function. Fail is expected and would be ok, if it were not always fatal error. It is used by default in RNDC key or might be used for dynamic updates. I have found any request authenticated by hmac-md5 will terminate server if such algorithm is accepted in configuration. Even if I do not know the secret, just the key name+algorithm, I am able to crash server on system running in FIPS mode. It might be request from nsupdate or rndc. I found no way to disable MD5 algorithm runtime. Options disable-algorithms and disable-ds-digests are not related to TSIG hmac keys. Is there something I am missing? I found hmac keys always first check matching algorithm, so I were not able to crash server accepting only hmac-sha256 keys by sending hmac-md5 request. Is there other known way to use isc_md5_init? Can it be somehow triggered by not having algorithm hmac-md5 key in configuration? Can be TKEY use to choose MD5 algorithm by the remote party, that cannot be prevented from configuration? It seems to me tkey-gssapi should not be affected. I am seeking configuration options that can prevent remote denial of service. I am aware it is possible to disable MD5 algorithm completely at compile time since 9.10.5. It would not work in our configuration, we would have to ship different package for fips mode. Configure option --enable-openssl-hash will use openssl digest functions directly, is default in current master. In our 9.9 version all openssl errors in digest function are ignored. In later versions it would crash any process if EVP_md5() usage returned error, just like PKCS#11 native variant. It seems clear to me built-in hash functions cannot ever fail, but both external digest providers can fail. I think it should be possible to catch such errors runtime and fail only one request, not the whole process. I made working patch to allow md5 and sha1 digest functions to return failures. It passes all system tests. Because it can be used against running systems in FIPS mode, I do not want it yet as public suggestion I would normally use. I would create bug for it if you want. Patch applies to last master. -- Petr Menšík Software Engineer Red Hat, http://www.redhat.com/ email: pemensik@redhat.com PGP: 65C6C973

---