Subject: Crash from Debian 9.9.5.dfsg-9+deb8u6 (likely PoD scenario) MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) Content-Disposition: inline X-RT-Interface: Web Message-ID: <rt-4.2.8-40397-1480961658-1321.0-0-0@isc.org> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: binary X-RT-Original-Encoding: utf-8 X-RT-Encrypt: 0 X-RT-Sign: 0 Content-Length: 5406 This is the crash reported to Ray - with the further detail that was requested. No core dumps, sadly. ----- From Robin Stevens <robin.stevens@it.ox.ac.uk>: The University of Oxford operates four central DNS resolvers, running Debian 8.6 (jessie, aka stable) and Debian's most recent bind9 package, version 9.9.5.dfsg-9+deb8u6 (incorporating patches for CVE-2016-8864). These servers provide service internally to c.30,000 hosts in the University, and are not open to queries from the outside world. Around 12:00 UTC on Tuesday 29 November, BIND crashed on all four servers within a few minutes of each other. BIND on one server crashed again around 13:40 the same day. At the time of writing we have seen no further crashes. Unfortunately core dumps were not enabled on the servers at the time. syslog on three of the servers recorded identical assertion failures as per the attached log; a configuration error on the fourth unfortunately prevented any similar error being recorded. We do log DNS queries on the servers but have identified nothing obviously unusual immediately prior to the crashes. Our current hypothesis is that a particular request from one of our clients, or subsequent response from an external server, triggered the first crash. Our use of anycast then meant that the query was then repeated to each of the other servers, causing each to fail in turn. We appreciate that this isn't a huge amount of information to go on, and we are only likely to be able to provide further information if the problem strikes again. Moreover, it's possible that the problem is specific to the Debian package and not to ISC's codebase. We have already had a couple of comments suggesting we use the stock ISC code rather than Debian's, and are considering whether that is appropriate in our environment. If there is anything else you can suggest that might yield useful information then please let me know, and I will pass it to our sysadmins. Kind regards, Robin -- ------------ Dr Robin Stevens Information Security Architect ------------- IT Services, University of Oxford, 16 Wellington Square, Oxford OX1 2HY, UK Tel: +44 1865 273212 ----------------------------- https://www.it.ox.ac.uk/ Fax: +44 1865 273275 ------------------------ https://www.infosec.ox.ac.uk/ ---- Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.086 general: critical: dispatch.c:3700: REQUIRE((disp->attributes & 0x00000020U) != 0) failed, back trace Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.086 general: critical: #0 0x7f6e6efe9a20 in ?? Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.086 general: critical: #1 0x7f6e6d1c68ea in ?? Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.086 general: critical: #2 0x7f6e6e7faa0a in ?? Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.086 general: critical: #3 0x7f6e6efddded in ?? Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.086 general: critical: #4 0x7f6e6d1e8d5b in ?? Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.087 general: critical: #5 0x7f6e6cb990a4 in ?? Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.087 general: critical: #6 0x7f6e6c56762d in ?? Nov 29 12:03:04 redness named[112611]: 29-Nov-2016 12:03:04.087 general: critical: exiting (due to assertion failure) Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: dispatch.c:3700: REQUIRE((disp->attributes & 0x00000020U) != 0) failed, back trace Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: #0 0x7f0240476a20 in ?? Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: #1 0x7f023e6538ea in ?? Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: #2 0x7f023fc87a0a in ?? Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: #3 0x7f024046aded in ?? Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: #4 0x7f023e675d5b in ?? Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: #5 0x7f023e0260a4 in ?? Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: #6 0x7f023d9f462d in ?? Nov 29 12:01:45 rudeness named[81730]: 29-Nov-2016 12:01:45.117 general: critical: exiting (due to assertion failure) Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: dispatch.c:3700: REQUIRE((disp->attributes & 0x00000020U) != 0) failed, back trace Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: #0 0x7fe6ff0fda20 in ?? Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: #1 0x7fe6fd2da8ea in ?? Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: #2 0x7fe6fe90ea0a in ?? Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: #3 0x7fe6ff0f1ded in ?? Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: #4 0x7fe6fd2fcd5b in ?? Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: #5 0x7fe6fccad0a4 in ?? Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: #6 0x7fe6fc67b62d in ?? Nov 29 12:01:20 rapidness named[5592]: 29-Nov-2016 12:01:20.246 general: critical: exiting (due to assertion failure)