Report information
The Basics
Id:
41707
Status:
resolved
Priority:
Medium/Medium
Queue:

People
Owner:
Nobody in particular
Requestors:
Cc:
AdminCc:

BugTracker
Version Fixed:
(no value)
Version Found:
(no value)
Versions Affected:
(no value)
Versions Planned:
(no value)
Priority:
(no value)
Severity:
(no value)
CVSS Score:
(no value)
CVE ID:
(no value)
Component:
BIND Server
Area:
bug

Dates
Created:Tue, 16 Feb 2016 08:19:55 -0500
Updated:Fri, 07 Jul 2017 19:35:37 -0400
Closed:Wed, 17 Feb 2016 13:22:59 -0500



This bug tracker is no longer active.

Please go to our Gitlab to submit issues (both feature requests and bug reports) for active projects maintained by Internet Systems Consortium (ISC).

Due to security and confidentiality requirements, full access is limited to the primary maintainers.

Subject: Recent change "improve performance of RBT (#41165)" magically broke dyndb
Date: Tue, 16 Feb 2016 14:19:45 +0100
To: bind-bugs@isc.org
From: "Petr Spacek" <pspacek@redhat.com>
Hello, I just found out that dyndb tests (git/bin/tests/system/dyndb) are failing on latest master. git bisect pointed me to following change: commit 5d79b60fc5e4dad4f04da39570517d20a2425f8b Author: Mukund Sivaraman <muks@isc.org> Date: Wed Dec 9 19:07:20 2015 +0530 Improve performance of RBT (#41165) I do not see an obvious reason what would be wrong with the test as the test logic itself did not change. According to named.run, some queries from clients are being refused: client 10.53.0.1#52201 (10.0.53.10.in-addr.arpa): query failed (REFUSED) for 10.0.53.10.in-addr.arpa/IN/PTR at query.c:6770 Zones are loaded in this order: loading DynDB instance 'sample' driver '../driver/lib/sample.so' param: 'ipv4.example.nil.' param: 'in-addr.arpa.' number of params: 2 zone ipv4.example.nil/IN: loaded serial 0 zone in-addr.arpa/IN: loaded serial 0 loading DynDB instance 'sample2' driver '../driver/lib/sample.so' param: 'ipv6.example.nil.' param: '8.b.d.0.1.0.0.2.ip6.arpa.' number of params: 2 zone ipv6.example.nil/IN: loaded serial 0 zone 8.b.d.0.1.0.0.2.ip6.arpa/IN: loaded serial 0 none:100: 'max-cache-size 90%' - setting to 14163MB (out of 15737MB) Interestingly, queries for zone ipv4.example.nil. (the first loaded zone) work, but queries for any other zones fail. Further digging in gdb pointed me to following piece of code: #0 query_getzonedb (client=0x7fffec042e80, name=0x7ffff7e93010, qtype=dns_rdatatype_soa, options=0, zonep=0x7ffff4ebd568, dbp=0x7ffff4ebe278, versionp=0x7ffff4ebdd78) at query.c:911 > result = dns_zt_find(client->view->zonetable, name, ztoptions, NULL, &zone); Variables *before* the function call look like this (please note that I'm using custom pretty printers for RBT and dns_name_t). (gdb) p *name $8 = 10.0.53.10.in-addr.arpa. (DNS_NAMEATTR_ABSOLUTE) (gdb) p ztoptions $9 = 0 (gdb) p *client->view->zonetable->table $10 = RBT node count = 7 . (black; node 0x7ffff7f91710; data 0x0) ++ BEG down from . example.nil (black; node 0x7ffff7fa12e0; data 0x0) ++ BEG down from example.nil ipv4 (black; node 0x7ffff7e2af90; data 0x7fffec289b00) NULL ipv6 (RED from ipv4; node 0x7ffff7f917f0; data 0x7fffec294a30) NULL NULL -- END down from example.nil arpa (RED from example.nil; node 0x7ffff7f918d0; data 0x0) ++ BEG down from arpa in-addr (black; node 0x7ffff7fa1268; data 0x7fffec2935f0) NULL 8.b.d.0.1.0.0.2.ip6 (RED from in-addr; node 0x7ffff7f98f88; data 0x7fffec295a00) NULL NULL -- END down from arpa NULL NULL NULL -- END down from . NULL NULL Apparently the zone is loaded and present in the zone table: (gdb) print ((dns_zone_t *)0x7fffec2935f0)->origin $14 = in-addr.arpa. (DNS_NAMEATTR_ABSOLUTE | DNS_NAMEATTR_DYNAMIC) (gdb) print *((dns_zone_t *)0x7fffec2935f0)->db $16 = {magic = 1145983812, impmagic = 1397572674, methods = 0x7ffff35b24e0 <sampledb_methods>, attributes = 0, rdclass = dns_rdataclass_in, origin = in-addr.arpa. (DNS_NAMEATTR_READONLY | DNS_NAMEATTR_ABSOLUTE | DNS_NAMEATTR_DYNAMIC | DNS_NAMEATTR_DYNOFFSETS), ondest = {magic = 1147491188, events = { head = 0x0, tail = 0x0}}, mctx = 0x9b4390} For zone reason, dns_rbt_findname() call from #0 dns_zt_find (zt=0x7ffff7e0b010, name=0x7ffff7e93010, options=0, foundname=0x0, zonep=0x7ffff4ebd4c8) at zt.c:172 returns (gdb) p result $19 = ISC_R_NOTFOUND Apparently this comes from underlying dns_rbt_findnode() call from #0 dns_rbt_findname (rbt=0x7ffff7f7f250, name=0x7ffff7e93010, options=0, foundname=0x0, data=0x7ffff4ebd470) at rbt.c:1974 (gdb) p options $23 = 0 (gdb) n (gdb) p result $24 = ISC_R_NOTFOUND Again, I'm able to see in the debugger that in-addr.arpa. is in the RBT as shown above. I have no idea what is happening, but it does not look like a problem with dyndb driver itself. Could you have a look? Thank you! -- Petr Spacek @ Red Hat
CC:
Subject: Re: [ISC-Bugs #41707] Recent change "improve performance of RBT (#41165)" magically broke dyndb
Date: Tue, 16 Feb 2016 18:01:51 +0000
To: "Petr Spacek via RT" <bind9-bugs@isc.org>
From: "Evan Hunt" <each@isc.org>
> I just found out that dyndb tests (git/bin/tests/system/dyndb) are failing on > latest master. git bisect pointed me to following change: Yep, thanks, we just discovered the same problem here last week. It's the same class of problem we had back while we were developing it: inconsistent namespace resolution for global variables referenced inside the dyndb driver. It all works fine on ubuntu, but it's failing on your system (which I guess is redhat?), and also on BSD. We fixed it in the original dyndb work by passing a pointer to named's hash context (isc_hashctx) to the dyndb driver, so that hashes would be consistent. But the new hash function introduced a while ago deprecates the hash context, and so hashes started coming out differently inside and outside of the driver, causing RBT lookups to fail. A fix is attached, can you try it? I'm also looking into an intermittent reference leak that seems to happen sometimes when a dyndb driver is reloaded. That one's making slower progress.

Message body is not shown because sender requested not to inline it.

Subject: Re: [ISC-Bugs #41707] Recent change "improve performance of RBT (#41165)" magically broke dyndb
Date: Wed, 17 Feb 2016 14:28:20 +0100
To: bind9-bugs@isc.org
From: "Petr Spacek" <pspacek@redhat.com>
On 16.2.2016 19:01, Evan Hunt via RT wrote: >> I just found out that dyndb tests (git/bin/tests/system/dyndb) are failing on >> latest master. git bisect pointed me to following change: > > Yep, thanks, we just discovered the same problem here last week. > > It's the same class of problem we had back while we were developing > it: inconsistent namespace resolution for global variables referenced > inside the dyndb driver. It all works fine on ubuntu, but it's failing > on your system (which I guess is redhat?), and also on BSD. > > We fixed it in the original dyndb work by passing a pointer to named's > hash context (isc_hashctx) to the dyndb driver, so that hashes would be > consistent. But the new hash function introduced a while ago deprecates > the hash context, and so hashes started coming out differently inside and > outside of the driver, causing RBT lookups to fail. A fix is attached, > can you try it? It works for me on Fedora 23. Thank you for your help! > I'm also looking into an intermittent reference leak that seems to > happen sometimes when a dyndb driver is reloaded. That one's making > slower progress. How do you detect the leak? I can try and have a look, too. -- Petr Spacek @ Red Hat
CC:
Subject: Re: [ISC-Bugs #41707] Recent change "improve performance of RBT (#41165)" magically broke dyndb
Date: Wed, 17 Feb 2016 16:10:10 +0000
To: "Petr Spacek via RT" <bind9-bugs@isc.org>
From: "Evan Hunt" <each@isc.org>
On Wed, Feb 17, 2016 at 01:28:28PM +0000, Petr Spacek via RT wrote: > > I'm also looking into an intermittent reference leak that seems to > > happen sometimes when a dyndb driver is reloaded. That one's making > > slower progress. > > How do you detect the leak? I can try and have a look, too. An assertion failure when cleaning up memory contexts. But I wonder now if it was something I'd done wrong when building, because I haven't been able to get it to happen again this week. I have jury duty and haven't been able to focus much time on it. I'll let you know what I find out when I'm back. Meantime I'm resolving this ticket, thankis.
Subject: Re: [ISC-Bugs #41707] Recent change "improve performance of RBT (#41165)" magically broke dyndb
Date: Wed, 17 Feb 2016 17:28:19 +0100
To: bind9-bugs@isc.org
From: "Petr Spacek" <pspacek@redhat.com>
On 17.2.2016 17:10, Evan Hunt via RT wrote: > On Wed, Feb 17, 2016 at 01:28:28PM +0000, Petr Spacek via RT wrote: >>> I'm also looking into an intermittent reference leak that seems to >>> happen sometimes when a dyndb driver is reloaded. That one's making >>> slower progress. >> >> How do you detect the leak? I can try and have a look, too. > > An assertion failure when cleaning up memory contexts. But I wonder > now if it was something I'd done wrong when building, because I haven't > been able to get it to happen again this week. Okay, I will watch out for this kind of failure. > I have jury duty and haven't been able to focus much time on it. I'll let > you know what I find out when I'm back. Meantime I'm resolving this > ticket, thankis. BTW I do not see the patch in master branch yet. Do not forget to commit it :-) Have a nice day. -- Petr Spacek @ Red Hat