Hello,

I just found out that dyndb tests (git/bin/tests/system/dyndb) are failing on
latest master. git bisect pointed me to following change:

commit 5d79b60fc5e4dad4f04da39570517d20a2425f8b
Author: Mukund Sivaraman <muks@isc.org>
Date:   Wed Dec 9 19:07:20 2015 +0530
    Improve performance of RBT (#41165)

I do not see an obvious reason what would be wrong with the test as the test
logic itself did not change.


According to named.run, some queries from clients are being refused:
client 10.53.0.1#52201 (10.0.53.10.in-addr.arpa): query failed (REFUSED) for
10.0.53.10.in-addr.arpa/IN/PTR at query.c:6770


Zones are loaded in this order:
loading DynDB instance 'sample' driver '../driver/lib/sample.so'
param: 'ipv4.example.nil.'
param: 'in-addr.arpa.'
number of params: 2
zone ipv4.example.nil/IN: loaded serial 0
zone in-addr.arpa/IN: loaded serial 0
loading DynDB instance 'sample2' driver '../driver/lib/sample.so'
param: 'ipv6.example.nil.'
param: '8.b.d.0.1.0.0.2.ip6.arpa.'
number of params: 2
zone ipv6.example.nil/IN: loaded serial 0
zone 8.b.d.0.1.0.0.2.ip6.arpa/IN: loaded serial 0
none:100: 'max-cache-size 90%' - setting to 14163MB (out of 15737MB)

Interestingly, queries for zone ipv4.example.nil. (the first loaded zone)
work, but queries for any other zones fail.


Further digging in gdb pointed me to following piece of code:

#0  query_getzonedb (client=0x7fffec042e80, name=0x7ffff7e93010,
qtype=dns_rdatatype_soa, options=0, zonep=0x7ffff4ebd568, dbp=0x7ffff4ebe278,
versionp=0x7ffff4ebdd78)
    at query.c:911

> result = dns_zt_find(client->view->zonetable, name, ztoptions, NULL, &zone);

Variables *before* the function call look like this (please note that I'm
using custom pretty printers for RBT and dns_name_t).

(gdb) p *name
$8 = 10.0.53.10.in-addr.arpa. (DNS_NAMEATTR_ABSOLUTE)

(gdb) p ztoptions
$9 = 0

(gdb) p *client->view->zonetable->table
$10 = RBT node count = 7
. (black; node 0x7ffff7f91710; data 0x0)
    ++ BEG down from .
    example.nil (black; node 0x7ffff7fa12e0; data 0x0)
        ++ BEG down from example.nil
        ipv4 (black; node 0x7ffff7e2af90; data 0x7fffec289b00)
            NULL
            ipv6 (RED from ipv4; node 0x7ffff7f917f0; data 0x7fffec294a30)
                NULL
                NULL
        -- END down from example.nil
        arpa (RED from example.nil; node 0x7ffff7f918d0; data 0x0)
            ++ BEG down from arpa
            in-addr (black; node 0x7ffff7fa1268; data 0x7fffec2935f0)
                NULL
                8.b.d.0.1.0.0.2.ip6 (RED from in-addr; node 0x7ffff7f98f88;
data 0x7fffec295a00)
                    NULL
                    NULL
            -- END down from arpa
            NULL
            NULL
        NULL
    -- END down from .
    NULL
    NULL


Apparently the zone is loaded and present in the zone table:
(gdb) print ((dns_zone_t *)0x7fffec2935f0)->origin
$14 = in-addr.arpa. (DNS_NAMEATTR_ABSOLUTE | DNS_NAMEATTR_DYNAMIC)

(gdb) print *((dns_zone_t *)0x7fffec2935f0)->db
$16 = {magic = 1145983812, impmagic = 1397572674, methods = 0x7ffff35b24e0
<sampledb_methods>, attributes = 0, rdclass = dns_rdataclass_in,
  origin = in-addr.arpa. (DNS_NAMEATTR_READONLY | DNS_NAMEATTR_ABSOLUTE |
DNS_NAMEATTR_DYNAMIC | DNS_NAMEATTR_DYNOFFSETS), ondest = {magic = 1147491188,
events = {
      head = 0x0, tail = 0x0}}, mctx = 0x9b4390}


For zone reason, dns_rbt_findname() call from
#0  dns_zt_find (zt=0x7ffff7e0b010, name=0x7ffff7e93010, options=0,
foundname=0x0, zonep=0x7ffff4ebd4c8) at zt.c:172

returns
(gdb) p	result
$19 = ISC_R_NOTFOUND


Apparently this comes from underlying dns_rbt_findnode() call from
#0  dns_rbt_findname (rbt=0x7ffff7f7f250, name=0x7ffff7e93010, options=0,
foundname=0x0, data=0x7ffff4ebd470) at rbt.c:1974
(gdb) p	options
$23 = 0
(gdb) n
(gdb) p	result
$24 = ISC_R_NOTFOUND

Again, I'm able to see in the debugger that in-addr.arpa. is in the RBT as
shown above. I have no idea what is happening, but it does not look like a
problem with dyndb driver itself.

Could you have a look? Thank you!

-- 
Petr Spacek  @  Red Hat