Report information
The Basics
Id:
45557
Status:
open
Priority:
Medium/Medium
Queue:

People
Owner:
Nobody in particular
Cc:
AdminCc:

BugTracker
Version Fixed:
(no value)
Version Found:
(no value)
Versions Affected:
(no value)
Versions Planned:
(no value)
Priority:
(no value)
Severity:
(no value)
CVSS Score:
(no value)
CVE ID:
(no value)
Component:
(no value)
Area:
(no value)

Dates
Created:Thu, 13 Jul 2017 19:09:29 -0400
Updated:Fri, 28 Jul 2017 12:42:05 -0400
Closed:Not set



This bug tracker is no longer active.

Please go to our Gitlab to submit issues (both feature requests and bug reports) for active projects maintained by Internet Systems Consortium (ISC).

Due to security and confidentiality requirements, full access is limited to the primary maintainers.

Date: Thu, 13 Jul 2017 23:09:23 +0000
Subject: Version 4.1-ESV-R8 - sigabrt with assert in dhcpd process
To: dhcp-bugs@isc.org
From: "Pradeep Ponnuchamy" <doors.pradeep@gmail.com>
Bug Report from www.isc.org: Name: Pradeep Ponnuchamy Email: doors.pradeep@gmail.com Software Version: Version 4.1-ESV-R8 OS: Linux Subject:sigabrt with assert in dhcpd process Bug Detail =========== #0 0xb6f8ae1c in raise () from lib/libc.so.0 (gdb) bt #0 0xb6f8ae1c in raise () from lib/libc.so.0 #1 0xb6f84ff8 in abort () from lib/libc.so.0 #2 0xb6f50b80 in __assert () from lib/libc.so.0 #3 0x000712e0 in sink_down (heap=0x18360b8, i=<optimized out>, elt=0x189cd38) at ../../common/heap.c:177 #4 0x000404f4 in renew_lease6 (pool=<optimized out>, lease=0x189cd38) at ../../server/mdb6.c:1282 #5 0x0003c4dc in reply_process_ia_na (ia=<optimized out>, reply=<optimized out>) at ../../server/dhcpv6.c:1844 #6 lease_to_client (reply_ret=0xbea906dc, packet=0x1898d70, client_id=<optimized out>, server_id=<optimized out>) at ../../server/dhcpv6.c:1326 #7 0x0003f154 in dhcpv6_rebind (packet=0x1898d70, reply=0xbea906dc) at ../../server/dhcpv6.c:4632 #8 build_dhcpv6_reply (reply=0xbea906dc, packet=0x1898d70) at ../../server/dhcpv6.c:5879 #9 0x0003fc10 in dhcpv6 (packet=0x1898d70) at ../../server/dhcpv6.c:5990 #10 0x00058098 in do_packet6 (interface=0x17e57b8, packet=0xbea9075c "6\261\260}", len=78, from_port=<optimized out>, from=0xbeaa0778, was_unicast=isc_boolean_false) at ../../common/options.c:3941 #11 0x0004abcc in got_one_v6 (h=<optimized out>) at ../../common/discover.c:1587 #12 0x000793a4 in omapi_one_dispatch (wo=<optimized out>, t=0xbeaa0b50) at ../../omapip/dispatch.c:539 #13 0x0004ccf0 in dispatch () at ../../common/dispatch.c:95 #14 0x0000fbc4 in main (argc=<optimized out>, argv=<optimized out>) at ../../server/dhcpd.c:874 --- This email was received through isc.org Bug Submission Form
Hello Pradeep: While back traces are often very helpful, in this case it really doesn't give us enough information. The assertion that occurs is detecting an inconsistent state within the lease heap and is something of a self-defense mechanism. The back trace really doesn't shed any light on what might have caused the inconsistency. First, I would recommend that you consider upgrading as you are quite a few releases behind. The current releases are 4.3.6 and 4.1-ESV-R14, and we are releasing 4.3.6 and 4.1-ESV-R15 betas later today. The final releases for those are due 7/31/2017. While you're considering upgrading, any contextual information you can provide such as log files, server config files, lease files, or pcaps would be helpful. Is this something you can reproduce, if so how? Can you describe the circumstances under which you see this and how often it occurs? The more information you can provide, the more likely it is that we will be able to find the issue. Thank you for taking the time to report this issue to us. Regards, Thomas Markwalder ISC Software Engineering On Thu Jul 13 23:09:29 2017, doors.pradeep@gmail.com wrote: > Bug Report from www.isc.org: > > Name: Pradeep Ponnuchamy > Email: doors.pradeep@gmail.com > Software Version: Version 4.1-ESV-R8 > OS: Linux > Subject:sigabrt with assert in dhcpd process > > > Bug Detail > =========== > #0 0xb6f8ae1c in raise () from lib/libc.so.0 > (gdb) bt > #0 0xb6f8ae1c in raise () from lib/libc.so.0 > #1 0xb6f84ff8 in abort () from lib/libc.so.0 > #2 0xb6f50b80 in __assert () from lib/libc.so.0 > #3 0x000712e0 in sink_down (heap=0x18360b8, i=<optimized out>, > elt=0x189cd38) at ../../common/heap.c:177 > #4 0x000404f4 in renew_lease6 (pool=<optimized out>, lease=0x189cd38) > at ../../server/mdb6.c:1282 > #5 0x0003c4dc in reply_process_ia_na (ia=<optimized out>, > reply=<optimized out>) at ../../server/dhcpv6.c:1844 > #6 lease_to_client (reply_ret=0xbea906dc, packet=0x1898d70, > client_id=<optimized out>, server_id=<optimized out>) at > ../../server/dhcpv6.c:1326 > #7 0x0003f154 in dhcpv6_rebind (packet=0x1898d70, reply=0xbea906dc) > at ../../server/dhcpv6.c:4632 > #8 build_dhcpv6_reply (reply=0xbea906dc, packet=0x1898d70) at > ../../server/dhcpv6.c:5879 > #9 0x0003fc10 in dhcpv6 (packet=0x1898d70) at > ../../server/dhcpv6.c:5990 > #10 0x00058098 in do_packet6 (interface=0x17e57b8, packet=0xbea9075c > "6\261\260}", len=78, from_port=<optimized out>, from=0xbeaa0778, > was_unicast=isc_boolean_false) at ../../common/options.c:3941 > #11 0x0004abcc in got_one_v6 (h=<optimized out>) at > ../../common/discover.c:1587 > #12 0x000793a4 in omapi_one_dispatch (wo=<optimized out>, > t=0xbeaa0b50) at ../../omapip/dispatch.c:539 > #13 0x0004ccf0 in dispatch () at ../../common/dispatch.c:95 > #14 0x0000fbc4 in main (argc=<optimized out>, argv=<optimized out>) at > ../../server/dhcpd.c:874 > > --- > This email was received through isc.org Bug Submission Form
From: "Pradeep ponnuchamy" <doors.pradeep@gmail.com>
Date: Fri, 14 Jul 2017 16:51:08 -0500
To: dhcp-confidential@isc.org
Subject: Re: [ISC-Bugs #45557] Version 4.1-ESV-R8 - sigabrt with assert in dhcpd process
Hi Thomas,

Thanks for the response

I'm using dhcp-isc for DHCPv6 server. Any hint on bug fixes related to the below scenario in recent releases will be helpful.

Can u please explain how and when the lease heap memory can go to inconsistent state.How the lease->heap.array data strcuture is built. whats the maximum size of this array.

I don't have the reproduction steps for now, i try to get it. 
Getting continues REBIND messages from the DHCP client cause this assert failure. The DHCP client is not accepting the REPLY messages from server. Also, i see this assert failure only in long run, when DHCP clients keep sending REBIND messages.

From the coredump i see that the heap index is coming as 2. Is this valid index to process the heap index in sink_down fn ?

This is what the information i have for now.  I will keep you updated.

Regards,
Pradeep


On Fri, Jul 14, 2017 at 5:41 AM, Thomas Markwalder via RT <dhcp-confidential@isc.org> wrote:
Hello Pradeep:

While back traces are often very helpful, in this case it really doesn't give us enough information.  The assertion that occurs is detecting an inconsistent state within the lease heap and is something of a self-defense mechanism.  The back trace really doesn't shed any light on what might have caused the inconsistency.

First, I would recommend that you consider upgrading as you are quite a few releases behind.  The current releases are 4.3.6 and 4.1-ESV-R14, and  we are releasing 4.3.6 and 4.1-ESV-R15 betas later today.  The final releases for those are due 7/31/2017.

While you're considering upgrading, any contextual information you can provide such as log files, server config files, lease files, or pcaps would be helpful.  Is this something you can reproduce, if so how?   Can you describe the circumstances under which you see this and how often it occurs?

The more information you can provide, the more likely it is that we will be able to find the issue.  Thank you for taking the time to report this issue to us.


Regards,

Thomas Markwalder
ISC Software Engineering



On Thu Jul 13 23:09:29 2017, doors.pradeep@gmail.com wrote:
> Bug Report from www.isc.org:
>
> Name: Pradeep Ponnuchamy
> Email: doors.pradeep@gmail.com
> Software Version: Version 4.1-ESV-R8
> OS: Linux
> Subject:sigabrt with assert in dhcpd process
>
>
> Bug Detail
> ===========
> #0  0xb6f8ae1c in raise () from lib/libc.so.0
> (gdb) bt
> #0  0xb6f8ae1c in raise () from lib/libc.so.0
> #1  0xb6f84ff8 in abort () from lib/libc.so.0
> #2  0xb6f50b80 in __assert () from lib/libc.so.0
> #3  0x000712e0 in sink_down (heap=0x18360b8, i=<optimized out>,
> elt=0x189cd38) at ../../common/heap.c:177
> #4  0x000404f4 in renew_lease6 (pool=<optimized out>, lease=0x189cd38)
> at ../../server/mdb6.c:1282
> #5  0x0003c4dc in reply_process_ia_na (ia=<optimized out>,
> reply=<optimized out>) at ../../server/dhcpv6.c:1844
> #6  lease_to_client (reply_ret=0xbea906dc, packet=0x1898d70,
> client_id=<optimized out>, server_id=<optimized out>) at
> ../../server/dhcpv6.c:1326
> #7  0x0003f154 in dhcpv6_rebind (packet=0x1898d70, reply=0xbea906dc)
> at ../../server/dhcpv6.c:4632
> #8  build_dhcpv6_reply (reply=0xbea906dc, packet=0x1898d70) at
> ../../server/dhcpv6.c:5879
> #9  0x0003fc10 in dhcpv6 (packet=0x1898d70) at
> ../../server/dhcpv6.c:5990
>  #10 0x00058098 in do_packet6 (interface=0x17e57b8, packet=0xbea9075c
> "6\261\260}", len=78, from_port=<optimized out>, from=0xbeaa0778,
>     was_unicast=isc_boolean_false) at ../../common/options.c:3941
> #11 0x0004abcc in got_one_v6 (h=<optimized out>) at
> ../../common/discover.c:1587
> #12 0x000793a4 in omapi_one_dispatch (wo=<optimized out>,
> t=0xbeaa0b50) at ../../omapip/dispatch.c:539
> #13 0x0004ccf0 in dispatch () at ../../common/dispatch.c:95
> #14 0x0000fbc4 in main (argc=<optimized out>, argv=<optimized out>) at
> ../../server/dhcpd.c:874
>
> ---
> This email was received through isc.org Bug Submission Form



From: "Pradeep ponnuchamy" <doors.pradeep@gmail.com>
Date: Fri, 28 Jul 2017 11:41:55 -0500
To: dhcp-confidential@isc.org
Subject: Re: [ISC-Bugs #45557] Version 4.1-ESV-R8 - sigabrt with assert in dhcpd process
Hi Thomas,
Recently i found this problem is due to the lease file corruption and there is a difference with prefix delegated lease time (14 days), isc-dhcp configured with 30 days lease.
At some point the lease tile is getting corrupted, example:

DHCPv6 server lease table:
Host Name        IP Address                          MAC Address       State      Type   Expiration
<none>           2001:1890:xxxx:fa30::45/64          00:18:7d:xx:xx:xx active     IA_NA  29:23:22:37     2592000s     604800s
<none>           2001:1890:xxxx:fa30::48/64          78:88:6d:xx:xx:xx active     IA_NA  17387:20:50:39    1209600s    1209600s
<none>           2001:1890:xxxx:fa30::49/64          20:47:47:xx:xx:xx active     IA_NA  29:23:25:30     2592000s     604800s

I see that the assert condition based on the below scenario.

(gdb) p *(struct iasubopt *)pool->active_timeouts->array[1]
$16 = {refcnt = 3, addr = {__in6_u = {__u6_addr8 = " \001\030\220\022\250\372\060\000\000\000\000\000\000\000I", __u6_addr16 = {288, 36888, 43026, 12538, 0, 0, 0, 18688}, __u6_addr32 = {2417492256, 
        xxxxxxxxx, 0, 1224736768}}}, plen = 0 '\000', state = 2 '\002', scope = 0x0, hard_lifetime_end_time = 1503763415, soft_lifetime_end_time = 0, prefer = 604800, valid = 2592000, ia = 0x5d7dc0, 
  ipv6_pool = 0x5f6488, heap_index = 1}
(gdb) p *(struct iasubopt *)pool->active_timeouts->array[2]
$17 = {refcnt = 4, addr = {__in6_u = {__u6_addr8 = " \001\030\220\022\250\372\060\000\000\000\000\000\000\000H", __u6_addr16 = {288, 36888, 43026, 12538, 0, 0, 0, 18432}, __u6_addr32 = {2417492256, 
       xxxxxxxxx, 0, 1207959552}}}, plen = 0 '\000', state = 2 '\002', scope = 0x0, hard_lifetime_end_time = 1502385681, soft_lifetime_end_time = 0, prefer = 1209600, valid = 1209600, ia = 0x5f5d38, 
  ipv6_pool = 0x5f6488, heap_index = 2}


these problem can be resolved in latest code ? because no assert or heap.c present in latest code.


Thanks,
Pradeep

On Fri, Jul 14, 2017 at 4:51 PM, Pradeep Ponnuchamy via RT <dhcp-confidential@isc.org> wrote:
Hi Thomas,

Thanks for the response

I'm using dhcp-isc for DHCPv6 server. Any hint on bug fixes related to the
below scenario in recent releases will be helpful.

Can u please explain how and when the lease heap memory can go to
inconsistent state.How the lease->heap.array data strcuture is built. whats
the maximum size of this array.

I don't have the reproduction steps for now, i try to get it.
Getting continues REBIND messages from the DHCP client cause this assert
failure. The DHCP client is not accepting the REPLY messages from server.
Also, i see this assert failure only in long run, when DHCP clients keep
sending REBIND messages.

From the coredump i see that the heap index is coming as 2. Is this valid
index to process the heap index in sink_down fn ?

This is what the information i have for now.  I will keep you updated.

Regards,
Pradeep


On Fri, Jul 14, 2017 at 5:41 AM, Thomas Markwalder via RT <
dhcp-confidential@isc.org> wrote:

> Hello Pradeep:
>
> While back traces are often very helpful, in this case it really doesn't
> give us enough information.  The assertion that occurs is detecting an
> inconsistent state within the lease heap and is something of a self-defense
> mechanism.  The back trace really doesn't shed any light on what might have
> caused the inconsistency.
>
> First, I would recommend that you consider upgrading as you are quite a
> few releases behind.  The current releases are 4.3.6 and 4.1-ESV-R14, and
> we are releasing 4.3.6 and 4.1-ESV-R15 betas later today.  The final
> releases for those are due 7/31/2017.
>
> While you're considering upgrading, any contextual information you can
> provide such as log files, server config files, lease files, or pcaps would
> be helpful.  Is this something you can reproduce, if so how?   Can you
> describe the circumstances under which you see this and how often it occurs?
>
> The more information you can provide, the more likely it is that we will
> be able to find the issue.  Thank you for taking the time to report this
> issue to us.
>
>
> Regards,
>
> Thomas Markwalder
> ISC Software Engineering
>
>
>
> On Thu Jul 13 23:09:29 2017, doors.pradeep@gmail.com wrote:
> > Bug Report from www.isc.org:
> >
> > Name: Pradeep Ponnuchamy
> > Email: doors.pradeep@gmail.com
> > Software Version: Version 4.1-ESV-R8
> > OS: Linux
> > Subject:sigabrt with assert in dhcpd process
> >
> >
> > Bug Detail
> > ===========
> > #0  0xb6f8ae1c in raise () from lib/libc.so.0
> > (gdb) bt
> > #0  0xb6f8ae1c in raise () from lib/libc.so.0
> > #1  0xb6f84ff8 in abort () from lib/libc.so.0
> > #2  0xb6f50b80 in __assert () from lib/libc.so.0
> > #3  0x000712e0 in sink_down (heap=0x18360b8, i=<optimized out>,
> > elt=0x189cd38) at ../../common/heap.c:177
> > #4  0x000404f4 in renew_lease6 (pool=<optimized out>, lease=0x189cd38)
> > at ../../server/mdb6.c:1282
> > #5  0x0003c4dc in reply_process_ia_na (ia=<optimized out>,
> > reply=<optimized out>) at ../../server/dhcpv6.c:1844
> > #6  lease_to_client (reply_ret=0xbea906dc, packet=0x1898d70,
> > client_id=<optimized out>, server_id=<optimized out>) at
> > ../../server/dhcpv6.c:1326
> > #7  0x0003f154 in dhcpv6_rebind (packet=0x1898d70, reply=0xbea906dc)
> > at ../../server/dhcpv6.c:4632
> > #8  build_dhcpv6_reply (reply=0xbea906dc, packet=0x1898d70) at
> > ../../server/dhcpv6.c:5879
> > #9  0x0003fc10 in dhcpv6 (packet=0x1898d70) at
> > ../../server/dhcpv6.c:5990
> >  #10 0x00058098 in do_packet6 (interface=0x17e57b8, packet=0xbea9075c
> > "6\261\260}", len=78, from_port=<optimized out>, from=0xbeaa0778,
> >     was_unicast=isc_boolean_false) at ../../common/options.c:3941
> > #11 0x0004abcc in got_one_v6 (h=<optimized out>) at
> > ../../common/discover.c:1587
> > #12 0x000793a4 in omapi_one_dispatch (wo=<optimized out>,
> > t=0xbeaa0b50) at ../../omapip/dispatch.c:539
> > #13 0x0004ccf0 in dispatch () at ../../common/dispatch.c:95
> > #14 0x0000fbc4 in main (argc=<optimized out>, argv=<optimized out>) at
> > ../../server/dhcpd.c:874
> >
> > ---
> > This email was received through isc.org Bug Submission Form
>
>