Skip Menu |
Report information
The Basics
Id: 46767
Status: resolved
Estimated: 8 hours (480 minutes)
Worked: 3 hours (180 minutes)
Users:
tmark: 3 hours (180 minutes)
Priority: 10/10
Queue: dhcp-public

Bug Information
Version Fixed: 4.4.0 4.3.7
Version Found: (no value)
Versions Affected: (no value)
Versions Planned: 4.4.0 4.3.7
Priority: P2 Normal
Severity: S2 Normal
CVSS Score: (no value)
CVE ID: (no value)
Component: (no value)
Area: bug

Dates
Created:Mon, 04 Dec 2017 11:09:57 -0500
Updated:Tue, 12 Dec 2017 08:56:43 -0500
Closed:Thu, 07 Dec 2017 11:46:06 -0500



This bug tracker is no longer active.

Please go to our Gitlab to submit issues (both feature requests and bug reports) for active projects maintained by Internet Systems Consortium (ISC).

Due to security and confidentiality requirements, full access is limited to the primary maintainers.

Subject: [PATCH] DHCPd leaks sockets descriptors
To: dhcp-bugs@isc.org
Date: Mon, 04 Dec 2017 17:09:22 +0100
From: "Pavel Zhukov" <pzhukov@redhat.com>
commit ce17b1af08f3ea58d56c9168fd06f26a3094e95b Author: Pavel Zhukov <pzhukov@redhat.com> Date: Mon Dec 4 16:58:57 2017 +0100 Close the connecion if writer doesn't send data Some network monitoring tool check if service is alive by connect() to the port and close the connection without sending any data (ncat -z or nagios check_tcp are an example). As the result dhcpd leaks sockets and result is DoS. Signed-off: Martin Osvald <mosvald@redhat.com> diff --git a/omapip/dispatch.c b/omapip/dispatch.c index ba99889..d1fb57e 100644 --- a/omapip/dispatch.c +++ b/omapip/dispatch.c @@ -183,6 +183,10 @@ omapi_iscsock_cb(isc_task_t *task, if (status == ISC_R_INPROGRESS) { return (1); } + + /* Writer closes the connection without data sent */ + if (obj->reaper != NULL) + obj->reaper(obj->inner); } /* -- PAVEL ZHUKOV SOFTWARE ENGINEER, RHCE, RHCVA Red Hat pzhukov@redhat.com IM: IRC: landgraf
Hello Pavel: Thank you for bringing this matter to our attention. We are researching it now. Please keep this information confidential until we've had to time to assess the situation. Regards, Thomas Markwalder ISC Software Engineering
Hello Pavel: A few questions. First, what version of dhcpd are you testing? Secondly, could you describe the exact steps you are conducting to cause the leak? Thanks, Thomas Markwalder ISC Software Engineering
Subject: Re: [ISC-Bugs #46767] [PATCH] DHCPd leaks sockets descriptors
From: "Pavel Zhukov" <pzhukov@redhat.com>
To: "Thomas Markwalder via RT" <dhcp-confidential@isc.org>
Date: Tue, 05 Dec 2017 13:30:11 +0100
"Thomas Markwalder via RT" <dhcp-confidential@isc.org> writes: > Hello Pavel: Hello Thomas, > > A few questions. First, what version of dhcpd are you testing? > Secondly, could you describe the exact steps you are conducting to > cause the leak? dhcp-server-4.3.5-9.fc26.x86_64 I've tested both Fedora's and RHEL versions. RHEL has 4.2.5 and Fedora 4.3.5 released. Steps to Reproduce: 1. configure omapi port and start dhcpd 2. ncat -z <host> <port> (or telnet <hostname> <port> <<< "" ) 3. lsof | grep dhcpd | grep TCP | wc -l > > Thanks, > > Thomas Markwalder > ISC Software Engineering -- PAVEL ZHUKOV SOFTWARE ENGINEER, RHCE, RHCVA Red Hat pzhukov@redhat.com IM: IRC: landgraf
On Wed Dec 06 11:39:10 2017, tmark wrote: > The server does continue to respond to DHCP client traffic. => fortunately the server has no reconfigure/redetect so service sockets are opened are startup and remain until the program is stopped...
From: "Pavel Zhukov" <pzhukov@redhat.com>
Subject: Re: [ISC-Bugs #46767] [PATCH] DHCPd leaks sockets descriptors
To: "Francis Dupont via RT" <dhcp-confidential@isc.org>
Date: Wed, 06 Dec 2017 14:45:58 +0100
"Francis Dupont via RT" <dhcp-confidential@isc.org> writes: > On Wed Dec 06 11:39:10 2017, tmark wrote: >> The server does continue to respond to DHCP client traffic. > > => fortunately the server has no reconfigure/redetect so > service sockets are opened are startup and remain until > the program is stopped... Right it's good for sure. But DHCPd is used in products like Foreman/Satellite and unavailability to make new ip reservation using omshell is critical for production because new machines cannot be provisioned/reprovisioned using PXE. -- PAVEL ZHUKOV SOFTWARE ENGINEER, RHCE, RHCVA Red Hat pzhukov@redhat.com IM: IRC: landgraf
Hello Pavel: We were able to reproduce it under Fedora 23. Ubuntu and Centos 7 seem to be immune. We have an alternative patch that is under review and can forward it to you for testing once reviewed. Our testing here with 4.3.3 under Fedora does confirm an eventual inability to make further OMAPI connections, but the server does continue to service DHCP clients. The code limits the number of open FDs to 200. As 4.2* is EOL, we did not test it, though same patch could be applied to it. Regards, Thomas Markwalder ISC Sofware Engineering
From: "Pavel Zhukov" <pzhukov@redhat.com>
To: "Thomas Markwalder via RT" <dhcp-confidential@isc.org>
Date: Wed, 06 Dec 2017 15:42:58 +0100
Subject: Re: [ISC-Bugs #46767] [PATCH] DHCPd leaks sockets descriptors
"Thomas Markwalder via RT" <dhcp-confidential@isc.org> writes: > Hello Pavel: Hello Thomas, > > We were able to reproduce it under Fedora 23. Ubuntu and Centos 7 > seem to be immune. We have an alternative patch that is under review > and can forward it to you for testing once reviewed. This is strange. Centos uses same codebase as RHEL (binary compatible) and issue has been reported by RHEL Customer. Customer confirmed that RHEL6's version (4.1.1) is not affected but RHEL7's is > > Our testing here with 4.3.3 under Fedora does confirm an eventual > inability to make further OMAPI connections, but the server does > continue to service DHCP clients. The code limits the number of open > FDs to 200. As 4.2* is EOL, we did not test it, though same patch > could be applied to it. Thank you! > > Regards, > > Thomas Markwalder > ISC Sofware Engineering -- PAVEL ZHUKOV SOFTWARE ENGINEER, RHCE, RHCVA Red Hat pzhukov@redhat.com IM: IRC: landgraf
Date: Wed, 06 Dec 2017 16:07:22 +0100
Subject: Re: [ISC-Bugs #46767] [PATCH] DHCPd leaks sockets descriptors
To: "Thomas Markwalder via RT" <dhcp-confidential@isc.org>
From: "Pavel Zhukov" <pzhukov@redhat.com>
"Thomas Markwalder via RT" <dhcp-confidential@isc.org> writes: > Hello Pavel: > > We were able to reproduce it under Fedora 23. Ubuntu and Centos 7 > seem to be immune. We have an alternative patch that is under review > and can forward it to you for testing once reviewed. Reproduced with Centos 7 # rpm -q dhcp dhcp-4.2.5-58.el7.centos.x86_64 # lsof | grep dhcpd | grep TCP | wc -l 191 # omshell > server localhost > port 7911 > connect dhcpctl_connect: no more Note: in case of virtual environment with virtio drivers and local to dhcpd traffic the issue is not reproducible. > > Our testing here with 4.3.3 under Fedora does confirm an eventual > inability to make further OMAPI connections, but the server does > continue to service DHCP clients. The code limits the number of open > FDs to 200. As 4.2* is EOL, we did not test it, though same patch > could be applied to it. > > Regards, > > Thomas Markwalder > ISC Sofware Engineering
From: "Thomas Markwalder" <tmark@isc.org>
To: dhcp-confidential@isc.org
Subject: Re: [ISC-Bugs #46767] [PATCH] DHCPd leaks sockets descriptors
Date: Wed, 6 Dec 2017 10:10:28 -0500
On 12/06/2017 10:07 AM, Pavel Zhukov via RT wrote: > "Thomas Markwalder via RT" <dhcp-confidential@isc.org> writes: > >> Hello Pavel: >> >> We were able to reproduce it under Fedora 23. Ubuntu and Centos 7 >> seem to be immune. We have an alternative patch that is under review >> and can forward it to you for testing once reviewed. > Reproduced with Centos 7 > # rpm -q dhcp > dhcp-4.2.5-58.el7.centos.x86_64 > # lsof | grep dhcpd | grep TCP | wc -l > 191 > # omshell >> server localhost >> port 7911 >> connect > dhcpctl_connect: no more > > Note: in case of virtual environment with virtio drivers and local to dhcpd > traffic the issue is not reproducible. AHA! Well done. Always better to have an logical explanation than not. Thank you for solving that. Thomas
Hello Pavel: You mentioned this was reported by a RHEL customer, is there RHEL bug ticket for this? Thanks Thomas
Subject: Re: [ISC-Bugs #46767] [PATCH] DHCPd leaks sockets descriptors
Date: Wed, 6 Dec 2017 20:50:24 +0100
To: "Thomas Markwalder via RT" <dhcp-confidential@isc.org>
From: "Pavel Zhukov" <pzhukov@redhat.com>
Hello Thomas,

Yes. There's (private) bug report openedĀ https://bugzilla.redhat.com/1519363 .

On Wed, Dec 6, 2017 at 7:28 PM, Thomas Markwalder via RT <dhcp-confidential@isc.org> wrote:
Hello Pavel:

You mentioned this was reported by a RHEL customer, is there RHEL bug ticket for this?

Thanks

Thomas



--
Pavel Zhukov
Software Engineer
IRC: landgraf


Hello Pavel: Please find attached our patch for correcting the issue. It should apply to 4.3.3 and 4.2.5, just exclude RELNOTES change. This will be going into our next feature release, 4.4.0 due out Q1/2018 and maintenance release 4.3.7 whose date is TBD. I appreciate your quick responses to my questions and for your initial patch as it gave me some insights. Let me know how it works for you. Regards, Thomas Markwalder ISC Software Engineering
Subject: rt46767.patch
Download rt46767.patch
text/x-patch 2.1KiB
commit ccff9ed69d0b26d33ce9cac8e83dab535b64d627 Author: Thomas Markwalder <tmark@isc.org> Date: Tue Dec 5 15:12:34 2017 -0500 [46767] Plugged a socket descriptor leak in OMAPI If disconnect is triggered by the reader closing the socket, while there is data left to write, the socket would be orphaned. omapip/buffer.c omapi_connection_writea() - added logic to recall disconnect once pending data has been written omapip/message.c Removed static declaration from omapi_message_unregister so you can actually compile when DEBUG_PROTOCOL is defined. Added a release note diff --git a/RELNOTES b/RELNOTES index b93af61..595d18e 100644 --- a/RELNOTES +++ b/RELNOTES @@ -1181,6 +1181,11 @@ dhcp-users@lists.isc.org. to accomodate a change in behavior in BIND9 isc_heap_delete(). [ISC-bugs #46719] +- Plugged a socket descriptor leak in OMAPI, that can occur when there is + data pending to be written to an OMAPI connection, when the connection + is closed by the reader. + [ISC-Bugs #46767] + Changes since 4.2.0 (new features) - If a client renews before 'dhcp-cache-threshold' percent of its lease diff --git a/omapip/buffer.c b/omapip/buffer.c index 6e0621b..a21f0a8 100644 --- a/omapip/buffer.c +++ b/omapip/buffer.c @@ -565,6 +565,15 @@ isc_result_t omapi_connection_writer (omapi_object_t *h) omapi_buffer_dereference (&buffer, MDL); } } + + /* If we had data left to write when we're told to disconnect, + * we need recall disconnect, now that we're done writing. + * See rt46767. */ + if (c->out_bytes == 0 && c->state == omapi_connection_disconnecting) { + omapi_disconnect (h, 1); + return ISC_R_SHUTTINGDOWN; + } + return ISC_R_SUCCESS; } diff --git a/omapip/message.c b/omapip/message.c index ee15d82..37abbd2 100644 --- a/omapip/message.c +++ b/omapip/message.c @@ -339,7 +339,7 @@ isc_result_t omapi_message_unregister (omapi_object_t *mo) } #ifdef DEBUG_PROTOCOL -static const char *omapi_message_op_name(int op) { +const char *omapi_message_op_name(int op) { switch (op) { case OMAPI_OP_OPEN: return "OMAPI_OP_OPEN"; case OMAPI_OP_REFRESH: return "OMAPI_OP_REFRESH";