Bug #36101 for bind9-public: [PATCH] IDN support in host/dig/nslookup using GNU libidn(2)

# Mon, 26 May 2014 08:51:53 -0400 Tomas Hozza <thozza@redhat.com> - Ticket created [Reply] [Comment]

Subject:	IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Mon, 26 May 2014 08:51:51 -0400 (EDT)
To:	bind-suggest@isc.org
From:	Tomas Hozza <thozza@redhat.com>

Download (untitled) / with headers
text/plain 1.2KiB

Hi. We maintain like 5 Fedora/RHEL specific patches that are modifying host/dig/nslookup to use libidn instead of idnkit. The change was made by the previous maintainer Adam Tkac, but I guess the reason was that the idnkit is not included in Fedora/RHEL and we don't want to build software using bundled libraries (it is against Fedora policy). I'm not sure if there is already a ISC-Bugs # number for this. However I would like to know if you would be interested in adding support for libidn. I consider it more a feature request than a Bug. My suggestion is to: - introduce new configure option to turn on the usage of GNU libidn (something like --with-gnu-libidn) - use libidn2 for IDN -> ACE conversion (as libidn2 suppoerts IDNA2008 [1] whereas libidn supports only IDNA2003) - use libidn for ACE -> IDN conversion where needed (as libidn2 apparently does not include support for this) I am willing to make the contribution and work with you and incorporate your feedback to make the patch acceptable. Please let me know what do you think about this. Thank you. [1] http://www.icann.org/en/resources/idn/idn-guidelines-02sep11-en.htm Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D Red Hat Inc. http://cz.redhat.com

# Mon, 26 May 2014 12:07:53 -0400 Evan Hunt <each@isc.org> - Correspondence added [Reply] [Comment]

CC:	undisclosed-recipients: ;
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Mon, 26 May 2014 16:07:53 +0000
To:	Tomas Hozza via RT <bind-suggest@isc.org>
From:	Evan Hunt <each@isc.org>

Download (untitled) / with headers
text/plain 688B

> I'm not sure if there is already a ISC-Bugs # number for this. However > I would like to know if you would be interested in adding support > for libidn. Yes. > - introduce new configure option to turn on the usage of GNU libidn > (something like --with-gnu-libidn) > - use libidn2 for IDN -> ACE conversion (as libidn2 suppoerts IDNA2008 [1] > whereas libidn supports only IDNA2003) > - use libidn for ACE -> IDN conversion where needed (as libidn2 apparently > does not include support for this) Pardon my confusion here, but do you mean that two different versions of libidn would be in use at the same time? -- Evan Hunt -- each@isc.org Internet Systems Consortium, Inc.

# Mon, 26 May 2014 12:07:54 -0400 The RT System itself - Status changed from 'new' to 'open'

# Tue, 27 May 2014 12:33:14 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Tue, 27 May 2014 12:33:10 -0400 (EDT)
To:	bind-suggest@isc.org
From:	Tomas Hozza <thozza@redhat.com>

Download (untitled) / with headers
text/plain 2.2KiB

----- Original Message ----- > > I'm not sure if there is already a ISC-Bugs # number for this. However > > I would like to know if you would be interested in adding support > > for libidn. > > Yes. Great! > > - introduce new configure option to turn on the usage of GNU libidn > > (something like --with-gnu-libidn) > > - use libidn2 for IDN -> ACE conversion (as libidn2 suppoerts IDNA2008 [1] > > whereas libidn supports only IDNA2003) > > - use libidn for ACE -> IDN conversion where needed (as libidn2 apparently > > does not include support for this) > > Pardon my confusion here, but do you mean that two different versions > of libidn would be in use at the same time? Well I have to admit that I was also little bit confused when looking at the idnkit code in dighost.c. The process seems to work like this: 1. the domain name from the CLI is converted to UTF-8 2. the origin is converted to UTF-8 3. UTF-8 domain name and UTF-8 origin are joined. 4. the joined domain name is conveted to the translated IDN name (ACE) 5. there is output_format() function for converting the translated IDN ascii (ACE) name to locale when printing it to the CLI. Now the "old" a.k.a. IDNA2003 library "libidn" has all the necessary functions (part of stringprep.h) for conversions of locale <-> UTF-8 <-> ACE. However the libidn2 (IDNA2008) library has only functions for conversion of locale -> translated IDN as UTF-8 and UTF-8 -> translated IDN as UTF-8. But I think there is still need for the functions to convert the translated IDN in UTF-8 to ACE, locale -> UTF-8 without the IDN translation, ACE -> locale (in output_filter). There is library libunistring that could solve this, but it is also yet another library. Maybe it would make sense to add compile time option for libidn, which would be used by default if compiled with it. Then provide yet another compile time option to use also libidn2 in addition to the libidn library and make the usage of it dependent on some runtime option (like -8 meaning IDNA2008 or something different). I'm open to any suggestions. Maybe I missed something... Thanks! Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D Red Hat Inc. http://cz.redhat.com

# Wed, 28 May 2014 10:13:45 -0400 Brian Conry <bconry@isc.org> - Correspondence added [Reply] [Comment]

Download (untitled) / with headers
text/plain 1.9KiB

On Tue May 27 16:33:14 2014, thozza@redhat.com wrote: > > > - introduce new configure option to turn on the usage of GNU > libidn > > > (something like --with-gnu-libidn) > > > - use libidn2 for IDN -> ACE conversion (as libidn2 suppoerts > IDNA2008 [1] > > > whereas libidn supports only IDNA2003) > > > - use libidn for ACE -> IDN conversion where needed (as libidn2 > apparently > > > does not include support for this) > > > > Pardon my confusion here, but do you mean that two different > versions > > of libidn would be in use at the same time? > > Well I have to admit that I was also little bit confused when looking > at the idnkit > code in dighost.c. > > The process seems to work like this: > 1. the domain name from the CLI is converted to UTF-8 > 2. the origin is converted to UTF-8 > 3. UTF-8 domain name and UTF-8 origin are joined. > 4. the joined domain name is conveted to the translated IDN name (ACE) > > 5. there is output_format() function for converting the translated > IDN ascii (ACE) name to locale when printing it to the CLI. > > Now the "old" a.k.a. IDNA2003 library "libidn" has all the necessary > functions (part of stringprep.h) for conversions of locale <-> UTF-8 > <-> ACE. > > However the libidn2 (IDNA2008) library has only functions for > conversion > of locale -> translated IDN as UTF-8 and UTF-8 -> translated IDN as > UTF-8. > > But I think there is still need for the functions to convert the > translated IDN in UTF-8 > to ACE, locale -> UTF-8 without the IDN translation, ACE -> locale (in > output_filter). > > There is library libunistring that could solve this, but it is also > yet another > library. Looking at the official libidn2 web page[1], it says: "Libidn2 uses GNU libunistring for Unicode processing and GNU libiconv for character set conversion." So it seems that they intend you to use libiconv for converting between the locale and UTF-8. [1] https://www.gnu.org/software/libidn/#libidn2

# Wed, 28 May 2014 12:10:27 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Wed, 28 May 2014 12:10:23 -0400 (EDT)
To:	bind-suggest@isc.org
From:	Tomas Hozza <thozza@redhat.com>

Download (untitled) / with headers
text/plain 2.5KiB

----- Original Message ----- > On Tue May 27 16:33:14 2014, thozza@redhat.com wrote: > > > > - introduce new configure option to turn on the usage of GNU > > libidn > > > > (something like --with-gnu-libidn) > > > > - use libidn2 for IDN -> ACE conversion (as libidn2 suppoerts > > IDNA2008 [1] > > > > whereas libidn supports only IDNA2003) > > > > - use libidn for ACE -> IDN conversion where needed (as libidn2 > > apparently > > > > does not include support for this) > > > > > > Pardon my confusion here, but do you mean that two different > > versions > > > of libidn would be in use at the same time? > > > > Well I have to admit that I was also little bit confused when looking > > at the idnkit > > code in dighost.c. > > > > The process seems to work like this: > > 1. the domain name from the CLI is converted to UTF-8 > > 2. the origin is converted to UTF-8 > > 3. UTF-8 domain name and UTF-8 origin are joined. > > 4. the joined domain name is conveted to the translated IDN name (ACE) > > > > 5. there is output_format() function for converting the translated > > IDN ascii (ACE) name to locale when printing it to the CLI. > > > > Now the "old" a.k.a. IDNA2003 library "libidn" has all the necessary > > functions (part of stringprep.h) for conversions of locale <-> UTF-8 > > <-> ACE. > > > > However the libidn2 (IDNA2008) library has only functions for > > conversion > > of locale -> translated IDN as UTF-8 and UTF-8 -> translated IDN as > > UTF-8. > > > > But I think there is still need for the functions to convert the > > translated IDN in UTF-8 > > to ACE, locale -> UTF-8 without the IDN translation, ACE -> locale (in > > output_filter). > > > > There is library libunistring that could solve this, but it is also > > yet another > > library. > > Looking at the official libidn2 web page[1], it says: > > "Libidn2 uses GNU libunistring for Unicode processing and GNU libiconv > for character set conversion." > > So it seems that they intend you to use libiconv for converting between > the locale and UTF-8. > > > > > [1] https://www.gnu.org/software/libidn/#libidn2 My bad. I missed this one. However I'm not sure if it makes sense to support IDNA2003 and IDNA2008 at the same time and let the user to choose the standard by some runtime option. I think it makes some sense, but I'm not sure if it is worth the effort. I'll wait for your opinion and work on the separate support in the meantime. Thank you. Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D Red Hat Inc. http://cz.redhat.com

# Wed, 28 May 2014 12:30:48 -0400 Evan Hunt <each@isc.org> - Correspondence added [Reply] [Comment]

CC:	undisclosed-recipients: ;
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Wed, 28 May 2014 16:30:48 +0000
To:	Tomas Hozza via RT <bind-suggest@isc.org>
From:	Evan Hunt <each@isc.org>

Download (untitled) / with headers
text/plain 347B

> I'll wait for your opinion and work on the separate support in the meantime. Your opinion is probably at least as well-informed as ours. Nobody's asked us about IDN in a while, which is why it hasn't been touched in so long. What does your specific customer use case require? -- Evan Hunt -- each@isc.org Internet Systems Consortium, Inc.

# Thu, 29 May 2014 02:46:31 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Thu, 29 May 2014 02:45:29 -0400 (EDT)
To:	bind-suggest@isc.org
From:	Tomas Hozza <thozza@redhat.com>

Download (untitled) / with headers
text/plain 727B

----- Original Message ----- > > I'll wait for your opinion and work on the separate support in the > > meantime. > > Your opinion is probably at least as well-informed as ours. Nobody's > asked us about IDN in a while, which is why it hasn't been touched in > so long. What does your specific customer use case require? It is a Fedora request https://bugzilla.redhat.com/show_bug.cgi?id=1098783 The problem is that some IDN domains are translated differently using IDNA2003 and IDNA2008. So users have to translate some domains manually to get to the desired site. Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D Red Hat Inc. http://cz.redhat.com

# Thu, 12 Jun 2014 08:59:41 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Thu, 12 Jun 2014 08:59:36 -0400 (EDT)
To:	bind-suggest@isc.org
From:	Tomas Hozza <thozza@redhat.com>

Download (untitled) / with headers
text/plain 1KiB

----- Original Message ----- > ----- Original Message ----- > > > I'll wait for your opinion and work on the separate support in the > > > meantime. > > > > Your opinion is probably at least as well-informed as ours. Nobody's > > asked us about IDN in a while, which is why it hasn't been touched in > > so long. What does your specific customer use case require? > > It is a Fedora request https://bugzilla.redhat.com/show_bug.cgi?id=1098783 > > The problem is that some IDN domains are translated differently using > IDNA2003 and IDNA2008. So users have to translate some domains manually > to get to the desired site. Hi. I prepared a patch that is adding two new configure options --with-libidn and --with-libidn2. It also renames the existing --with-idn to --with-idnkit to be more clear on its purpose. Please let me know if you see anything that could be improved. Thank you in advance for your feedback. Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D Red Hat Inc. http://cz.redhat.com

Download 0001-Add-support-for-libidn-and-libidn2.patch
text/x-patch 34.2KiB

Message body is not shown because sender requested not to inline it.

# Thu, 12 Jun 2014 14:00:31 -0400 Evan Hunt <each@isc.org> - Correspondence added [Reply] [Comment]

CC:	undisclosed-recipients: ;
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Thu, 12 Jun 2014 18:00:31 +0000
To:	Tomas Hozza via RT <bind-suggest@isc.org>
From:	Evan Hunt <each@isc.org>

Download (untitled) / with headers
text/plain 370B

> Thank you in advance for your feedback. It looks okay at first glance, except for a few minor style issues (already fixed). I'll need to review it in greater detail, and I don't have the bandwidth today, but I will get to it as soon as I can. Do you have any kind of test setup that I could use to confirm correctness? eh

# Fri, 13 Jun 2014 02:39:15 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Fri, 13 Jun 2014 02:39:12 -0400 (EDT)
To:	bind-suggest@isc.org
From:	Tomas Hozza <thozza@redhat.com>

Download (untitled) / with headers
text/plain 933B

----- Original Message ----- > > Thank you in advance for your feedback. > > It looks okay at first glance, except for a few minor style issues > (already fixed). I'll need to review it in greater detail, and I > don't have the bandwidth today, but I will get to it as soon as I > can. > > Do you have any kind of test setup that I could use to confirm > correctness? > > eh I used only the test case we have in Red Hat Bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1098783 In other words translating the "straße.de" produces different results using IDNA2003 and IDNA2008. $ idn straße.de strasse.de $ idn2 straße.de xn--strae-oqa.de Hope this helps. If you need anything else from me, just let me know. Thank you. Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D Red Hat Inc. http://cz.redhat.com

# Fri, 13 Jun 2014 03:22:27 -0400 Evan Hunt <each@isc.org> - Correspondence added [Reply] [Comment]

CC:	undisclosed-recipients: ;
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Fri, 13 Jun 2014 07:22:26 +0000
To:	Tomas Hozza via RT <bind-suggest@isc.org>
From:	Evan Hunt <each@isc.org>

Download (untitled) / with headers
text/plain 235B

> Hope this helps. Yes, thank you. With libidn2, the LANG trick for disabling IDN mentioned in the man page (LANG=langauage[_terrotory].ASCII) doesn't seem to be working as described; I'm not sure yet why not. Can you try it out?

# Fri, 13 Jun 2014 03:24:25 -0400 Evan Hunt <each@isc.org> - Correspondence added [Reply] [Comment]

CC:	undisclosed-recipients: ;
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Fri, 13 Jun 2014 07:24:24 +0000
To:	Evan Hunt via RT <bind-suggest@isc.org>
From:	Evan Hunt <each@isc.org>

Download (untitled) / with headers
text/plain 315B

> With libidn2, the LANG trick for disabling IDN mentioned in the man page > (LANG=langauage[_terrotory].ASCII) doesn't seem to be working as described; > I'm not sure yet why not. Can you try it out? CHARSET=ASCII not working with libidn, either. In both cases I get: dig: convert textname to UTF-8: failure

# Fri, 13 Jun 2014 04:12:13 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Fri, 13 Jun 2014 04:11:11 -0400 (EDT)
To:	bind-suggest@isc.org
From:	Tomas Hozza <thozza@redhat.com>

Download (untitled) / with headers
text/plain 1.5KiB

----- Original Message ----- > > With libidn2, the LANG trick for disabling IDN mentioned in the man page > > (LANG=langauage[_terrotory].ASCII) doesn't seem to be working as described; > > I'm not sure yet why not. Can you try it out? > > CHARSET=ASCII not working with libidn, either. In both cases I get: > > dig: convert textname to UTF-8: failure For the disable IDN I deducted it from the original patch from Adam Tkac for libidn we had in Fedora. For libidn it can be tested as follows: $ CHARSET=ASCII dig axfr . @k.root-servers.net | grep '^xn--' | wc -l 576 $ dig axfr . @k.root-servers.net | grep '^xn--' | wc -l 0 It just make dig to display the output in punycode rather than in the international format. For libidn2 I was deducting the "workaround" based on the libidn doc [1] but for libidn2 [2] (look for Environment Variables part). However I can see now that it does not work for libidn2. Thinking more about it, the functions that are used for libidn2 compared to libidn don't to the IDN translation when producing output. [1] http://www.gnu.org/software/libidn/manual/libidn.html#Invoking-idn [2] http://www.gnu.org/software/libidn/libidn2/manual/libidn2.html#Invoking-idn2 If you don't have any hint I'll have to investigate it more thoroughly to be able to fix the man page or source accordingly. Thanks for finding this and sorry for the mistake. Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D Red Hat Inc. http://cz.redhat.com

# Fri, 13 Jun 2014 15:05:10 -0400 Evan Hunt <each@isc.org> - Correspondence added [Reply] [Comment]

CC:	undisclosed-recipients: ;
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Fri, 13 Jun 2014 19:05:09 +0000
To:	Tomas Hozza via RT <bind-suggest@isc.org>
From:	Evan Hunt <each@isc.org>

Download (untitled) / with headers
text/plain 1.9KiB

> $ CHARSET=ASCII dig axfr . @k.root-servers.net | grep '^xn--' | wc -l > 576 > $ dig axfr . @k.root-servers.net | grep '^xn--' | wc -l > 0 Ah you're right, that does work with IDN2003. I was testing the input side: "dig straße.de", and that's not working. Here's what I'm seeing, marked with (FAIL) where it didn't match expectations: No IDN support: "dig straße.de" sends a query for "stra\195\159e.de." "dig axfr ." contains xn-- names idnkit: "dig straße.de" sends a query for "strasse.de" "dig axfr ." contains names in non-ASCII charsets idnkit (with IDN_DISABLE): "dig straße.de" sends a query for "stra\195\159e.de." "dig axfr ." contains xn-- names GNU libidn: "dig straße.de" sends a query for "strasse.de" "dig axfr ." contains names in non-ASCII charsets GNU libidn (with CHARSET=ASCII): "dig straße.de" fails (FAIL) "dig axfr ." contains xn-- names GNU libidn2: "dig straße.de" sends a query for "xn--strae-oqa.de". "dig axfr ." contains xn-- names (FAIL) GNU libidn2 (with LANG=en_US.ASCII): "dig straße.de" fails (FAIL) "dig axfr ." contains xn-- names Note, something has gone wrong in configure with building idnkit; I had to build it from an unpatched source tree. It occurs to me that we could simplify things by checking for IDN_DISABLE in dighost.c itself. Then we could have the same documentation for all three libraries. (Though it might be worthwhile to add some text explaining differences between IDN 2003 and 2008.) That still leaves the problem that libidn2 doesn't seem to be working on the output side -- I only see xn-- names from libidn2, regardless of environment. Here's a diff showing the patch as I've modified it for ISC style compatibility. Files that are automatically generated (man pages, configure, config.h) have been omitted. (I thought it might be easier if we're both working in the same code.)

Download idn.diff
text/x-diff 24KiB

Message body is not shown because sender requested not to inline it.

# Wed, 17 Sep 2014 13:42:59 -0400 Vicky Risk <vicky@isc.org> - Subject changed from 'IDN support in host/dig/nslookup using GNU libidn(2)' to '[PATCH] IDN support in host/dig/nslookup using GNU libidn(2)'

# Mon, 29 Dec 2014 10:19:27 -0500 Brian Conry <bconry@isc.org> - Severity S2 Normal added

# Sun, 11 Jan 2015 07:10:52 -0500 Mukund Sivaraman <muks@isc.org> - Correspondence added [Reply] [Comment]

Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Sun, 11 Jan 2015 17:40:39 +0530
To:	Tomas Hozza via RT <bind-suggest@isc.org>
From:	Mukund Sivaraman <muks@isc.org>

Download (untitled) / with headers
text/plain 1.1KiB

Hi Tomas On Mon, May 26, 2014 at 12:51:54PM +0000, Tomas Hozza via RT wrote: > We maintain like 5 Fedora/RHEL specific patches that are modifying > host/dig/nslookup to use libidn instead of idnkit. This is somewhat related, but because we don't maintain these patches in the BIND tree yet, I'm mentioning it here instead of creating an ISC bug: Run Wireshark and try the following query in nslookup: ra4add\226a\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd.\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191.\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222.\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\238.4Co.example.org Repeat it a few times. It seems there is some misbehavior due to the patches to BIND in Fedora (probably libidn related) that causes it to send junk names (random every time). Vanilla BIND with and without idnkit configured don't show this issue. Mukund

Download (untitled)
application/pgp-signature 801B

Message body not shown because it is not plain text.

# Tue, 13 Jan 2015 07:24:49 -0500 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Tue, 13 Jan 2015 13:24:41 +0100
To:	bind-suggest@isc.org
From:	Tomas Hozza <thozza@redhat.com>

Download (untitled) / with headers
text/plain 1.7KiB

On 01/11/2015 01:10 PM, Mukund Sivaraman via RT wrote: > Hi Tomas > > On Mon, May 26, 2014 at 12:51:54PM +0000, Tomas Hozza via RT wrote: > > We maintain like 5 Fedora/RHEL specific patches that are modifying > > host/dig/nslookup to use libidn instead of idnkit. > > This is somewhat related, but because we don't maintain these patches in > the BIND tree yet, I'm mentioning it here instead of creating an ISC > bug: > > Run Wireshark and try the following query in nslookup: > > ra4add\226a\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd.\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191.\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222.\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\2384Cohd\191\222\238.4Co.example.org > > Repeat it a few times. It seems there is some misbehavior due to the > patches to BIND in Fedora (probably libidn related) that causes it to > send junk names (random every time). Vanilla BIND with and without > idnkit configured don't show this issue. > > Mukund > Hi Mukund. Thanks for the notice. There sure may be some issue related to those downstream patches. They were created long time ago. The work on the patches is still in my TODO. I haven't yet resolved issues catches by Evan. The work on IDN had low priority in the past few months. I'll make sure to look at the issue you're describing before sending a new patch. Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D Red Hat Inc. http://cz.redhat.com

# Wed, 23 Sep 2015 12:10:42 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

CC:	"Jakub Luzny" <jluzny@redhat.com>
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Wed, 23 Sep 2015 18:10:32 +0200
To:	bind-suggest@isc.org
From:	"Tomas Hozza" <thozza@redhat.com>

Download (untitled) / with headers
text/plain 3.8KiB

Hi Evan. Sorry for sooo late response, but I had more priority work to do. Finally I have one new colleague Jakub Luzny helping me with nice-to-have things in BIND. Jakub reworked the patch and split it in two. First patch reworks the code for IDN, which enables us to add other IDN implementations easily. As part of this it also adds the support for GNU libidn. The second patch adds the libidn2 support. We split it away, because libidn2 does not support translation from punycode to Unicode. This is a problem, because the IDN domains are then showed in punycode in BIND tools output. Nevertheless we think that libidn2 support may be useful and needed by some people, since we have also request for this in Fedora: https://bugzilla.redhat.com/show_bug.cgi?id=1098783 On 13.06.2014 21:05, Evan Hunt via RT wrote: >> $ CHARSET=ASCII dig axfr . @k.root-servers.net | grep '^xn--' | wc -l >> 576 >> $ dig axfr . @k.root-servers.net | grep '^xn--' | wc -l >> 0 > > Ah you're right, that does work with IDN2003. I was testing the input > side: "dig straße.de", and that's not working. Here's what I'm seeing, > marked with (FAIL) where it didn't match expectations: > > No IDN support: > "dig straße.de" sends a query for "stra\195\159e.de." > "dig axfr ." contains xn-- names > > idnkit: > "dig straße.de" sends a query for "strasse.de" > "dig axfr ." contains names in non-ASCII charsets > > idnkit (with IDN_DISABLE): > "dig straße.de" sends a query for "stra\195\159e.de." > "dig axfr ." contains xn-- names Unfortunately I was not able to get idnkit working on Fedora, but I used your results for comparison. > GNU libidn: > "dig straße.de" sends a query for "strasse.de" > "dig axfr ." contains names in non-ASCII charsets Works the same way also in the current patch set. However I was not able to transfer the root zone. So if you can test this, it would be great. > GNU libidn (with CHARSET=ASCII): > "dig straße.de" fails (FAIL) sends query for "stra\195\159e.de." with (IDN_DISABLE defined) > "dig axfr ." contains xn-- names > > GNU libidn2: > "dig straße.de" sends a query for "xn--strae-oqa.de". works the same way > "dig axfr ." contains xn-- names (FAIL) I was not able to test this. > > GNU libidn2 (with LANG=en_US.ASCII): > "dig straße.de" fails (FAIL) sends query for "stra\195\159e.de." with (IDN_DISABLE defined) > "dig axfr ." contains xn-- names > > Note, something has gone wrong in configure with building idnkit; I > had to build it from an unpatched source tree. I'm not sure what exactly broke. The build fails for me with vanilla BIND and also with patched source. If you still see the issue, please send us some more information on what exactly is the problem. > It occurs to me that we could simplify things by checking for > IDN_DISABLE in dighost.c itself. Then we could have the same > documentation for all three libraries. (Though it might be worthwhile > to add some text explaining differences between IDN 2003 and 2008.) Jakub reworked the code in a way that only IDN_DISABLE is checked. > That still leaves the problem that libidn2 doesn't seem to be > working on the output side -- I only see xn-- names from libidn2, > regardless of environment. Right. I would not make it the default for this reason, however I think it would be beneficial to at least include it in the upstream source, so people wanting this may use IDN 2008 at least when doing queries. > Here's a diff showing the patch as I've modified it for ISC style > compatibility. Files that are automatically generated (man pages, > configure, config.h) have been omitted. (I thought it might be easier > if we're both working in the same code.) Thanks, I'm attaching two patches. Thank you! Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D UTC+2 (CEST) Red Hat Inc. http://cz.redhat.com

Download 0001-Add-support-for-GNU-libidn.patch
text/x-patch 18.6KiB

Message body is not shown because sender requested not to inline it.

Download 0002-Add-support-for-libidn2.patch
text/x-patch 7.8KiB

Message body is not shown because sender requested not to inline it.

# Wed, 23 Sep 2015 13:09:59 -0400 Evan Hunt <each@isc.org> - Correspondence added [Reply] [Comment]

CC:
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Wed, 23 Sep 2015 17:09:59 +0000
To:	"Tomas Hozza via RT" <bind-suggest@isc.org>
From:	"Evan Hunt" <each@isc.org>

Download (untitled) / with headers
text/plain 385B

On Wed, Sep 23, 2015 at 04:10:42PM +0000, Tomas Hozza via RT wrote: > Works the same way also in the current patch set. However I was > not able to transfer the root zone. So if you can test this, > it would be great. If you use "dig @f.root-servers.net" it should work, it's configured to allow transfer for anyone. (More detailed response to follow when I have a bit more time.)

# Thu, 24 Sep 2015 03:34:31 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

CC:	"Jakub Luzny" <jluzny@redhat.com>
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Thu, 24 Sep 2015 09:34:21 +0200
To:	bind-suggest@isc.org
From:	"Tomas Hozza" <thozza@redhat.com>

Download (untitled) / with headers
text/plain 1.1KiB

On 23.09.2015 19:10, Evan Hunt via RT wrote: > > On Wed, Sep 23, 2015 at 04:10:42PM +0000, Tomas Hozza via RT wrote: > > Works the same way also in the current patch set. However I was > > not able to transfer the root zone. So if you can test this, > > it would be great. > > If you use "dig @f.root-servers.net" it should work, it's configured > to allow transfer for anyone. > > (More detailed response to follow when I have a bit more time.) Thank you for the hint. I retested the code. With libidn: dig @f.root-servers.net axfr . --> there are domains with Unicode characters in the output IDN_DISABLE="" dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- With libidn2: dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- IDN_DISABLE="" dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- As noted before, the behavior with libidn2 is expected, since libidn2 does not support translation from punycode to Unicode. Thanks! Regards, -- Tomas Hozza Software Engineer - EMEA ENG Developer Experience PGP: 1D9F3C2D UTC+2 (CEST) Red Hat Inc. http://cz.redhat.com

# Wed, 22 Jun 2016 19:19:05 -0400 Vicky Risk <vicky@isc.org> - Versions Planned 9.11 added

# Wed, 06 Jul 2016 12:27:26 -0400 Vicky Risk <vicky@isc.org> - Versions Planned 9.11 deleted

# Tue, 28 Mar 2017 04:54:52 -0400 Tomas Hozza <thozza@redhat.com> - Correspondence added [Reply] [Comment]

CC:	"Petr Mensik" <pemensik@redhat.com>
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
Date:	Tue, 28 Mar 2017 10:54:44 +0200
To:	bind-suggest@isc.org
From:	"Tomas Hozza" <thozza@redhat.com>

Download (untitled) / with headers
text/plain 1.7KiB

On 24.09.2015 09:34, Tomas Hozza via RT wrote: > On 23.09.2015 19:10, Evan Hunt via RT wrote: > > > > On Wed, Sep 23, 2015 at 04:10:42PM +0000, Tomas Hozza via RT wrote: > >> Works the same way also in the current patch set. However I was > >> not able to transfer the root zone. So if you can test this, > >> it would be great. > > > > If you use "dig @f.root-servers.net" it should work, it's configured > > to allow transfer for anyone. > > > > (More detailed response to follow when I have a bit more time.) > > Thank you for the hint. I retested the code. > > With libidn: > dig @f.root-servers.net axfr . --> there are domains with Unicode characters in the output > IDN_DISABLE="" dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- > > With libidn2: > dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- > IDN_DISABLE="" dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- > > As noted before, the behavior with libidn2 is expected, since > libidn2 does not support translation from punycode to Unicode. > > Thanks! > > Regards, > Hello. Fedora plans to move to libidn2 and get completely rid of libidn since Fedora 27 (https://fedoraproject.org/wiki/Changes/IDNA2008). Rather than carrying another downstream patch in Fedora, we would like to work with you on the inclusion of the functionality into BIND. Can you please review the last version of patch set we sent to you and let us know in case you need anything from us? Thank you in advance! Regards, -- Tomas Hozza Associate Manager, Software Engineering - EMEA ENG Mainstream RHEL PGP: 1D9F3C2D UTC+1 (CET) Red Hat Inc. http://cz.redhat.com

# Wed, 28 Jun 2017 15:31:29 -0400 Vicky Risk <vicky@isc.org> - Queue changed from #3 to bind9-public

# Wed, 28 Jun 2017 15:31:29 -0400 Vicky Risk <vicky@isc.org> - Area feature added

# Wed, 02 Aug 2017 17:51:05 -0400 Vicky Risk <vicky@isc.org> - Given to Mark Andrews <marka@isc.org>

# Wed, 16 Aug 2017 13:19:15 -0400 Petr Menšík <pemensik@redhat.com> - Correspondence added [Reply] [Comment]

From:	"Petr Menšík" <pemensik@redhat.com>
Date:	Wed, 16 Aug 2017 19:18:56 +0200
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
To:	"Tomas Hozza" <thozza@redhat.com>, bind-suggest@isc.org

To:	"Tomas Hozza" <thozza@redhat.com>, bind-suggest@isc.org
Subject:	Re: [ISC-Bugs #36101] IDN support in host/dig/nslookup using GNU libidn(2)
From:	"Petr Menšík" <pemensik@redhat.com>

Download (untitled) / with headers
text/plain 3.8KiB

Hello Evan, Hello Tomas. I reworked IDN support of dig to better support (recent) libidn2. First motivation was always failing test in tests/system/nslookup. I think there was many differences and duplicate code when handling relative_domain+origin with and without IDN. I am not 100% confident, but I think both IDNA 2003 and IDN 2008 do parse each label separately. I could not find a reason why would relative textname and origin be first converted to utf-8, then put together. And as a last step converted into ACE encoding. There is only single reason I found, checking of lengths of labels and wire length of domain name. Idnkit checks that, libidn does not and libidn2 checks that again. However there is already checking for this in dns_name_fromtext. Because it was handled in different way, the mentioned test was failing. Libidn2 since v2.0 supports also punny code decoding. I separated idn support for input processing and output translation of responses. It supports libidn2 before 2.0, where punny code will not be decoded. Supported versions would enable output decoding auto-magically from configure. Because libidn2 already implements decoding from locale, I left it to libidn2 and removed iconv calls from dighost.c. I made input decoding errors fatal to help display relevant error message from idn library. Because in master is already support for +idnout option, I removed support for original environment variable IDN_DISABLED. I thought there is no reason to implement turning off IDN support for input names at first. There was a bug in libidn2 fixed only in recently released 2.0.3 however, which stripped any invalid characters from domain names. But because in such versions name "_kerberos.fedoraproject.org." is silently transformed into "kerberos.fedoraproject.org.", I think +noidnin still has some use cases, so I added new option. It is now possible to turn off both input and output IDN handling runtime in dig. nslookup and host will always use defaults. Attached patch applies on top of both patches from Tomas. Included are modified patches that applies cleanly on current master. Could you please review it? Thank you in advance! Dne 28.3.2017 v 10:54 Tomas Hozza napsal(a): > On 24.09.2015 09:34, Tomas Hozza via RT wrote: >> On 23.09.2015 19:10, Evan Hunt via RT wrote: >>> >>> On Wed, Sep 23, 2015 at 04:10:42PM +0000, Tomas Hozza via RT wrote: >>>> Works the same way also in the current patch set. However I was >>>> not able to transfer the root zone. So if you can test this, >>>> it would be great. >>> >>> If you use "dig @f.root-servers.net" it should work, it's configured >>> to allow transfer for anyone. >>> >>> (More detailed response to follow when I have a bit more time.) >> >> Thank you for the hint. I retested the code. >> >> With libidn: >> dig @f.root-servers.net axfr . --> there are domains with Unicode characters in the output >> IDN_DISABLE="" dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- >> >> With libidn2: >> dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- >> IDN_DISABLE="" dig @f.root-servers.net axfr . --> there are no domains with Unicode characters, only xn-- >> >> As noted before, the behavior with libidn2 is expected, since >> libidn2 does not support translation from punycode to Unicode. >> >> Thanks! >> >> Regards, >> > > Hello. > > Fedora plans to move to libidn2 and get completely rid of libidn since Fedora 27 (https://fedoraproject.org/wiki/Changes/IDNA2008). > > Rather than carrying another downstream patch in Fedora, we would like to work with you on the inclusion of the functionality into BIND. > > Can you please review the last version of patch set we sent to you and let us know in case you need anything from us? > > Thank you in advance! > > Regards, > Regards, -- Petr Menšík Software Engineer Red Hat, http://www.redhat.com/ email: pemensik@redhat.com PGP: 65C6C973

Download 0001-Add-support-for-GNU-libidn.patch
text/x-patch 17.6KiB

Message body is not shown because sender requested not to inline it.

Download 0002-Add-support-for-libidn2.patch
text/x-patch 7.8KiB

Message body is not shown because sender requested not to inline it.

Download 0003-Removed-iconv-convert-directly-from-locale-to-ACE.patch
text/x-patch 27.7KiB

Message body is not shown because sender requested not to inline it.

Download signature.asc
application/pgp-signature 473B

Message body not shown because it is not plain text.

# Mon, 25 Sep 2017 11:31:45 -0400 Mukund Sivaraman <muks@isc.org> - Priority P1 High added

# Mon, 25 Sep 2017 11:31:46 -0400 Mukund Sivaraman <muks@isc.org> - Severity S2 Normal changed to S1 High

Bug #36101 for bind9-public: [PATCH] IDN support in host/dig/nslookup using GNU libidn(2)

This bug tracker is no longer active.