Relax requests’ IDNA2008 requirement?

(Also filed as psf/requests#5845 on GitHub.)

The venerable requests Python HTTP library currently uses the idna library to check input URLs for IDNA2008 compliance, and rejects URLs that don’t comply. This breaks non-compliant URLs with emoji characters, like http://โ˜ƒ.net/, which you all said was intentional (more background), since those domains’ time is arguably limited, ie they’re effectively “dead domains walking.” Understood.

However, not all TLDs require IDNA2008 compliance. Unlike gTLDs, ccTLDs generally get to choose their own domain policies – background from Wikipedia, ICANN, a GoDaddy representative – and a handful of them have stuck with IDNA2003, UTS#46, or related variants. (Not to mention older proprietary schemes like ThaiURL ๐Ÿ˜.) For example, .ws, .la, .ai, .to, and .fm evidently explicitly allow emoji.

Similarly, afaik domain owners can do whatever they want with their own subdomains. So thanks to Punycode, third level (and beyond) hostnames like ๐ŸŒโžกโžกโค๐Ÿ”’.ayeshious.com and ๐Ÿ”’๐Ÿ”’๐Ÿ”’.scotthelme.co.uk seem to not be at risk of breaking due to gTLD registries enforcing IDNA2008 on pay-level domain registrations.

Any chance you all could relax the IDNA2008 requirement so that you support both of those kinds of domains?

Right now, I’m working around this with code like this, using the domain2idna library, to support at least IDNA2003 in addition to IDNA2008. It’d be nice not to have to.

try:
  resp = requests.get(url, ...)
except requests.exceptions.InvalidURL:
  punycode = domain2idna(url)
  if punycode != url:
    # the domain is valid idna2003 but not idna2008. encode and try again.
    resp = requests.get(punycode, ...)

Thanks again for listening, and for maintaining requests!

One thought on “Relax requests’ IDNA2008 requirement?

  1. Seth Larson replied:

    This is not likely to land as there are additional security requirements to be mindful of when using IDNA2003, hence why it’s preferable to use IDNA2008. My recommendation in this case is to do the normalization yourself and pass Requests an ASCII-only host.

Leave a Reply

Your email address will not be published.