I run a number of projects on App Engine. I love it. However, it’s a bit embarrassing that the standard environment still only supports Python 2, not Python 3.

I recently migrated two Python 2 libraries with matching App Engine projects, granary and oauth-dropins, to be Python 3 compatible. It worked, but it took a lot of effort, and a fair amount of monkey patching. Here are a few App Engine standard warts I had to handle.

• Bumped my version of webapp2 up to 3.0.0b1, which includes Python3 support. (You can ignore this if you don’t use webapp2.) Note that App Engine has older versions of webapp2 built in, but not 3.0.0b1, so you’ll need to bundle it with your app’s code and use google.appengine.ext.vendor to load it.

• Monkey patched google.appengine.runtime.wsgi to allow unicode header values, since I added from __future__ import unicode_literals to all of my code:

  from google.appengine.runtime.wsgi import WsgiRequest
  orig_response = WsgiRequest._StartResponse
  
  def unicode_start_response(self, status, response_headers, exc_info=None):
    headers = [(name.encode('utf-8'), val.encode('utf-8'))
               for name, val in response_headers]
    return orig_response(self, status, headers, exc_info)
  
  WsgiRequest._StartResponse = unicode_start_response
  

• I used the future package (basically a better six) to do most of the heavy lifting for Python 2/3 compatibility, aka straddling. It’s great! However, it has a couple outstanding bugs that bit me on App Engine. I forked it and fixed them, but it’s evidently abandoned, so they haven’t merged my PRs or cut a new release.

  No matter, I’d usually just bundle my app and point my pip packages at my fork…except setuptools doesn’t allow specifying a dependency on GitHub instead of PyPi. Yes, you can kind of get there with dependency_links and pip install --process-dependency-links, but both of those are flaky and deprecated, and I didn’t want to depend on users always reading the docs and adding that unusual flag every time.

  So instead, I monkey patch these fixes in too:

  • First, future assumes httplib has a few constants that don’t exist on App Engine: _CS_IDLE, _CS_REQ_STARTED, and _CS_REQ_SENT. Add them:

    import httplib
    if not hasattr(httplib, '_CS_IDLE'):
    httplib._CS_IDLE = 'Idle'
    if not hasattr(httplib, '_CS_REQ_STARTED'):
    httplib._CS_REQ_STARTED = 'Request-started'
    if not hasattr(httplib, '_CS_REQ_SENT'):
    httplib._CS_REQ_SENT = 'Request-sent'
    

  • Second, future‘s backported urllib.parse module is overly strict about type checking its parameters, and doesn’t allow mixing types like Python 2’s unicode and str. Relax it:

    def _coerce(obj):
    return (unicode(obj) if isinstance(obj, basestring)
            else [_coerce(elem) for elem in obj] if isinstance(obj, (list, tuple, set))
            else {k: _coerce(v) for k, v in obj.items()} if isinstance(obj, dict)
            else obj)
    
    orig = {}
    def wrapper(name):
    def wrapped(*args, **kwargs):
      return orig[name](*_coerce(args), **_coerce(kwargs))
    return wrapped
    
    from future.backports.urllib import parse
    for name in ('urlparse', 'urlunparse', 'urlsplit', 'urlunsplit', 'urljoin',
               'urldefrag', 'parse_qsl'):
    orig[name] = getattr(parse, name)
    setattr(parse, name, wrapper(name))
    

• App Engine’s protobuf library chokes on future’s newstr property values, so I used an ndb.Model mixin that converts them to Python 2 strs:

  from future.types.newstr import newstr
  
  class FutureModel(ndb.Model):
    def put(self, *args, **kwargs):
      for name, val in self.to_dict().items():
        if isinstance(val, newstr):
          setattr(self, name, native_str(val))
  
      return super(FutureModel, self).put(*args, **kwargs)