Using PyCrypto instead of M2Crypto on Google App Engine

If you've ever tried to leverage the M2Crypto library on the Google App Engine platform, you'll notice that you're prevented from doing so. One of the reasons is that M2Crypto depends on C-compiled openssl libraries that are not allowed to be used on the platform. The major alternative that Google provides is the Python Cryptography Toolkit, or pycrypto, which is a Python-based version of various encryption algorithms.

Performance wise, the pycrypto library may not run as fast since it implements the encryption algorithms primarily in Python. Furthermore, the current version that is supported is version 2.3, which doesn't provide RSA PKCS#1 v1.5 encryption support (if you find yourself getting "Block type 01 not supported, you need to handle the padding/encoding format -- see information below) nor does it provide PEM format (you need to encode your keys as DER format, which are basically the base64-unencoded versions of the PEM).

Nonetheless, you can get pycrypto working on Google App Engine. Here's how you can do it inside your App Engine projects. I'll also discuss some of the extra steps needed to get RSA signatures working too (using the PKCS#1 v1.5 standard). It's not completely obvious so some of the stack traces you see will hopefully explain why these steps were necessary.

1. Inside the app.yaml configuration, you need to specify a runtime of python27, and specify the pycrypto librarie. Note: you either have to install python2.7 on your own local dev instance, or you simply have to deploy constantly to Google App Engine to test your app. Note that you also need to specify the threadsafe: parameter too for the Python 2.7 experimental version (see the Google App Engine Python 2.7 docs for more information).

application: YOUR_APP_NAME_HERE
version: 1runtime: python27
api_version: 1
threadsafe: false
handlers:- url: /static  
static_dir: static

...
libraries:
- name: lxml  
version: "2.3"
- name: pycrypto  
version: "2.3"

2. You need to generate your RSA keys with the DER format. If you see this stack trace, chances are that you're trying to provide key formats in PEM. The Pycrypto v2.3 version on Google App Engine assumes you're feeding things in DER format. In other words, these two lines will not work:

>>> from Crypto.PublicKey import RSA
>>> privatekey = open("privkey.pem", "r")
>>> m = RSA.importKey(privatekey)

Stack trace:File 
"/base/python27_runtime/python27_lib/versions/third_party/pycrypto-2.3/Crypto/PublicKey/RSA.py", line 247, in importKey    return self._importKeyDER(der)  
File "/base/python27_runtime/python27_lib/versions/third_party/pycrypto-2.3/Crypto/PublicKey/RSA.py", line 234, in _importKeyDER    raise ValueError("RSA key format is not supported")

You can use the openssl command to convert your public/private key PEM file to DER format. You'll also need to the same if you're using a PEM certificate too:

openssl rsa -outform der privkey.pem > privkey.der
openssl rsa -outform der < cert.pem > cert.der

These lines should the work:

>>> from Crypto.PublicKey import RSA 
>>> privatekey = open("privkey.der", "r")
>>> m = RSA.importKey(privatekey)

If you really want to verify your keys are working correctly, you can also use m.decrypt() and m.encrypt() to encode and encode the data. You can click this link for another good overview about how to use pycrypto.

>>> from Crypto.PublicKey import RSA
>>> from Crypto import Random
>>> plaintext = "abcd"
>>> random_generator = Random.new().read
>>> encrypted_data = m.encrypt(plaintext, random_generator)
>>> m.decrypt(encrypted_data)

3. Next, it's not completely obvious how to use pycrypto to handle RSA signature signing. Most of the heavy lifting is done by the M2Crypto library, but if you're using pycrypto to do you may find yourself having to do some extra work to implement some of the PKCS algorithms.

For PKCS#1 v1.5 (which M2Crypto seems to be using), the private key is signed by the sender using the decryption process (not encryption) while the public key is used with the encryption process by the receiver to verify a signature, which is somewhat the opposite if you wanted to encrypt/decrypt data. (You can confirm this point by reviewing the Latex-based pycrypt.tex file).

In the M2Crypto world, RSA signature can be performed with the following lines:

>>> m = M2Crypto.RSA.load_key_string(privatekey)>>> digest = hashlib.sha1(plaintext).digest()>>> signature = m.sign(digest, "sha1")

In the PyCrypto world, you have to implement PKCS#1 v1.5 yourself. If you get "block type is not 01" when using M2Crypto to verify a signature create by PyCrypto, chances are M2Crypto is expecting the data to be encoded. You can review the RFC2313 spec to see what needs to be done to convert the signature with the correct padding. There is a padding spacing (ps) parameter that determines how many FF bytes should be added, which can be calculated by doing a number.size() call on your private key with the PyCrypto library and dividing by 8 bits (i.e. 2048 bits / 8 = 256 bytes)

The code below demonstrates how you would implement PKCS#1 v1.5 padding. Note that the private key is used to decrypt the data:

import hashlib

   def _emsa_pkcs1_v1_5_encode(M, emLen):
        SHA1DER = '\x30\x21\x30\x09\x06\x05\x2b\x0e\x03\x02\x1a\x05\x00\x04\x14'
        SHA1DERLEN = len(SHA1DER) + 0x14

        H = hashlib.sha1(M).digest()
        T = SHA1DER + H
        if emLen < (SHA1DERLEN + 11):
            raise Exception('intended encoded message length too short (%s)' % emLen)
        ps = '\xff' * (emLen - SHA1DERLEN - 3)
        if len(ps) < 8:
            raise Exception('ps length too short')
        return '\x00\x01' + ps + '\x00' + T

4. Currently, Google App Engine only supports pycrypto v2.3. You may encounter issues with using openssl-generated keys to do RSA signature verification. The bug reports for pycrypto v2.3 are posted here: https://bugs.launchpad.net/pycrypto/+bug/898466 https://bugs.launchpad.net/pycrypto/+bug/702835 A bug report has been issued here: http://code.google.com/p/googleappengine/issues/detail?id=5302 If you find issues trying to decrypt/encrypt data, you can bypass this issue by importing your own custom RSA.py version patched from the 2.3 version. Here is the diff that is needed to patch the v2.3 RSA.py version.

--- a/RSA.py
+++ b/RSA.py
@@ -206,7 +206,7 @@ class RSAImplementation(object):
def generate(self, bits, randfunc=None, progress_func=None):
if bits < 1024 or (bits & 0xff) != 0:              # pubkey.getStrongPrime doesn't like anything that's not a multiple of 128 and > 512
-            raise ValueError("RSA modulus length must be a multiple of 256 and > 1024")
+            raise ValueError("RSA modulus length must be a multiple of 256 and >= 1024")
rf = self._get_randfunc(randfunc)
obj = _RSA.generate_py(bits, rf, progress_func)    # TODO: Don't use legacy _RSA module
key = self._math.rsa_construct(obj.n, obj.e, obj.d, obj.p, obj.q, obj.u)
@@ -221,7 +221,8 @@ class RSAImplementation(object):
der.decode(externKey, True)
if len(der)==9 and der.hasOnlyInts() and der[0]==0:
# ASN.1 RSAPrivateKey element
-               del der[6:8]    # Remove d mod (p-1) and d mod (q-1)
+               del der[6:]     # Remove d mod (p-1), d mod (q-1), and q^{-1} mod p
+                der.append(inverse(der[4],der[5])) # Add p^{-1} mod q
del der[0]      # Remove version
return self.construct(der[:]

Hopefully this explains how you can get RSA encryption and signature signing working on Google App Engine without the M2Crypto library! (The last issue was not easy to find. In fact, it seemed very puzzling that RSA signature verification worked fine on local hosts, but never seemed to work on GAE instances.)