Thursday, August 28, 2014

Patching OpenSSL in Python

Ubuntu 14.04 comes with a newer version of OpenSSL 1.0.1, which can cause TLS Client Hello messages on some web servers to break (, especially when sending a handshake of more than 255 bytes.  With additional cipher suites and extensions, this number can easily exceed the expected amount.  By doing a tshark capture on port 443, we can see that the actual payload is 267 bytes:
tshark port 443

SSL Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.0 (0x0301)
        Length: 267
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 263
            Version: TLS 1.2 (0x0303)
                gmt_unix_time: Aug 26, 1973 10:35:41.000000000 UTC
                random_bytes: 46f445e55a639f227706f911bac2c2b312e64fa6f95dd630...
            Session ID Length: 0
            Cipher Suites Length: 86
            Cipher Suites (43 suites)

How to avoid this issue?  If you're using the Python requests library, one way is to take advantage of the PyOpenSSL extensions that will patch the underlying SSL implementation (  This way, you don't need to get into the nitty/gritty details of patching the code yourself.  

In order to do so, you must first install the libffi-dev package before doing a pip install:
pip install pyopenssl ndg-httpsclient pyasn1
The requests library automatically tries to import this library (see, but in order to fix the above issue, you may have to set the DEFAULT_SSL_CIPHER_LIST to MEDIUM instead of DEFAULT. Note: do not make this change unless you understand the implications to reduce the cipher strength. For more options, see
# Attempt to enable urllib3's SNI support, if possible
from requests.packages.urllib3.contrib import pyopenssl
One other issue to note: pip installs also rely on the requests library, so adding this change will also enable certificate validation.  Be sure to be using pip v1.5.6 since older pip installs have 301/302 redirect issues when using this package.

Addendum: you can download and use to probe the supported ciphers (i.e. python --sslv3 AES-128 and AES-256 based encryption may need to be added to your cipher list.

Saturday, August 23, 2014

Fixing the Jenkins scm-sync-configuration plugin

I've often found that anytime you upgrade Jenkins versions, the plug-ins that accompany it often break.  The other day, the scm-sync-configuration plugin that helps manage and store the XML-based configuration files stopped working after upgrading to Ubuntu 14.04 and the Jenkins package included with it.  The plugin appeared to copy files to the scm-sync-configuration/ directory but never seemed to commit the changes to the Git repo.  What was wrong?

It turns out that the issue was already reported since April 2014.  The workaround for most people was to downgrade from Git 1.9, which comes with the Ubuntu 14.04 installs, to Git 1.8 versions.   It so happens that the maven-scm-provider-git plugin on which this Jenkins plugin depends attempts to parse the output of the "git status".  When the formatting change, it broke the ability to ascertain what changes had been made.  The related ticket is here:

The issue for this Maven project was fixed subsequently, but the dependent packages that the scm-sync-configuration plugin had to be updated.  Furthermore, you had to check that the maven-scm-api that has this fix actually got updated to 1.9.1, since missing one upgrade dependency meant that it would stay at an older version.

The changes for this fix are here:

The other issue, which was much less obvious, was that there's a related XML file that determines what source code control systems are supported.  For some reason, only Subversion (SVN) was declared, which broke tests for the Git-related ones.   It wasn't so obvious until several hours of tracing through the source code and noticing that the ScmManager class only had one ScmProvider declared.  Apparently this plugin uses a package called Plexus to load this information, which relies on XML configurations such as this file.

Wednesday, August 20, 2014

Running RabbitMQ on Ubuntu 14.04?

Recently during a few test runs we noticed that our AMQP clients would often lock up when interfacing with a RabbitMQ machine running on Ubuntu 14.04 hosted on Amazon AWS.  By installing tshark and capturing the network traces on the host running the AMQP client, I noticed there seemed to be issues reported as "Unknown frame type".  Upon further investigation of the unknown frame packet, I noticed that we seemed to be missing the AMQP header, causing the packet to be classified as unknown by Wireshark.  Normally there should have been Basic.Deliver like all the previous packets:

I noticed on the machine running on RabbitMQ that there were "xen_netfront: xennet: skb rides the rocket" error messages.  It turns out that AMQP is very sensitive to missing frame headers and caused both RabbitMQ and the AMQP client to wait on each other.  The broker thinks there are more messages waiting to be acknowledged, while the client sees no new messages.  The reason appears to be that the missing AMQP header caused by the dropped transmit packet causes this issue:

The solution appears to be disable scatter/gather I/O on the network card, since the Xen driver may take longer to resolve:

sudo ethtool -k eth0

The patch that appears to have started contributing to these issues:

Tuesday, August 19, 2014

Stepping through a running Python program through gdb

The main part is that you need to have your custom Python and gdb build.   The Ubuntu Python version has compiler optimizations removed, which makes it difficult to use. You'll also need to install all your package dependencies using the virtual environment that you've setup.

1. Build your own custom Python version.  Instructions are here.

2. If you're using Ubuntu 14.04, you need to compile your own gdb too, since the one that comes is linked to Python 3:

a. Download the source from

b. sudo apt-get install libncurses5-dev texinfo

c. ./configure --with-python=python2 --prefix=/home/rhu/special (or wherever you intend to do make install).

d. make install

3. Locate the from your Python install.

a. sudo /home/rhu/special/bin/gdb -p <python process>

b. python
sys.path.insert(0, "/home/rhu/projects")
import libpython

You should now be able to do py-up, py-down, py-list, py-locals to step through the various call stacks.

Solving the "had used a different Landroid/support/v4/app/FragmentActivity; during pre-verification" error

This Stack Overflow pointed to issues with including the support-v4.jar twice.

The solution?

If you're using Roboelectric, the support-v4.jar will likely be specified as a compilation dependency.  If you see "(Lcom/example/android/app/FragmentLayoutSupport; had used a different Landroid/support/v4/app/FragmentActivity; during pre-verification) error if it is not excluded.", it's likely it needs to be defined as part of your exclude settings.

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])

    // borrowed from
    androidTestCompile 'org.hamcrest:hamcrest-integration:1.1'
    androidTestCompile 'org.hamcrest:hamcrest-core:1.1'
    androidTestCompile 'org.hamcrest:hamcrest-library:1.1'

    androidTestCompile('junit:junit:4.+') {
        exclude module: 'hamcrest-core'
    androidTestCompile('org.robolectric:robolectric:2.3') {
        exclude module: 'classworlds'
        exclude module: 'commons-logging'
        exclude module: 'httpclient'
        exclude module: 'maven-artifact'
        exclude module: 'maven-artifact-manager'
        exclude module: 'maven-error-diagnostics'
        exclude module: 'maven-model'
        exclude module: 'maven-project'
        exclude module: 'maven-settings'
        exclude module: 'plexus-container-default'
        exclude module: 'plexus-interpolation'
        exclude module: 'plexus-utils'
        exclude module: 'support-v4' // avoid support v4 DEX conflicts
        exclude module: 'wagon-file'
        exclude module: 'wagon-http-lightweight'
        exclude module: 'wagon-provider-api'

Wednesday, July 23, 2014

What every engineer should know about Gradle when using Android Studio

Having used Android Studio since it was version 0.3, I've dealt with the growing pains of the integration of the new build system, Gradle, for quite a bit of time.  It used to be that you could create Android Studio projects without using Gradle -- now you have no choice but to use the build system.  However, thanks to recent improvements dating back to recent releases since February 2014, you also no longer have to modify the Gradle file directly to accommodate any additional library files since Android Studio tries to handle things for you.

Regardless, here are a few things that I thought are useful to know if you're using Gradle and Android Studio:

1. There are multiple build.gradle files.  When you create a new project, Android Studio creates two build.gradle files. The first is the parent that encompasses all modules in your project, while the 2nd is the one that pertains directly to the app. You will in most cases be dealing with this second build.gradle file. You do not really need to make any changes to the 1st one, which is blank by default.

2. Adding JAR files: When you drop a file into the designated libs/ directory, the changes are usually updated in the local build.gradle file.  You can see the changes reflected in the dependencies section.  For instance, if you drop the android-async-http-1.4.3.jar, it gets explicitly added to the dependency tree, even though the first statement should suffice:

dependencies {
    compile fileTree(dir: 'libs', include: ['*.jar'])
    compile files('libs/android-async-http-1.4.3.jar')
The addition of the 2nd is redundant because of the 1st statement -- however, if you add a library file outside the libs/ directory, presumably it will be automatically added to this list.

3. Removing JAR files: When you delete the JAR file, Android Studio currently not does handle removal of these .JAR files in the build.gradle file.  If you intend to remove one, you should check the build.gradle to see if you also need to delete the line that deals with importing the JAR file.

4. Adding library dependencies: If you're trying to add another library such as the Facebook SDK, which will usually reside outside the scope of your main app, you need to update the settings.gradle in the parent directory to reference this module.  For instance, if you copy the contents into a Libraries/facebook directory, you need to have the 2nd extra include statement:
include ':app'
include ":Libraries:facebook"
If your app intends to depend on this SDK, you also need to reference it inside the build.gradle of your app:
dependencies {
    compile project(':Libraries:facebook')

If you try to use the compile project statement without the changes to settings.gradle, you will likely see an error about "Project with path :Libraries:facebook could not be found in project :app".  If you want a working example, check out Jonathan Azoff's repo:

For more context, see the section on Multi project setup at

5. Use the Gradle wrapper option. When you select the option, you are effectively generating a few files that enable other people who check out your project not to have to install Gradle themselves once you've generated the files needed to bootstrap the process.  It is the recommended that you check these files into your source code to allow others to be able to compile your program successfully.

The exact Gradle version that will be downloaded and used to build your app is defined in the gradle/wrapper/ file. The downloaded version is usually cached in ~/.gradle/wrapper/dists.

6. Unsure about whether Android Studio is creating build issues?  If you'd like to see if the gradle build succeeds, you can always run things by executing Gradle at the command line.  Within your project, you can type "bash gradlew assembleDebug" or "gradlew.bat assembleDebug", which will attempt to compile and build your Android app.

7. Seeing Duplicate files copied in APK errors?  Double-check your build.gradle files in both your main Android app and your libraries.  Your main app should be using the 'android' plug-in, while your libraries should be using the android-library plugin.

I made the mistake of trying to define all libraries with the 'android' plugin, triggering this issue possibly since Gradle didn't know how to resolve duplicate AndroidManifest.xml and other shared library files.

For instance, your main app should have the android plugin:

buildscript {
    repositories {
    dependencies {
        classpath ''
apply plugin: 'android'

Your other libraries should be using the android-library plugin:
buildscript {
    repositories {
    dependencies {
        classpath ''
apply plugin: 'android-library'
You can also take a look at the explanation at that covers the different between a project and a library project.  A project will generate an Android Application Package (APK) file, while a library project will generate an Android ARchive (AAR) file.  Since you can only have one APK file, it makes sense that you're only allowed to define one project.  Android Studio currently doesn't prevent you from making this mistake however.

8. New versions of Gradle may require the Android Gradle plugin to be updated too.  

When you add the "apply plugin: 'android'", you are in fact instructing Gradle to use the Android Gradle plugin, which Google maintains at   This plugin helps inform Gradle where to find your Android SDK, which is usually defined in

The evolving nature of Gradle and the Android Gradle plugin means that the versions you need may need to be updated. You should always check the gradle/wrapper/ and the build.gradle file to verify that the versions you are attempting to use are compatible.  Check the Android tools link to verify.

9. Seeing cryptic "failed to find Build Tools revision" errors?

Double-check the Android SDK Build Tools version specified in your buildToolsVersion reflects the one that's installed i Android SDK Manager.  You may need to tweak your buildToolsVersion to be updated to one that you have installed.    Similarly, you may also need to do the same for the compileSdkVersion definition.

android {
    buildToolsVersion "19.1.0"

10. Still confused?  Read the writeup of new Android Gradle plugin build system.

Friday, July 18, 2014

Where Google keeps the Gradle source code..

Go to:

Instead of downloading the entire Android source code base, you can click on one of the branches and click on the 'tgz' link to download a snapshot of the tree.