Saturday, April 12, 2014

Twilio and Heartbleed

Seeing these errors? It seems that when Twilio changed certificates after the Heartbleed incident, they also may have created issues with older httplib2 libraries that do incorrect cert validation.
CertificateHostnameMismatch: Server presented certificate that does not match host {'notAfter': 'Apr 10 23:59:59 2015 GMT', 'subjectAltName': (('DNS', ''), ('DNS', '*')), 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'California'),), (('localityName', u'San Francisco'),), (('organizationName', u'Twilio, Inc.'),), (('commonName', u'*'),))}
It turns out that the logic for the host check is pretty faulty. Since there are two alternate names defined ( and *, the for loop needs to iterate across the second one. However, because the regexp pattern fails on the 1st entry, it returns false and fails the validation.
(Pdb) host
(Pdb) hosts
['', '*']

for host in hosts:
    host_re = host.replace('.', '\.').replace('*', '[^
    if'^%s$' % (host_re,), hostname, re.I):
        return True
    return False
Upgrading to httplib2 v0.8 seems to have done the trick. Why? Note where the return call is made now:
for host in hosts:
    host_re = host.replace('.', '\.').replace('*', '[^.]*')
    if'^%s$' % (host_re,), hostname, re.I):
        return True
return False

Sunday, March 23, 2014

How RabbitMQ computes the name of its message queue directories...

One popular open source implementation of the AMQP messaging standard is RabbitMQ, which is currently supported by Pivotal/VmWare.  The code itself is written in Erlang, which is a programming language developed internally by Ericsson and released as open source later in 1994.  One of the best introductions about the language is a paper written by its initial creator, Joel Armstrong.

RabbitMQ relies on the Mnesia database system, which is a distributed store written in Erlang. RabbitMQ uses the system to persist information about virtual hosts, message queues, and exchanges. We can obviously use the rabbitmqctl command to query this information, but I wanted to understand how the queue directory names mapped to the ones listed in /var/lib/rabbitmq/mnesia:
$ ls -al /var/lib/rabbitmq/mnesia/rabbit@hostname/queues

drwxr-xr-x 2 rabbitmq rabbitmq 4096 Mar 17 04:27 3RG15Y3PJT7OHGG08CCU0Y7Z6
drwxr-xr-x 2 rabbitmq rabbitmq 4096 Mar 17 04:27 8LSP3194PK9RGC9PQTVOKKMQW
drwxr-xr-x 2 rabbitmq rabbitmq 4096 Mar 17 04:27 8XEM9YWU4AWY8YNC9KIW62NJW
To do so required learning a bit how Erlang works.  Through the process of trial-and-error, reading through several Erlang books and whatever could information posted online, I was able to understand how the language worked at a basic level.  The documentation inside the RabbitMQ source code was incredibly valuable, and using the Erlang shell allowed me to experiment and understand how RabbitMQ is implemented.

One of the big advantages of Erlang is its concurrency model.  When Erlang starts up a process, it creates a file called ~/.erlang_cookie. The contents of this file acts as a shared secret.  Erlang is built with concurrency in mind, so another Erlang process can exchange messages with other processes so long as they share this same cookie value.  When starting up a new Erlang process, we can use the -setcookie argument that should match.  In addition, we need to provide the short-hand name for this process using the -sname parameter to differentiate the name of the node (using the same node as another one will generate a conflict error).
erl -mnesia dir '"/tmp/tst"' -setcookie [COOKIE_CONTENTS] -sname tst
(An alternative is to simply copy the .erlang_cookie created by another process to your own home directory.  The RabbitMQ Ubutnu PPA repository provided also sets up a username 'rabbitmq'.  Any Erlang commands using this username will use this file.  The rabbitmqctl program switches to the rabbitmq user, which allows is to communicate with other Erlang processes running under that username).

We also want to make use of the functions that are available from the RabbitMQ. The compiled Erlang modules are located in /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/ebin, so we can also specify additional compiled code by adding this directory to the codepath search directory using the -pa argument.
erl -mnesia dir '"/tmp/tst"' -setcookie [COOKIE_CONTENTS] -sname tst -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/ebin
Once we've started up the process, we can communicate with the RabbitMQ process.  Inside our local Erlang process, we first need to load the data structures declared by RabbitMQ using the 'rr' command (read records).   These records are the equivalent of typedef struct declarations in C and declared as "-record" in the include .hrl files.
Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

Eshell V5.8.5  (abort with ^G)
1> rr ("/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/include/rabbit_msg_store.hrl").
The results returned are all the various data structures available to use.  If we want to see what records have been loaded, we can use the rl() function:
2> rl().
-record(amqp_error,{name,explanation = "",method = none}).
                  exclusive_owner = none,
                       routing_keys = [],
-record(binding,{source,key,destination,args = []}).
These record definitions are the data structures used by RabbitMQ.   We can use this information to make RPC calls to the rabbitmq node (the rabbit@hostname should match whatever name of the RabbitMQ process uses) This function call is equivalent of calling mnesia:schema(rabbit_durable_queue) locally:
rpc:call( 'rabbit@hostname', mnesia, schema, [rabbit_durable_queue] ).
To query the table of the rabbit_durable table,  we would use the mnesia:table() RPC call:
rpc:call( 'rabbit@hostname', mnesia, table, [rabbit_durable_queue] ). 
However, this query returns results that depend on using the qlc library.  We declare a lambda function that will query the table and generate a list.  The statement below amounts to a query result "Q such that Q equals instances of the amqqueue data structure instantiated by the Name and Pid columns from the rabbit_durable_queue table."  We then evaluate this query with the qlc:e() function, which converts the results to a list and use the function generated to make the RPC call.
Fun = fun() ->
                qlc:e(qlc:q([Q || Q = #amqqueue{name = Name,
                                                pid  = Pid}
                                      <- mnesia:table=(rabbit_durable_queue)]))

rpc:call( "rabbit@hostname", mnesia, transaction, [Fun]).
The result are instances of amqqueue instances returned from this query (The # in Erlang represents the instance of the amqqueue record with properties defined within the {} block):
#amqqueue{name = #resource{virtual_host = <<"myvhost_rhu">>,
                                                kind = queue,name = <<"archive">>},
                               durable = true,auto_delete = false,exclusive_owner = none,
                               arguments = [],pid = <5827.978.0="">,slave_pids = [],...},
We can use this information to figure out how the queue directory names in generated in /var/lib/rabbitmq/, which at first glance seem to be a string of random 25-byte characters.  Upon further inspection of the source code, there is a function called queue_name_to_dir_name in rabbit_queue_index.erl which takes as an input a resource record of type 'queue':
queue_name_to_dir_name(Name = #resource { kind = queue }) ->
    <<Num:128>> = erlang:md5(term_to_binary(Name)),
    rabbit_misc:format("~.36B", [Num]).
Note the use of the term_to_binary() call here.  It appears to be a way of serializing the data structure using the Erlang term format.  The binary result is then formatted into base36 format.   There is a helper function in rabbit_misc.erl that will generate an instance of a resource instance:
r(VHostPath, Kind, Name) ->
    #resource{virtual_host = VHostPath, kind = Kind, name = Name}.
What to input for these 3 parameters?  We only need to look at the #resource declaration of the results from the previous query.  We can use this information to compute the MD5 hash of the queue directory:
1> <<Num:128>> = erlang:md5(term_to_binary(rabbit_misc:r(<<"myvhost_rhu">>, queue, <<"archive">>))).
2> rabbit_misc:format("~.36B", [Num]).
Assuming messages are been sent to this queue, we should be able to find this matching directory inside the queues directory.  Inside each directory are .idx files that are the queue index that records the order of the messages within the disk.  We can use this information to study the internals of RabbitMQ's queue index defined in rabbit_queue_index.erl, which will be a subject for a later posting.

Wednesday, March 19, 2014

Installing IE7-IE11 on VirtualBox

Microsoft has long provided virtual images for IE6-IE11, but the installation often requires using a combination of curl and manual steps to setup.  The script provides an automated way to create VirtualBox images.   In addition, it also provides the option to save download time by reusing previously downloaded virtual images and upgrade to a specific browser version.  For instance, rather than using Windows Vista for IE7 image provided by Microsoft, the script can download the Windows XP for IE6 image and auto-upgrade the browser version to IE7.

To install, you can run:
IEVMS_VERSIONS="7 8 9 10 11" ./

If you abort your install or need to reinstall, there are a few places to check:

1. Go to File -> Virtual Media Manager.  Remove any of the .vmdk that you are trying to install.
2. Remove the .vbox file ( i.e. /Users/johnsmith/VirtualBox VMs/IE8 - WinXP/IE8 - WinXP.vbox)
3.  Remove the ~/.ievms directory.

This PR enhancement should helpfully improve the checking of incomplete downloads:

Wednesday, February 5, 2014

What socket error 536871025 means..

We started seeing these errors:

_librabbitmq.ConnectionError: Error opening socket: Unknown error 536871025

We use the same approach to determine what this number means
>>> hex(int(536871025))
>>> hex(int(536871025))
>>> int(0x71)

The socket error refers to No route to host..

Sunday, February 2, 2014

Troubleshooting Jenkins plugins in IntellIJ...

Having spent many sleepless nights over the past 2 years trying to fix random issues with Jenkins plug-ins that constantly are breaking, I finally got tired of the standard logging and reading source approach and decided to try to setup an environment that allowed for easy debugging. Jenkins's Wiki about setting up plug-ins was fairly unhelpful.  For instance, for JetBrains' IntelliJ IDE users, there are 2 sentences explaining how to setup a test environment:

"IntelliJ 7.0 (or later) users can load pom.xml directly from IDE, and you should see all the source code of libraries and Jenkins core all the way to the bottom. Consider installing IntelliJ IDEA plugin for Stapler to make the development easier."

Here is what I ended up doing with IntelliJ v13.  The nice part is that you can setup a Jenkins test environment for the plug-in very quickly.  You don't need to install Jenkins and then reinstall plugins each time you make a change.  You can even setup breakpoints that can be triggered while running a test Jenkins site.

1. Install Maven3 and create ~/.m2/settings.xml according to

2. git clone (or whatever plug-in you want to troubleshoot)

3. Edit the pom.xml to match the current Jenkins release you're using.  I found that I needed to test on v1.549 simply older versions were not able to support some of the newer plugins that have been breaking.
3. Open up IntelliJ and import the pom.xml file as a new Project.  The dependent modules should be listed.

4. Go to IntelliJ IDEA -> Preferences.  Find Maven.  Make sure all the appropriate directories are setup.  You shouldn't have to make any changes but the Maven home directory should be setup if you've correctly installed it.

5. Go to Tool Windows -> Maven Projects.

6. The Maven Projects window should appear on the right.  Make sure that the Jenkins profile is clicked.  Click on the package Lifestyle, which should start triggering the dependencies to be downloaded.

7. Once dependencies have been retrieved, the Plugins section should appear.  Find the hpi Plugin, which will allow you to spawn a Jetty server and setup breakpoints for your plug-in. Click on the hpi:run option in the dropdown.

8. You can then access http://localhost:8080/jenkins, which should give you the landing screen for Jenkins.

All the Jenkins-specific configurations are done within a work/ directory, so if you decide to open a new project for a different plug-in, you can setup a completely new Jenkins environment.  If you decide to make a code-change, just stop the debugging and restart.  The plug-in will be recompiled and the Jetty web server will be restarted.  Any breakpoints and variable inspection can also be done.

I used this approach to diagnose issues with Jenkins' git-client-plugin, which has had a whole host of problems recently.  For instance, there were several revisions leading up to v1.6.1 (which is currently the latest release) that simply didn't work -- the syntax to call Git was plain wrong.

The latest v1.6.2 snapshot has problems with Git repos that rely on HTTP-based credentials. One issue occurs in the shared use of a function that attempts to validate a repo URL. There are assumptions that an existing Git workspace exists, which aren't true if you're first trying to configure a new build job.  By setting breakpoints to where the offending issue was occurring, I was able to pinpoint quickly and introduce a PR to the discussion below:

Tuesday, January 28, 2014

Jenkins git-client-plugin v1.6.1 is broken..

If you've upgraded to the latest git-client-plugin (v1.6.1) and are reliant on HTTP-based credentials, you may have noticed this error message:
stderr: error: cannot run /bin/echo : No such file or directory
fatal: could not read Username for '': No such device or address

 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(
 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(
 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$200(
 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(
 at hudson.plugins.git.GitSCM.fetchFrom(
 ... 10 more
The issue has already been reported and the fix has already been implemented in the GitHub repo, but it hasn't made its way to an official release.

 The current workaround is to download the latest plugin here:

The red error message within Jenkins "Failed to connect to repository : Command "git config --local credential.helper store" may appear, but so long as the Console Output shows the checkout process working, you can safely ignore it.

Monday, December 30, 2013

Android NinePatch Image Pain

Since 2010, Android has provided 9-patch image support. It enables app developers to specify bitmap areas that can be stretched in order to allow text to fit inside graphics. This format allows custom graphics to be used as buttons with fairly quick ease.  Even iOS appears to have borrowed this concept and third-party libraries are available to leverage this format.

There are a multitude of sites explaining how to use the tool to create your own 9 patch images.  The documentation on Android's site provides a walkthrough about how to use the draw9patch tool to create new ones.  The idea of 9-patch is to define the areas that can be stretched both horizontally and vertically by drawing a line along the outside border.  If you also want to specify where the text should fix, you also need to define the region by creating lines along the right and bottom lines.

The draw9patch tool provides the ability to show how these areas are defined as you change the borders, but it often doesn't show you what the gotchas are.  I encountered some of these hiccups when trying to create custom speech bubbles with the Android Map Utils library.  Here was my first attempt to define the stretchable region:

Here's the result:

The arrow gets stretched too, so we actually want to define two stretchable regions:

The actual result?

Android's documentation says that if the bottom and right lines are not included, it uses the left and top lines.   It doesn't say what happens what happens if two stretchable regions are defined.   To avoid this confusion, we add a black line to the border below:

Now we get:

Also, if you're trying to understand how NinePatch internals works, you actually have to dig into NinePatchImpl.cpp.  The NinePatch implementation is actually a JNI interface to C++ code. The header file ResourceTypes.h is actually the best documented source of how the stretchable regions are defined internally.    The implementation code handles the magic of stretching the regions and proportionately scaling them.  In the example above,  only 1 pixels are defined on both left and right so they will be stretched uniformly.

Underlying all these internals is that Android relies on the Skia 2D library, so many of the esoteric units that start with Sk has to deal with this fact that much of the library deals in SkScalar metrics.   The magic happens with the calculateStretch() function, which figures out how many pixels to scale based on the total boundary size and the remaining number of stretchable regions.  It just turns out that much of Android's graphics library is built with Skia, so trying to understand how any of it works leads you down the path of reviewing the C++ code

Finally, keep in mind that the only way the draw9patch/Android knows that your images are 9-patch drawable is by the file extension.  If you save your custom button with rounded corners with the 9.png file extension, it's likely you'll find your corners cut off since it assumes that the outer 1px borders have been created to define the stretchable and content regions.   There's no magic number type, much less any type of header prefix, so be careful about naming of files.

Furthermore, there's also a source and compiled version of 9.png files.   If you try to retrieve the 9.png file from the .apk file, chances are it will not look like what's in your source code.