Experiences with Celery v3.1

Our experiences with Celery v3.1 so far:

http://engineering.hearsaysocial.com/2014/05/21/experiences-with-celery-v31/

Celery v3.1 asynchronous loops

I've written an entry here about Celery v3.1 and issues with the current async loop design:

http://engineering.hearsaysocial.com/2014/05/02/celery-v31-and-async-loops/

Pull request here:

https://github.com/celery/kombu/pull/353

Twilio and Heartbleed

Seeing these errors? It seems that when Twilio changed certificates after the Heartbleed incident, they also may have created issues with older httplib2 libraries that do incorrect cert validation.

CertificateHostnameMismatch: Server presented certificate that does not match host api.twilio.com: {'notAfter': 'Apr 10 23:59:59 2015 GMT', 'subjectAltName': (('DNS', 'twilio.com'), ('DNS', '*.twilio.com')), 'subject': ((('countryName', u'US'),), (('stateOrProvinceName', u'California'),), (('localityName', u'San Francisco'),), (('organizationName', u'Twilio, Inc.'),), (('commonName', u'*.twilio.com'),))}

It turns out that the logic for the host check is pretty faulty. Since there are two alternate names defined (twilio.com and *.twilio.com), the for loop needs to iterate across the second one. However, because the regexp pattern fails on the 1st entry, it returns false and fails the validation.

(Pdb) host
'twilio.com'
(Pdb) hosts
['twilio.com', '*.twilio.com']

for host in hosts:
    host_re = host.replace('.', '\.').replace('*', '[^
    if re.search('^%s$' % (host_re,), hostname, re.I):
        return True
    return False

Upgrading to httplib2 v0.8 seems to have done the trick. Why? Note where the return call is made now:

for host in hosts:
    host_re = host.replace('.', '\.').replace('*', '[^.]*')
    if re.search('^%s$' % (host_re,), hostname, re.I):
        return True
return False

How RabbitMQ computes the name of its message queue directories...

One popular open source implementation of the AMQP messaging standard is RabbitMQ, which is currently supported by Pivotal/VmWare.  The code itself is written in Erlang, which is a programming language developed internally by Ericsson and released as open source later in 1994.  One of the best introductions about the language is a paper written by its initial creator, Joel Armstrong.

RabbitMQ relies on the Mnesia database system, which is a distributed store written in Erlang. RabbitMQ uses the system to persist information about virtual hosts, message queues, and exchanges. We can obviously use the rabbitmqctl command to query this information, but I wanted to understand how the queue directory names mapped to the ones listed in /var/lib/rabbitmq/mnesia:

$ ls -al /var/lib/rabbitmq/mnesia/rabbit@hostname/queues

drwxr-xr-x 2 rabbitmq rabbitmq 4096 Mar 17 04:27 3RG15Y3PJT7OHGG08CCU0Y7Z6
drwxr-xr-x 2 rabbitmq rabbitmq 4096 Mar 17 04:27 8LSP3194PK9RGC9PQTVOKKMQW
drwxr-xr-x 2 rabbitmq rabbitmq 4096 Mar 17 04:27 8XEM9YWU4AWY8YNC9KIW62NJW

To do so required learning a bit how Erlang works.  Through the process of trial-and-error, reading through several Erlang books and whatever could information posted online, I was able to understand how the language worked at a basic level.  The documentation inside the RabbitMQ source code was incredibly valuable, and using the Erlang shell allowed me to experiment and understand how RabbitMQ is implemented.

One of the big advantages of Erlang is its concurrency model.  When Erlang starts up a process, it creates a file called ~/.erlang_cookie. The contents of this file acts as a shared secret.  Erlang is built with concurrency in mind, so another Erlang process can exchange messages with other processes so long as they share this same cookie value.  When starting up a new Erlang process, we can use the -setcookie argument that should match.  In addition, we need to provide the short-hand name for this process using the -sname parameter to differentiate the name of the node (using the same node as another one will generate a conflict error).

erl -mnesia dir '"/tmp/tst"' -setcookie [COOKIE_CONTENTS] -sname tst

(An alternative is to simply copy the .erlang_cookie created by another process to your own home directory.  The RabbitMQ Ubutnu PPA repository provided also sets up a username 'rabbitmq'.  Any Erlang commands using this username will use this file.  The rabbitmqctl program switches to the rabbitmq user, which allows is to communicate with other Erlang processes running under that username).

We also want to make use of the functions that are available from the RabbitMQ. The compiled Erlang modules are located in /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/ebin, so we can also specify additional compiled code by adding this directory to the codepath search directory using the -pa argument.

erl -mnesia dir '"/tmp/tst"' -setcookie [COOKIE_CONTENTS] -sname tst -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/ebin

Once we've started up the process, we can communicate with the RabbitMQ process.  Inside our local Erlang process, we first need to load the data structures declared by RabbitMQ using the 'rr' command (read records).   These records are the equivalent of typedef struct declarations in C and declared as "-record" in the include .hrl files.

Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0] [kernel-poll:false]

Eshell V5.8.5  (abort with ^G)
1> rr ("/usr/lib/rabbitmq/lib/rabbitmq_server-3.2.4/include/rabbit_msg_store.hrl").
[amqp_error,amqqueue,basic_message,binding,content,delivery,
 event,exchange,exchange_serial,internal_user,listener,
 message_properties,msg_location,permission,plugin,resource,
 reverse_binding,reverse_route,route,runtime_parameters,
 ssl_socket,topic_trie_binding,topic_trie_edge,
 topic_trie_node,trie_binding,trie_edge,trie_node,user,
 user_permission|...]
2>

The results returned are all the various data structures available to use.  If we want to see what records have been loaded, we can use the rl() function:

2> rl().
-record(amqp_error,{name,explanation = "",method = none}).
-record(amqqueue,{name,
                  durable,
                  auto_delete,
                  exclusive_owner = none,
                  arguments,
                  pid,
                  slave_pids,
                  sync_slave_pids,
                  policy,
                  gm_pids,
                  decorators}).
-record(basic_message,{exchange_name,
                       routing_keys = [],
                       content,
                       id,
                       is_persistent}).
-record(binding,{source,key,destination,args = []}).
-record(content,{class_id,
                 properties,
                 properties_bin,
                 protocol,
                 payload_fragments_rev}).
-record(delivery,{mandatory,sender,message,msg_seq_no}).
-record(event,{type,props,timestamp}).
.
.
.

These record definitions are the data structures used by RabbitMQ.   We can use this information to make RPC calls to the rabbitmq node (the rabbit@hostname should match whatever name of the RabbitMQ process uses) This function call is equivalent of calling mnesia:schema(rabbit_durable_queue) locally:

rpc:call( 'rabbit@hostname', mnesia, schema, [rabbit_durable_queue] ).

To query the table of the rabbit_durable table,  we would use the mnesia:table() RPC call:

rpc:call( 'rabbit@hostname', mnesia, table, [rabbit_durable_queue] ). 

However, this query returns results that depend on using the qlc library.  We declare a lambda function that will query the table and generate a list.  The statement below amounts to a query result "Q such that Q equals instances of the amqqueue data structure instantiated by the Name and Pid columns from the rabbit_durable_queue table."  We then evaluate this query with the qlc:e() function, which converts the results to a list and use the function generated to make the RPC call.

Fun = fun() ->
                qlc:e(qlc:q([Q || Q = #amqqueue{name = Name,
                                                pid  = Pid}
                                      <- mnesia:table(rabbit_durable_queue)]))
end,
rpc:call( "rabbit@hostname", mnesia, transaction, [Fun]).

The result are instances of amqqueue instances returned from this query (The # in Erlang represents the instance of the amqqueue record with properties defined within the {} block):

#amqqueue{name = #resource{virtual_host = <<"myvhost_rhu">>,
                                                kind = queue,name = <<"archive">>},
                               durable = true,auto_delete = false,exclusive_owner = none,
                               arguments = [],pid = <5827.978.0="">,slave_pids = [],...},

We can use this information to figure out how the queue directory names in generated in /var/lib/rabbitmq/, which at first glance seem to be a string of random 25-byte characters.  Upon further inspection of the source code, there is a function called queue_name_to_dir_name in rabbit_queue_index.erl which takes as an input a resource record of type 'queue':

queue_name_to_dir_name(Name = #resource { kind = queue }) ->
    <<Num:128>> = erlang:md5(term_to_binary(Name)),
    rabbit_misc:format("~.36B", [Num]).

Note the use of the term_to_binary() call here.  It appears to be a way of serializing the data structure using the Erlang term format.  The binary result is then formatted into base36 format.   There is a helper function in rabbit_misc.erl that will generate an instance of a resource instance:

r(VHostPath, Kind, Name) ->
    #resource{virtual_host = VHostPath, kind = Kind, name = Name}.

What to input for these 3 parameters?  We only need to look at the #resource declaration of the results from the previous query.  We can use this information to compute the MD5 hash of the queue directory:

1> <<Num:128>> = erlang:md5(term_to_binary(rabbit_misc:r(<<"myvhost_rhu">>, queue, <<"archive">>))).
2> rabbit_misc:format("~.36B", [Num]).
"EUVCFMQ3KCK9L8KMFN5Q0WBQR"

Assuming messages are been sent to this queue, we should be able to find this matching directory inside the queues directory.  Inside each directory are .idx files that are the queue index that records the order of the messages within the disk.  We can use this information to study the internals of RabbitMQ's queue index defined in rabbit_queue_index.erl, which will be a subject for a later posting.

Installing IE7-IE11 on VirtualBox

Microsoft has long provided virtual images for IE6-IE11, but the installation often requires using a combination of curl and manual steps to setup.  The iemvs.sh script provides an automated way to create VirtualBox images.   In addition, it also provides the option to save download time by reusing previously downloaded virtual images and upgrade to a specific browser version.  For instance, rather than using Windows Vista for IE7 image provided by Microsoft, the script can download the Windows XP for IE6 image and auto-upgrade the browser version to IE7.

To install, you can run:

IEVMS_VERSIONS="7 8 9 10 11" ./ievms.sh

If you abort your install or need to reinstall, there are a few places to check:

1. Go to File -> Virtual Media Manager.  Remove any of the .vmdk that you are trying to install.
2. Remove the .vbox file ( i.e. /Users/johnsmith/VirtualBox VMs/IE8 - WinXP/IE8 - WinXP.vbox)
3.  Remove the ~/.ievms directory.

This PR enhancement should helpfully improve the checking of incomplete downloads:

https://github.com/xdissent/ievms/pull/213

What socket error 536871025 means..

We started seeing these errors:

_librabbitmq.ConnectionError: Error opening socket: Unknown error 536871025

We use the same approach to determine what this number means http://hustoknow.blogspot.com/2013/03/what-536871023-means.html...

>>> hex(int(536871025))
'0x20000071'
>>> hex(int(536871025))
'0x20000071'
>>> int(0x71)
113

The socket error refers to No route to host..

Troubleshooting Jenkins plugins in IntellIJ...

Having spent many sleepless nights over the past 2 years trying to fix random issues with Jenkins plug-ins that constantly are breaking, I finally got tired of the standard logging and reading source approach and decided to try to setup an environment that allowed for easy debugging. Jenkins's Wiki about setting up plug-ins was fairly unhelpful.  For instance, for JetBrains' IntelliJ IDE users, there are 2 sentences explaining how to setup a test environment:

"IntelliJ 7.0 (or later) users can load pom.xml directly from IDE, and you should see all the source code of libraries and Jenkins core all the way to the bottom. Consider installing IntelliJ IDEA plugin for Stapler to make the development easier."

Here is what I ended up doing with IntelliJ v13.  The nice part is that you can setup a Jenkins test environment for the plug-in very quickly.  You don't need to install Jenkins and then reinstall plugins each time you make a change.  You can even setup breakpoints that can be triggered while running a test Jenkins site.

1. Install JDK 8.  JRE installed will not allow you to debug or run tests.

2. Install Maven3 according to https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial.

brew install maven

3. git clone git@github.com:jenkinsci/git-client-plugin.git (or whatever plug-in you want to troubleshoot)

4. Edit the pom.xml to match the current Jenkins release you're using.  I found that I needed to test on v1.549 simply older versions were not able to support some of the newer plugins that have been breaking.

  <parent>
    <groupId>org.jenkins-ci.plugins</groupId>
    <artifactId>plugin</artifactId>
    <version>1.549</version>
  </parent>

3. Open up IntelliJ and import the pom.xml file as a new Project.  The dependent modules should be listed.

5. Go to IntelliJ IDEA -> Preferences.  Find Maven.  Make sure all the appropriate directories are setup.  You shouldn't have to make any changes but the Maven home directory should be setup if you've correctly installed it.

6. Go to Tool Windows -> Maven Projects.

7. The Maven Projects window should appear on the right.  Make sure that the Jenkins profile is clicked.  Click on the package Lifestyle, which should start triggering the dependencies to be downloaded.

8. Once dependencies have been retrieved, the Plugins section should appear.  Find the hpi Plugin, which will allow you to spawn a Jetty server and setup breakpoints for your plug-in.  Right click on the hpi:run option in the menu and run as Debug mode.

9. You can then access http://localhost:8080/jenkins, which should give you the landing screen for Jenkins.

All the Jenkins-specific configurations are done within a work/ directory, so if you decide to open a new project for a different plug-in, you can setup a completely new Jenkins environment.  If you decide to make a code-change, just stop the debugging and restart.  The plug-in will be recompiled and the Jetty web server will be restarted.  Any breakpoints and variable inspection can also be done.

I used this approach to diagnose issues with Jenkins' git-client-plugin, which has had a whole host of problems recently.  For instance, there were several revisions leading up to v1.6.1 (which is currently the latest release) that simply didn't work -- the syntax to call Git was plain wrong.

The latest v1.6.2 snapshot has problems with Git repos that rely on HTTP-based credentials. One issue occurs in the shared use of a function that attempts to validate a repo URL. There are assumptions that an existing Git workspace exists, which aren't true if you're first trying to configure a new build job.  By setting breakpoints to where the offending issue was occurring, I was able to pinpoint quickly and introduce a PR to the discussion below:

https://issues.jenkins-ci.org/browse/JENKINS-21016