Hus to Know?

Tricks of using the SGE2010 switch

2012-02-21T09:09:00.003-08:00

1. Telnet to the IP address.

2. Enable the SSH server.

3. SSH into the switch.

4. Hit Ctrl-Z. You will then be dropped into the shell. Type 'lcli'.

5. You can now access the Cisco switch using the CLI interface. To save config changes, type "copy running-config startup-config".

X-Forwarded-Host to X_FORWARDED_HOST in mod_wsgi..

2012-02-18T16:17:00.001-08:00

Ever wondered how the parameters in Django's request.META, such as X-Forwarded-Host or X-Forwarded-For get set in the os.environ objects? The headers often get transformed to entire upper-case and the dashes get replaced by underscores (i.e. X-Forwarded-Host becomes X_FORWARDED_HOST). Most of the work gets done inside mod_wsgi.c using the wsgi_http2env() function.


static PyObject *Auth_environ(AuthObject *self, const char *group)
{
            PyDict_SetItemString(vars, wsgi_http2env(r->pool, hdrs[i].key),
                                 object);

The wsgi_http2env() function basically performs this conversion to HTTP environment variables:


static char *wsgi_http2env(apr_pool_t *a, const char *w)
{
    char *res = (char *)apr_palloc(a, sizeof("HTTP_") + strlen(w));
    char *cp = res;
    char c;

    *cp++ = 'H';
    *cp++ = 'T';
    *cp++ = 'T';
    *cp++ = 'P';
    *cp++ = '_';

    while ((c = *w++) != 0) {
        if (!apr_isalnum(c)) {
            *cp++ = '_';
        }
        else {
            *cp++ = apr_toupper(c);
        }
    }
    *cp = 0;

    return res;
}

Integrating Google Apps with Apache module

2012-02-18T16:13:00.000-08:00

Recntly, we've been using https://github.com/epotocko/apache-google-apps-sso to implement single-sign-on for a lot of our Apache-based resources for Google Apps/OpenID integration. The documentation are pretty clear and discuss how to go about protecting certain URL paths, but what if you want to require authentication to occur for the entire site?

The trick is to use a RewriteRule and mod_rewrite to basically stop Auth_memCookie from being needed for /auth/ logins, which is where the PHP code will be executed to perform the authentication. The 'L' flag in the RewriteRule will stop the rest of the rulesets from executing.

RewriteEngine on
RewriteRule ^/auth/(.*)$ /auth/$1 [L]

<IfModule mod_auth_memcookie.c>
<Location />
AuthType Cookie
.
.
.
ProxyPass http://127.0.0.1:9999/
ProxyPassReverse http://127.0.0.1:9999/

</Location>
</IfModule>
The ProxyPass is used to proxy traffic to an internal webserver host, and the ProxyPassReverse rewrites Location: headers. If you are using Django, an HttpResponseRedirect() function call will usually set the Location: header based on the X-Forwarded-Host but because if you are not using mod_wsgi(), the header may not be correctly set. The ProxyPassReverse provides a way to rewrite redirection requests for this reason.
The other thing is to make sure that your base URL is also set with this /auth/ path:

// Base url to protect
GApps_Session::$BASE_URL = 'https://myhost.com/auth/';

How Browsers Work

2012-02-14T10:23:00.000-08:00

http://www.html5rocks.com/en/tutorials/internals/howbrowserswork/

Replacing the motor on a Maytag MDE3000AYW dryer...

2012-01-28T22:46:00.000-08:00

About two weeks ago, one of these 1999 Maytag dryer models suddenly stopped working. You could push the Start button but all you would get is a low-hum. Suspecting that the motor died, I decided to open up the unit up to see if I could confirm the issue myself. NOTE: apparently if you try to call a Maytag service person at this point, they'll actually charge you extra for being forced to figure out what you did! It actually took at least 5+ hours to get this whole unit disassembled and reassembled, so also don't try if you really have a pressing need to clean your dirty underwear. A technician may work faster too if you don't have all the tools are your disposal.

Maytag still keeps its information about these models online, and this particular model the links are posted below. However, you won't find any repair guide, since the company normally recommends that you contact an authorized technician. Nonetheless, you can get a list of parts information in case you want to get a rough estimate about the cost. Pandora OEM, which sells off Amazon.com, can deliver your parts from Fresno, CA within a few days, and they even respond to email inquiries on weekends so double-check with them about your model before you order your parts.

https://www.maytag.com/digitalassets/MLPDF/Use%20and%20Care%20Guide%20-%206-3702450.pdf (Owner's Manual)

https://www.maytag.com/digitalassets/MLPDF/Dimension%20Guide%20-%20DNDRY.pdf (Dimensions Guide)

https://www.maytag.com/digitalassets/MLPDF/Repair%20Part%20List%20-%20MDE3000AYW.pdf (Parts List)

You can however find the service manual by doing a Google search for the Dryer Service Manual (16023110.pdf) for this model. Background info: Apparently the way these models are named is that M represents the brand, DE represents an electric dryer (DG is gas dryer), 3000 represents the model no, A is the production code, Y = 240 volts, and W represents the color (white on white).

If you want some context about how to disassemble dryers for Maytag models, there are several YouTube clips that provide a good overview (PartSelect.com and RepairClinic.com do an excellent job explaining the step-by-step process). A dryer consists of a pulley system, a belt, motor, a cynlindrical tumbler for turning your clothes, a heating element, and a fan (along with a thermal fuses to keep you from burning the whole unit up). The videos make things a lot easier than they seem, but many of the aspects of taking apart a dryer especially related to the retaining clips and clamps were fairly consistent even with this older 1999 model.

A brief overview of many of the tools that I ended up using:

- Safety goggles and worker gloves
- Ohm-meter (for continuity testing)
- 5/16" socket wrench (and 7/8" if you actually want to pull off the motor pulley)
- Screwdriver
- Adjustable wrench
- Snap ring pliers (for removing the snap ring on the blower wheel)
- Vice grips (for removing the blower wheel clamp)
- Replacement motor w/ a motor pulley
- Optional: blower wheel & replacement belt (the blower wheel can be a pain to remove since it may be fused to the motor shaft after many years of use).

Disclaimer: do not try any of this work without the plug for the dryer disconnected. You may also want to verify that your circuit breaker didn't accidentally trip and therefore is actually supplying no power to your unit (if you see a lamp on when the lid is open, chances are there is power). In addition, verify that the thermal fuse simply needs replacing. The thermal fuse can be seen at the bottom of the dryer, and you can use an ohm-meter to verify that there is still connectivity. In other words, check everything before you decide to replace the motor since it's a fairly involved procedure.

In order to get to the motor, you need to first remove the hinges from the door (see Chapter 5 - Cabinet Assembly Components). The hinges can be removed with a screwdriver, and the door can be pulled from the slots in the wall. You can then remove the outer shell of the once these hinges and the screws attached to the door are removed.

Once you've removed the front door (see Section 6 on Tumbler & Bulkheads in the service manual) you can remove the front bulkhead by starting to remove many of the screws that you see. What's confusing is that the instruction manual says "Remove the lowest outside screw at each corner and then the top screw on each side") in step 4. What the manual is really saying is that there are another set of screws holding the tumbler in place, so the manual wants you to remove the screws that aren't attached directly to the tumbler. By doing so, you'll disconnect the first layer of the dryer before you get to the tumbler.

There are also "hold down brackets" shaped as a horse-shoe that keep you from being able to lift the lid of the dryer. Once you unscrew the bolt from the bracket with a 5/6" socket wrench, you swing the bracket to the side (see Figure 5-5). Chances are the bracket will fall out, so it's better just to stow them away somewhere.

The manual covers how to remove most of the front assembly, but basically you need to remove a lot of screws to even get to the tumbler and then to the motor. You'll notice that the tumbler actually is held in the front and the moment you start removing the screws, it may start to no longer align with the back of the dryer cabinet.

The tumbler itself may need to be removed, which requires you to crouch down and try to locate several pulleys in the back. One is the motor pulley, which is attached the motor. The other is known as the idler pulley, which is attached a spring and helps provide the tension of the belt. You should be able to pull the idler pulley to the left, which will remove the tension from the belt and allow you to take it. Consult the YouTube clips to see how it's done. This model's instruction manual specifies explicitly that the belt should be ribbed-side down on the tumbler, whereas other models seem to indicate the opposite.

Once the tumbler is removed, you'll eventually get to the blower wheel, which amounts to a small fan at the bottom of the dryer. You need to remove the blower wheel since it's attached to the motor shaft and the motor can't be removed until you detach this blower wheel. Basically there are two clamps attached to the blower wheel: the retaining clip and clamp. The retaining clamp usually requires snap-ring pliers to remove (Home Depot sells them for $20, but you have to make sure that the pliers can fit into the small holes of this retaining clip. Orchard Supply Hardware carried a pair but the attachments were a bit too large.) The clamp itself can be removed with vise grips. If you squeeze the outer protruding edges, the clamp will actually expand (the video clips may show this point, but it's not exactly obvious since it happens so fast).

Make sure to keep the retaining clip and clamp, since they can be reused when reinstalling the blower wheel. Once you've successfully removed the retaining clip and clamp, you'll need to remove the blower wheel itself. The instructions in the service manual say to "work blower wheel back and forth while pulling off shaft...[and] may require considerable effort to remove the blower wheel." The wheel itself may be somewhat fused to the shaft after so many years of use, so adding some lubricant (careful not to spray it elsewhere in the dryer, since the substance may be flammable) to help remove the wheel. A replacement blower wheel can cost $20-$30, but makes it much easier to attach to a replacement motor so I'd highly recommend replacing it too if you're also planning to change the motor.

Once the tumbler blower wheel is removed, you should be able to see the motor. There should be a plastic connector adapter that is attached to the motor that needs to be removed by pushing on both sides. The connector is keyed to only plug-in a certain way, so you shouldn't have to remember which way to reinsert the connector when replacing the motor. The service manual contains the resistance that you could measure from the controller panel to the terminal (i.e 3.00 Ohms from terminal 3 of the start switch to terminal 4), but I found opening another panel to be extra hassle and decided to just replace the battery to see if it would solve the problem. There are YouTube clips that also show how you can create a test cord to verify the motor works independently, but there isn't an off-the-shell unit so you'd have to do soldering work and find a 220-volt three-prong adapter so I decided to opt against making one.

After the plastic connector is removed, the next step is to remove the motor. There are two retaining clips on the two sides of the motor, which keep the motor in place. You need to remove them by pushing down on one side of the tab, which will hopefully unlatch the tab. Again, the YouTube links are a good place to see how to remove these retaining clips. You shouldn't have to remove the screws attached to the base of the dryer cabinet, but it may be easier to manuever.

At this point you should check if you have the same replacement motor as the one you purchased. Some motors don't include the motor pulley. If you don't have the motor pulley with the replacement motor, you either ordered the wrong motor, especially for this model. Other MayTag units seem to require that you take it from the old motor using an Allen wrench. (For this particular brand, if you wanted to remove the motor pulley, you could use a 7/8" socket wrench and a wrench to hold one side of the motor shaft steady you loosen the screw attached to the motor pulley. I wouldn't recommend spending the time since the replacement motor you ordered should come with it.)

At this point, you should have successfully removed the motor from the dryer. Now you have to reverse the process and reinstall the replacement one. There are several things that are key to know. First, you may find it easier to remove the blower wheel casing that sits adjacent to the motor before installing the motor, since the motor has two cylindrical circles that need to be aligned with the motor housing. The retaining clips need to be added, and you may find one side easier to latch than the other. The plastic connector on the battery terminal should then be reattached to this new motor.

Reattaching the belt also proved to be different from how the YouTube clips demonstrated, which often call for looping the belt around the idler pulley first before attaching to the motor pulley. For this particular model, you actually want to follow the instructions specified in "Installing Drive Belt" which call for looping the belt around the motor pulley first. You want to position the ribbed-side of the belt against the tumbler and you should loop the belt around the motor pulley first ribbed-side. You then want to pull the idler pulley left and up (only pull to the maximum spring tension allowed) and attach the

belt. In this way, the remaining slack from the belt is taken up by the tension in the idler pulley. You should be able to rotate the tumbler now and have everything rotated correctly.

You shouldn't have had to remove any electrical wiring, but if you inadvertently detached a cable, you should also check that wiring connections are correct. For the dryer lamp, there are two connectors (one ground and one power), and one plug is bigger than the other, which makes it easier to distinguish between the two. Since the door light controls whether the start switch can be used, you actually need to reinstall most of the unit to verify that your motor is successfully work. You can obviously take shortcuts and attach the door without screwing everything back, but do it with caution.

The rest of the installation pretty much involves reversing what you just did. Remember that the lint is flammable, so you would be well-advised to try to clean up all the excess lint in the dryer. Remember that installing the clamp on the blower wheel requires that you use vice grips and squeezing the sides to expand the clamp. In addition, adding the retaining clip on the blower wheel will require using the snap-ring pliers again. Finally, you should verify the felt seals are resting on the outside of the tumbler before reattaching everything, which appear to be used to help reduce the friction between the metal cabinet and the tumbler. (You can check by rotating the tumbler and watching where the felt pieces rest).

You should make sure that everything works by carefully monitoring the dryer. Although there is a circuit breaker that will trip, it's a good idea not to leave the dryer on when you're out of the house after this install process. You may also wish to find a service technician instead, since this information should only be used as a reference.

Using PyCrypto instead of M2Crypto on Google App Engine

2012-01-25T09:41:00.000-08:00

If you've ever tried to leverage the M2Crypto library on the Google App Engine platform, you'll notice that you're prevented from doing so. One of the reasons is that M2Crypto depends on C-compiled openssl libraries that are not allowed to be used on the platform. The major alternative that Google provides is the Python Cryptography Toolkit, or pycrypto, which is a Python-based version of various encryption algorithms.

Performance wise, the pycrypto library may not run as fast since it implements the encryption algorithms primarily in Python. Furthermore, the current version that is supported is version 2.3, which doesn't provide RSA PKCS#1 v1.5 encryption support (if you find yourself getting "Block type 01 not supported, you need to handle the padding/encoding format -- see information below) nor does it provide PEM format (you need to encode your keys as DER format, which are basically the base64-unencoded versions of the PEM).

Nonetheless, you can get pycrypto working on Google App Engine. Here's how you can do it inside your App Engine projects. I'll also discuss some of the extra steps needed to get RSA signatures working too (using the PKCS#1 v1.5 standard). It's not completely obvious so some of the stack traces you see will hopefully explain why these steps were necessary.

1. Inside the app.yaml configuration, you need to specify a runtime of python27, and specify the pycrypto librarie. Note: you either have to install python2.7 on your own local dev instance, or you simply have to deploy constantly to Google App Engine to test your app. Note that you also need to specify the threadsafe: parameter too for the Python 2.7 experimental version (see the Google App Engine Python 2.7 docs for more information).

application: YOUR_APP_NAME_HERE
version: 1runtime: python27
api_version: 1
threadsafe: false
handlers:- url: /static  
static_dir: static

...
libraries:
- name: lxml  
version: "2.3"
- name: pycrypto  
version: "2.3"

2. You need to generate your RSA keys with the DER format. If you see this stack trace, chances are that you're trying to provide key formats in PEM. The Pycrypto v2.3 version on Google App Engine assumes you're feeding things in DER format. In other words, these two lines will not work:

>>> from Crypto.PublicKey import RSA
>>> privatekey = open("privkey.pem", "r")
>>> m = RSA.importKey(privatekey)

Stack trace:File 
"/base/python27_runtime/python27_lib/versions/third_party/pycrypto-2.3/Crypto/PublicKey/RSA.py", line 247, in importKey    return self._importKeyDER(der)  
File "/base/python27_runtime/python27_lib/versions/third_party/pycrypto-2.3/Crypto/PublicKey/RSA.py", line 234, in _importKeyDER    raise ValueError("RSA key format is not supported")

You can use the openssl command to convert your public/private key PEM file to DER format. You'll also need to the same if you're using a PEM certificate too:

openssl rsa -outform der privkey.pem > privkey.der
openssl rsa -outform der < cert.pem > cert.der

These lines should the work:

>>> from Crypto.PublicKey import RSA 
>>> privatekey = open("privkey.der", "r")
>>> m = RSA.importKey(privatekey)

If you really want to verify your keys are working correctly, you can also use m.decrypt() and m.encrypt() to encode and encode the data. You can click this link for another good overview about how to use pycrypto.

>>> from Crypto.PublicKey import RSA
>>> from Crypto import Random
>>> plaintext = "abcd"
>>> random_generator = Random.new().read
>>> encrypted_data = m.encrypt(plaintext, random_generator)
>>> m.decrypt(encrypted_data)

3. Next, it's not completely obvious how to use pycrypto to handle RSA signature signing. Most of the heavy lifting is done by the M2Crypto library, but if you're using pycrypto to do you may find yourself having to do some extra work to implement some of the PKCS algorithms.

For PKCS#1 v1.5 (which M2Crypto seems to be using), the private key is signed by the sender using the decryption process (not encryption) while the public key is used with the encryption process by the receiver to verify a signature, which is somewhat the opposite if you wanted to encrypt/decrypt data. (You can confirm this point by reviewing the Latex-based pycrypt.tex file).

In the M2Crypto world, RSA signature can be performed with the following lines:

>>> m = M2Crypto.RSA.load_key_string(privatekey)>>> digest = hashlib.sha1(plaintext).digest()>>> signature = m.sign(digest, "sha1")

In the PyCrypto world, you have to implement PKCS#1 v1.5 yourself. If you get "block type is not 01" when using M2Crypto to verify a signature create by PyCrypto, chances are M2Crypto is expecting the data to be encoded. You can review the RFC2313 spec to see what needs to be done to convert the signature with the correct padding. There is a padding spacing (ps) parameter that determines how many FF bytes should be added, which can be calculated by doing a number.size() call on your private key with the PyCrypto library and dividing by 8 bits (i.e. 2048 bits / 8 = 256 bytes)

The code below demonstrates how you would implement PKCS#1 v1.5 padding. Note that the private key is used to decrypt the data:

import hashlib

   def _emsa_pkcs1_v1_5_encode(M, emLen):
        SHA1DER = '\x30\x21\x30\x09\x06\x05\x2b\x0e\x03\x02\x1a\x05\x00\x04\x14'
        SHA1DERLEN = len(SHA1DER) + 0x14

        H = hashlib.sha1(M).digest()
        T = SHA1DER + H
        if emLen < (SHA1DERLEN + 11):
            raise Exception('intended encoded message length too short (%s)' % emLen)
        ps = '\xff' * (emLen - SHA1DERLEN - 3)
        if len(ps) < 8:
            raise Exception('ps length too short')
        return '\x00\x01' + ps + '\x00' + T

4. Currently, Google App Engine only supports pycrypto v2.3. You may encounter issues with using openssl-generated keys to do RSA signature verification. The bug reports for pycrypto v2.3 are posted here:https://bugs.launchpad.net/pycrypto/+bug/898466 https://bugs.launchpad.net/pycrypto/+bug/702835A bug report has been issued here:http://code.google.com/p/googleappengine/issues/detail?id=5302If you find issues trying to decrypt/encrypt data, you can bypass this issue by importing your own custom RSA.py version patched from the 2.3 version. Here is the diff that is needed to patch the v2.3 RSA.py version.

--- a/RSA.py
+++ b/RSA.py
@@ -206,7 +206,7 @@ class RSAImplementation(object):
def generate(self, bits, randfunc=None, progress_func=None):
if bits < 1024 or (bits & 0xff) != 0:              # pubkey.getStrongPrime doesn't like anything that's not a multiple of 128 and > 512
-            raise ValueError("RSA modulus length must be a multiple of 256 and > 1024")
+            raise ValueError("RSA modulus length must be a multiple of 256 and >= 1024")
rf = self._get_randfunc(randfunc)
obj = _RSA.generate_py(bits, rf, progress_func)    # TODO: Don't use legacy _RSA module
key = self._math.rsa_construct(obj.n, obj.e, obj.d, obj.p, obj.q, obj.u)
@@ -221,7 +221,8 @@ class RSAImplementation(object):
der.decode(externKey, True)
if len(der)==9 and der.hasOnlyInts() and der[0]==0:
# ASN.1 RSAPrivateKey element
-               del der[6:8]    # Remove d mod (p-1) and d mod (q-1)
+               del der[6:]     # Remove d mod (p-1), d mod (q-1), and q^{-1} mod p
+                der.append(inverse(der[4],der[5])) # Add p^{-1} mod q
del der[0]      # Remove version
return self.construct(der[:]

Hopefully this explains how you can get RSA encryption and signature signing working on Google App Engine without the M2Crypto library! (The last issue was not easy to find. In fact, it seemed very puzzling that RSA signature verification worked fine on local hosts, but never seemed to work on GAE instances.)

Forcing Ubuntu 10.0.4 to renew a DHCP client after hibernate/sleep mode...

2012-01-18T10:06:00.000-08:00

If you're on a network that has a short DHCP lease and your Ubuntu machine suspends/hibernates, you may find that you often may be IP conflicting with another system on the same network. You can run /etc/init.d/networking but often times you just want to force the DHCP renewal yourself.

One way is to modify /etc/default/acpi-support and add the "networking" module to the STOP_SERVICES declaration for the /etc/default/acpi-support file. However, if you have multiple networking interfaces that need to be restarted on bootup, this could cause more delays after resuming. It also seems strange that you have to taken down an entire networking module just to get things working again.

Another approach is to execute dhclient -r after resuming. As of Ubuntu 10.04, the power management modules actually use pm-utils, though there is a legacy /etc/acpi/ directory that often invokes pm-sleep, pm-hibernate and setups events depending on laptop keys pressed. You can find learn how to write your own power management hooks.

Here's how I got things to work:

1. sudo vi /etc/pm/sleep.d/00_network

#!/bin/bash
case $1 in
 hibernate)
     echo "Hey guy, we are going to suspend to disk!"
     ;;
 suspend)
     echo "Oh, this time we are doing a suspend to RAM. Cool!"
     ;;
 thaw)
     /sbin/dhclient -r
     /sbin/dhclient 
     ;;
 resume)
     /sbin/dhclient -r
     /sbin/dhclient 
     ;;
 *)  echo "Somebody is calling me totally wrong."
     ;;
esac

2. sudo chown root /etc/pm/sleep.d/00_network

To test, run sudo pm-sleep or sudo pm-hibernate. On bootup, take a look at tail -30 /var/log/pm-suspend.log and see if you see the 00_network script resume. If so, you should be good to go!

/etc/pm/sleep.d/00_network thaw hibernate:Internet Systems Consortium DHCP Client V3.1.3
Copyright 2004-2009 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Upgrading from jQuery v1.4 to v1.5..

2012-01-14T17:47:00.000-08:00

Apparently jQuery v1.5+ no longer allows extra data parameters. Instead, it's passed in as extra.data.

$("#mydiv").click(my_func, {'abc' : 123});  (old)

my_func = function(event) { alert(event.data); }

http://code.jquery.com/jquery-1.4.2.js

jQuery.each( ("blur focus focusin focusout load resize scroll unload click dblclick " +
"mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave " +
"change select submit keydown keypress keyup error").split(" "), function( i, name ) {

// Handle event binding
jQuery.fn[ name ] = function( fn ) {
return fn ? this.bind( name, fn ) : this.trigger( name );
};

if ( jQuery.attrFn ) {
jQuery.attrFn[ name ] = true;
}
});

Note that the data parameter is no longer passed as an extra parameter:
http://code.jquery.com/jquery-1.7.js

jQuery.each( ("blur focus focusin focusout load resize scroll unload click dblclick " +
"mousedown mouseup mousemove mouseover mouseout mouseenter mouseleave " +
"change select submit keydown keypress keyup error contextmenu").split(" "), function( i, name ) {

// Handle event binding
jQuery.fn[ name ] = function( data, fn ) {
if ( fn == null ) {
 fn = data;
 data = null;
}

return arguments.length > 0 ?
 this.bind( name, data, fn ) :
 this.trigger( name );
};

Port scanning for web servers on your network..

2012-01-13T19:10:00.000-08:00

sudo apt-get install nmap
sudo nmap -sS -O -p80 192.168.0.1/24

Dealing with zombie Facebook cookies...

2012-01-12T18:40:00.000-08:00

Not able to logout in your Facebook Connect applications? Facebook recently pushed a change in the past day or so such that the domain= parameter is now added to your fbsr_ cookies. If you find that your cookies are not being removed after a user attemps to logout, chances are you're experiencing the repercussions of these recent changes.

In the timestamped version (1326413659,169943147), there is a section that was added to the Facebook Connect Library:

2340	2348	}
2341	2349	return b;
2342	2350	},
	2351	loadMeta: function() {
	2352	var a = document.cookie.match('\\bfbm_' + FB._apiKey + '="([^;]*)\\b'),
	2353	b;
	2354	if (a) {
	2355	b = FB.QS.decode(a[1]);
	2356	if (!FB.Cookie._domain) FB.Cookie._domain = b.base_domain;
	2357	}
	2358	return b;
	2359	},
2343	2360	loadSignedRequest: function() {
2344	2361	var a = document.cookie.match('\\bfbsr_' + FB._apiKey + '=([^;]*)\\b');
2345	2362	if (!a) return null;

The regexp apparently fails to match because there is an extra quotation mark. Instead of:

var a = document.cookie.match('\\bfbm_' + FB._apiKey + '="([^;]*)\\b'), b;

It should be:

var a = document.cookie.match('\\bfbm_' + FB._apiKey + '=([^;]*)\\b'), b;

If you inspect document.cookie in a JavaScript console, you'll see no sign of how this regexp could match (i.e. the regexp would match fbm_1234="abcde" but not fm_1234=abcde). You can also use the Chrome/Safari Web Inspector, put breakpoints on this function, and use the deminifier feature (look for the {} icon at the bottom) to double-check.

Background info: Before the OAuth2 migration, the fbs_ cookie was used. Included in the fbs_cookie was a query string that needed to be decoded and the base_domain parameter used for the domain= cookie parameter.(For more background about how to set or delete cookies in JavaScript, see: http://www.quirksmode.org/js/cookies.html.)

Cookies be cleared by setting the expiration date to 01/01/1970 GMT. However, most browsers won't know how to delete the cookie unless the path= and domain= parameters are set correctly too. In other words, if you had a cookie named fbsr_1234, with domain=abc.com, the browser would not be able to delete it unless you also specified this parameter.

Until now, OAuth2 didn't include the domain= parameter in fbsr_ cookies. But with today's recent push, it is now being used. The result? If you had an old cookie without this domain= parameter and attempted to logout with this new JavaScript code, you might find that you're unable to clear them. You may also encounter strange logout issues in general and not see the fbsr_ cookie cleared correctly.

When you logout, a cross-domain request gets sent to Facebook to invalidate the session. When you hit reload and invoke another FB.init() command, another request gets sent to Facebook's site -- since FB recognizes the existing cookie as invalid, it correctly deletes the cookie from your browser (the cross-domain response appears to pass in the domain= parameter correctly). Unfortunately the clearing of the cookie initially doesn't work because the fbm_ cookie isn't parsed correctly because of this regexp bug.

Facebook will most likely fix this issue soon, though in the interim your users may not be able to logout of your app. One thing we've done is to use a server-side code to instruct the browser to clear the cookie, though it may not always work unless your page invokes FB.init() properly and receives back a cross-domain request from Facebook to set the domain= properly.

You can examine this code for how you can delete cookies from the server-side:

https://github.com/rogerhu/facebook_oauth2/blob/master/views.py

Want to track Facebook Connect JavaScript changes? Check out:

https://github.com/rogerhu/connect-js

Using FTP/TLS with Python 2.6...

2012-01-10T23:16:00.000-08:00

FTP over SSL/TLS can be supported on Python 2.6. If you have prod machines that are running Python 2.6 and you want to avoid upgrading, you can do the following:

wget http://www.python.org/ftp/python/2.7.1/Python-2.7.1.tgz

...untar the file and grab the Lib/ftplib.py file.

You can then do:

from ftplib import FTP_TLS

ftps = FTP_TLS('ftp.mysite.com')
ftps.login('userid', 'pw')
ftps.prot_p()
ftps.retrlines('LIST')

To download a file, you would do:

ftps.retrbinary('RETR file.bin', open('output.bin', 'wb').write)

or to upload a file, you would do something on the order of (borrowed from http://www.daniweb.com/software-development/python/threads/110195)

def upload(ftp, file):
  ext = os.path.splitext(file)[1]
  if ext in (".txt", ".htm", ".html"):
     ftp.storlines("STOR %s" % (file), open(file))
  else:
     print ftp.storbinary("STOR %s" % (file), open(file, "rb"), 1024)

jQuery UI 1.9

2012-01-10T15:08:00.001-08:00

Cool slide presentation about jQuery UI 1.9..

http://ajpiano.com/widgetfactory/

Cisco Unified Communications 500 Series - Cisco Smart Business Communication Systems

2012-01-10T13:27:00.000-08:00

https://supportforums.cisco.com/docs/DOC-9772

Can the T1/E1 module on UC500 terminate Voice and Data?
The VWIC-2MFT-T1/E1 card on UC500 can only be used for Voice T1 termination. It cannot be used for terminating Data T1s. On UC500, this card does not support multiplexing of Voice and Data. For Data connectivity, using an ISR based T1/E1 is recommended.

What are the total number of PSTN interfaces that the UC500 can support?
Depending on the UC500 model or SKU - can have any of the below:
4 to 12 FXO ports or 2 to 6 BRI ports (VIC slot can be used to add interfaces in certain model)
T1/E1 VWIC interface card for use in the 8-, 16-, and 32-user UC500 SKUs (VIC slot can be used to add this T1/E1 interface card)
UC500 48-user SKU is also available with integrated T1/E1 interface.

What T1 parameters are supported on the UC500 for voice termination?
The below parameters are supported:
T1 line coding = B8ZS
T1 framing = ESF
Interface switch Type for PRI - National with FAS (User side)
Hunt Type = two way, descending (channel 23 to 1)
Switch Protocol = See ISDN PRI switch-types above

Tracking the SWF file..

2012-01-03T18:16:00.000-08:00

Facebook doesn't often change the SWF file that is used for its cross-domain handler, but sometimes it's useful to download automatically the right binary file:


SWF_FILE=`grep --color=none all_deminified.js | "_swfPath\": " | sed -E "s/(.*?)swfPath\": \"rsrc.php(.*?.swf)\"/\2/" | sed 's/\\\\//g'`
/usr/bin/wget "http://static.ak.fbcdn.net/rsrc.php/${SWF_FILE}" -O ${SWF_OUTPUT}

https://github.com/rogerhu/connect-js/blob/master/update_fb_github.sh

Cisco routers can't do PPTP client..

2012-01-02T13:35:00.000-08:00

https://supportforums.cisco.com/thread/2010872


#config t
#vpdn enable
#vpdn-group 1
#accept-dialout
#protocol ?
  l2tp  Use L2TP

Changes in Facebook's swf flash handler..

2011-12-23T15:31:00.000-08:00

Ever since Facebook introduced a change in their Facebook Connect Library that caused severe login issues for IE users that lasted for more than a week, we've created scripts to monitor Facebook's JavaScript Connect Library to detect for any changes that might affect our users. Nate Frieldy first created the first version to monitor for diffs, and I soon forked it here to monitor for diffs that span more than just the changes that happen on the 1st line that indicates timestamp changes.

On December 8, 2011, our diff detection scripts picked up this change:

1		/1323218538,169893481,JIT Construction: v482006,en_US/
	1	/1323305592,169920374,JIT Construction: v482779,en_US/
2	2
3	3	if (!window.FB) window.FB = {
4	4	_apiKey: null,
…
5361	5361	[10, 3, 181, 34],
5362	5362	[11, 0, 0]
5363	5363	],
5364		"_swfPath": "rsrc.php\/v1\/yK\/r\/RIxWozDt5Qq.swf"
	5364	"_swfPath": "rsrc.php\/v1\/yD\/r\/GL74y29Am1r.swf"
5365	5365	}, true);
5366	5366	FB.provide("XD", {
5367	5367	"_xdProxyUrl": "connect\/xd_proxy.php?version=3"

This SWF file is Facebook's cross-domain handler for web browsers that don't implement HTML5 but can use a Flash-based version of HTML5's postMessage() function that allows messages to be passed between different domains. Facebook doesn't often recompile the SWF file, so this diff caught my attention. The most reliable decompiler I've found is Sothink's SWF decompiler, which can be used to export the ActionScript files with a 30-day trial (for more context about how to decompile, see http://hustoknow.blogspot.com/2011/06/facebooks-flash-xdcomm-receiver.html).

I've decompiled the SWF file and ActionScript files from http://static.ak.fbcdn.net/rsrc.php\/v1\/yD\/r\/GL74y29Am1r.swf and reviewed the diffs between the previously decompiled SWF with this one. If you were to compare the diff changes for the XdComm.as file, you would see:

15a16,17
>         private static var initialized:Boolean = false;
>         private static var origin_validated:Boolean = false;
20,21c22,29
<             Security.allowDomain("*"); <             Security.allowInsecureDomain("*"); --- >             if (XdComm.initialized)
>             {
>                 return;
>             }
>             XdComm.initialized = true;
>             var _loc_1:* = PostMessage.getCurrentDomain();
>             Security.allowDomain(_loc_1);
>             Security.allowInsecureDomain(_loc_1);
51a60
>             ExternalInterface.addCallback("postMessage_init", this.initPostMessage);
60a70,76
>         private function initPostMessage(param1:String, param2:String) : void
>         {
>             origin_validated = true;
>             this.postMessage.init(param1, param2);
>             return;
>         }// end function
>
164a181,189
>         public static function proxy(param1:String, param2:String) : void
>         {
>             if (origin_validated)
>             {
>                 ExternalInterface.call(param1, param2);
>             }
>             return;
>         }// end function
>

The changes indicate that Facebook has tightened the cross-domain security policies. Instead of using wildcard domains to accept messages in its allowDomain() function, it now invokes a call to getCurrentDomain(), which is a function defined in the PostMessage.as file used to execute a call to document.domain, relying more on the browser to define the security restrictions.

Most of these change should not affect your users...just wished Facebook would discuss more what's going on behind the scenes since your apps may very well be using the Facebook Connect Library without realizing these changes are happening beneath you!

I've started to post the decompiled SWF files here:
https://github.com/rogerhu/connect-js/tree/master/swf

Note that these updates are only manually. If someone knows of an open-source SWF decompiler, then the diffs could be much more automated!

Setting up IPSec with racoon and a Cisco router..

2011-12-14T16:36:00.000-08:00

Tools on Linux v2.6

The Linux 2.6 kernel already comes with IPSec support (Ubuntu distributed appears to have AH/ESP support), so you need two packages to get it working. First, you need ipsec-tools and racoon. You can install them by doing apt-get install ipsec-tools and apt-get install racoon respectively. You'll need to setup /etc/ipsec-tools.conf to define what IP subnets/hosts will be connected via VPN (and whether to use ESP and/or AH tunnel or transport mode, as well as the gateway IP's that are used to bridge the connections). Racoon has other parameters about Phase 1 and Phase 2 negotiation that you need to setup too, which are defined in /etc/racoon/racoon.conf. You use the remote {} configuration parameters for Phase 1, and sainfo parameters for Phase 2.). See http://lists.freebsd.org/pipermail/freebsd-net/2006-June/010975.html for more detailed info.

Note the unique allows for multiple security associations to be used over the same host, See http://pardini.net/blog/2008/08/21/ipsec-with-setkeyracoon-and-multiple-single-host-spds/. Apparently using the keyword 'unique' instead of 'require' fixes the issue:

flush;spdflush;spdadd ${LOCAL_NETWORK} ${STJUST_NETWORK} any -P out ipsec esp/tunnel/${LOCAL_OUTSIDE}-${STJUST_OUTSIDE}/unique;spdadd ${STJUST_NETWORK} ${LOCAL_NETWORK} any -P in  ipsec esp/tunnel/${STJUST_OUTSIDE}-${LOCAL_OUTSIDE}/unique;

More documentation here about unique in the setkey man page:

The protocol/mode/src-dst/level part specifies the rule how to process the packet. Either ah, esp, or ipcomp must be used as protocol. mode is either transport or tunnel. If mode is tunnel, you must specify the end-point addresses of the SA as src and dst with '-' between these addresses, which is used to specify the SA to use. If mode is transport, both src and dst can be omitted. level is to be one of the following: default, use, require, or unique. If the SA is not available in every level, the kernel will ask the key exchange daemon to establish a suitable SA. default means the kernel consults the system wide default for the protocol you specified, e.g. the esp_trans_deflev sysctl variable, when the kernel processes the packet. use means that the kernel uses an SA if it's available, otherwise the kernel keeps normal operation. require means SA is required whenever the kernel sends a packet matched with the policy. unique is the same as require; in addition, it allows the policy to match the unique out-bound SA. You just specify the policy level unique, racoon(8) will configure the SA for the policy.

Racoon works by basically listening for commands from the Linux kernel. The tunnels get setup the first time you attempt to negotiate a connection to a host. If you have certain routes established that are defined in /etc/ipsec-tools.conf and do a setkey -f /etc/ipsec-tools.conf, this information will be loaded as the security policy database (SPD) and the kernel will send a trigger that causes Racoon to attempt to establish the connection.

The two commands you will use to initially test are:

sudo setkey -f /etc/ipsec-tools.confsudo racoon -F -f /etc/racoon/racoon.conf -v -ddd -l /etc/racoon/racoon.log

The most secure (but complicated) is to use is a Internet Key Exchange (IKE) authentication approach. In this approach, both VPN client/server sides announce that they will use a pre-shared key authentication mechanism and their authentication and hash algorithm. The pre-shared key is just some hard-coded value that both sides decide before setting up the VPN connection. Once the connection is established, both sides use a Diffie-Hellman key approach to generate a public/private key so that future exchanges will be encrypted. It so happens in this approach both sides exchange a public key and are able to decode the packet using their own private key! All this negotiation happens during what's called Phase 1 negotiation.

Alternate approaches to the IKE implementation call call for setting up manual keys. In other words, both sides have to know how to encrypt the data beforehand instead of this intricate key exchange approach. A lot of the Racoon documentation mention setting up security association (SA)'s within /etc/ipsec-tools.conf, but this approach is un-needed if we are going to use the IKE-based approach, which is presumably more secure. If the IKE implementation is used, then Phase 2 negotiations must also occur.

A great guide to troubleshooting IPSec connections is here:

http://vpn.precision-guesswork.com/vpn/ipsec_troubleshooting.pdf

Want to know how all the nomencalture is laid out? Click here:

http://www.unixwiz.net/techtips/iguide-ipsec.html

Phase 1
If you read the IPSec documents, you'll see there are 3 rounds of this Phase 1. You can use tshark/wireshark to watch the network dumps:

Round 1: agree on authentication, encryption, hash payload algorithm

Round 2: key exchanges w/ nonce values (to avoid replay attacks).

Round 3: validation of hash/identification payloads using the secret keys completes successfully.

Phase 2 (IKE only)
The second step, also known as Quick Mode, in the IKE approach is to negotiate a security association (SA) policy. These policies not only define what encryption/authentication algorithms should be used, what encryption keys should be used for data transferring and for what IP subnets.

The ipsec-tools package comes with an /etc/ipsec-tools.conf that defines the security association (SA) policies. This policy must match against the information provided by the customer side. In the Northwestern Mutual's case, their IT department set their Cisco router with an access control list that defines that's allowed to connect. You will notice in the ISAKMP protocol during Phase 2 negotation that the packet structure for Phase 2 also includes an IDci and IDcr identity payload. You can watch Racoon and see what bits get passed through:

2011-12-14 01:12:56: DEBUG: IDci:
2011-12-14 01:12:56: DEBUG:
04000000 <IP address here> Data exchanges
Assuming everything is setup correctly, you need to setup your route table for the specific IP blocks to which you are connecting. Make sure do netstat -rn and then do route add's to add the correct routes. Unless you're bridging Ethernet interfaces, you need to be sure that you are always sending packets over the same Ethernet interface.

You can confirm packets that go over the wire by using either tcpdump or tshark -i eth0 not port 22 (to exclude traces from your current SSH connection from being dumped out). If you are using ESP encryption, then you should also see that the kernel encrypting packets destined for those IP's to the appropriate place). Again, the Linux kernel is handling most of the work, so long as the routes are correctly defined.

Ways to debug:
Watch isakmp packets:

1. sudo tshark -i eth0 udp port 500 -V or
sudo tshark -i eth0 udp not port 22

2. ssh -X
sudo wireshark

(X11Forwarding needs to be temporarily enabled on /etc/ssh/sshd_config, then do /etc/init.d/ssh restart &. You then need to make sure X11Forwarding is setup in your /etc/ssh/ssh_config on hte client side).

Wireshark can actually decrypt ESP/AH authentication assuming you provide the Security Parameter Indexes (SPI) generated on-the-fly and encryption keys. Most of this data you can observe via running Racoon in debug mode.

FYI - You may also notice "next payload" in Racoon dumps. The ISAKMP standard appears to define multiple types of payloads. Often times you will see vendor ID and other data such as the following:

Vendor ID: RFC 3706 Detecting Dead IKE Peers (DPD)        Next payload: Vendor ID (13)        Payload length: 20        Vendor ID: RFC 3706 Detecting Dead IKE Peers (DPD)    Vendor ID: XXXXXX        Next payload: Vendor ID (13)        Payload length: 20        Vendor ID: XXXXXX    Vendor ID: draft-beaulieu-ike-xauth-02.txt        Next payload: NONE (0)        Payload length: 12        Vendor ID: draft-beaulieu-ike-xauth-02.txt

Setting up a VPN between two DD-WRT routers..

2011-12-11T21:53:00.000-08:00

This setup worked for two WRT54GL routers running DD-WRT v24-sp1.

PPTP server (192.168.0.1):
First, one machine needs to be setup as a PPTP server.
1. Go to Services->PPTP to enable the PPTP server.
2. Set the server IP (should be a virtual lAN IP address different than your LAN IP address -- i.e. 192.168.0.2),
3. Set the Client IP block (192.168.0.50-192.168.0.70), and then setup the CHAP Secrets (johndoe* mypassword *).

PPTP client (192.168.1.1) :
1. Enable PPTP client.
2. Set the PPTP Server IP.
3. Set the Remote Subnet (192.168.0.0) and Remote Subnet Mask (255.255.255.0)
4. Set the MPPE Encryption to "mppe required".
5. Set the MTU/MRU to be 1450.
6. Disable NAT mdoe.
7. Set the username and password to the CHAP secret set in the PPTP Server.

You should verify the PPTP connection is established by telnetting into the PPTP client box and attempting to ping the private IP address of the PPTP server (i.e. 192.168.0.2 or 192.168.0.1). If this succeeds, then you may be able to ping the routers but other machines on the network are not able to talk with each other. In this case, you may wish to confirm that the PPTP server has not setup a route of 192.168.1.0. To do it automatically, you need to do the following:

1. Go to Administration->Commands.
2. Add the following firewall commands. Usually what happens is that the /tmp/pptpd_client/ip-up script is created. A delay is inserted before adding the route and then re-executing the ip-up bash script again.

sleep 40
/bin/sh -c 'echo "ip route add 192.168.1.0/24 dev ppp0" >> /tmp/pptpd/ip-up'
/tmp/pptpd_client/ip-up

Click on Save->Firewall after saving.

If you want to reinitiate the PPTP connection, try to click Save/Apply Settings and waiting for the VPN connection to be re-established. If you really want to check things out, you can configure a VPN client on Ubuntu 10.04 through the Network Manager (make sure to click Point-to-Point MPPE Encryption and allow stateful encryption, send PPP echo packets to help keep the connection alive). (Note: If you forget to enable the MPPE encryption/stateful encryption options, you may find that the VPN connection is flaky. It seems as if there are CHAP requests/rejects that keep happening without these two options). This VPN client will help you verify that the PPTP server is responding correctly.

You should also telnet to both DD-WRT routers and verify the routes have been established between the two subnets. You should also cat /tmp/pptpd_client/ip-up on the PPTP server to verify that the IP route was added correctly.

Upgrading a Compaq Presario C700 from Vista to Windows 7..

2011-12-10T21:35:00.000-08:00

Recently, I upgraded an old Compaq C714NR Presario laptop that had been running Windows Vista to Windows 7. Coffee had been spilled on the touchpad, rendering it inoperable. The keyboard still worked but the spacebar was sort of sticky so needed to be replaced. You may have older machines that might be worth upgrading, especially if you took advantage of Microsoft offers that allowed .edu email addresses to get a copy of Windows 7 Professional for $25.

1. Although your machine may be running 32-bit Windows Vista, chances are that if the machine runs a dual-core processor, which means it can run 64-bit Windows Professional. Most of the graphics, networks, and sound drivers already come with Windows 7, so there usually isn't a need to download extra drivers. Windows 7 should install right out of the box.

2. If you're burning a copy of Windows 7, you may encounter issues about "Required cd/dvd drive device driver is missing". If you observe this case, chances are likely the DVD you burned actually has problems, especially if you were using a copy that hadn't been used for awhile. Originally you may be led to think that there are some driver incompatibility issues with the 64-bit Windows 7 version, but try to reburn the DVD and see if the install works.

3. The touchpad can be replaced, but you have to buy one that comes with the laptop casing too. Since most of the casing + touchpads parts are sold off Ebay for $30+, it may easier to simply attach a USB-mouse instead. The picture below shows an example of the touchpad + upper casing:

4. There are web sites that sell spare keyboard keys (i.e. laptopkey.com), but buying one part can easily cost $8 and you can usually buy the entire keyboard replacement for $12. The HP service manual for replacing the keyboard is fairly straightforward, but there are a few key things to know. In the case of the C700, there were 3 screws at the bottom of the laptop, each with a keyboard icon at the bottom. One of these screws was obscured by the memory lid, so you may have to remove the lid first.

Second, there is the Zero-Insertion Force (ZIF) connector that attaches to the keyboard and laptop. What this usually means is that the sides of the connector need to be pushed out.

You should avoid pulling the ribbon cable out until the connector is released. The picture below shows one example of how the ZIF connector is pushed out. You can usually use your fingers and push the connector out slightly before inserting the ribbon cable. You should push down on the sides to fasten the ribbon cable securely.
Keep in mind that you should verify that all keys work. If the connector is not fully fastened, you may find some keys do not respond. You can try to boot up the computer with the keyboard installed, but be careful if any components are exposed.

5. Finally, if you need to replace any keys, you first have to figure out how the keyboard mechanism works.. There are a bunch of YouTube demos for replacing the keys in HP laptops, but none of the videos I found pointed out that it's easier to attach one of the plastic hinges to the key, and the other smaller plastic hinge to the laptop. If you were to setup the hinges on the laptop first, the plastic hinges should move up and down if you were to apply pressure to them, supplementing the spring-like action in the button.

Once you figure out the right way to place them, take the large hinge and attach it to the key before attaching the other part. For this keyboard, I couldn't just put the key over the two plastic hinges since the pressure of the keys would cause both plastic to be pushed down without snapping into place. You have to be careful with this part since the plastic hooks can break, so avoid trying to force the keys to attach to the plastic hinges.

Celery 2.3+ crashes when channel.close() commands are issued...

2011-12-10T13:14:00.001-08:00

This AMQPChannelException issue has happened for us over the last 3 weeks, so I decided to dig-in to understand why we were getting AMQPChannelException's that caused our Celery workers to simply die. Well, it turns out this exception often appeared in our logs:


(404, u"NOT_FOUND - no exchange 'reply.celeryd.pidbox' in vhost 'myhost'", (60, 40), 'Channel.basic_publish')

The basic problem I suspect is that we have a task that is checking for long running Celery tasks. It relies on the celeryctl command, which is a special mechanism used by Celery to broadcast messages to all boxes running Celery (i.e. celeryctl inspect active)

The celeryctl command implements a multicast protocol by leveraging the AMQP standard (see RabbitMQ tutorial). All Celery machines on startup bind to the exchange celery.pidbox. When you send a celeryctl command, RabbitMQ receives this message, and then goes and delivers to all machines listening to this exchange celery.pidbox.

The machines also send back their replies on a separate exchange called reply.celery.pidbox, which is used by the main process that issued the celeryctl command to collect all the responses. Once all program completes, it will delete the celery.pidbox since it's no longer needed. Unfortunately, if a worker receives this command and attempts to respond but it's too late, it can trigger an exchange not found, causing RabbitMQ to issue a channel.close() command. I suspect this happens especially during heavy loads and/or during intermittent network failures, since the problem often shows up during these times.

Celery handles connection failures fine, but doesn't seem to deal with situations where the AMQP host issues a close command. I solved it in two ways: first, allowing Celery to gracefully reset the entire connection when such an event happens (PR request to the Celery framework), and increasing the window in which we check for replies so the exchange isn't deleted quickly thereafter (i.e. celery inspect active --timeout=60). The latter may be the quicker way to solve it, though the former should probably be something that would help avoid the situation altogether (although it may cause other issues).

The result may appear to trap the exception and try to establish a new connection. This approach is already being used when for Celery control commands (an error msg "Error occurred while handling control command" gets printed out but the underlying Exception gets caught). It seems this exception occurs when the RabbitMQ sends a close() command to terminate the connection, causing the entire process to die.


    def on_control(self, body, message):
        """Process remote control command message."""
        try:
            self.pidbox_node.handle_message(body, message)
        except KeyError, exc:
            self.logger.error("No such control command: %s", exc)
        except Exception, exc:
            self.logger.error(
                "Error occurred while handling control command: %r\n%r",
                    exc, traceback.format_exc(), exc_info=sys.exc_info())
            self.reset_pidbox_node()

We'll see in this pull-request whether the authors of Celery think this is a good idea...I suspect that it would be better to create a new channel than to restart the connection altogether.

https://github.com/ask/celery/pull/564

Moving to Nose..

2011-12-09T10:09:00.000-08:00

We've started to have our developers use Nose, a much more powerful unit test discovery package for Python. For one, in order to generate JUnit XML outputs for Hudson/Jenkins, the test runner comes with a --with-xunit command that lets you dump these results out.

Here are a few things that might help you get adjusted to using Nose:

1. As mentioned in the Testing Efficiently with Nose tutorial, the convention is slightly different for running tests has changed. The format has changed:


python manage.py test app.tests:YourTestCaseClass
python manage.py test app.tests:YourTestCaseClass.your_test_method

One way to force the test results to use the same format is to create a class that inherits from unittest that overrides the __str__, __id__, and shortDescription methods. The __str__() method is used by the Django test runner to display what tests are running and which ones have failed, enabling you to copy/paste the test to re-run the test. The __id__() method is used by the XUnit plug-in to generate the test name, enabling you to swap out the class name with the Nose convention. Finally, the shortDescription() will prevent docstrings from replacing the test name when running the tests.


class BaseTestCase(unittest.TestCase):

    def __str__(self):
        # Use Nose testing format (colon to differentiate between module/class name)

        if 'django_nose' in settings.INSTALLED_APPS or 'nose' in settings.TEST_RUNNER.lower():
            return "%s:%s.%s" % (self.__module__, self.__class__.__name__, self._testMethodName)
        else:
            return "%s.%s.%s" % (self.__module__, self.__class__.__name__, self._testMethodName)

    def id(self):  # for XUnit outputs
        return self.__str__()

    def shortDescription(self):  # do not return the docstring
        return None

2. For SauceLabs jobs, you can also expose the URL of the job run in which you are running. WebDriverException's inherit from the Exception class, and add a 'msg' property that we can use to insert the SauceLabs URL. You want to avoid adding the URL in the id() and __str__() methods, since those routines are used to dump out the names of classes that Hudson/Jenkins may used to compare against between builds.


    def get_sauce_job_url(self):
        # Expose SauceLabs job number for easy reference if an error occurs.
        if getattr(self, 'webdriver', None) and hasattr(self.webdriver, 'session_id') and self.webdriver.session_id:
            session_id = "(http://saucelabs.com/jobs/%s)" % self.webdriver.session_id
        else:
            session_id = ''

        return session_id

    def __str__(self):
        session_id = self.get_sauce_job_url()
        return " ".join([super(SeleniumTestSuite, self).__str__(), session_id])

    def _exc_info(self):
        exc_info = super(SeleniumTestSuite, self)._exc_info()

        # WebDriver exceptions have a 'msg' attribute, which gets used to be dumped out.
        # We can take advantage of this fact and store the SauceLabs jobs URL in there too!
        if exc_info[1] and hasattr(exc_info[1], 'msg'):
            session_id = self.get_sauce_job_url()
            exc_info[1].msg = session_id + "\n" + exc_info[1].msg
        return exc_info

3. Nose has a bunch of nifty commands that you should try, including --with-id & --failed, which lets you run your entire test suite and then run only the ones that failed. You can also use the attrib decorator, which lets you to decorate certain test suites/test methods with various attributes, such as network-based tests or slow-running ones. The attrib plugin seems to support a limited boolean logic, so check the documentation carefully if you intend to use the -a flag (you can use --collect-only in conjunction to verify your tests are correctly running with the right logic).

Hover states with Selenium 2..

2011-12-08T20:45:00.000-08:00

We do a lot of automated tests on Selenium 2, and recently we noticed that some of our tests that attempt to verify mouseover/hover states intermittently break. We could see the mouseover event get fired, but then suddenly it would stop.

The phenomenon at first made us suspect there were possible race conditions in our JavaScript code. Were we accidentally clearing/hiding elements before WebDriver could reach them? Was the web browser loading multiple times and clearing out the original state? Or was it an issue in Selenium 2, since it attempts to calculate the center position of the element, scroll into view, and then move the mouse cursor to the center of the element (see http://www.google.com/codesearch#2tHw6m3DZzo/trunk/javascript/atoms/dom.js). You can look at how the Windows driver's mouseMove is implemented here:
http://www.google.com/codesearch#2tHw6m3DZzo/trunk/cpp/webdriver-interactions/interactions.cpp

It turns out that hover states for Windows in Selenium 2 are just problematic right now. You can even see inside the Selenium 2 test suite that hover states are not skipped: for Windows-based systems:

http://www.google.com/codesearch#2tHw6m3DZzo/trunk/java/client/test/org/openqa/selenium/RenderedWebElementTest.java&q=2067%20package:http://selenium\.googlecode\.com


if (!supportsHover()) {       
  System.out.println("Skipping hover test: Hover is very short-lived on Windows. Issue 2067.");       
  return;
}

The full bug report is here:
http://code.google.com/p/selenium/issues/detail?id=2067

In Selenium v2.15.0, even the Internet Explorer wiki was updated to remind people that hover states (using mousemove) have issues:

http://code.google.com/p/selenium/source/diff?spec=svn14947&r=14947&format=side&path=/wiki/InternetExplorerDriver.wiki

The workaround seems to be generating your own synthetic events:


$('#myelem').trigger('mouseenter');

AT&T Uverse Internet 18Mbps solution..

2011-12-03T16:27:00.000-08:00

You have two modem/router choices for the AT&T Internet Uverse plan: Motorola NVG510 or the Motorola 2210-02-1ATT. The NVG510 doesn't have a bridge mode, and there are a wide variety of hacks to use the IP Passthrough option for the Motorola NVG510 modem:

http://forums.att.com/t5/Features-and-How-To/NVG510-Bridge-Mode/td-p/2890841/page/2

If you read through the discussions though, you'll notice that the discussion gravitates towards using the IP Passthrough with a Fixed-DHCPS set for the router's IP address: "i'm using the IP Passthrough functionality. According to its own instructions on the right-hand side of the tab, the RG is supposed to pass its public IP address through to another device, using its own DHCP - but it won't do it. I had to manually enter the information into my regular router (an Airport Extreme Base Station, aka AEBS) to use the Passthrough functionality.

...it seems more better if you had a choice to go with the Motorola 2210 option. Also, don't confuse the Motorola 2210-02-1TT model with the Motorola 2210-02-1022 model, the latter of which was an old DSL modem that you can buy off Ebay. For one thing, the 2210-02-1022 models only speak PPPoE, and since the Uverse solution relies on authentication based on the MAC address, you won't be able to plug this modem in place of things. You can check your diagnostics tool on the router to verify that you're connecting via IP-DSLAM, which is synonymous with the AT&T Uverse Internet 18Mbps solution.

Fujitsu C2010 specs

2011-12-02T23:43:00.001-08:00

http://implitech.com/manufacturers/fujitsu-siemens/lifebook-c2010/

Intel 845MP/MZ chipset

Fujitsu Siemens LifeBook C2010 Graphics - ATI Radeon Mobility M6 LY Driver
Fujitsu Siemens LifeBook C2010 Human Interface - Belkin MI 2150 Trust Mouse Driver
Fujitsu Siemens LifeBook C2010 IEEE 1394 - Texas Instruments TSB43AB21 IEEE-1394a-2000 Controller Driver
Fujitsu Siemens LifeBook C2010 Modem - Intel 82801CA CAM AC97 Modem Driver
Fujitsu Siemens LifeBook C2010 Multimedia - Intel 82801CA CAM AC97 Audio Controller Driver
Fujitsu Siemens LifeBook C2010 Network - Realtek RTL 8139 8139C 8139C Driver
Fujitsu Siemens LifeBook C2010 PCMCIA - Texas Instruments PCI1520 PC Card Cardbus Driver
Fujitsu Siemens LifeBook C2010 Storage - Intel 82801CAM IDE U100 Driver
Fujitsu Siemens LifeBook C2010 Intel Chipset Drivers

Device manager for Fujitsu Siemens LifeBook C2010 Laptop
Radeon Mobility M6 LY - ATI
PCI\VEN_1002&DEV_4C59&SUBSYS_113E10CF&REV_00\4&34C0AFA1&0&0008
MI 2150 Trust Mouse - Belkin
USB\VID_1241&PID_1166\5&95D66C5&0&2
TSB43AB21 IEEE-1394a-2000 Controller - Texas Instruments
PCI\VEN_104C&DEV_8026&SUBSYS_116210CF&REV_00\4&6B54384&0&30F0
82801CA CAM AC97 Modem - Intel
PCI\VEN_8086&DEV_2486&SUBSYS_10D110CF&REV_02\3&61AAA01&0&FE
82801CA CAM AC97 Audio Controller - Intel
PCI\VEN_8086&DEV_2485&SUBSYS_117710CF&REV_02\3&61AAA01&0&FD
RTL 8139 8139C 8139C - Realtek
PCI\VEN_10EC&DEV_8139&SUBSYS_111C10CF&REV_10\3&61AAA01&0&68
PCI1520 PC Card Cardbus - Texas Instruments
PCI\VEN_104C&DEV_AC55&SUBSYS_116410CF&REV_01\4&139E449D&0&50F0
82801CAM IDE U100 - Intel
PCI\VEN_8086&DEV_248A&SUBSYS_113D10CF&REV_02\3&61AAA01&0&F9
Brookdale (82845 845 Chipset Host Bridge) - Intel
PCI\VEN_8086&DEV_1A30&SUBSYS_00000000&REV_04\3&61AAA01&0&00
Brookdale (82845 845 Chipset AGP Bridge) - Intel
PCI\VEN_8086&DEV_1A31&SUBSYS_00000000&REV_04\3&61AAA01&0&08
82801CA CAM USB Controller #1 - Intel
PCI\VEN_8086&DEV_2482&SUBSYS_113D10CF&REV_02\3&61AAA01&0&E8
82801CA CAM USB Controller #2 - Intel
PCI\VEN_8086&DEV_2484&SUBSYS_113D10CF&REV_02\3&61AAA01&0&E9
82801 Mobile PCI Bridge - Intel
PCI\VEN_8086&DEV_2448&SUBSYS_00000000&REV_F3\3&33FD14CA&0&F0
82801CAM ISA Bridge (LPC) - Intel
PCI\VEN_8086&DEV_248C&SUBSYS_00000000&REV_02\3&61AAA01&0&F8
82801CA CAM SMBus Controller - Intel
PCI\VEN_8086&DEV_2483&SUBSYS_113D10CF&REV_02\3&61AAA01

Setting up wake-on LAN in your own home

2011-11-26T14:31:00.000-08:00

One of the drawbacks of using Slicehost or Amazon EC2 is that you're pretty much paying $16-$20/month for a VPS server even when you're not accessing the server that much. If you have data that you also don't want being stored on a cloud, you may opt to setup your own home server to store this information. You may want to have the ability to access this data but not have your machine turned on all the time adding to your electricity bill.

First, your computer(s) needs to have wake-on-LAN enabled in the BIOS and your computer's Ethernet port must be plugged directly into a switch/router (wireless wake-on LAN NIC cards may also be possible). For Toshiba laptops, for instance, you need to reboot, hit Esc and then F1 to enter the BIOS. You can then enable Wake-on-LAN support and save the changes.

You also need a router that can run the DD-WRT or Tomoto open source firmware. If you use the VPN version of the DD-WRT, you can also setup a PPTP server with DynDNS so that you can VPN into your server even though you're using an ISP that does not provides static IP's. You'll want to setup your DD-WRT server to assign a static IP address to your remote computer (inside the Services tab, look for static leases).

If have one of the Windows Professional versions, you can also take advantage of the Remote Desktop servers. You can enable Remote Desktop Sever by going to the Control Panel > System > Remote. You can also right-click My Computer (if the icon is shown on the desktop) and choose Properties.

To verify that Wake-on-LAN works, you can use Netcat/socat to test things out. This blog posting explains how the magic packet for Wake-on-LAN is constructed: 6 bytes of FF followed by the LAN MAC address repeated 16 times. The bash script used to generate a hex version is listed as follows -- you just need to substitute the ETHER parameter with the MAC address of the machine that will need to be woken up:

ETHER="aa:bb:cc:dd:ee:ff"
ETHER2=`echo $ETHER | sed "s/://g"`
ETHER3="${ETHER2}${ETHER2}${ETHER2}${ETHER2}"
ETHER4="FFFFFFFFFFFF${ETHER3}${ETHER3}${ETHER3}${ETHER3}"
echo ${ETHER4} | xxd -r -p > wake.packet

The blog posting points out that you can use netcat to send to a router IP, but what about sending a broadcast address from within the same network? For this purpose, the socat utility seems to work better (not sure if netcat allows sending to broadcast IP addresses?)

socat - UDP-DATAGRAM:192.168.0.255:7,broadcast < wake.packet

You can also use DD-WRT to wake up any device (check the WoL section) too, but the above information is more background information for how it actualy works!

To setup dynamic DNS, you'll need to go to the DDNS page of the DD-WRT firmware and input your username/pw. The DynDNS service is now $20/year, though it used to be free in recent years. The instructions for setting DynDNS are posted at: http://www.dd-wrt.com/wiki/index.php/DDNS_-_How_to_setup_Custom_DDNS_settings_using_embedded_inadyn_-_HOWTO

To setup the PPTP server for VPN, go to the PPTP section and follow the instructions according to http://www.dd-wrt.com/wiki/index.php/PPTP_Server_Configuration. One thing to note is that the PPTP server IP should be different from your router's internal IP, and the CHAP secrets format is ( * *). Yes, there is an asterisk after username and password.

Ubuntu comes with 'tsclient', which is a Remote Desktop Connection. Once you've setup a VPN connection correctly, you can To get your Windows machine to hibernate, you can also create this batch file and add to your Desktop:


%windir%\system32\rundll32.exe powrprof.dll,SetSuspendState Hibernate

You can also get an iPad/Android app to wake up your computer remotely -- we found Mocha WoL, a free app that lets you wake up the machine remotely. You can also set the hostname to a DynDNS machine, and assuming you have UDP port forwarding setup to relay WoL broadcasts, it should be able to wake up the machine remotely.

WebDriverWait and SauceLabs

2011-11-25T14:07:00.000-08:00

If you've used the WebDriverWait class with SauceLabs, you may have noticed that it actually will run beyond the designated timeout period. I first noticed this issue when seeing that this function took 40 minutes to complete with a 20 second implicit wait time and a 60 second timeout w/ .5 poll frequency! (60 / .5 = 120 loops * 20 seconds/implicit wait = 2400 seconds total)

The problem appears to be that WebDriverWait attempts to run self._timeout / self._poll times before returning a Timeout Exception:


for _ in xrange(max(1, int(self._timeout/self._poll))):

Since this code could wait until an ElementNotFound exception is found (and if the implicit wait time is increased), then the total time this routine takes is a function of the total # of loops * poll frequency * the implicit wait time.

The correct patch appears to be recording the start time and looping until the maximum timeout period has been exceeded:


--- a/selenium/webdriver/support/wait.py
+++ b/selenium/webdriver/support/wait.py
@@ -24,7 +24,7 @@ class WebDriverWait(object):
 
     def __init__(self, driver, timeout, poll_frequency=POLL_FREQUENCY):
         """Constructor, takes a WebDriver instance and timeout in seconds.
-           
+
            :Args:
             - driver - Instance of WebDriver (Ie, Firefox, Chrome or Remote)
             - timeout - Number of seconds before timing out
@@ -43,7 +43,8 @@ class WebDriverWait(object):
     def until(self, method):
         """Calls the method provided with the driver as an argument until the \
         return value is not Falsy."""
-        for _ in xrange(max(1, int(self._timeout/self._poll))):
+        end_time = time.time() + self._timeout
+        while(time.time() < end_time):
             try:
                 value = method(self._driver)
                 if value:

The bug report is filed here:

http://code.google.com/p/selenium/issues/detail?id=2939

How the celeryctl command works in Celery..

2011-11-12T11:53:00.000-08:00

One of the most popular Django apps out there is the Celery task queue framework. It allows you to build your own task queue system for offline processing and provides an elegant framework for interfacing with message brokers such as AMQP, Redis, etc.

The celeryctl command in the Celery task queue is an amazing tool. It allows you send commands to your Celery works to figure out which ones are actively processing tasks, revoke tasks that have been dispatched so that workers will skip over procesing them, and see which ones are scheduled. We use it here to monitor long-running tasks, such as this script:

from celery.bin import celeryctl
import datetime

def probe_celeryctl():
    results = celeryctl.inspect().run("active")
    check_old_celery_tasks(results)


def check_old_celery_tasks(results):
    bad_tasks = []

    MAX_TIMEDELTA = {'hours': 1}
    for host, tasks in results.items():
        for task in tasks:
            task_start = task.get('time_start')
            timestamp = float(task_start)
            task_timestamp = datetime.datetime.fromtimestamp(timestamp)
            time_diff = abs(datetime.datetime.utcnow() - task_timestamp)
            if time_diff > datetime.timedelta(**MAX_TIMEDELTA):
                print "Hmm..%s %s (on %s)" % (time_diff, task, host)
                bad_tasks.append("Task %s elapsed (%s): name=%s, PID=%s, args=%s, kwargs=%s" % (time_diff, host, task.get('name'), task.get('worker_pid'), task.get('args'), task.get('kwargs')))

    if len(bad_tasks) > 0:
        message = "You'd better check on these tasks...they are slowing things down.\n\n"
        message += "\n".join(bad_tasks)
        print message

if __name__ == "__main__":
    probe_celeryctl()

How does celeryctl work? Assuming you're using the AMQP backend with Celery, celeryctl relies on the same concepts used in the AMQP open standard (click here for a basic intro). When you first startup, Celery will create an AMQP exchange called "celeryd.pidbox" on the AMQP host. You can confirm by using rabbitmqctl to list exchanges:

$ sudo rabbitmqctl -p fanmgmt_prod list_exchanges
Listing exchanges ...
celeryd.pidbox fanout
.
.
.

You'll notice that celeryd.pidbox is created as a fanout exchange (see the AMQP intro for more details). This way, using celeryctl on one machine will broadcast a message to this exchange. The AMQP host will deliver messages to each queue that is bound to this celeryd.pidbox exchange. On startup, every Celery worker will also create a queue (queue.hostname.celery.pidbox) bound to this exchange, which can be used to respond to celeryctl commands.

Replies from each Celery worker are passed back via direct exchange using celeryd.reply.pidbox. When you startup celeryctl, it sends messages to the celeryd.pidbox and listens for messages to arrive from the celeryd.reply.pidbox queue. The timeout period to wait for replies is 1 second, so if you wish to increase the time to wait for replies from Celery workers, you can increase this number with the -t parameter (i.e. celeryctl inspect active -t )

Note: the reply exchange gets deleted after celeryctl command exists (usually set to auto_delete=True). Since all the Celery workers are still bound to the other celeryd.pidbox exchange, it should still persist until you shutdown all Celery workers.

Installing HipChat in Jenkins/Hudson..

2011-11-10T09:45:00.000-08:00

1. git clone https://github.com/jlewallen/jenkins-hipchat-plugin

2. Compile and build (See http://hustoknow.blogspot.com/2010/09/installing-java-on-ubuntu-1004.html and http://hustoknow.blogspot.com/2011/04/hudson-becomes-jenkins-compling-plugins.html)

3. Copy the target/.hpi file that got generated into your hudson/plugins dir.

4. Restart Hudson.

5. Go into the Configure Hudson and provide the API key/conference room.

6. Make sure to enable HipChat notifications in each of your build projects!

FYI - this plug-in uses the HipChat API to publish information to the respective room that you deisgnate:

https://github.com/jlewallen/jenkins-hipchat-plugin/blob/master/src/main/java/jenkins/plugins/hipchat/StandardHipChatService.java

   public void publish(String message, String color) {
      for(String roomId : roomIds) {
         logger.info("Posting: " + from + " to " + roomId + ": " + message + " " + color);
         HttpClient client = new HttpClient();
         String url = "https://api.hipchat.com/v1/rooms/message?auth_token=" + token;
         PostMethod post = new PostMethod(url);
         try {
            post.addParameter("from", from);
            post.addParameter("room_id", roomId);
            post.addParameter("message", message);
            post.addParameter("color", color);
            client.executeMethod(post);
         }
         catch(HttpException e) {
            throw new RuntimeException("Error posting to HipChat", e);
         }
         catch(IOException e) {
            throw new RuntimeException("Error posting to HipChat", e);
         }
         finally {
            post.releaseConnection();
         }
      }
   }

   public void rooms() {
      HttpClient client = new HttpClient();
      String url = "https://api.hipchat.com/v1/rooms/list?format=json&auth_token=" + token;
      GetMethod get = new GetMethod(url);
      try {
         client.executeMethod(get);
         logger.info(get.getResponseBodyAsString());
      }
      catch(HttpException e) {
         throw new RuntimeException("Error posting to HipChat", e);
      }
      catch(IOException e) {
         throw new RuntimeException("Error posting to HipChat", e);
      }
      finally {
         get.releaseConnection();
      }
   }

}

Installing HipChat on 64-bit Linux.

2011-11-10T09:34:00.001-08:00

The instructions for HipChat are pretty straightforward, though you do need to run th commands from root.

https://www.hipchat.com/help/page/how-do-i-install-hipchat-on-64-bit-linux


# Adobe's instructions forget to mention ia32-libs-gtk
$ apt-get install lib32asound2 lib32gcc1 lib32ncurses5 lib32stdc++6 \
  lib32z1 libc6 libc6-i386 ia32-libs-gtk lib32nss-mdns

$ wget http://frozenfox.freehostia.com/cappy/getlibs-all.deb
$ dpkg -i getlibs-all.deb

# incorrect in adobe's instructions (corrected)
$ getlibs -p gnome-keyring
$ getlibs -p libhal-storage1

Seems like these commands were not needed on Ubuntu 10.04:


$ ln -s /usr/lib32/libnss3.so.1d /usr/lib32/libnss3.so
$ ln -s /usr/lib32/libssl3.so.1d /usr/lib32/libssl3.so
$ ln -s /usr/lib32/libnspr4.so.0d /usr/lib32/libnspr4.so
$ ln -s /usr/lib32/libsmime3.so.1d /usr/lib32/libsmime3.so # missing from adobe's instructions


wget http://airdownload.adobe.com/air/lin/download/2.6/AdobeAIRInstaller.bin
chmod 777 AdobeAIRInstaller.bin
./AdobeAIRInstaller.bin

The final step is to download the Adobe AIR client here:


$ wget http://downloads.hipchat.com/hipchat.air
$ /opt/Adobe\ AIR/Versions/1.0/Adobe\ AIR\ Application\ Installer hipchat.air

Debugging Facebook's JavaScript code

2011-11-03T07:56:00.000-07:00

In reviewing Facebook's JavaScript code, there apparently is a way to enable debugging of the JavaScript. If you set fb_debug=1, then the logging option will be enabled:

if (!options.logging &&        window.location.toString().indexOf('fb_debug=1') < 0) {      FB._logging = false;    }

Celerybeat and celerybeat-schedule

2011-10-27T20:19:00.000-07:00

In my effort to attempt to replace /etc/init.d/celerybeat with a version that worked more reliably with fabric, one of the discoveries is that celerybeat keeps firing off all tasks from the scheduled task list because Celerybeat stores the last_run time of a scheduled task usually in a celerybeat-schedule in the default dir)...if it hasn't been run in awhile (by virtue of using an older celerybeat-schedule) then you may see a lot of "Sending due task".

You can check the last_run_at apparently by using the shelve library from Python, which celerybeat uses to store all the scheduled tasks defined in your Celery configurations. Each time you restart Celerybeat, this celerybeat-schedule gets merged with your Celery scheduled tasks. Those that don't exist already are added.

sudo python
>>> import shelve
>>> shelve.open("/var/run/celerybeat-schedule")
>>> a['entries']['my_task'].last_run_at
datetime.datetime(2011, 10, 28, 2, 1, 57, 717454)

The key is to specify explicitly define the celerybeat-schedule:

/etc/default/celerybeat:

export CELERYBEAT_OPTS="--schedule=/var/run/celerybeat-schedule"

IPython: interactive Python

2011-10-27T16:02:00.000-07:00

- who: shows you what var
- store: stores a variable to a file (%store foo > a.txt)
- reset: clears namespace
- logstart, logon, logoff
- lsmagic

- run -d <file>: run python code step-by-step
- run -p <file>:

- xmode Context (xmode Verbose: shows the call values)
- pdb: turns on uncaught exception
- time (func): times run

Django caches all its models:

http://stackoverflow.com/questions/890924/how-do-you-reload-a-django-model-module-using-the-interactive-interpreter-via-m/903943#903943

Reasons for double-forking....

2011-10-26T20:29:00.001-07:00

http://stackoverflow.com/questions/881388/what-is-the-reason-for-performing-a-double-fork-when-creating-a-daemon

Adobe Flash Player 11 for Ubuntu

2011-10-25T10:34:00.001-07:00

Adobe finally has a 64-bit version of Adobe Flash Player for Ubuntu now!

http://get.adobe.com/flashplayer/completion/?installer=Flash_Player_11_for_Ubuntu_(apt)

Integrating OpenID Google Apps Single Sign On with Hudson/Jenkins....

2011-10-22T13:55:00.000-07:00

A not so well-documented aspect of using Hudson is that you can integrate OpenID single-sign on (SSO) with your Google Apps domain. You could implement SSO using the Jenkins Crowd plugin that comes pre-packaged with Hudson, but then you'd have to do custom integration work. Since the Crowd protcol is all SOAP-based, just getting the SOAP bindings right can be a big pain. Then you'd have to go about either setting up Crowd identity server or creating your own version via the Crowd API.

The OpenId plugin does not seem to be provided with the Hudson/Jenkins v2.1.2 release, but you can download and install it yourself. You do need the Sun version of Java (not OpenJDK), since there seems to be some sun.com dependencies existing in the Jenkins code base (the instructions for setting up on Ubuntu are listed here). You also need to install Maven too (sudo apt-get install maven2) and configure your ~/.m2/settings.xml.

Once Java and maven are setup, you can clone the OpenID repo and compile:

1) git clone https://github.com/jenkinsci/openid-plugin.git

2) mvn

If the compile was successful, the openid.hpi plugin should have been compiled into the target/ dir. You need to copy this open.hpi into your Hudson plugins/ dir (i.e. /var/lib/hudson/plugins). You don't appear to need to add an openid.hpi.pinned to avoid Hudson from overwriting this package, since the OpenID does come with Jenkins by defualt.

3) The OpenID plugin expects that the URL that a user connects to your continuous integration ends with a trailing slash ('/'). In your Apache2 config, you may find that you need to add a rewrite rule to force connections to your server always to end with a '/'. If your server is just http://hudson.myhost.com, the rewrite rule becomes:

RewriteEngine on
  RewriteRule  ^$  /  [R]

(The major reason is that the getRootUrl() command in the Jenkins code base borrows from the request URL). The OpenID plugin, when concatenates the OpenID finish callbacks, assumes that there will be a trailing slash at the end. Without it, your OpenID authorization flows may not work):

src/main/java/hudson/plugins/openid/OpenIdSession.java:
receivingurl="Hudson.getInstance().getRootUrl()+finishUrl";

If you notice that the OpenID callbacks (i.e federatedLoginService/openid/finish) are not prefixed with a '/', it means that you are missing this trailing slash!

4) Inside the Hudson configuration screen, the OpenID SSO will be https://www.google.com/accounts/o8/id. Your permissions will be defined by the email address of the SSO. If you do not wish anonymous users to be able to login, you should make sure that they do not have any types of permissions.

5) Make sure to enable OpenID SSO support in your Google Apps domain. The checkbox should be enabled inside "Manage this domain"->"Advanced Tools"->"Federated Authenticatin using OpenID".

One extra bonus...if you're using the Git plugin with Hudson, you may have also noticed, depending on which version of the Git plugin, that User accounts were based either on the full name or the e-mail username of the Git committer. If you want the user accounts associated with your Git committers to also be linked to your SSO solution, then this pull-request may also be useful.

https://github.com/rogerhu/git-plugin/pull/new/fix_git_email

(If you have pre-existing users, you may wish to convert their user directories from "John Doe" to jdoe@myhost.com to be consistent.)

(Interesting note: the Git plugin used in the Jenkins/Hudson 2.1.2 release is located at https://github.com/hudson-plugins/git-plugin, whereas the older v1 versions are at https://github.com/jenkinsci/git-plugin. The code base appears to have diverged a little bit, so one commit patch incorporated in https://github.com/jenkinsci/git-plugin.git 3607d2ec90f69edcf8cedfcb358ce19a980b8f1a that attempted to create accounts based on the Git commiter's username is not included in the v2.1.2 Jenkins release.)

Also, if you use automated build triggers, it appears they still work even if you turned on the OpenID SSO on too!

Update: it looks like the Git plug-in will start to expose an option to use the username's entire email address as a Hudson/Jenkins option. See the PR below:

https://github.com/hudson-plugins/git-plugin/pull/31/files

start-stop-daemon

2011-10-20T23:06:00.001-07:00

Wondering what the internals of the start-stop-daemon source code are?

http://doxygen.kannel.org/d6/d8e/start-stop-daemon_8c-source.html

Crowd and WSDL

2011-10-20T22:47:00.000-07:00

Need the latest copy of the Crowd WSDL file?

1. Visit http://www.atlassian.com/software/crowd/CrowdDownloadCenter.jspa

2. Download a copy.

3. Unpack the .jar files, and go into the atlassian-x.x.x directory.

4. vi crowd-webapp/WEB-INF/classes/crowd-init.properties

crowd.home=/home/myuser/projects/atlassian-crowd/data

5. ./start_crowd.sh

6. Go to http://localhost:8095 (or your dev server IP).

7. You should be able to connect and setup the Crowd service.

8. Go through the setup flow, and get a license key from Atlassian .

9. wget http://yourhost.com:8095/crowd/services/SecurityServer?wsdl

Need to get the WSDL working in Python? Either use ZSI (which is Google App Engine compatible) or the Python suds library:

https://jira.atlassian.com/browse/CWD-159

The instructions below will show you how to do Crowd authentication using ZSI:

http://tearsoffire.org/twiki/bin/view/Projects/CrowdSoapApi

from SecurityServer_services import SecurityServerLocator, SecurityServerHttpBindingSOAP
import SecurityServer_services as sss
from SecurityServer_services_types import ns0

loc = SecurityServerLocator()
server = loc.getSecurityServerPortType()

#build up the application authentication token
r = ns0.ApplicationAuthenticationContext_Def('ApplicationAuthenticationContext')
cred = ns0.PasswordCredential_Def('_credential').pyclass()
req = sss.authenticateApplicationRequest()
r._name='soaptest'
cred._credential = 'passwordGoesHere'
r._credential=cred
req._in0 = r
token = server.authenticateApplication( req )

#Look up a principle from the 'soaptest' application
prin = sss.findPrincipalByNameRequest()
prin._in0 = token._out
prin._in1 = 'cpepe'
me = server.findPrincipalByName( prin )
for i in me._out._attributes._SOAPAttribute:
    print '%s: %s' % (str(i._name), str(i._values.__dict__))

Using Fabric with sudo

2011-10-20T20:00:00.000-07:00

If you've ever had to use Fabric, one of the issues is that your scripts must return an error code of 0 in order for the sudo() command to assert that the command executed successfully. Any non-zero error code will result in an error message.

Fatal error: sudo() encountered an error (return code 1) while executing 'sudo"...

If you're using bash scripts, this means that any "set -e" or "bash -e" statements that trigger an abnormal exit. The "kill -0 ", which allows you to test whether a process exists and can be killed, suffers from a flaw in that if you provide a PID that does not exist, it will trigger an exception and cause bash to break out if "set -e" or "bash -e" is set (normally you can use $? to check the return value).

You also should check the integer value (if [ $? -eq 0 ]; then or if [ ?! -eq 1]; then) to determine which step to use.

Minus sign

2011-10-19T14:50:00.000-07:00

The minus sign is the default value if the variable isn't set. This line has two sets of fallbacks:

CELERYBEAT_PID_FILE=${CELERYBEAT_PID_FILE:-${CELERYBEAT_PIDFILE:-$DEFAULT_PID_FILE}}

value_for_platform

2011-10-19T09:34:00.001-07:00

http://wiki.opscode.com/display/chef/Recipes#Recipes-valueforplatform

Chef template specificity

2011-10-19T09:22:00.001-07:00

http://wiki.opscode.com/display/chef/Templates

Template Location Specificity

Cookbooks are often designed to work on a variety of hosts and platforms. Templates often need to differ depending on the platform, host, or function of the node. When the differences are minor, they can be handled with a small amount of logic within the template itself. When templates differ dramatically, you can define multiple templates for the same file. Chef will decide which template to render based on the following rules.

Within a Cookbook's template directory, you might find a directory structure like this:

templates
- host-foo.example.com
- ubuntu-8.04
- ubuntu
- default

For a node with FQDN of foo.example.com and the sudoers.erb resource above, we would match:

host-foo.example.com/sudoers.erb
ubuntu-8.04/sudoers.erb
ubuntu/sudoers.erb
default/sudoers.erb

In that order.

Then, for example: sudoers.rb placed under the files/host-foo.example.com/ directory, means it will be only copied to the machine with the domain name foo.example.com. (Note the "host-" prefix to the directory name)

So, the rule distilled:

host-node[:fqdn]
node[:platform]-node[:platform_version]
node[:platform]
default

Dealing with IOError and mod_wsgi

2011-10-19T09:01:00.000-07:00

http://permalink.gmane.org/gmane.comp.python.django.devel/30886

Open source version of Facebook's JavaScript library

2011-10-17T21:32:00.001-07:00

https://github.com/facebook/connect-js/

coroutines

2011-10-17T14:56:00.001-07:00

David Beazley has an awesome set of PDF slides about Python co-routines:

http://www.dabeaz.com/coroutines/Coroutines.pdf

http://www.dabeaz.com/coroutines/

PyCon talk too:
http://blip.tv/pycon-us-videos-2009-2010-2011/pycon-2011-an-outsider-s-look-at-co-routines-4899200

get_task_logger() in Celery...

2011-10-16T21:56:00.000-07:00

If you looked at the Celery documentation, you'll notice that the get_task_logger() examples
constantly show up.

@task
def add(x, y):
    logger = add.get_logger()
    logger.info("Adding %s + %s" % (x, y))
    return x + y

What does this function do? Well, it turns out that it will create a separate logger instance specifically tied to the task name (submitted as a PR on https://github.com/ask/celery/issues/129). The propagate=False is always set, so that any messages passed to it will not move up the parent/ancestor chain.

Instead, a handler is always added to this task. If you wish to adjust the logger level,
you could do:

import logging
logging.getLogger('myproject.add').setLevel(logging.DEBUG)

If no loglevel is specified in get_logger(), then the default log level defined in CELERYD_LOG_LEVEL is used. Be careful though! The right way is to set the level number (not the level name) if you are modifying directly through Python:

from celery import current_app
from celery.utils import LOG_LEVELS
current_app.conf.CELERYD_LOG_LEVEL = LOG_LEVELS['DEBUG']  # pretty much the same as logging.DEBUG

What's the purpose of get_task_logger()? Well it appears the motivation is to allow logging by task names. If we were just to import the standard logging module, Celery will patch the logger module to add process-aware information (ensure_process_aware_logger()), and then add format/handlers to both the root logger and the logger defined by the multiprocessing module (the multiprocessing get_logger() does not use process shared-logs but it allows you to login things to the "multiprocessing" namespace, which adds SUBDEBUG/SUBWARNING debug levels).

def setup_logging_subsystem(self, loglevel=None, logfile=None, format=None, colorize=None, **kwargs):                                                             
        if Logging._setup:                   
            return                                                                                            
        loglevel = loglevel or self.loglevel      
        format = format or self.format                                                                         
        if colorize is None:                
            colorize = self.supports_color(logfile)                                                            
                                                                                                              
        if mputil and hasattr(mputil, "_logger"):                    
            mputil._logger = None                                                                              
        ensure_process_aware_logger()                               
        receivers = signals.setup_logging.send(sender=None,                    
                        loglevel=loglevel, logfile=logfile,                                 
                        format=format, colorize=colorize)                      
        if not receivers:                                           
            root = logging.getLogger()                              
                                                                                                              
            if self.app.conf.CELERYD_HIJACK_ROOT_LOGGER:                       
                root.handlers = []                                                                             
                                                                    
            mp = mputil.get_logger() if mputil else None                 
            for logger in filter(None, (root, mp)):                         
                self._setup_logger(logger, logfile, format, colorize, **kwargs)               
                logger.setLevel(loglevel)                        
                signals.after_setup_logger.send(sender=None, logger=logger,                                    
                                        loglevel=loglevel, logfile=logfile,                                    
                                        format=format, colorize=colorize)

Debugging Celery tasks locally

2011-10-16T21:45:00.000-07:00

Want to make sure your Celery tasks work correctly before you deploy? Here are a bunch of useful tips you can do:

First, set the root logger and "celery.task.default" to use DEBUG mode:

import logging 
logging.getLogger('celery.task.default').setLevel(pythonLogging.DEBUG)
logging.getLogger().setLevel(pythonLogging.DEBUG)

Set ALWAYS_EAGER mode so that Celery will always invoke tasks locally instead of dispatching to the Celery machine.

Set EAGER_PROPAGATES_EXCEPTION so that any exceptions within tasks will be bubbled up so that you can actually see any exceptions that may cause your batch calls to fail (i.e. any uncaught exception can cause a fatal error!)

from celery import current_app
current_app.conf.CELERY_ALWAYS_EAGER = True
current_app.conf.CELERY_EAGER_PROPAGATES_EXCEPTIONS = True

from celery.utils import LOG_LEVELS
current_app.conf.CELERYD_LOG_LEVEL = LOG_LEVELS['DEBUG']  # pretty much the same as logging.DEBUG

Finally, if you are invoking a task from the same Python script, you should import the task_name as if it were being imported, even if the function is declared within the same file. The reason is that when running the Celeryd daemon and looks for registered tasks, Celery will consider the task function you invoked to come from the "__main__" class. The way to get around it is to import the task residing in the same file, assuming your PYTHONPATH is set correctly.

from celery.decorators import task

@task
def task_name():
   print "here"
   return 1

if "__name__ == "__main__":
  from  import task_name

  task_name.apply_async()

(Note: This information has been updated to reflect Celery v2.3.3 inner-workings).

Celery and the big instance refactor

2011-10-15T16:59:00.000-07:00

One of the strange parts in Celery is that if you want a logger that will write to celery.task.default instead of its own default name, you can do:

import celery.task import Task
logger = Task.get_logger()

The Task class appears to be a global instantiation of Celery. Normally, the task logger is setup via the get_logger() method, which then calls setup_task_logger(), which in turn calls get_task_logger. If you invoke get_logger() within a Task class, the name of the task name is used:

def setup_task_logger(self, loglevel=None, logfile=None, format=None,
            colorize=None, task_name=None, task_id=None, propagate=False,
            app=None, **kwargs):
        logger = self._setup_logger(self.get_task_logger(loglevel, task_name),
                                    logfile, format, colorize, **kwargs)

If you use Task.get_logger(), no name is used and the logger namespace is set to celery.task.default.

def get_task_logger(self, loglevel=None, name=None):                                                                               
      logger = logging.getLogger(name or "celery.task.default")  
      if loglevel is not None:
         logger.setLevel(loglevel)                                                                                  
      return logger

This Task appears to be part of the “The Big Instance” Refactor. It appears that there are plans for multiple instances of the Celery object to be instantiated.

Also, one thing to note:

http://ask.github.com/celery/userguide/tasks.html#logging

Instantiation
A task is not instantiated for every request, but is registered in the task registry as a global instance.

This means that the __init__ constructor will only be called once per process, and that the task class is semantically closer to an Actor.

If you have a task,

FQL is not being deprecrated

2011-10-14T22:53:00.000-07:00

You can now do FQL queries via the Open Graph API....

One of the most common misconceptions we hear from developers is the belief that FQL will be deprecated with the REST API. The primary reason for this misunderstanding is that you can only issue FQL queries using the fql.query or fql.multiquery methods. We want to make it clear that FQL is here to stay and that we do not have any plans to deprecating it. Today, we are enabling developers to issue FQL queries using the Graph API. With this change, FQL is now just an extension of Graph API.

You can issue an HTTP GET request to https://graph.facebook.com/fql?q=QUERY. The ‘q’ parameter can be a single FQL query or a multi-query. A multi-query is a JSON-encoded dictionary of queries.

http://developers.facebook.com/blog/post/579/

Pylint on Ubuntu

2011-10-13T09:58:00.000-07:00

Ever see this issue?

>> from logilab.common.compat import builtins
Traceback (most recent call last):
  File "", line 1, in 
ImportError: cannot import name builtins

Chances are you have an old version of logilab that is stored inside /usr/lib/pymodules:

>>> import logilab
>>> logilab.common

>>> logilab.common.compat

>>> logilab.common.compat.__file__
'/usr/lib/pymodules/python2.6/logilab/common/compat.pyc'

The solution is to delete the logilab directory in /usr/lib/pymodules, or do:

sudo apt-get remove python-logilab-common
sudo apt-get remove python-logilab-astng

Then you can do:

pip install -U pylint

Paper on Linux's pseudo-random number generator

2011-10-03T13:44:00.001-07:00

http://eprint.iacr.org/2006/086.pdf
http://www.linuxfromscratch.org/hints/downloads/files/entropy.txt
http://www.linuxcertified.com/hw_random.html

Changing end of line in Emacs

2011-09-29T11:32:00.000-07:00

http://mike.kruckenberg.com/archives/2004/08/replace_m_in_em.html

M-x set-buffer-file-coding-system RET unix

Facebook Python code for OAuth2

2011-09-23T11:25:00.001-07:00

Facebook recently announced that on October 1st, 2011, all Facebook third-party apps will need to transition to OAuth2. The JavaScript and PHP SDK code is posted, but how would you make the change if you're using Python/Django? To help others make the transition, we've released our own set of Python code at this GitHub repo:

https://github.com/rogerhu/facebook_oauth2

One of the pain points is that users may have existing OAuth cookies set on their browser, which you may use in your current application to authenticate. However, because Facebook Connect's JavaScript library requires an apiKey change parameter, it makes it hard to use their existing library to force these fbs_ cookie deletions. Furthermore, you'd have to write your own JS since the Facebook JS SDK is hard-coded to use only the new apiKey parameter.

We also show in this code how you can force these fbs_ cookie deletions on the server-side, primarily by setting the expiration date and providing the correct domain= parameter back to the client. It's worked well for us in managing the transition to OAuth2, so we hope you will find the same approach useful.

Good luck!

Lessons Learned: Migrating Tests to Selenium v2

2011-09-21T08:21:00.000-07:00

The slides from yesterday's talk, Lessons Learned: Migrating Tests to Selenium v2, are now posted. Remember the game Simon Says? Check out the game Hearsay Social Says, which shows how Selenium v2 can be used to automate web browser testing.

How Selenium v1 bypasses cross-origin domain security policies...

2011-09-17T10:10:00.000-07:00

The writeup by Simon Stewart from the Architecture of Open Source Applications discusses how the architecture of Selenium v2 was guided by the lessons drawn from Selenium 1. The figures and discussions on the Selenium http://seleniumhq.org/docs/05_selenium_rc.html also mention the fact that a proxy-server is used to get past cross-domain security policies. But how does Selenium v1 work under the covers?

Here's the general steps of what happens when you go from launching a Selenium RC (v1) server.
Suppose you wrote this Python script (let's call test1.py) on your local Linux box and launched a Selenium RC server on a Windows 7 machine.

from selenium import selenium

selenium = selenium(“10.10.10.1", 4444, "*iexplore", "http://www.google.com/")
selenium.start()
selenium.open(“/”)

1. The Selenium RC server starts and waits for an HTTP connection from a client. When the client connects and sends an instruction to initiate new Selenium instance, the Selenium RC server will launch the browser and configure the browser's proxy settings.

2. Selenium v1 JavaScript code is also loaded into the browser. This JavaScript code is what is used by Selenium for performing all its automated testing (i.e. used to query DOM elements, generate mouse clicks, etc.). You can click View Source on the Selenium RC tab to see what JavaScript code is loaded.

Note that there's also a runTest() JavaScript command executed when this page is loaded. When this happens, an Ajax connection is also initiated in selenium-remoterunner.js:

nextCommand: function () {this.xmlHttpForCommandsAndResults = XmlHttp.create();
sendToRC(postResult, urlParms, fnBind(this._HandleHttpResponse, this), this.xmlHttpForCommandsAndResults);
};

During the configuration of this proxy, all /selenium-server URL's are intercepted and routed to the RC server. By using an Ajax connection that uses the same URL as the one that is being tested, Selenium v1 can therefore avoid triggering browser cross-domain security policies and establish establish a connection back to the Selenium RC server to sending/receiving future commands. You'll notice how this browser proxy configuration works if you tried to add a /selenium-server to any URL (i.e. http://www.google.com/selenium-server would normally return back a Page Not Found error).

3. The RC server then opens a URL connection specified by the client API with a /selenium-server/core/Blank.html?start=true. (Note that when creating a Selenium instance, a specific URL must also be provided.) If this connection was successful, it also helps to verify that the proxy configuration was setup properly.

4. What happens if this Ajax connection from the browser to the RC server times out? If the request timeouts (i.e. no command), another Ajax request is sent and the channel is re-established. This is performed within the selenium-executionloop.js, which initiates a new Ajax connection back to the Selenium server.

continueTest : function() {LOG.debug("currentTest.continueTest() - acquire the next command");                                            
        if (! this.aborted) {                                                                                          
            this.currentCommand = this.nextCommand();                                                                  
        }

6. When this communication channel is setup, Selenium clients sending an v1 API command will be sent to the server, which in turns relays the communicate on this Ajax channel to send/receive commands. Selenium v1 relies on JavaScript to simulate all browser interactions, and the appropriate command is executed.

Selenium v2 avoids this issue entirely by using native drivers that bind tightly to the operating system...but this approach introduces new issues and complexities, which will be discussed in this week's upcoming Meetup.com event.

BeautifulSoup v4

2011-09-13T10:59:00.000-07:00

See this warning msg?
bs4/builder/_html5lib.py:60: DataLossWarning: namespaceHTMLElements not supported yet
DataLossWarning)
bs4/builder/_html5lib.py:77: DataLossWarning: BeautifulSoup cannot represent elements in any namespace
warnings.warn("BeautifulSoup cannot represent elements in any namespace", DataLossWarning)

Apparently here is the reason for this msg:
https://bugs.launchpad.net/beautifulsoup/+bug/727014

Leonard Richardson (leonardr) wrote on 2011-03-01: #1
html5lib supports namespaced elements (like ), and Beautiful Soup doesn't yet. These warnings are mostly a reminder to myself that I need to add namespace support. Unless you're actually parsing code that has namespaced tags, there won't be any real data loss.

Changed in beautifulsoup:
status: New → Confirmed

lxml and removing nodes

2011-09-13T00:58:00.000-07:00

The lxml library for Python represents a really effective tool for parsing and manipulating XML-based data. You can manipulate the XML documents to deal with the W3C standards for Inclusive and Exclusive Canonicalization, which deals with all messy details of adjusting namespaces as you extract sections of the data.

XML is inherently a difficult data structure to manipulate. The white spaces, return lines, and new lines make a big difference in validating signatures and/or digest values. If you accidentally miss a character in your text manipulation, perform the wrong canonicalization, etc. your one-way SHA hash can easily be affected, causing you to be unable to verify the signature of the data.

One of the idiosyncracies of the lxml library, described best in this lxml document, is that the internal data structures are stored as Element objects with a .text and .tail property. The .text represents all the underlying value within the tag, while the .tail property represents the text between tags. This data structure differs from the DOM-model in that the text after an element is represented by the parent. For example, consider this XML-structure:

<a>aTEXT
<b>bTEXT</b>bTAIL
</a>aTAIL

This can be represented with the following lxml code:

import etree

a = etree.Element('a')
a.text = "aTEXT"
a.tail = "aTAIL"

b = SubElement(a, 'b')
b.text = "bTEXT"
b.tail = "bTAIL"

What happens if you remove the 'b' node? Ideally, the text with the 'b' tag disappears, while the bTAIL gets moved up. The structure would look like the following:

<a>aTEXTbTAIL</a>aTAIL

The command to remove the lxml node would be:

a.remove(b)

Upon making this change, however, it appears in lxml v2.3, the output appeared as: <a>aTEXT</a>aTAIL</a>

In order to understand what's going on, I had to download the source for the lxml, install the Cython library that converts the .pyx code to .C bindings, recompile, and link the new etree.so binary. If you're curious, the instructions for doing so here are posted here.

Upon inspecting the etree.pyx, I noticed the code to move the tail occured after unlinking the node. What we really wanted is that the tail to be moved before the node is unlinked. Otherwise, the information about the tail would also be potentially be removed, which may have explained why the tail was never copied.

def remove(self, _Element element not None):
-        tree.xmlUnlinkNode(c_node)
         _moveTail(c_next, c_node)
+        tree.xmlUnlinkNode(c_node)

Examining the _moveTail code also points to something interesting. The .tail is represented internally by XML-based text-based nodes, which are siblings of the current node (denoted by the .next pointer). Text nodes are also XML-based text-nodes, but appear to be children of the node. There is a loop that traverses the linked list of nodes, such that there can be multiple text-nodes, which could could happen if multiple subelements were removed, and you were left with a chain of XML-based .tail nodes.

cdef void _moveTail(xmlNode* c_tail, xmlNode* c_target):
    cdef xmlNode* c_next
    # tail support: look for any text nodes trailing this node and
    # move them too
    c_tail = _textNodeOrSkip(c_tail)
    while c_tail is not NULL:
        c_next = _textNodeOrSkip(c_tail.next)
        tree.xmlUnlinkNode(c_tail)
        tree.xmlAddNextSibling(c_target, c_tail)
        c_target = c_tail
        c_tail = c_next

Upon fixing this code, the text_xinclude_test started failing. If I recompiled and reverted back to the original etree.pyx, the test passed fine. One even more unusual aspect was the invocation of the self.include(), which appeared to be overriden depending on whether the lxml library would rely on the native implementation of the xinclude() routine, or rely on its Python-based version that allows external URL's to referenced in ElementInclude.py.

def test_xinclude_text(self):
        filename = fileInTestDir('test_broken.xml')
        root = etree.XML(_bytes('''\
        
          
        
        ''' % filename))
        old_text = root.text
        content = read_file(filename)
        old_tail = root[0].tail

        self.include( etree.ElementTree(root) )
        self.assertEquals(old_text + content + old_tail,
                          root.text)

The test_xinclude_text() is a routine to verify that one can use <:xi:include> directives to incorporate other files within an XML-document. When such a tag is discovered, the contents of the file is read (in this case, the contents of test_broken.xml) and the entire node is substituted with this text. The parent node's .text property will then be set and the <xi:include> is removed.

It appears that code within the ElementInclude.py the text appeared to mask this issue by appending the tail before removing it:

@@ -204,7 +204,8 @@ def _include(elem, loader=None, _parent_hrefs=None, base_url=None):
                 elif parent is None:
                     return text # replaced the root node!
                 else:
-                    parent.text = (parent.text or "") + text + (e.tail or "")
+                    parent.text = (parent.text or "") + text 
                 parent.remove(e)

The entire pull request for this fix is located here:

https://github.com/lxml/lxml/pull/14/files

Update on this PR:

Note that this is a deliberate design choice. It will not change.

http://lxml.de/FAQ.html#what-about-that-trailing-text-on-serialised-elements

http://lxml.de/tutorial.html#elements-contain-text

In other words, if you remove a subelement, you have to take care of the .tail and move it to the right tag. The lxml library will not change so this PR request was rejected.

Compiling and testing lxml2..

2011-09-12T00:23:00.000-07:00

The instructions for recompiling lxml seem pretty straightforward. You have to pip install Cython, which is used to convert the .pyx file into a .c file, which then can be gcc compiled. The libxml2-dev and libxslt-dev must be packaged installed.

http://lxml.de/1.3/build.html

git clone https://github.com/lxml/lxml/
sudo apt-get install libxml-dev
sudo apt-get install libxslt-dev
pip install Cython
python setup.py build_ext

If you want to test that the unit tests still pass, you can link the uld type:

cd src/lxml
ln -s ../../build/lib.linux-x86_64-2.6/lxml/etree.so 
cd ../..
python test.py test_etree.py

lxml

2011-09-09T19:16:00.000-07:00

One of the best documentation for using lxml in Python is located here:

infohost.nmt.edu/tcc/help/pubs/pylxml/pylxml.pdf

One interesting tidbit:

In the DOM, trees are build out of nodes represented as Node instances. Some nodes are Element instances, representing whole elements. Each Element has an assortment of child nodes of various types: Element nodes for its element children; Attribute nodes for its attributes; and Text nodes for textual content.

The lxml view of an XML document, by contrast, builds a tree of only one node type: the Element.

The text following the element. This is the most unusual departure. In the DOM model, any text
following an element E is associated with the parent of E; in lxml, that text is considered the “tail” of E.

Getting the latest version of RabbitMQ v2.5.0+ via Ubuntu..

2011-09-01T11:30:00.001-07:00

http://www.rabbitmq.com/debian.html#apt

The trouble with Protected Mode in Selenium 2 for setting cookies...

2011-08-31T13:33:00.000-07:00

If you try to set cookies using the WebDriver API through Selenium 2, you may find that Internet Explorer fails to even set the cookie. The issue has been reported here:

http://code.google.com/p/selenium/issues/detail?id=1227&q=cookie&colspec=ID%20Stars%20Type%20Status%20Priority%20Milestone%20Owner%20Summary#makechanges

Selenium has a bunch of test suites to verify the behavior so it seemed strange that there would be an issue. Examining the IEDriver code too shows that the AddCookieCommandHandler is very similar to behavior of DeleteAllCookiesHandler and other IE command handlers, so I didn't really find an issue. Nor did the AddCookie() method in DocumentHost.cpp handles the dispatching of adding cookies.

If we do:

>>> driver = webdriver.Remote(desired_capabilities=current_env,

                                        command_executor="http://localhost:4444/wd/hb")

>>> driver.execute_script("document.cookie='a=1';")

>>> driver.get_cookies()

[{u'name': u'a', u'value': u'1', u'path': u'/', u'hCode': 97, u'class': u'org.openqa.selenium.Cookie', u'secure': False}, {u'name': u'sessionid', u'value': u'e1257265399f35b5c7ae4cf630581c90', u'path': u'/', u'hCode': 607797809, u'class': u'org.openqa.selenium.Cookie', u'secure': False}]

>>> driver.delete_cookie('a')

>>> driver.get_cookies()

[{u'name': u'sessionid', u'value': u'e1257265399f35b5c7ae4cf630581c90', u'path': u'/', u'hCode': 607797809, u'class': u'org.openqa.selenium.Cookie', u'secure': False}]

>>> driver.add_cookie({'name' : 'a' , 'value' : '2', 'secure' : False})

(Pdb) driver.get_cookies()

[{u'name': u'sessionid', u'value': u'e1257265399f35b5c7ae4cf630581c90', u'path': u'/', u'hCode': 607797809, u'class': u'org.openqa.selenium.Cookie', u'secure': False}]

Both delete commands will work successfully. But doing an add_cookie() fails to work if IE7/IE8 are in Protected mode in Selenium 2.5.0, even though Internet Explorer/IEDriver does not report any error. However, I was able to get cookies to be set once I disabled Protected mode.

Since Selenium 2.5.0 requires all Protected Mode settings to be consistent, you have to go into your Internet Options and uncheck the Protected Mode for every single zone (i.e. click through the icons for Internet, Local Internet, Trusted sites, Restricted sites). Then cookies can be correctly set.

It appears that IE7 and IE8 have this issue. IE9 may not have this problem. I was not able to get cookies set in either Protected/non-Protected mode using Selenium v2.0.0b3 though so you may still need to upgrade to Selenium 2 beyond this version to get cookie support working in IE7/IE8.

Using add_cookie in Selenium 2

2011-08-30T09:41:00.000-07:00

The documentation in the Python bindings for using the add_cookie() function Selenium 2 are unclear. The add_cookie() appears to take in a simple key/value pair:

def add_cookie(self, cookie_dict):

        """Adds a cookie to your current session.

        Args:

            cookie_dict: A dictionary object, with the desired cookie name as the key, and

            the value being the desired contents.

        Usage:

            driver.add_cookie({'foo': 'bar',})

        """

        self.execute(Command.ADD_COOKIE, {'cookie': cookie_dict})

If you're encountering NullPointerExceptions similar to a bug it's possible the problem is that your dictionary needs to include name, value, path, and secure keys. The tests in selenium/webdriver/common/cookies_test.py appear to back this point up:

self.COOKIE_A = {"name": "foo",

                         "value": "bar",

                         "path": "/",

                         "secure": False}



    def testAddCookie(self):

        self.driver.execute_script("return document.cookie")

        self.driver.add_cookie(self.COOKIE_A)

Even the section posted at http://readthedocs.org/docs/selenium-python/en/latest/navigating.html#cookies suggest that adding cookie just a matter of connecting to using a key/value pair too:

Before we leave these next steps, you may be interested in understanding how to use cookies. First of all, you need to be on the domain that the cookie will be valid for:



# Go to the correct domain

driver.get("http://www.example.com")



# Now set the cookie. This one's valid for the entire domain

cookie = {"key": "value"})

driver.add_cookie(cookie)



# And now output all the available cookies for the current URL

all_cookies = driver.get_cookies()

for cookie_name, cookie_value in all_cookies.items():

    print "%s -> %s", cookie_name, cookie_value

For disabling the Django debug toolbar in Selenium 2, then the command should be:

self.selenium.add_cookie({"name" : "djdt",

                          "value" : "true",

                          "path" : "/",

                          "secure" : False})

As of Selenium v2.5.0, It appears that all name/value and secure must be specified to avoid triggering the NullPointerException error.

An issue report has been filed here:

http://code.google.com/p/selenium/issues/detail?id=2367

Extracting audio clips from YouTube videos

2011-08-28T19:53:00.000-07:00

If you use the stock Ubuntu v10.04 youtube-dl version, you may encounter this error message when trying to download a YouTube clip:

youtube-dl http://www.youtube.com/watch?v=

ERROR: no fmt_url_map or conn information found in video info

The solution is to git clone the youtube-dl repo and use the latest youtube-dl version:

git clone https://github.com/rg3/youtube-dl.git

Extracting only the audio portion means that you should set the -vn option, which disables video encoding. The -acodec option determines the output format. So you would execute the command (depending if you want Ogg bitstream or Mp3 format)

ffmpeg -i  -vn -acodec vorbis 

ffmpeg -i  -vn -acodec mp3

Getting branch support to work with Nose/Coverage

2011-08-27T00:19:00.000-07:00

While Ned Batchelder's coverage utility has supported branch measurements for sometime, it hasn't been supported in the main line of the nose unit discovery util. We can see in the upcoming v1.1.3 release that the --cover-branches option will be supported:

http://readthedocs.org/docs/nose/en/latest/news.html

Currently nose v1.1.3 is still labeled as a development version, so you'd have to get pip install nose==dev in order to install this copy.

pip install --upgrade coverage

pip install --upgrade nosexcover

pip install --upgrade nose==dev (1.1.3)

An alternative, which has long been suggested in discussion groups, is to create a .coveragerc file to enable branch coverage by default. This file must be placed in the location where coverage.py is run, not necessarily in your home directory:

[run] 

     branch=True

If you've enabled things correctly, you should see the header (instead of the default) as follows:

Name                                              Stmts   Miss Branch BrPart  Cover   Missing

---------------------------------------------------------------------------------------------

If you're also using --with-xunit to generate Cobertura-style XML reports, hopefully you should also see the branch conditionals also being tallied correctly too!

Facebook's OAuth2 support for Python

2011-08-17T07:47:00.000-07:00

Facebook recently announced that they will be phasing in OAuth 2.0 support and require its use starting October 1, 2011. On the JavaScript SDK side, there are several changes on the JavaScript code that have to be done, which are listed as follows.

You can download the Python code here:

https://github.com/rogerhu/facebook_oauth2

1. FB.init has to be initialized with the Facebook APP ID instead of the API Key, though the apiKey parameter still is used.

2. The oauth: true options. must be set in the FB.init() calls.

In other words, the code changes would be:

FB.init({apiKey: facebook_app_id,
             oauth: true,
             cookie: true});

3. Instead of response.session, the response should now be response.authResponse. Also,
make note that scope: should be used instead of perms:

FB.login(function(response) {
    if (response.authResponse) {
    },
    {scope: 'email,publish_stream,manage_pages'}
    });

Also, if you need to retrieve the user id on the JavaScript, the value is stored as response.authResponse.userID instead of response.session.uid:

FB.api(
       { method: 'fql.query',
        query: 'SELECT ' + permissions.join() + ' FROM permissions WHERE uid=' + response.authResponse.userID},
        function (response) { });

If you see yourself not being able to logout, it means you haven't set the right APP ID or forgot to set oauth: true in both your login and logout code. If you're going to make the change, you should make it everywhere in your code!

On the Python/Django side, you need to implement a few helper routines. If Facebook authenticates properly, a cookie with the prefix fbsr_ will be set as a cookie (instead of fbs_). This signed request includes an encoded signature and payload, which must be separated and verified. You can look at the PHP SDK code to understand how it's implemented, or you can review this Python version of the code (see http://developers.facebook.com/docs/authentication/signed_request/)

def parse_signed_request(signed_request, secret):

    encoded_sig, payload = signed_request.split('.', 2)

    sig = base64_urldecode(encoded_sig)
    data = json.loads(base64_urldecode(payload))

    if data.get('algorithm').upper() != 'HMAC-SHA256':
        return None
    else:
        expected_sig = hmac.new(secret, msg=payload, digestmod=hashlib.sha256).digest()

    if sig != expected_sig:
        return None

    return data

In the PHP SDK code, there is a base64_url_decode function that automatically adds the correct number of "=" characters to the end of the Base64 encoded string. The basic problem is that Base64 encodes 3 bytes for every 4 characters, so the total length will be 4*len(string)/3. We can use this knowledge to realize that the total length will be a multiple of 4 and then insert the appropriate number of '=' characters to the end of the string. Facebook also appears to use a Base64-uRL variant in which the '+' and '/' characters of standard Base64 are respectively replaced by '-' and '_', which then must be replaced during the decode process (see http://en.wikipedia.org/wiki/Base64#URL_applications). The code looks like the following:

def base64_urldecode(data):
    # http://qugstart.com/blog/ruby-and-rails/facebook-base64-url-decode-for-signed_request/                       
    # 1. Pad the encoded string with "+".                                                                          
    # See http://fi.am/entry/urlsafe-base64-encodingdecoding-in-two-lines/                                         
    data += "=" * (4 - (len(data) % 4) % 4)

    return base64.urlsafe_b64decode(data)

If you're using the old Python SDK implementation, you may wish to implement code that mimics the way in which the Python SDK implemented get_user_from_cookie, since the expires, session_key, and oauth_token can be derived from retrieving the access token. We also set an fbsr_signed parameter in case you have debugging statements in your code and want to differentiate between your old get_user_from_cookie from this code.

Note: in order to make things backward-compatible, you need to make an extra URL request back to Facebook to retrieve the access token. This code was also inspired from the Facebook PHP SDK code too:

def get_access_token_from_code(code, redirect_url=None):
    """ OAuth2 code to retrieve an application access token. """

    data = {
        'client_id' : settings.FACEBOOK_APP_ID,
        'client_secret' : settings.FACEBOOK_SECRET_KEY,
        'code' : code,
        }

    if redirect_url:
        data['redirect_uri'] = redirect_url
    else:
        data['redirect_uri'] = ''


   return get_app_token_helper(data)

BASE_LINK = "https://graph.facebook.com"

def get_app_token_helper(data=None):
   
    if not data:
        data = {}

    try:
        token_request = urllib.urlencode(data)

        app_token = urllib2.urlopen(BASE_LINK + "/oauth/access_token?%s" % token_request).read()
    except urllib2.HTTPError, e:
        logging.debug("Exception trying to grab Facebook App token (%s)" % e)
        return None

    matches = re.match(r"access_token=(?P.*)", app_token).groupdict()

    return matches.get('token')

How to redirect bash time outputs..

2011-08-16T03:16:00.001-07:00

http://mywiki.wooledge.org/BashFAQ/032

How can I redirect the output of 'time' to a variable or file?

Bash's time keyword uses special trickery, so that you can do things like

   time find ... | xargs ...

and get the execution time of the entire pipeline, rather than just the simple command at the start of the pipe. (This is different from the behavior of the external command time(1), for obvious reasons.)

Because of this, people who want to redirect time's output often encounter difficulty figuring out where all the file descriptors are going. It's not as hard as most people think, though -- the trick is to call time in a SubShell or block, and then capture stderr of the subshell or block (which will contain time's results). If you need to redirect the actual command's stdout or stderr, you do that inside the subshell/block. For example:

File redirection:

   bash -c "time ls" 2>time.output      # Explicit, but inefficient.

   ( time ls ) 2>time.output            # Slightly more efficient.

   { time ls; } 2>time.output           # Most efficient.



   # The general case:

   { time some command >stdout 2>stderr; } 2>time.output

Vizio 42" LCD TV Tivo remote control code

2011-08-13T17:07:00.000-07:00

Tivo remote code is 0128.

http://www.tivocommunity.com/tivo-vb/archive/index.php/t-263308.html

Bash tests

2011-08-13T10:26:00.000-07:00

http://wiki.bash-hackers.org/commands/classictest

Apache's mod_rewrite and changing URL cases...

2011-08-13T10:06:00.000-07:00

Suppose we want to use Apache's mod_rewrite RewriteMap:

http://example.com/id to http://example.com/ID

RewriteMap uppercase int:toupper

RewriteRule [a-z] %{uppercase:%{REQUEST_URI}} [L,R=301]

The list of internal functions are listed here:
http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html#mapfunc

Internal Function

MapType: int, MapSource: Internal Apache function

Here, the source is an internal Apache function. Currently you cannot create your own, but the following functions already exist:



toupper:

Converts the key to all upper case.

tolower:

Converts the key to all lower case.

escape:

Translates special characters in the key to hex-encodings.

unescape:

Translates hex-encodings in the key back to special characters.

You can also use this approach to map static content to a list of servers:

Example:



Rewrite map file



##

##  map.txt -- rewriting map

##



static   www1|www2|www3|www4

dynamic  www5|www6

Configuration directives



RewriteMap servers rnd:/path/to/file/map.txt



RewriteRule ^/(.*\.(png|gif|jpg)) http://${servers:static}/$1 [NC,P,L]

RewriteRule ^/(.*) http://${servers:dynamic}/$1 [P,L]

Hash File

Celery v2.3.1

2011-08-09T10:35:00.000-07:00

Celery v2.3.0 has a new setting for the CELERY_RESULT_BACKEND that allows you to store the results of your apply_async() and dispatch() calls in something other than an AMQP-based backend. In previous versions Celery (without the ignore_result=True) would store these results as a message created by a separate queue corresponding to the taskset ID (a UUID). If you had a lot of tasks without consuming them (i.e. checking he result), you would eventually exhausting the memory usage.

The problem is well-described here. One of the issues was using an older version of RabbitMQ, which used a different persister that would try to keep everything in memory and would crash. With recent changes in RabbitMQ, which allow task results to be expired, the problem is much more mitigated. Nonetheless, setting ignore_result=True also helps with this respect. With the recent Celery v2.3.0 release you can also use a different backend (i.e. Redis) to store these task set results!

http://packages.python.org/celery/userguide/tasks.html#result-backends

Note: Celery is still highly dependent on an AMQP host. Just because you can change the CELERY_RESULT_BACKEND doesn't mean you can use a completely different messaging system.

Using find with --regex option

2011-08-09T03:02:00.000-07:00

http://www.gnu.org/software/findutils/manual/html_mono/find.html#posix_002degrep-regular-expression-syntax

Note that you need to specify .*/ in the beginning because find matches the whole path.

http://stackoverflow.com/questions/6844785/how-to-use-regex-with-find-command

Experimenting with the Kombu framework

2011-08-07T23:36:00.000-07:00

The Kombu framework is an excellent way to work with RabbitMQ/AMQP message brokers. The documentation has several different ways of doing so by instantiating the exchange, queue, and then the RabbitMQ (AMQP broker) connection (See http://packages.python.org/kombu/introduction.html). The documentation at http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ shows how you can use the py-amqplib to talk to RabbitMQ hosts, but the Kombu framework with its ability to support multiple back-ends provides a much more elegant approach.

Here's a simple code that we can use to talk to our queue if we also have our Celery configuration settings defined too. In this example, we only use the Queue class to consume messages. We can bind the default queue and exchange to the channel and then register a callback that will dump the message to stdout.

from celery.conf import settings

from kombu.connection import BrokerConnection

from kombu.messaging import Exchange, Queue, Consumer



connection = BrokerConnection(settings.BROKER_HOST, settings.BROKER_USER, settings.BROKER_PASSWORD, settings.BROKER_VHOST)



# RabbitMQ connection

channel = connection.channel()



default_exchange = Exchange("default", "direct", durable=True)

default_queue = Queue("default", exchange=default_exchange, key="default")

bound_default_queue = default_queue(channel)



def process_msg(msg):

    print "%s" % repr(msg)



bound_default_queue.consume(callback=process_msg)



while True:

    connection.drain_events()

We can also do the same by declaring a Consumer class too and calling the consume() to register the Consumer:


from celery.conf import settings

from kombu.connection import BrokerConnection

from kombu.messaging import Exchange, Queue, Consumer



connection = BrokerConnection(settings.BROKER_HOST, settings.BROKER_USER, settings.BROKER_PASSWORD, settings.BROKER_VHOST)



# RabbitMQ connection                                                                                                                                                                                                                          

channel = connection.channel()



default_exchange = Exchange("default", "direct", durable=True)



default_queue = Queue("default", exchange=default_exchange, key="default")



def process_msg(body, msg):

    print "body %s, msg %s" % (repr(body), repr(msg))



consumer = Consumer(channel, default_queue, callbacks=[process_msg])

consumer.consume()



while True:

    connection.drain_events()

How scheduled tasks and Celery and Kombu interfaces with RabbitMQ

2011-08-07T00:28:00.000-07:00

A really good overview of how PBS Education Technology uses Celery:

http://www.slideshare.net/tarequeh/life-in-a-queue-using-message-queue-with-django

If you want to learn the AMQP interface and try building consumers, producers, exchanges, queues, the Python code here is extremely useful to learning how the basic standard works with the amqplib, which is the basis on which Celery is built. Celery handles a lot of the higher-level functionality of building task queues, but the underlying plumbing is built on the Kombu framework (originally Carrot but completely rewritten):

http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/

The internals of how scheduled tasks are outlined in the Celery documentation:

http://ask.github.com/celery/internals/worker.html

When you create a scheduled task in Celery (using either the eta= or countdown= parameter), a message gets created that gets pickled (assuming you're using the default CELERY_TASK_SERIALIZER as pickle) that includes an 'eta' keyword:

print pickle.loads(msg.body).keys()
['retries', 'task', 'args', 'expires', 'eta', 'kwargs', 'id']
(Pdb) print pickle.loads(msg.body)['eta']
2012-08-02T17:03:00.593141

When you startup a Celery worker, it will create a Consumer that will begin to receive messages from the RabbitMQ broker. If it sees a message with an eta parameter, then the task is moved into an ETA scheduler queue instead of the ready queue. The message itself is not acknowledged until the task is executed, though since it is received by the Consumer, any other workers will not receive the same task. If the Consumer disconnects or loses connection from the RabbitMQ worker, then RabbitMQ will attempt to redeliver this message. The message itself will not be deleted until an acknowledge is sent back to the broker:

kombu/transport.py:

def ack(self):
       """Acknowledge this message as being processed.,                                     
       This will remove the message from the queue.                                         
                                                                                            
       :raises MessageStateError: If the message has already been                           
           acknowledged/requeued/rejected.                                                  
                                                                                            
       """
       if self.channel.no_ack_consumers is not None:
           consumer_tag = self.delivery_info["consumer_tag"]
           if consumer_tag in self.channel.no_ack_consumers:
               return
       if self.acknowledged:
           raise self.MessageStateError(
               "Message already acknowledged with state: %s" % self._state)
       self.channel.basic_ack(self.delivery_tag)
       self._state = "ACK"

When a message is received by Celery, it invokes a function called from_message(), which then passes on to the on_task() to insert into the queue. Notice that the 'ack' function, which will be used to acknowledge a message once it has been processed, is passed along to this routine.

celery/worker/listener.py:

task = TaskRequest.from_message(message, message_data, ack,
                                                logger=self.logger,
                                                hostname=self.hostname,

The ready queue itself (assuming no rate limits) uses a basic Python queue and uses the process_task function, which will then call self.acknowledge() and invoke the ack function that was passed into the initial TaskRequest.from_message creation.
celery/worker/__init.py:

if disable_rate_limits:
            self.ready_queue = FastQueue()
            self.ready_queue.put = self.process_task

Also, it appears that revoked tasks are not persistent if you do not setup a CELERYD_STATE_DB (defaults to None). Celery appears to keep all revoked tasks in memory and skips tasks if they are in this list of revoked task ID's without this setting. Without this configuration variable, all revoked tasks will be forgotten if you restart Celery.

Changes in Facebook's SWF code.

2011-08-05T08:16:00.000-07:00

Yesterday evening (8/4/2011) at 9:55 pm, Facebook changed some code that affects its Flash code which is used by Internet Explorer to handle cross-domain communication:

1		/1312412724,169546110,JIT Construction: v416050,en_US/
	1	/1312520159,169918336,JIT Construction: v416929,en_US/
2	2
3	3	if (!window.FB) window.FB = {
4	4	_apiKey: null,
…
30	30	return FB._domain.api_read;
31	31	case 'cdn':
32	32	return (window.location.protocol == 'https:' \|\| FB._https) ? FB._domain.https_cdn : FB._domain.cdn;
	33	case 'cdn_foreign':
	34	return FB._domain.cdn_foreign;
33	35	case 'https_cdn':
34	36	return FB._domain.https_cdn;
35	37	case 'graph':
…
246	248	for (var a = 0, b = FB.Flash._callbacks.length; a < b; a++) FB.Flash._callbacks[a]();
247	249	FB.Flash._callbacks = [];
248	250	};
249		FB.Flash.embedSWF('XdComm', FB.getDomain('cdn') + FB.Flash._swfPath);
	251	FB.Flash.embedSWF('XdComm', FB.getDomain('cdn_foreign') + FB.Flash._swfPath);
250	252	},
251	253	embedSWF: function(d, e, b) {
252	254	var a = !! document.attachEvent,
…
4952	4954	"api": "https:\/\/api.facebook.com\/",
4953	4955	"api_read": "https:\/\/api-read.facebook.com\/",
4954	4956	"cdn": "https:\/\/s-static.ak.fbcdn.net\/",
	4957	"cdn_foreign": "https:\/\/connect.facebook.net\/",
4955	4958	"graph": "https:\/\/graph.facebook.com\/",
4956	4959	"https_cdn": "https:\/\/s-static.ak.fbcdn.net\/",
4957	4960	"https_staticfb": "https:\/\/s-static.ak.facebook.com\/",
…
4968	4971	"_minVersions": [
4969	4972	[10, 0, 22, 87]
4970	4973	],
4971		"_swfPath": "rsrc.php\/v1\/yx\/r\/WFg56j28XFs.swf"
	4974	"_swfPath": "rsrc.php\/v1\/yK\/r\/RIxWozDt5Qq.swf"
4972	4975	}, true);
4973	4976	FB.provide("XD", {
4974	4977	"_xdProxyUrl": "connect\/xd_proxy.php?version=3"

You can fetch the new SWF file at (note though that the diff above indicates that the SWF must now be downloaded by the browser from http://connect.facebook.net/RIxWozDt5Qq.swf):

wget http://static.ak.fbcdn.net/rsrc.php/v1/yk/r/RIxWozDt5Qq.swf

By decompiling the SWF file using Sothink's SWF Decompiler (the unregistered version allows you to export up to the first two FLA files you designate to save), you can review the changes that were made.

diff PostMessage.as PostMessage_old.as

143,147d142
<         public static function extractPathAndQuery(param1:String) : String
<         {
<             return /^\w+:\/\/[^\/]+(.*)$""^\w+:\/\/[^\/]+(.*)$/.exec(param1)[1
];
<         }// end function
<

It also appears that the XDComm receiver must be downloaded/loaded from connect.facebook.net now, or at least originate from the facebook.com with an /intern/ URL specified. Otherwise, the cross-domain receiver will not initiate.

diff XDComm.as XDComm_old.a

19c19
<             XdComm.fbTrace("XdComm Constructor", {url:stage.loaderInfo.url});
---
>             XdComm.fbTrace("XdComm Initialized", {});
28d27
<             var _loc_4:String = null;
31,50d29
<             var _loc_2:* = stage.loaderInfo.url;
<             var _loc_3:* = PostMessage.extractDomain(_loc_2);
<             if (_loc_3 != "connect.facebook.net")
<             {
<                 XdComm.fbTrace("XdComm is not loaded from connect.facebook.net
", {swfDomain:_loc_3});
<                 if (_loc_3.substr(-13) == ".facebook.com")
<                 {
<                     _loc_4 = PostMessage.extractPathAndQuery(_loc_2);
<                     if (_loc_4.substr(0, 8) != "/intern/")
<                     {
<                         XdComm.fbTrace("XdComm is NOT in intern mode", {swfPat
h:_loc_4});
<                         return;
<                     }
<                     XdComm.fbTrace("XdComm is in intern mode", {swfPath:_loc_4
});
<                 }
<                 else
<                 {
<                     return;
<                 }
<             }
188c167,174
<             return param3;
---
>             if (param2 == 0)
>             {
>             }
>             else
>             {
>                 return param3;
>             }
>             return;
192a179
>             traceObject(param2);

The different versions of PostMessage.as and XdComm.as are posted here:

http://bit.ly/kAJ7AJ (PostMessage.as)
http://bit.ly/ltGkTF (XdComm.as)

What the prefetch in Celery means...

2011-08-05T00:48:00.001-07:00

http://ask.github.com/celery/userguide/optimizing.html

Prefetch Limits

Prefetch is a term inherited from AMQP that is often misunderstood by users.

The prefetch limit is a limit for the number of tasks (messages) a worker can reserve for itself. If it is zero, the worker will keep consuming messages, not respecting that there may be other available worker nodes that may be able to process them sooner[#], or that the messages may not even fit in memory.

The workers’ default prefetch count is the CELERYD_PREFETCH_MULTIPLIER setting multiplied by the number of child worker processes[#].

If you have many tasks with a long duration you want the multiplier value to be 1, which means it will only reserve one task per worker process at a time.

However – If you have many short-running tasks, and throughput/roundtrip latency[#] is important to you, this number should be large. The worker is able to process more tasks per second if the messages have already been prefetched, and is available in memory. You may have to experiment to find the best value that works for you.
Values like 50 or 150 might make sense in these circumstances. Say 64, or 128.

If you have a combination of long- and short-running tasks, the best option is to use two worker nodes that are configured separatly, and route the tasks according to the run-time. (see Routing Tasks).

[†]

RabbitMQ and other brokers deliver messages round-robin, so this doesn’t apply to an active system. If there is no prefetch limit and you restart the cluster, there will be timing delays between nodes starting. If there are 3 offline nodes and one active node, all messages will be delivered to the active node.

[‡]	This is the concurrency setting; `CELERYD_CONCURRENCY` or the -c option to celeryd.

Reserve one task at a time

When using early acknowledgement (default), a prefetch multiplier of 1 means the worker will reserve at most one extra task for every active worker process.

When users ask if it’s possible to disable “prefetching of tasks”, often what they really want is to have a worker only reserve as many tasks as there are child processes.

But this is not possible without enabling late acknowledgements acknowledgements; A task that has been started, will be retried if the worker crashes mid execution so the task must be idempotent (see also notes at Should I use retry or acks_late?).

You can enable this behavior by using the following configuration options:

CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

Using the git completion bash script

2011-08-04T23:52:00.000-07:00

http://repo.or.cz/w/git.git/blob_plain/HEAD:/contrib/completion/git-completion.bash

Within the file, the description is listed here:

3) Consider changing your PS1 to also show the current branch:

#         Bash: PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '

PS1 is the bash custom prompt:
http://www.cyberciti.biz/tips/howto-linux-unix-bash-shell-setup-prompt.html
The default bash prompt is usually:

echo $PS1
${debian_chroot:+($debian_chroot)}\u@\h:\w\$

...which is defined in /etc/bash.bashrc:

# set variable identifying the chroot you work in (used in the prompt below)
if [ -z "$debian_chroot" ] && [ -r /etc/debian_chroot ]; then
    debian_chroot=$(cat /etc/debian_chroot)
fi

The ${debian_chroot} is defined in /etc/bash.bashrc if the directory is chrooted, which it normally isn't so is left blank. The \u @ \h refers to the username and the host, the \w refers to the current working directory, and \$ will show '$' if the UID is not 0 (in which case, will show '#' to indicate root-level access)

time.mktime() versus calendar.timegm()

2011-08-02T16:30:00.001-07:00

http://stackoverflow.com/questions/2956886/python-calendar-timegm-vs-time-mktime

time.mktime() assumes that the passed tuple is in local time, calendar.timegm() assumes its in GMT/UTC. Depending on the interpretation the tuple represents a different time, so the functions return different values (seconds since the epoch are UTC based).

The difference between the values should be equal to the time zone offset of your local time zone.

Same-origin policy..

2011-08-02T01:09:00.000-07:00

The same origin policy requires that the protocol, port, and host are the same for both pages. Otherwise, Firefox apparently "preflights" the HTTP request:

https://developer.mozilla.org/en/HTTP_access_control

https://developer.mozilla.org/en/Same_origin_policy_for_JavaScript

Mozilla considers two pages to have the same origin if the protocol, port (if one is specified), and host are the same for both pages. The following table gives examples of origin comparisons to the URL http://store.company.com/dir/page.html:

URL	Outcome	Reason
`http://store.company.com/dir2/other.html`	Success
`http://store.company.com/dir/inner/another.html`	Success
`https://store.company.com/secure.html`	Failure	Different protocol
`http://store.company.com:81/dir/etc.html`	Failure	Different port
`http://news.company.com/dir/other.html`	Failure	Different host

jQuery events

2011-08-02T01:08:00.000-07:00

Ever curious to know what events are bound to DOM elements?   You can access DOM elements by its DOM data storage element:

http://james.padolsey.com/javascript/things-you-may-not-know-about-jquery/

// List bound events:
console.dir( jQuery('#elem').data('events') );
 
// Log ALL handlers for ALL events:
jQuery.each($('#elem').data('events'), function(i, event){
    jQuery.each(event, function(i, handler){
        console.log( handler.toString() );
    });
});
 
// You can see the actual functions which will occur
// on certain events; great for debugging!

Selenium 2.0 migration lessons

2011-07-19T16:00:00.000-07:00

At Hearsay Social, we've upgraded our testing environment to use Selenium 2. We made the switch because there was enough evidence to suggest a huge 2-4x performance increase. Having learned a few lessons along the way, we thought it would be helpful to share what we found, especially for those who are considering making the transition.

Since Selenium 2 is redesigned to leverage what works best for the browser, whether it's a NPAPI plugin in Firefox or a DLL module for IE, we've discovered a huge performance gain, especially in Internet Explorer (IE) browsers that have much slower JavaScript engines. The new approach seems to allow us to run Selenium more conveniently on Internet Explorer browsers without the hassle of changing the security options because of all the exceptions that were thrown as a result of the older JavaScript-based architecture.
Selenium 2 gets closer to simulating the behavior of a user on a browser. In Selenium 2, the DOM element that is actually clicked is determined by the X/Y coordinates of the mouse event. Therefore, if you attempt to search for a DOM element that is hidden or obstructed by another element, the top element will always be fired and you might encounter ElementNotVisibleException errors from the Selenium server. You need to keep this issue in mind when rewriting your tests, since Selenium 1 versions may not have had this restriction. (We use the Django web framework and the popular django-debug-toolbar, which adds a popup overlay in our web application that has to be disabled in our application during Selenium tests.)

We've found that the new Selenium 2 WebDriver-based API is easier to train our developers to use. The documentation for Selenium 2 is still somewhat sparse, especially for the updated Python bindings, so digging into the source code (in our case, remote/webdriver.py and remote/webelement.py code) is still the best way to learn what API commands are available. While Java developers may have access to WebDriverBackedSeleniumclass that can use existing Selenium 1 code while leveraging the WebDriver-based API, we didn't find any similar support for Python. So we took the plunge and refactored most of our tests.
webdriver/remote/webelement.py:

@property
    def tag_name(self):
        """Gets this element's tagName property."""
        return self._execute(Command.GET_ELEMENT_TAG_NAME)['value']

    @property
    def text(self):
        """Gets the text of the element."""
        return self._execute(Command.GET_ELEMENT_TEXT)['value']

    def click(self):
        """Clicks the element."""
        self._execute(Command.CLICK_ELEMENT)

    def submit(self):
        """Submits a form."""
        self._execute(Command.SUBMIT_ELEMENT)

    def clear(self):
        """Clears the text if it's a text entry element."""
        self._execute(Command.CLEAR_ELEMENT)

On the server-end, it's important to study how the client API is sending remote commands by reviewing the JsonWireProtocol document posted on the Selenium Wiki, especially since Sauce Labs provides you with the raw logs to see what commands are actually being issued by the client.

While experimenting with Selenium 2, we found it much easier to test out the new WebDriver API by downloading and running the Selenium server locally. This way, your connection won't constantly timeout as a result of using your Sauce Labs account, giving you more freedom to experiment with all the various commands. If you need to run browser tests against an external site while using your own machine to drive the browser interactions, you can setup a reverse SSH tunnel and then experiment with Selenium 2 API by setting debugger breakpoints and testing out the API bindings. In the long-term, though, you definitely want to use Sauce Labs for hosting all the virtual machines in the cloud for running your browser tests!
If you're interested in using Firebug to help debug your application, Selenium 2 also provides a way to inject Firefox profiles. You can create a Firefox profile with this plug-in extension, and Selenium 2 includes an API that will base-64, zip-encode the profile that will be downloaded by the remote host. Note that this approach works best if you're running the Selenium server locally, since using it over a Sauce Labs instance only gives you access to view the video.
Selenium 2 continues to be a moving target with its API, so you'll want to keep up to date with any release notes posted on the Selenium HQ blog. Most recently, we found that the toggle() and select() commands have not only been deprecated but removed completely from the implementation. If you try to issue these commands, the Selenium server simply doesn't recognize the commands and WebDriverExceptions are raised. The best thing to do is look at the Selenium version number. In this particular example, version 2.0.0 (three decimal places) are used to represent the release candidate of the latest Selenium build. You may also instantiate your .jar files with the -debug flag to watch how your client bindings execute API commands to the Selenium server.

20:38:02.687 INFO - Java: Sun Microsystems Inc. 20.1-b02
20:38:02.687 INFO - OS: Windows XP 5.1 x86
20:38:02.703 INFO - v2.0.0, with Core v2.0.0. Built from revision 12817

Selenium 1 users will find that is_text_present() and wait_for_condition() commands no longer exist, and are replaced by a more DOM-driven approach of selecting the element first before firing click() events or retrieving attribute values through get_attribute(). You no longer have to have wait_for_condition() for page loads. Instead, you set implicitly_wait() to a certain timeout limit to rely on find_element_by_id() to wait for the presence of DOM elements to appear to between page loads.
Lastly, we've noticed in the Selenium discussion groups that often there are questions about how to deal with concurrent Ajax requests during your tests. In many test frameworks, there's the concept of setup and tear down of the database between each set of tests. One issue that we encountered is that if your browser is issuing multiple requests, you're better off waiting for the Ajax requests to complete in your tear down function since the requests could arrive when the database is an unknown state. If this happens, then your Selenium tests will fail and you're going to spend extra time trying to track down these race conditions. If you're using jQuery, you can check the ajax.global state to determine whether to proceed between pages (i.e. execute_script("return jQuery.active === 0")). You'll want to keep looping until this condition is satisfied (for an example of implementing your own wait_for_condition() command, click here.)

Hope you find these tips helpful for migrating over to Selenium 2. Happy testing!

Issues with Vista/Windows 7 with the Kinesis Keyboard

2011-07-19T07:25:00.000-07:00

Kinesis has a few suggestions that may help resolve issues with the Kinesis on Vista/Windows 7.

The suggestions seem to be:

1. Hit Shift, both Ctrl keys, and both Alt keys to reset.

2. Change advanced power settings (USB) and set to disabled.

3. Replace the firmware (only through Kinesis) and/or firmware board.

http://www.kinesis-ergo.com/tech_support/trouble.htm#q8

2. Advantage keyboard not working after Windows computer goes to sleep mode
We have learned of a widely reported problem with Windows operating systems (XP & Vista) using third party keyboards (even though the problem has been reported with the Microsoft Natural keyboard as well) where the keyboard does not function after the computer enters sleep or hibernation mode. Fortunately there is a easy solution:
Go to Start-Settings-Control Panel, double click Keyboard. Click the Hardware Tab, highlight the "HID Keyboard device" and select Properties. Click Power Management. UNCHECK the box that says "Allow this device to bring the computer out of standby." Click OK. Now when your computer goes to standby or sleep mode, you will simply need to wake the computer by moving your pointing device. Your keyboard should now work perfectly fine.
For Vista users, go to Control Panel, Power Options, click "change plan settings" then "change advanced power settings" then go to "USB settings" and select "disabled."
Back to Top

8. Weird behavior with contoured keyboard
Stuck modifier key. Sometimes the computer misses the upstroke from keys that are held down in key combinations. The result is a "stuck" key. Try pressing both Shift keys, bothCtrl keys, and both Alt keys. If this problem happens to your contoured keyboard more than once every few weeks, you may need a firmware upgrade or new main circuit board.

Check your firmware version (serial numbers 20,000 and higher). Open a text editor other than Microsoft Word (e.g notepad, wordpad or equivalent). Press both Shift keys plus F12. The keyboard will produce a sentence which ends with the firmware version number and version date, such as:
copyright 1986 - 1998 by interfatron-bbc, ltd., rev 2.48 08/13/98.

Circuit board may need replacement. If your contoured keyboard was built before April, 1998, it may need a new main circuit board. The old boards look brown if you look past the thumb keys at the underlying circuit board. The new circuit boards look green with a gold grid.

Back to Top

sudo race condition in Fabric 1.0.0

2011-07-13T01:40:00.000-07:00

If you've used Fabric with sudo, you might notice keystrokes during the password phase sometimes doesn't work. The problem with Fabric for sudo() command is tracked here:

http://code.fabfile.org/issues/show/320

The diff for Fabric 1.0.2 is here:
http://code.fabfile.org/repositories/diff/fabric/efd9fef820dd6f2a593c172eac7e500c5e8f0602

It appears that the fix basically shuts off the input_loop when the connection is attempting to use get_pass() when performing an SSH connection to another machine with Fabric. The problem is
that while Fabric claims it is single-threaded, but it still creates thread handlers for standard input/output:

        workers = (
            ThreadHandler('out', output_loop, channel, "recv", stdout),
            ThreadHandler('err', output_loop, channel, "recv_stderr", stderr),
            ThreadHandler('in', input_loop, channel, using_pty)
        )

The fix disables input checking so that the getpass inside the prompt_for_password() command can work correctly.

Pruning GitHub / merged

2011-07-12T15:15:00.001-07:00

Tired of all your outdated branches? Now there's a way..

git branch --merged | grep -v "^* " | awk '{print "git branch -d " $1 "; git push origin :" $1}' | sh

What it boils down to:

git branch --merged = shows all the merged ones
grep -v "^*" = excludes the current branch (can't delete a branch you're currently on)
awk '{print "git branch -d " $1 "; git push origin :" $1}' = delete merged branch and delete it on GitHub.

If you want to double-check, take away the last "| sh" to verify everything works. You have been warned.

You may get some errors if the branch that has already been merged no longer exists on your GitHub origin..

Django and large datasets

2011-07-12T14:00:00.000-07:00

If you've worked with large data sets with Python/Django, memory consumption can hit 2GB on dev, even if settings.DEBUG=False..

Django's documentation about querysets hints about the issue does little to mention this issue about big datasets:

https://docs.djangoproject.com/en/dev/topics/db/queries/#

Caching and QuerySets
Each QuerySet contains a cache, to minimize database access. It's important to understand how it works, in order to write the most efficient code.

In a newly created QuerySet, the cache is empty. The first time a QuerySet is evaluated -- and, hence, a database query happens -- Django saves the query results in the QuerySet's cache and returns the results that have been explicitly requested (e.g., the next element, if the QuerySet is being iterated over). Subsequent evaluations of the QuerySet reuse the cached results.

Keep this caching behavior in mind, because it may bite you if you don't use your QuerySets correctly. For example, the following will create twoQuerySets, evaluate them, and throw them away:

...so if you do queries against big datasets (i.e.):

for item in MyTable.objects.all():

...memory consumption can hit the roof. The culprit was iterating through all objects (475,000)...as the size increases, the self._result_cache in the QuerySet object just keeps appending. Even though Django fetches 100 rows at a time from MySQL, each time you iterate, it just adds to the result_cache array such that you will end up caching all 475K objects. No clearing, nada.

Disqus approached the problem by implementing a SkinnyQuerySet to avoid storing extra data in memory, though it's not clear to me that the changes they discussed attempted to query against a large dataset (i.e. not using LIMIT range queries) See p. 25 at http://www.scribd.com/doc/37072033/DjangoCon-2010-Scaling-Disqus (or the source code for SkinnyQuerySet at https://gist.github.com/550438). FYI -- the code also changes to prevent you from doing list() over a QuerySet.

The code that does all this stuff is here (ITER_CHUNK_SIZE is 100):
external/django/db/models/query.py(757)_fill_cache()

 def _result_iter(self):
     pos = 0
     while 1:
         upper = len(self._result_cache)
         while pos < upper:
             yield self._result_cache[pos]
             pos = pos + 1
         if not self._iter:
             raise StopIteration
         if len(self._result_cache) <= pos:
             self._fill_cache()

def _fill_cache(self, num=None):
    """
    Fills the result cache with 'num' more entries (or until the results
    iterator is exhausted).
    """
    if self._iter:

        try:
            for i in range(num or ITER_CHUNK_SIZE):
                self._result_cache.append(self._iter.next())
        except StopIteration:
            self._iter = None

The iterator() fixes the problem with Django caching over time, but I realized that the problem happened on the first iteration of a queryset, and looking at this object reference graph (you have to install the objgraph library) I inspected one of these objects and you can see that it's part of all the objects already loaded in MySQLdb's _rows dictionary after just one QuerySet iteration. So basically any QuerySet even with iterator() will store the entire SQL result in memory.

(Pdb) import objgraph
(Pdb) objgraph.show_most_common_types(limit=20)
tuple                      41682

(Pdb) next
-> if not result_type:
(Pdb) import objgraph
(Pdb) objgraph.show_most_common_types(limit=20)
tuple                      419858

What ultimately is happening is that it generates SQL command, sends it over to the Python MySQLdb wrapper, which eventually calls cursor.execute(), which calls store_result(), which stores the entire result set in memory (see store_result() reference in http://mysql-python.sourceforge.net/MySQLdb.html) The tradeoff is either to load everything in memory or one row at a time. Obviously we have more control and do LIMIT ranges to try to balance between too much memory and too many DB connections.

django/db/models/sql/compiler.py:
728  ->         cursor = self.connection.cursor()
729             cursor.execute(sql, params)

/usr/local/lib/python2.6/dist-packages/MySQLdb/cursors.py(313)
"""This is a MixIn class which causes the entire result set to be
stored on the client side, i.e. it uses mysql_store_result(). If the
result set can be very large, consider adding a LIMIT clause to your
query, or using CursorUseResultMixIn instead."""

def _get_result(self): return self._get_db().store_result()

The solution seems to use an implementation similar to query_iterator where you do LIMIT's upfront on QuerySet (you can also use iterator() to avoid caching the result set...so it seems a combination of Disqus and this snippet http://djangosnippets.org/snippets/1949/...

Selenium 2 RC3 and Python bindings have removed toggle() and select()...

2011-07-12T03:16:00.000-07:00

-    def toggle(self):
76   
-        """Toggles the element state."""
77   
-        resp = self._execute(Command.TOGGLE_ELEMENT)
78   
-        return resp['value']
79   
-
80 68 
     def is_selected(self):
81 69 
         """Whether the element is selected."""
82 70 
         return self._execute(Command.IS_ELEMENT_SELECTED)['value']
83 71 
 
84   
-    def select(self):
85   
-        """Selects an element."""
86   
-        self._execute(Command.SET_ELEMENT_SELECTED)
87   
-

IEDriver.dll changes:
http://code.google.com/p/selenium/source/browse/trunk/cpp/IEDriver/CommandHandlers/ClickElementCommandHandler.h?spec=svn12477&r=12477

How Facebook invalidates fbs_ cookies through all.js...

2011-07-08T17:22:00.000-07:00

If you've ever had to integrate with Facebook Connect, which allows third-party web sites to rely on Facebook for user authentication, you may find yourself wanting to test whether your integration works. You can obviously use Selenium to automate the login process through the popup window, but what if you just wanted to try to skip this step and set the cookie yourself?

Each time you login through the Facebook popup window, the JavaScript code will set an fbs_ cookie. The xxx in the fbs_xxxx cookie refers to the Facebook API key, and is signed with your application's secret key. If you have an infinite session key and a valid application access token, you can create your own fbs_xxx cookie. You can basically reverse the process that is normally done by the get_user_from_cookie from the Python Graph API library:

def get_user_from_cookie(cookies, app_id, app_secret):
    cookie = cookies.get("fbs_" + app_id, "")
    if not cookie: return None
    args = dict((k, v[-1]) for k, v in cgi.parse_qs(cookie.strip('"')).items())
    payload = "".join(k + "=" + args[k] for k in sorted(args.keys())
                      if k != "sig")
    sig = hashlib.md5(payload + app_secret).hexdigest()
    expires = int(args["expires"])
    if sig == args.get("sig") and (expires == 0 or time.time() < expires):
        return args
    else:
        return None

However, even when using a valid cookie signed by your app, it appears that Facebook invalidates the token. You end up seeing the same login popup window, but if you remove the all.js library and use a fake substitute, you'll notice that your app all of a suddenly works. So Facebook is doing something internally to disallow self-signed fbs_cookies.

function do_nothing(a, b, c, d, e, f) {
    return true;
}

var FB = {
    'init': do_nothing,
    'getLoginStatus': function () {},
    'login': do_nothing
};

When you first run the init() function for the Facebook JavaScript SDK code, it first loads the cookie that you set (fbs_ + API_KEY) and sets FB_session to correspond to these cookie values. The problem is that another auth.request command is sent to Facebook. This auth.request command tries to verify whether a session key was obtained from Facebook. If it fails, then the FB._session object is cleared and that pop-up window appears asking you to login. The Facebook JavaScript does this auth.request by injecting an <iframe src="http://www.facebook.com/extern/login_status.php?"> with three different callback functions in the URL query string (no_user, no_session, and ok_session).

The result from this IFrame request basically used to invoke the response, which is handled by either the no_user, no_session, or ok_session xdResponseWrappers. If the no_user/no_session response wrappers, then the 'connected' state will not be set internally and the session will be cleared. The code that invalidates this is located inside the cross-domain (aka 'xd') xdResponseWrapper function inside all.js:

xdResponseWrapper: function(a, c, b) {
   return function(d) {
     try {
       b = FB.JSON.parse(d.session);
     } catch (f) {}
     if (b) c = 'connected';
     var e = FB.Auth.setSession(b || null, c);
     e.perms = d && d.perms || null;
     a && a(e);
   };

When the no_user response is called, d.session apparently is null, which will then be used by FB.Auth.setSession() to clear the FB._session object. You can't easily set a breakpoint on how this happens because the Facebook all.js code creates a function that does so, so it isn't apparent unless you insert alert() statements. But basically if you put alert statements, you will observe that the no_user result is returned when you attempt to just set the cookie and login to your site.

It looks like Facebook has implemented this policy to try to prevent people from using robots to directly login to their site. It also appears that there are additional cookies that must be set (datr, lsd, c_user, h_user, lxe, and xs) in order for the login_status.php to return back the ok_session callback. You can observe this behavior by monitoring the cookie traffic if you were to login through the popup window.

Apparently the 'xs' cookie is a well-known Facebook cookie too: http://www.duke.edu/~jyw2/wwwsecurity.html

Deconstructing Facebook's Flash Cross Domain Handler in IE..

2011-06-25T05:01:00.001-07:00

I got tired of trying to understand why Facebook kept breaking on their IE support, so decided to peer directly into how they use Flash to implement the cross-domain receiver (you can see inside the .js file they supply that it fetches the SWF file http://static.ak.fbcdn.net/rsrc.php/v1/yx/r/WFg56j28XFs.swf I now suspect the problems Fri. morning that other people reported had to do with 64-bit Internet Explorer, since the Adobe Flash version is still highly experimental, and rebooting my machine (and my other machines at work had no problems) solved the issue.

Nonetheless, the hard part was to find a decompiler that could convert the SWF file that gets loaded in Internet Explorer back into bytecode. I tried to use flasm but this file was encoded using Adobe Flex 4, which was released around March 2010 so the open source tools are a bit scant. I ended up using an unregistered version of Sothink's SWFDecompiler, which wants $70.00 for a license. An older 6.0 version seems to have some quirky bugs where it actually lets you export the ActionScript files back to the original format. Regardless, the program preserved all the original variable names, so you can pretty much have the original format, and I found a way to post the decompiled files in their entirety here:
http://hustoknow.blogspot.com/2011/06/facebooks-flash-xdcomm-receiver.html

The way in which Facebook has implemented cross-domain message passing in Internet Explorer mimics the use of the HTML5 postMessage. Instead of creating an event sender/listener for messages from a different window to pass, Facebook relies on SWF objects to talk with other in Internet Explorer. In Flash, there is the concept of LocalConnection objects, where SWF objects can talk with other even when they are on different domains. It injects this SWF file into both your site as well as after you login through the window popup (facebook.com).

<object type="application/x-shockwave-flash" id="XdComm" name="XdComm" classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" allowscriptaccess="always"><param name="movie" value="https://s-static.ak.fbcdn.net/rsrc.php/v1/yx/r/WFg56j28XFs.swf"></param><param name="allowscriptaccess" value="always"></param></object>

So when you go to any Facebook Connect-enabled web site, you instantiate a SWF object that creates a XdComm object, along with an extra postMessage object. The XdComm class holds a PostMessage() object, which in turns instantiates a LocalConnection() object, which is basically becomes a port sender/receiver for messages. Adobe SWF objects, which are contained within ActiveX can be accessed via JavaScript, so the object tag gets added with the XdComm ID and Facebook's JavaScript code can access the postMessage_init() function, which is initialized in the SWF file.

   public function PostMessage()
        {
            this.connection = new LocalConnection();
            this.connection.client = this;
            this.connection.connect(Math.random().toString());
            ExternalInterface.addCallback("postMessage_init", this.init);
            ExternalInterface.addCallback("postMessage_send", this.send);
            return;
        }// end function

When you do an FB.init() on your localhost, it instantiates FB.XD._openerOrigin (i.e. http://dev.myhost.com/ + 16-character random string), which is used to define the connection name that SWF objects will be use to communicate. The FB.XD_openerOrign was the key to the problems this past week -- Facebook was messing around with stuff for IE8, and broke the connection name that the SWF objects would agree on listening. They later fixed it based on the reviews of the diff I examined. When you type your login and password, the URL that is used when you submit goes to http://static.ak.fbcdn.net/connect/xd_proxy.php, along with the origin= query string parameter. So origin must be the same on the window popup connecting to facebook.com as the original one that was initialized on your own Facebook Connect-enabled site.

  } else if (f.transport == 'flash') {
                    var d = window.location.toString() + Math.random();
                    window.FB_OnFlashXdCommReady = function() {
                        document.XdComm.postMessage_init('dummy', d);
                        document.XdComm.postMessage_send(a, f.origin);
                    };
                    createXdCommSwf();
                }

In addition to setting up the origin= query parameter, the FB.init() functions also sets up 3 separate callback functions: ok_session, cancel_session, and no_user, which are instantiated into a callback dictionary each accessible by a random 16-character signature. So if you've ever wondered why the query string on the Facebook login is so long, it's partially because of all the info they encode to handle the results of different authentications. Regardless, the message that gets passed includes a bunch of different URL-encoded string, but one of them is a cb= parameter, which specifies which callback function to run.

Once that callback function is run, then the FB JavaScript calls FB.Auth.setSession() and fires off events...but the message between facebook.com and your site has been successfully exchanged...all via SWF objects in Internet Explorer!

Initial investigations of the XD.Flash...

2011-06-25T02:42:00.000-07:00

One of the first things the Facebook Connect library does when a user on Internet Explorer clicks the "Facebook Connect" is to insert this Adobe Flash file:

<object type="application/x-shockwave-flash" id="XdComm" name="XdComm" classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" allowscriptaccess="always"><param name="movie" value="https://s-static.ak.fbcdn.net/rsrc.php/v1/yx/r/WFg56j28XFs.swf"></param><param name="allowscriptaccess" value="always"></param></object>

(The classid refers to the ActiveX object of the Adobe Flash player, which apparently remains constant. The allowscriptaccess option apparently allows outbound URL requests.)

Within the Facebook Connect all.js file, there is a line that refers to initializing the XdComm object. It passes in two parts, one the actual name of the callback handler, and the second is the connection name on which to listen for messages. For messages to be received, the connection name must be the same.

document.XdComm.postMessage_init('FB.XD.Flash.onMessage', FB.XD._openerOrigin ? FB.XD._openerOrigin : FB.XD._origin);

When you login to Facebook Connect, you're POST'ng to xd_proxy.php, which also loads a SWF file (assuming transport=flash) and sets the origin to the one provided in the origin=. By doing so, both the window that was opened as well as the original document can now communicate with each other. The xd_proxy.php if you were to download it looks like:

<!doctype html>
<html>
    
    <head>
        <title>
            XD Proxy
        </title>
    </head>
    
    <body onload="doFragmentSend()">
        <div id="swf_holder" style="position: absolute; top: -10000px; width: 1px; height: 1px">
        </div>
        <script>
            var XdCommUrl = "http:\/\/static.ak.fbcdn.net\/rsrc.php\/v1\/yx\/r\/WFg56j28XFs.swf";
            var AllowFbCom = false;

            function resolveRelation(b) {
                var g, d, f = b.split('.'),
                    e = window;
                for (var a = 0, c = f.length; a < c; a++) {
                    g = f[a];
                    if (g === 'opener' || g === 'parent' || g === 'top') {
                        e = e[g];
                    } else if (d = /^frames\[['"]?([a-zA-Z0-9-_]+)['"]?\]$/.exec(g)) {
                        e = e.frames[d[1]];
                    } else throw new SyntaxError('Malformed id to resolve: ' + b + ', pt: ' + g);
                }
                return e;
            }
            function createXdCommSwf() {
                var a = !! document.attachEvent,
                    c = /(?:MSIE.(\d+\.\d+))|(?:(?:Firefox|GranParadiso|Iceweasel).(\d+\.\d+))|(?:Opera(?:.+Version.|.)(\d+\.\d+))|(?:AppleWebKit.(\d+(?:\.\d+)?))/.exec(navigator.userAgent),
                    b = (c[1] && (parseFloat(c[1]) >= 9)),
                    e = 'XdComm' + (b ? (Math.random() * (1 << 30)).toString(16).replace('.', '') : ''),
                    d = ('<object ' + 'type="application/x-shockwave-flash" ' + 'id="' + e + '" ' + (a ? 'name="XdComm" ' : '') + (a ? '' : 'data="' + XdCommUrl + '" ') + (a ? 'classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" ' : '') + 'allowscriptaccess="always">' + '<param name="movie" value="' + XdCommUrl + '"></param>' + '<param name="allowscriptaccess" value="always"></param>' + '</object>');
                if (b) window.attachEvent('onunload', function() {
                    document.getElementById(e).removeNode(true);
                });
                document.getElementById('swf_holder').innerHTML = d;
            }
            function doFragmentSend() {
                if (!AllowFbCom && /(^|\.)facebook.com$/.test(document.domain.toString())) return;
                var c = window.location.toString(),
                    a = c.substr(c.indexOf('#') + 1),
                    f = {},
                    g = a.split('&'),
                    b, e;
                for (b = 0; b < g.length; b++) {
                    e = g[b].split('=', 2);
                    f[decodeURIComponent(e[0])] = decodeURIComponent(e[1]);
                }
                if (f.transport == 'postmessage') {
                    var h = resolveRelation(f.relation);
                    if (h) {
                        h.postMessage(a, f.origin);
                    } else if (f.relation == 'opener') {
                        var i = 0;
                        i = setInterval(function() {
                            if (window.fbrpc) {
                                clearInterval(i);
                                if (fbrpc.postMessage) {
                                    fbrpc.postMessage(a, f.origin);
                                } else fbrpc.call('postMessage', JSON.stringify({
                                    target: f.relation,
                                    message: a
                                }));
                            }
                        }, 10);
                    }
                } else if (f.transport == 'flash') {
                    var d = window.location.toString() + Math.random();
                    window.FB_OnFlashXdCommReady = function() {
                        document.XdComm.postMessage_init('dummy', d);
                        document.XdComm.postMessage_send(a, f.origin);
                    };
                    createXdCommSwf();
                }
            }
        </script>
    </body>

</html>

Remember in the previous XdComm file that the postMessage_init() and postMessae_send() were added as registered callbacks. In ActionScript, you can invoke commands directly in JavaScript by calling them as if there were functions that are part of the Active-X module. The loading of the 'dummy' and postMessage_send() gets sent to the LocalConnection() origin name, which your original page is also listening. In this way, this message will be received by the onMessage handler with the Flash object of the all.js library, which is URI decoded before passing back so that the query string to be decoded and the callback specified in the cb= is provided.

Flash: {
        init: function() {
            FB.Flash.onReady(function() {
                document.XdComm.postMessage_init('FB.XD.Flash.onMessage', FB.XD._openerOrigin ? FB.XD._openerOrigin : FB.XD._origin);
            });
        },
        onMessage: function(a) {
            FB.XD.recv(decodeURIComponent(a));
        }
    },

(Also, there is also a postMessage included in the code, but presumably if not event listener is added, it will be ignored by Internet Explorer): FYI -- The old REST API relied on using the oauthRequest() code to sendXdHttpRequest() to make the external Facebook request via the FB.ApiServer.flash() command, which makes the external request, but this section of the code seems deprecated. When the request is finished, the XdCom library creates an event on FB_OnXdHttpResult using the ExternalInterfaceclass that communicates back to the JavaScript/HTML container.

oauthRequest: function(b, f, c, e, a) {
        try {
            FB.ApiServer.jsonp(b, f, c, FB.JSON.flatten(e), a);
        } catch (g) {
            if (FB.Flash.hasMinVersion()) {
                FB.ApiServer.flash(b, f, c, FB.JSON.flatten(e), a);
            } else throw new Error('Flash is required for this API call.');
        }

See the http://hustoknow.blogspot.com/2011/06/facebooks-flash-xdcomm-receiver.html for an examination on the Flash XDComm .swf file decompiled....

Decompiling Facebook's Flash XDComm Receiver..

2011-06-24T20:07:00.000-07:00

Within Facebook's all.js file, there is a reference to a _swfPath that refers to http://static.ak.fbcdn.net/rsrc.php/v1/yx/r/WFg56j28XFs.swf. It's used for Internet Explorer installed with Flash to handle cross-domain requests. Ever wonder how it works? Well, you can do a wget and decompile it.

Decompiling the XDComm.swf file: None of the open source Flash decompilers work to decompile Facebook's XDComm Receiver, since it's compiled in Adobe Flex v4 (released apparently in March 2010). You can use the Sothink SWF Decompiler tool, but without paying $70.00, the program doesn't even allow you to export the files to ActionScript. Nonetheless, the unregistered version (as of the 6.00 version) limits you from decrypting more than the first two resources, but there's enough in the tool that allows you to save parts of the code.

Here are flashutils\PostMessage.as and XdComm.as. Hopefully it gives you a better idea of how the Flash cross-domain receiver works.

The files can also be downloaded from here:

http://bit.ly/kAJ7AJ
http://bit.ly/ltGkTF

Apparently the XdComm utilizes the LocalConnection library to communicate with the HTML page. It used the send/receive functionality and the FB.XD._origin variable to determine what the connection name should be used.

In http://hustoknow.blogspot.com/2011/06/how-flash-xdcomm-works-in-facebook.html, we'll walk through in more detail how the Flash XDComm Receiver works with Facebook.

XdComm.as:

package 
{
    import flash.display.*;
    import flash.events.*;
    import flash.external.*;
    import flash.net.*;
    import flash.system.*;
    import flashutils.*;

    public class XdComm extends Sprite
    {
        private var _cache:SharedObject;
        private var _cacheContext:String = "unknown";
        private var postMessage:PostMessage;
        private static var requestIdCount:int = 0;

        public function XdComm()
        {
            XdComm.fbTrace("XdComm Initialized", {});
            Security.allowDomain("*");
            Security.allowInsecureDomain("*");
            this.addEventListener(Event.ENTER_FRAME, this.init);
            return;
        }// end function

        private function init(event:Event) : void
        {
            XdComm.fbTrace("XdComm.init", {});
            this.removeEventListener(Event.ENTER_FRAME, this.init);
            this.postMessage = new PostMessage();
            ExternalInterface.addCallback("sendXdHttpRequest", this.sendXdHttpRequest);
            ExternalInterface.addCallback("setCache", this.setCache);
            ExternalInterface.addCallback("getCache", this.getCache);
            ExternalInterface.addCallback("setCacheContext", this.setCacheContext);
            ExternalInterface.addCallback("clearAllCache", this.clearAllCache);
            ExternalInterface.call("FB_OnFlashXdCommReady");
            return;
        }// end function

        public function sendXdHttpRequest(param1:String, param2:String, param3:String, param4) : int
        {
            var loader:URLLoader;
            var reqId:int;
            var loaded:Function;
            var key:String;
            var value:String;
            var method:* = param1;
            var url:* = param2;
            var requestBody:* = param3;
            var extraHeaders:* = param4;
            loaded = function (event:Event) : void
            {
                var _loc_2:* = loader.data.toString();
                XdComm.fbTrace("Requested completed", {data:_loc_2});
                ExternalInterface.call("FB_OnXdHttpResult", reqId, encodeURIComponent(_loc_2));
                return;
            }// end function
            ;
            XdComm.fbTrace("SendXdHttpRequest", {method:method, url:url, requestBody:requestBody, extraHeaders:extraHeaders});
            if (url.indexOf("https://") != 0 && url.indexOf("http://") != 0)
            {
                url = "http://" + url;
            }
            var host:* = PostMessage.extractDomain(url);
            if (host != "api.facebook.com" && host != "graph.facebook.com" && !/^(api|graph)\.[A-Za-z0-9\-\.]+\.facebook\.com$""^(api|graph)\.[A-Za-z0-9\-\.]+\.facebook\.com$/.test(host))
            {
                return 0;
            }
            var _loc_6:* = XdComm;
            var _loc_7:* = XdComm.requestIdCount + 1;
            _loc_6.requestIdCount = _loc_7;
            var req:* = new URLRequest(url);
            loader = new URLLoader();
            reqId = XdComm.requestIdCount;
            req.method = method;
            req.data = requestBody;
            if (extraHeaders != null)
            {
                var _loc_6:int = 0;
                var _loc_7:* = extraHeaders;
                while (_loc_7 in _loc_6)
                {
                    
                    key = _loc_7[_loc_6];
                    value = extraHeaders[key];
                    req.requestHeaders.push(new URLRequestHeader(key, value));
                }
            }
            loader.addEventListener(Event.COMPLETE, loaded);
            loader.load(req);
            return reqId;
        }// end function

        private function setCacheContext(param1:String) : void
        {
            if (param1 == null)
            {
                param1 = "unknown";
            }
            this._cacheContext = param1;
            return;
        }// end function

        private function clearAllCache() : void
        {
            this.cache.clear();
            this.cache.flush();
            return;
        }// end function

        private function getCache(param1:String) : String
        {
            return this.contextCache[param1];
        }// end function

        private function setCache(param1:String, param2:String) : void
        {
            var _loc_3:* = this.contextCache;
            _loc_3[param1] = param2;
            this.cache.flush();
            return;
        }// end function

        private function get cache() : SharedObject
        {
            if (this._cache == null)
            {
                this._cache = SharedObject.getLocal("cache");
            }
            return this._cache;
        }// end function

        private function get contextCache() : Object
        {
            var _loc_1:* = this.cache.data[this._cacheContext];
            if (_loc_1 == null)
            {
                _loc_1 = new Object();
                this.cache.data[this._cacheContext] = _loc_1;
            }
            return _loc_1;
        }// end function

        public static function traceObject(param1:Object, param2:int = 0, param3:String = "")
        {
            var _loc_6:* = undefined;
            var _loc_7:String = null;
            var _loc_4:String = "";
            var _loc_5:int = 0;
            while (_loc_5 < param2)
            {
                
                _loc_4 = _loc_4 + "\t";
                _loc_5++;
            }
            for (_loc_6 in param1)
            {
                
                param3 = param3 + (_loc_4 + "[" + _loc_6 + "] => " + param1[_loc_6]);
                _loc_7 = traceObject(param1[_loc_6], (param2 + 1));
                if (_loc_7 != "")
                {
                    param3 = param3 + (" {\n" + _loc_7 + _loc_4 + "}");
                }
                param3 = param3 + "\n";
            }
            if (param2 == 0)
            {
            }
            else
            {
                return param3;
            }
            return;
        }// end function

        public static function fbTrace(param1:String, param2:Object) : void
        {
            traceObject(param2);
            return;
        }// end function

    }
}

...and postMessage.as:

package flashutils
{
    import flash.events.*;
    import flash.external.*;
    import flash.net.*;
    import flash.utils.*;

    public class PostMessage extends Object
    {
        private var currentDomain:String;
        private var callback:String;
        private var connection:LocalConnection;
        private var connectionName:String;

        public function PostMessage()
        {
            this.connection = new LocalConnection();
            this.connection.client = this;
            this.connection.connect(Math.random().toString());
            ExternalInterface.addCallback("postMessage_init", this.init);
            ExternalInterface.addCallback("postMessage_send", this.send);
            return;
        }// end function

        public function getCurrentDomain() : String
        {
            if (!this.currentDomain)
            {
                try
                {
                    this.currentDomain = ExternalInterface.call("self.document.domain.toString");
                    this.fbTrace("getCurrentDomain", {currentDomain:this.currentDomain});
                }
                catch (e)
                {
                    logError("getCurrentDomain error", e);
                }
            }
            return this.currentDomain;
        }// end function

        public function onFacebookDomain() : Boolean
        {
            return /(^|\.)facebook\.com$""(^|\.)facebook\.com$/.test(this.getCurrentDomain()) || /(^|\.)fbcdn\.net$""(^|\.)fbcdn\.net$/.test(this.getCurrentDomain());
        }// end function

        public function init(param1:String, param2:String) : void
        {
            var cb:* = param1;
            var name:* = param2;
            this.fbTrace("init", {cb:cb, name:name});
            try
            {
                if (!this.onFacebookDomain() && PostMessage.extractDomain(name) != this.getCurrentDomain())
                {
                    this.logError("init", "name must be a URL on the current domain: " + name);
                }
                else
                {
                    this.callback = cb;
                    if (name == this.connectionName)
                    {
                        return;
                    }
                    this.connectionName = name;
                    name = encodeURIComponent(name);
                    this.connection = new LocalConnection();
                    this.connection.client = this;
                    this.connection.connect(name);
                }
            }
            catch (e)
            {
                logError("init", e.toString());
            }
            return;
        }// end function

        public function send(param1:String, param2:String) : void
        {
            var msg:* = param1;
            var name:* = param2;
            this.fbTrace("send", {name:name, msg:msg});
            if (!this.connection)
            {
                this.logError("send", "connection has not been initialized.");
                return;
            }
            try
            {
                name = encodeURIComponent(name);
                this.connection.send(name, "recv", msg);
            }
            catch (e)
            {
                logError("send", e.toString() + ". name: " + name + ", msg: " + msg);
            }
            return;
        }// end function

        public function recv(param1:String) : void
        {
            var deliverMessage:Function;
            var msg:* = param1;
            deliverMessage = function (event:TimerEvent) : void
            {
                var evt:* = event;
                try
                {
                    ExternalInterface.call(callback, encodeURIComponent(msg));
                }
                catch (e)
                {
                    logError("recv", e.toString());
                }
                return;
            }// end function
            ;
            this.fbTrace("recv", {msg:msg});
            var timer:* = new Timer(1, 1);
            timer.addEventListener(TimerEvent.TIMER, deliverMessage);
            timer.start();
            return;
        }// end function

        private function logError(param1:String, param2:Object) : void
        {
            XdComm.fbTrace("Error: XdComm.PostMessage." + param1, param2);
            return;
        }// end function

        private function fbTrace(param1:String, param2:Object) : void
        {
            XdComm.fbTrace("XdComm.PostMessage." + param1, param2);
            return;
        }// end function

        public static function extractDomain(param1:String) : String
        {
            return /^\w+:\/\/([^\/:]*)""^\w+:\/\/([^\/:]*)/.exec(param1)[1];
        }// end function

    }
}

Daily snapshots of Facebook Connect Library

2011-06-23T12:49:00.000-07:00

Nathan Friedly from Sociable Labs put together a set of utils to grab daily snapshots of Facebook's connect.js here:

https://github.com/nfriedly/connect-js

One thing you can do to keep yourself informed of changes that Facebook makes is to fork the repo, git clone the repo to your dev server, install the crontab script, and put a GitHub post-receive email hook to notify you of all changes. Then the changes made by Facebook can more quickly diagnosed by developers like you!

Often times Facebook only updates the Unix timestamp on the file, so here is a revised version of the script to only download/update the all_deminified.js file if there is more than one line of change made.

#!/bin/bash  -ex
cd `dirname $0`
TODAY=`date --rfc-3339=date`
JS_OUTPUT="js/all_${TODAY}.js"
if [ ! -d "js" ] 
then
  mkdir "js"
fi
/usr/bin/wget https://connect.facebook.net/en_US/all.js -O ${JS_OUTPUT}
/usr/bin/python jsbeautifier.py -o all_deminified.js ${JS_OUTPUT}
# Avoid sending out unnecessary updates if only the timestamp has changed.
ALL_JS_DIFF=`git diff --shortstat all_deminified.js | grep -v "1 insertions"`
if [ ! -z "$ALL_JS_DIFF" ]                            
then                                                  
  echo "Commit has changed..."                        
  git --no-pager diff . # Just to see what changed...turn off the pager.
  /usr/bin/git add all_deminified.js
  /usr/bin/git commit -m "Facebook Connect changes for $TODAY"
  /usr/bin/git push origin
fi

Since the GitHub post-receive hook does not provide nice friendly diffs, we can also add a post-commit hook in our local repository and use the Ruby git-commit notifier to notify us whenever a change has been made. This script dumps out last two revisions made in the commit(older to newer revision first). Since the git-commit-notifier will only trigger if there is something more than the timestamp, we should expect diff emails only when there is a more substantial change made in the Facebook JavaScript. You have to setup your own git-commit-notifier.yml and point it to an SMTP host (it apparently does not work with Gmail hosts since STARTTLS may be required).

vi .git/hooks/post-commit

#!/bin/bash -ex
HASH_DIFF=`git log --pretty=%H --reverse | tail -2 | xargs echo`
git-commit-notifier /home/rhu/fb/connect-js/git-commit-notifier.yml ${HASH_DIFF} refs/heads/master

Useful command to examine overall network stats

2011-06-23T10:30:00.001-07:00

netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n