Thursday, March 31, 2011

Celery v2.3 and chords..

Juust saw this on the Celery development branch for v2.3....

This is how the taskset callback stuff is done:

By default the synchronization step is implemented by having a recurring task poll the completion of the taskset every second, applying the subtask when ready.

Example implementation:
def unlock_chord(taskset, callback, interval=1, max_retries=None):
    if taskset.ready():
        return subtask(callback).delay(taskset.join())
    unlock_chord.retry(countdown=interval, max_retries=max_retries)
This is used by all result backends except Redis, which increments a counter after each task in the header, then applying the callback when the counter exceeds the number of tasks in the set.

The Redis approach is a much better solution, but not easily implemented in other backends (suggestions welcome!)

With statement...

The best explanation of the motivations behind the with statement..

Fabric obviously uses a lot of with statements and provides its own context managers:

Fabric also uses the nested function with the contextlib ilbrary:

Saturday, March 19, 2011

Internet Connection Sharing in Windows 7

Ever had the problem of installing a new desktop but don't have an Ethernet connection?  Well, you can turn your machine into an Internet router for this purpose.

1. Go to Administrative Tools and make sure the "Application Layer Gateway" and "Windows Firewall" service is started (and/or set to Automatic).  If you don't,  you'll get a "null" when trying to do the Internet Connection Sharing setup.

2. Go to view Network Connections and right-click Properties on the wireless Internet connection.  Click on the "Sharing" tab and make sure both checkboxes to enable sharing -- these screenshots still apply even though it's Vista:

3. Connect the Ethernet cable from your laptop to the desktop.  Make sure to do an ipconfig /renew on the desktop.  You should also verify that the IP address of your laptop has been changed to a private IP address.

If you need to ever use the laptop's Ethernet connection again, you need to change back to "Obtain DHCP address automatically".  You may also wish to disable the Internet Connection Sharing option for the wireless too.

Why does my Windows 7 machine accidentally turn on?

Three things to check..

1. Disable Wake-on LAN options in your BIOS. If they don't exist (i.e. HP machines), then you're OK.

2. Verify that you've turned off "wake timers".
  1. Control Panel -> System and Security -> Power Options
  2. Then click "Change plan settings" for your currently selected plan
  3. Click "Change advanced power settings"
  4. Scroll down to "Sleep" and expand the menu tree
  5. Expand "Allow wake timers"
  6. Click the word "enable" then drop-down and select "Disable" on both options if not already disabled
  7. Finally Click "OK"
Disabling "Allow wake timers" helped me get a better night sleep ... I hope it helps you!
Disable wake timers
Disable wake timers


How Do I fix the problem ?

To fix the problem go to Control Panel > Network and Internet > Network Connections
You should see your local area connection
Go to properties
Then go to Configure … (be sure to have the right one highlighted)
Now under the Power Management MAKE SURE uncheck “Allow this device to wake the computer” and it’s sub option
And you should make it looks something like this, and you should be good to go.
No more frustration find your computer being awake and knowing that you have put it into sleep.

Resetting primary key in Postgres

alter sequence "rsvp_table_id_seq" RESTART;
select * from rsvp_table_id_seq;

Concatenating multiple cells in Excel

Ever want to do the equivalent of ",".join(['foo', 'bar', 'baby'] in Excel?

One way is use the Excel formula:
=A1 & ", " & A2 & ", " & A3...

One other cool way is to add this Visual Basic function. Inside View->Macros, you can add this one:
Public Function MultiCat( _
        ByRef rRng As Excel.Range, _
        Optional ByVal sDelim As String = "") _
             As String
     Dim rCell As Range
     For Each rCell In rRng
         MultiCat = MultiCat & sDelim & rCell.Text
     Next rCell
     MultiCat = Mid(MultiCat, Len(sDelim) + 1)
  End Function

Then you can use:
= MultiCat(A1:C1," ")
Thanks to for this info!

Friday, March 18, 2011


The best way to see how the CannotSendRequest error works is to call request() multiple times:

>>> import httplib
>>> a = httplib.HTTPSConnection('')
>>> a.request('GET', '/')
>>> a.request('GET', '/')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.6/", line 898, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.6/", line 915, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib/python2.6/", line 802, in putrequest
    raise CannotSendRequest()

Then you can try:
>>> a.getresponse()

>>> a.request('GET', '/')

One thing observed is that Twitter seems to close the SSL connection after the 2nd time we ask for another request token, so we should avoid reusing the same HTTPS connection.

Also, if we reuse the connection a second time, BadStatus lines will start triggering because connection.getresponse() did not fully run and therefore does not sets the _CSIDLE state for the HTTPSConnection. As a result, no subsequent calls can be made. You may not see this issue as much on Apache running separate threads, but the issue is much more noticeable on Django's dev app server that runs on a single thread and the issue can easily be repeated.

Note: this problem does not happen with LinkedIn OAuth, so it seems specific to Twitter.

Friday, March 11, 2011


A great table of the 8-bit representation for ASCII:

UTF-8 uses the 7-bit character set and then uses the higher order bit to denote between standard characaters and international ones.

Great background article here:

Sunday, March 6, 2011

Issues uploading with quick review.

One of the issues with using is that it relies on SWFUploader, an open-source Adobe Flash plugin for uploading images/video clips. Normally it has defined the following acceptable file types in its JavaScript declaration:

file_types : "*.gif;*.jpg;*.png"

The problem is that for case sensitive file systems, such as Ubuntu Linux or MacOS X, the Adobe file browser expects these files to be lowercase .gif, .jpg, and .png files. But many cameras take photos with upper-case .GIF, .JPG, and .PNG files, and on machines besides Windows boxes, filenames are case sensitive. Therefore, some of your guests who run some version of Linux simply will not be able to upload photos without renaming the extensions to lowercase.

The temporary fix for the will need include both upper-case and lower case parameters (i.e. *.gif;*.jpg;*.png;*.GIF;*.JPG;*.PNG)". The root cause is actually located deep within the SWFUpload ActionScript, which relies on the class from Adobe's ActionScript code, which doesn't appear to have any regexp capabilities when searching for file extensions.

One such way is to do something similar within SWFUpload as shown in

var filterExtensions:String = "*.gif;*.jpeg;*.jpg;*.png";

// On Linux, file extensions are case-sensitive, so we need to include the
 // upper-case versions of all of these.  It seems to be important to specify
 // the upper-case versions at the beginning of the filter string rather than
 // at the end.
 if (Host.isLinux)
  filterExtensions = filterExtensions.toUpperCase() + ";" + filterExtensions;

The same problem applies to the upload video section too, which only scans for *.asf, *.avi, *.mov, etc. extensions.

So until makes these corrections, some your guests may simply not be able to share photos and videos with their sites!

Update: they've fixed it within a matter of 1-2 days! You can now upload without any issues now.

Friday, March 4, 2011

The difference between escape(), encodeURI(), and encodeURIComponent()...

escape: convert non-ASCII chars to hex string encoding

Use of the encodeURI() method is a bit more specialized than escape() in that it encodes for URIs [REF] as opposed to the querystring, which is part of a URL. Use this method when you need to encode a string to be used for any resource that uses URIs and needs certain characters to remain un-encoded. Note that this method does not encode the ' character, as it is a valid character within URIs.

encodeURIComponent: encodes everything

decodeURIComponent: converts back

The common way to UTF-8 encode a string is:

How does this work? If we simply did an escape(mystring), then all the spaces would get replaced by %20. We could use encodeURIComponent() to first convert the entire string, but doing a decodeURIComponent would get us back to where we started. To leverage the UTF-8 encoding aspects of URIComponent, we use encodeURIComponent and then use unescape().

Answer: To convert a string from URL-encoded form, use the JavaScript function unescape(string). This function works as follows: if the string contains escape-sequences of the form %XX, where XX stands for two hexadecimal digits, each escape-sequence is replaced by the character whose ASCII code is XX. Otherwise, the string remains unchanged.

(In Unicode-aware browsers, in addition to escape-sequences %XX, the unescape function also processes sequences of the form %uXXXX; see also escape.)

Using Django pagelets...

One issue that I encountered when using django-pagelets is that clicking on the "Add pagelet" would add a pagelet but it would never render within the page. The problem appears to be that we're missing a content_area parameter, which indicates to the create_pagelet view to apply the pagelet to a section. If you look at The Django admin section, it appears to show a default section, but checking things in the SQL database you'll notice that content_area is set to blank.

The solution is to include these changes into django-pagelets:

Or at the very least, put this _render_content_area.html in your own templates/pagelets.html to prevent content_area from ever being blank:

Thursday, March 3, 2011

Debugging Flymake

While trying to debug an issue with>JSLint/Rhino/Flymake, I noticed there were several ways to debug Lisp programs in Emacs:

1. Check out the *Messages* buffer. All the results from this Lisp program are dumped into here.

2. M-x ielm : you can verify your Lisp code yourself.

3. You can play with your config settings:
M-x customize-group 

Wednesday, March 2, 2011

Using Hudson's email-ext..

One of the issues in using the Hudson email extended notification is that the html.jelly file contains inline <style> that gets ignored by Gmail and other webmail clients. The way to get around this restriction is to use inline styles within the HTML tags. An example is shown here:

So the extended email notification html.jelly file, which shows up as this, doesn't appear quite right.

One way to get around this issue is to use a converter to take your HTML styles and apply them to all your HTML elements. One such web site that can do something is the following:

...but the problem I encountered was copying and pasting could easily introduce bugs in the final HTML code.

Another approach is to use Pynliner. However, while using Pynliner with the Hudson html.jelly file to do inline CSS styles, I found a bunch of issues using it with this file, most notably the BODY and TD.bg1 tags (the HTML itself had lowercase tags). Some styles were not added as a result of these case sensitivity mismatches. The html.jelly for Hudson is located here:

There are a few issues with using Pynliner:
1. It currently is case sensitive with respect to tags and classes used in the <style>. If you have upper-case style tags (i.e. <BODY>, then lower case <body> tags will not match and inline styles will not be generated for those tags.

2. It will introduce a newline for every multiple CSS styles you use (i.e. <h1 style="font-weight (newline); color: red">)

3. It doesn't have any tabs output (prettify=True option in BeautifulSoup) to help verify your HTML output is correct.

4. Finally, more related to the Hudson html.jelly file, but it does not auto-escape ampersand's, which means that Hudson will throw an exception on line 3 that contains two nbsp characters with a preceding ampersand. You have to swap the ampersand (\&) for the \& character after generating the HTML with inline styles.

I was able to correct it with the following GitHub fork:

Assuming you pull these GitHub changes (or they eventually get merged into Pynliner), you can then do:
import pynliner
p = pynliner.Pynliner().from_string(open("html.jelly", "r").read())
open("/tmp/html_gmail.jelly", "w").write(

Once you're done (and you've fixed the ampersand issues), you can copy/paste the html_gmail.jelly file and copy it in your HUDSON_HOME (specified in /etc/default/hudson) into ~/email-templates/dir, and then configure the Hudson email-extended notification to do:


FYI - Hudson will first try to look within its own WEB-INF directory for html_gmail.jelly. If it can't find it, Hudson will then try to look inside your HUDSON_HOME/email-templates dir. The html.jelly file, which is used to generate this file, should already be in your your HUDSON_HOME/plugins/email-ext/WEB-INF/classes/hudson/plugins/emailext/templates dir.

Finally, one last change that can be done is to configure Hudson to output ${FAILED_TESTS} so that you can quickly review the results instead of having to go to the web console. The following is what you can set inside you default Hudson Default Content inside the "Manage Hudson" link:
<H3>Failed Tests</H3>>
<pre style="font-size: 12px">
Note the instructions for using Jelly tokens (i.e. ${JELLY_SCRIPT, template="xx"}):
"Arguments may be given for each token in the form name="value" for strings and in the form name=value for booleans and numbers. The {'s and }'s may be omitted if there are no arguments.

Addendum (03/08): within Pynliner, you may also need to patch the pyliner/ code to handle Jelly tags. We use the BeautifulSoup XML parser instead of the HTML one since the latter introduces a bunch of restrictions on how nested tags can be interprted.
def _get_soup(self):
         """Convert source string to BeautifulSoup object. Sets it to self.soup.
-        self.soup = BeautifulSoup(self.source_string)
+        from BeautifulSoup import BeautifulStoneSoup
+        BeautifulStoneSoup.NESTABLE_TAGS['j:if'] = []
+        BeautifulStoneSoup.NESTABLE_TAGS['j:foreach'] = []
+        BeautifulStoneSoup.NESTABLE_TAGS['table'] = []
+        self.soup = BeautifulStoneSoup(self.source_string, selfClosingTags=['j:set', 'j:getstatic', 'br'])

Since BeautifulSoup lowercases many of the tags/classes, you may also find that some entries from the generated file also need to be adjusted too (i.e. varstatus to varStatus, classname to className). You must make sure that the tags are replaced to forEach and getStatic -- otherwise, the Jelly script will not run correctly.
<   <j:set var="spc" value="&nbsp; " />
>   <j:set var="spc" value="&nbsp;&nbsp;" />
<     <j:forEach var="cs" items="${changeSet.logs}" varstatus="loop">
>     <j:forEach var="cs" items="${changeSet.logs}" varStatus="loop">
<   <j:getStatic var="resultFailure" field="FAILURE" classname="hudson.model.Result" />
>   <j:getStatic var="resultFailure" field="FAILURE" className="hudson.model.Result" />
If you really don't want to go through these steps, you can download the html_gmail.jelly file from here:
The README.TXT file is here:

Grapelli, Django, and objects_name

KeyError exception that says "objects_name" is not defined.

The issue has already been reported here:

The problem appears to be that the objects_name template context variable has been changed in the latest Django dev distribution (but apparently not in Django 1.3 beta that's currently on the site). You have to copy the templates/admin/delete_selected_confirmation.html from the django-grappelli directory into your own templates/admin directory, and then change objects_name variable to object_name. This patch appears to fix the issue:
Index: templates/admin/delete_selected_confirmation.html
--- templates/admin/delete_selected_confirmation.html (revision 1401)
+++ templates/admin/delete_selected_confirmation.html (working copy)
@@ -18,7 +18,7 @@
     <div class="container-grid delete-confirmation">
         {% if perms_lacking %}
             <div class="module">
-                <h2>{% blocktrans %}Deleting the {{ objects_name }} would result in deleting related objects, but your account doesn't have permission to delete the following types of objects:{% endblocktrans %}</h2>
+                <h2>{% blocktrans %}Deleting the {% endblocktrans %}{{ object_name }}{% blocktrans %} would result in deleting related objects, but your account doesn't have permission to delete the following types of objects:{% endblocktrans %}</h2>
                 <div class="row">
                     <ul class="rte">
                         {% for obj in perms_lacking %}
@@ -29,7 +29,7 @@
         {% else %}
             <div class="module">
-                <h2>{% blocktrans %}Are you sure you want to delete the selected {{ objects_name }} objects? All of the following objects and their related items will be deleted:{% endblocktrans %}</h2>
+                <h2>{% blocktrans %}Are you sure you want to delete the selected {% endblocktrans %}{{ object_name }}{% blocktrans %} objects? All of the following objects and their related items will be deleted:{% endblocktrans %}</h2>
                 {% for deleteable_object in deletable_objects %}
                     <div class="row">
                         <ul class="rte">{{ deleteable_object|unordered_list }}</ul>

Random tracings through Pynliner and cssutils..



cssText = property(_getCssText, _setCssText, ...)

def getCssText(self, separator=None):
serialized property cssText, each property separated by                                                                                             
given `separator` which may e.g. be ``u''`` to be able to use                                                                                       
cssText directly in an HTML style attribute. ``;`` is part of                                                                                       
each property (except the last one) and **cannot** be set with                                                                                      
return cssutils.ser.do_css_CSSStyleDeclaration(self, separator)

Normally cssText is defined as a propety, but we can use the getCssText and pass in a separator for ' ' instead of the default '\n' line.

Tuesday, March 1, 2011

Using Hudson webhooks and wget for automated builds.

This command works pretty well for Hudson builds. It pipes the stdout results to /dev/null and remaps stderr too:

wget -O - --auth-no-challenge --user=hudsonuser --password=mypassword > /dev/null 2>&1


1. You need to setup a Hudson user that has access permission to create new builds.

2. The --user and --password need to be set in the wget.

One puzzling issue that I encountered is that on wget 1.11.1, you will get Forbidden 403 errors when using this URL format:
wget -O - 
But you still get a Forbidden 403 error when using this command:
wget -O - --auth-no-challenge
Why is this case? To find out, you can download the code from to understand. The issue actually turns out to be a bug in wget 1.11.1 that was later fixed (perhaps in 1.12?). Parsing of the URL happens in wget.c by setting the uname_b pointer and then storing the results of parse_credentials into the &user and &passwd, which then get stored as u->user and u->passwd.

uname_b = p;
  p = url_skip_credentials (p);
  uname_e = p;

 if (uname_b != uname_e)
      /* http://user:pass@host */
      /*        ^         ^    */
      /*     uname_b   uname_e */
      if (!parse_credentials (uname_b, uname_e - 1, &user, &passwd))
          error_code = PE_INVALID_USER_NAME;
          goto error;

In Wget 1.11.1, there is an extra condition such that if the username is encoded in the URL format, then it will not issue the basic authentication code. This explains why having anything encoded in the URL string for username/password does not work:

/* Find the username and password for authentication. */
  user = u->user;
  passwd = u->passwd;
  search_netrc (u->host, (const char **)&user, (const char **)&passwd, 0);
  user = user ? user : (opt.http_user ? opt.http_user : opt.user);
  passwd = passwd ? passwd : (opt.http_passwd ? opt.http_passwd : opt.passwd);

  if (user && passwd
      && !u->user) /* We only do "site-wide" authentication with "global"
                      user/password values; URL user/password info overrides. */
      /* If this is a host for which we've already received a Basic
       * challenge, we'll go ahead and send Basic authentication creds. */
      basic_auth_finished = maybe_send_basic_creds(u->host, user, passwd, req);

We can see in Wget 1.10.2 (src/http.c), there is no such check:

/* Find the username and password for authentication. */
  user = u->user;
  passwd = u->passwd;
  search_netrc (u->host, (const char **)&user, (const char **)&passwd, 0);
  user = user ? user : (opt.http_user ? opt.http_user : opt.user);
  passwd = passwd ? passwd : (opt.http_passwd ? opt.http_passwd : opt.passwd);

  if (user && passwd)
      /* We have the username and the password, but haven't tried
         any authorization yet.  Let's see if the "Basic" method
         works.  If not, we'll come back here and construct a
         proper authorization method with the right challenges.

         If we didn't employ this kind of logic, every URL that
         requires authorization would have to be processed twice,
         which is very suboptimal and generates a bunch of false
         "unauthorized" errors in the server log.

         #### But this logic also has a serious problem when used
         with stronger authentications: we *first* transmit the
         username and the password in clear text, and *then* attempt a
         stronger authentication scheme.  That cannot be right!  We
         are only fortunate that almost everyone still uses the
         `Basic' scheme anyway.

         There should be an option to prevent this from happening, for
         those who use strong authentication schemes and value their
         passwords.  */
      request_set_header (req, "Authorization",
                          basic_authentication_encode (user, passwd),

But on wget-1.12, extra allowances are made for the --auth-no-challenge flag.

/* Find the username and password for authentication. */
  user = u->user;
  passwd = u->passwd;
  search_netrc (u->host, (const char **)&user, (const char **)&passwd, 0);
  user = user ? user : (opt.http_user ? opt.http_user : opt.user);
  passwd = passwd ? passwd : (opt.http_passwd ? opt.http_passwd : opt.passwd);

  /* We only do "site-wide" authentication with "global" user/password
   * values unless --auth-no-challange has been requested; URL user/password
   * info overrides. */
  if (user && passwd && (!u->user || opt.auth_without_challenge))
      /* If this is a host for which we've already received a Basic
       * challenge, we'll go ahead and send Basic authentication creds. */
      basic_auth_finished = maybe_send_basic_creds(u->host, user, passwd, req);

Note: it turns out Wget 1.10.2 and previous versions always attempts basic HTTP authentication, which can be a security hole because username/passwords are sent in the clear. Wget 1.11.1 does not implement this behavior. Hudson does not try to send an HTTP negotiation and immediately throws a 403 Forbidden error (See The reason we need --auth-no-challenge is to force later wget versions to send HTTP authentication info regardless of negotiation:

           If this option is given, Wget will send Basic HTTP authentication information (plaintext username and password) for all requests, just like Wget 1.10.2 and prior did by default.

           Use of this option is not recommended, and is intended only to support some few obscure servers, which never send HTTP authentication challenges, but accept unsolicited auth info, say, in
           addition to form-based authentication.

So even if we provide the --auth-no-challenge, we still can't use the URL format specified by wget in for Hudson automated builds, at least for wget 1.11.1. You have to upgrade to wget 1.12 to avoid this issue.

Using celerybeat and Django

1. If you're using Django with celerybeat, make sure to include your DJANGO_SETTINGS_MODULE inside your /etc/default/celeryd. Celerybeat is rather unforgiving in not complaining if there are tasks defined in your CELERYBEAT_SCHEDULE that depend on Django. Here is a Pull request committed just now that would help solve the issue:

You may find that no matter what environment variables that are set, Celerybeat never seems to choose the correct AMQP host/password. That's a sign to you that there is an import issue.

2. Celerybeat comes with an /etc/init.d/celerybeat. If you find yourself constantly trying to use start-stop-daemon and seeing that a PID file gets created but no daemon, it's most likely because you need to set the path to CELERYBEAT to /usr/local/bin/ By default, the contrib/debian/init.d sets CELERYBEAT to 'celerybeat', which causes the start-stop-daemon not to be able to find it.

3. The --time-limit options set a hard limit on tasks that can run. Be careful if this is not your desired behavior! See for more info.

# Extra arguments to celeryd
#   CELERYD_OPTS="--time-limit=300"

4. If you're migrating from apscheduler, be careful about not running both celerybeat and apscheduler on the same machine when doing the transition. Celerybeat also tries to load Celery config, which will call get_all_tasks() to set CELERY_IMPORTS, which then will import each individual file, which will register another set of cron tasks to APScheduler (see code below), possibly causing 2x the number of tasks to occur at the same cron interval.

def import_task_module(self, module):
        return self.import_from_cwd(module)

    def import_default_modules(self):
        imports = getattr(self.conf, "CELERY_IMPORTS", None) or []
        imports = set(list(imports) + BUILTIN_MODULES)
        return map(self.import_task_module, imports)