Saturday, November 26, 2011

Setting up wake-on LAN in your own home

One of the drawbacks of using Slicehost or Amazon EC2 is that you're pretty much paying $16-$20/month for a VPS server even when you're not accessing the server that much. If you have data that you also don't want being stored on a cloud, you may opt to setup your own home server to store this information. You may want to have the ability to access this data but not have your machine turned on all the time adding to your electricity bill.

First, your computer(s) needs to have wake-on-LAN enabled in the BIOS and your computer's Ethernet port must be plugged directly into a switch/router (wireless wake-on LAN NIC cards may also be possible). For Toshiba laptops, for instance, you need to reboot, hit Esc and then F1 to enter the BIOS. You can then enable Wake-on-LAN support and save the changes.

You also need a router that can run the DD-WRT or Tomoto open source firmware. If you use the VPN version of the DD-WRT, you can also setup a PPTP server with DynDNS so that you can VPN into your server even though you're using an ISP that does not provides static IP's. You'll want to setup your DD-WRT server to assign a static IP address to your remote computer (inside the Services tab, look for static leases).

If have one of the Windows Professional versions, you can also take advantage of the Remote Desktop servers. You can enable Remote Desktop Sever by going to the Control Panel > System > Remote. You can also right-click My Computer (if the icon is shown on the desktop) and choose Properties.

To verify that Wake-on-LAN works, you can use Netcat/socat to test things out. This blog posting explains how the magic packet for Wake-on-LAN is constructed: 6 bytes of FF followed by the LAN MAC address repeated 16 times. The bash script used to generate a hex version is listed as follows -- you just need to substitute the ETHER parameter with the MAC address of the machine that will need to be woken up:
ETHER="aa:bb:cc:dd:ee:ff"
ETHER2=`echo $ETHER | sed "s/://g"`
ETHER3="${ETHER2}${ETHER2}${ETHER2}${ETHER2}"
ETHER4="FFFFFFFFFFFF${ETHER3}${ETHER3}${ETHER3}${ETHER3}"
echo ${ETHER4} | xxd -r -p > wake.packet

The blog posting points out that you can use netcat to send to a router IP, but what about sending a broadcast address from within the same network? For this purpose, the socat utility seems to work better (not sure if netcat allows sending to broadcast IP addresses?)
socat - UDP-DATAGRAM:192.168.0.255:7,broadcast < wake.packet 

You can also use DD-WRT to wake up any device (check the WoL section) too, but the above information is more background information for how it actualy works!

To setup dynamic DNS, you'll need to go to the DDNS page of the DD-WRT firmware and input your username/pw. The DynDNS service is now $20/year, though it used to be free in recent years. The instructions for setting DynDNS are posted at: http://www.dd-wrt.com/wiki/index.php/DDNS_-_How_to_setup_Custom_DDNS_settings_using_embedded_inadyn_-_HOWTO

To setup the PPTP server for VPN, go to the PPTP section and follow the instructions according to http://www.dd-wrt.com/wiki/index.php/PPTP_Server_Configuration. One thing to note is that the PPTP server IP should be different from your router's internal IP, and the CHAP secrets format is ( * *). Yes, there is an asterisk after username and password.

Ubuntu comes with 'tsclient', which is a Remote Desktop Connection. Once you've setup a VPN connection correctly, you can To get your Windows machine to hibernate, you can also create this batch file and add to your Desktop:

%windir%\system32\rundll32.exe powrprof.dll,SetSuspendState Hibernate


You can also get an iPad/Android app to wake up your computer remotely -- we found Mocha WoL, a free app that lets you wake up the machine remotely. You can also set the hostname to a DynDNS machine, and assuming you have UDP port forwarding setup to relay WoL broadcasts, it should be able to wake up the machine remotely.

Friday, November 25, 2011

WebDriverWait and SauceLabs

If you've used the WebDriverWait class with SauceLabs, you may have noticed that it actually will run beyond the designated timeout period. I first noticed this issue when seeing that this function took 40 minutes to complete with a 20 second implicit wait time and a 60 second timeout w/ .5 poll frequency! (60 / .5 = 120 loops * 20 seconds/implicit wait = 2400 seconds total)

The problem appears to be that WebDriverWait attempts to run self._timeout / self._poll times before returning a Timeout Exception:

for _ in xrange(max(1, int(self._timeout/self._poll))):

Since this code could wait until an ElementNotFound exception is found (and if the implicit wait time is increased), then the total time this routine takes is a function of the total # of loops * poll frequency * the implicit wait time.

The correct patch appears to be recording the start time and looping until the maximum timeout period has been exceeded:

--- a/selenium/webdriver/support/wait.py
+++ b/selenium/webdriver/support/wait.py
@@ -24,7 +24,7 @@ class WebDriverWait(object):

def __init__(self, driver, timeout, poll_frequency=POLL_FREQUENCY):
"""Constructor, takes a WebDriver instance and timeout in seconds.
-
+
:Args:
- driver - Instance of WebDriver (Ie, Firefox, Chrome or Remote)
- timeout - Number of seconds before timing out
@@ -43,7 +43,8 @@ class WebDriverWait(object):
def until(self, method):
"""Calls the method provided with the driver as an argument until the \
return value is not Falsy."""
- for _ in xrange(max(1, int(self._timeout/self._poll))):
+ end_time = time.time() + self._timeout
+ while(time.time() < end_time):
try:
value = method(self._driver)
if value:

The bug report is filed here:

http://code.google.com/p/selenium/issues/detail?id=2939

Saturday, November 12, 2011

How the celeryctl command works in Celery..

One of the most popular Django apps out there is the Celery task queue framework. It allows you to build your own task queue system for offline processing and provides an elegant framework for interfacing with message brokers such as AMQP, Redis, etc.

The celeryctl command in the Celery task queue is an amazing tool. It allows you send commands to your Celery works to figure out which ones are actively processing tasks, revoke tasks that have been dispatched so that workers will skip over procesing them, and see which ones are scheduled. We use it here to monitor long-running tasks, such as this script:
from celery.bin import celeryctl
import datetime

def probe_celeryctl():
    results = celeryctl.inspect().run("active")
    check_old_celery_tasks(results)


def check_old_celery_tasks(results):
    bad_tasks = []

    MAX_TIMEDELTA = {'hours': 1}
    for host, tasks in results.items():
        for task in tasks:
            task_start = task.get('time_start')
            timestamp = float(task_start)
            task_timestamp = datetime.datetime.fromtimestamp(timestamp)
            time_diff = abs(datetime.datetime.utcnow() - task_timestamp)
            if time_diff > datetime.timedelta(**MAX_TIMEDELTA):
                print "Hmm..%s %s (on %s)" % (time_diff, task, host)
                bad_tasks.append("Task %s elapsed (%s): name=%s, PID=%s, args=%s, kwargs=%s" % (time_diff, host, task.get('name'), task.get('worker_pid'), task.get('args'), task.get('kwargs')))

    if len(bad_tasks) > 0:
        message = "You'd better check on these tasks...they are slowing things down.\n\n"
        message += "\n".join(bad_tasks)
        print message

if __name__ == "__main__":
    probe_celeryctl()
How does celeryctl work? Assuming you're using the AMQP backend with Celery, celeryctl relies on the same concepts used in the AMQP open standard (click here for a basic intro). When you first startup, Celery will create an AMQP exchange called "celeryd.pidbox" on the AMQP host. You can confirm by using rabbitmqctl to list exchanges:
$ sudo rabbitmqctl -p fanmgmt_prod list_exchanges
Listing exchanges ...
celeryd.pidbox fanout
.
.
.
You'll notice that celeryd.pidbox is created as a fanout exchange (see the AMQP intro for more details). This way, using celeryctl on one machine will broadcast a message to this exchange. The AMQP host will deliver messages to each queue that is bound to this celeryd.pidbox exchange. On startup, every Celery worker will also create a queue (queue.hostname.celery.pidbox) bound to this exchange, which can be used to respond to celeryctl commands.

Replies from each Celery worker are passed back via direct exchange using celeryd.reply.pidbox. When you startup celeryctl, it sends messages to the celeryd.pidbox and listens for messages to arrive from the celeryd.reply.pidbox queue. The timeout period to wait for replies is 1 second, so if you wish to increase the time to wait for replies from Celery workers, you can increase this number with the -t parameter (i.e. celeryctl inspect active -t )

Note: the reply exchange gets deleted after celeryctl command exists (usually set to auto_delete=True). Since all the Celery workers are still bound to the other celeryd.pidbox exchange, it should still persist until you shutdown all Celery workers.

Thursday, November 10, 2011

Installing HipChat in Jenkins/Hudson..

1. git clone https://github.com/jlewallen/jenkins-hipchat-plugin

2. Compile and build (See http://hustoknow.blogspot.com/2010/09/installing-java-on-ubuntu-1004.html and http://hustoknow.blogspot.com/2011/04/hudson-becomes-jenkins-compling-plugins.html)

3. Copy the target/.hpi file that got generated into your hudson/plugins dir.

4. Restart Hudson.

5. Go into the Configure Hudson and provide the API key/conference room.

6. Make sure to enable HipChat notifications in each of your build projects!

FYI - this plug-in uses the HipChat API to publish information to the respective room that you deisgnate:

https://github.com/jlewallen/jenkins-hipchat-plugin/blob/master/src/main/java/jenkins/plugins/hipchat/StandardHipChatService.java

   public void publish(String message, String color) {
      for(String roomId : roomIds) {
         logger.info("Posting: " + from + " to " + roomId + ": " + message + " " + color);
         HttpClient client = new HttpClient();
         String url = "https://api.hipchat.com/v1/rooms/message?auth_token=" + token;
         PostMethod post = new PostMethod(url);
         try {
            post.addParameter("from", from);
            post.addParameter("room_id", roomId);
            post.addParameter("message", message);
            post.addParameter("color", color);
            client.executeMethod(post);
         }
         catch(HttpException e) {
            throw new RuntimeException("Error posting to HipChat", e);
         }
         catch(IOException e) {
            throw new RuntimeException("Error posting to HipChat", e);
         }
         finally {
            post.releaseConnection();
         }
      }
   }

   public void rooms() {
      HttpClient client = new HttpClient();
      String url = "https://api.hipchat.com/v1/rooms/list?format=json&auth_token=" + token;
      GetMethod get = new GetMethod(url);
      try {
         client.executeMethod(get);
         logger.info(get.getResponseBodyAsString());
      }
      catch(HttpException e) {
         throw new RuntimeException("Error posting to HipChat", e);
      }
      catch(IOException e) {
         throw new RuntimeException("Error posting to HipChat", e);
      }
      finally {
         get.releaseConnection();
      }
   }

}

Installing HipChat on 64-bit Linux.

The instructions for HipChat are pretty straightforward, though you do need to run th commands from root.

https://www.hipchat.com/help/page/how-do-i-install-hipchat-on-64-bit-linux

# Adobe's instructions forget to mention ia32-libs-gtk
$ apt-get install lib32asound2 lib32gcc1 lib32ncurses5 lib32stdc++6 \
lib32z1 libc6 libc6-i386 ia32-libs-gtk lib32nss-mdns

$ wget http://frozenfox.freehostia.com/cappy/getlibs-all.deb
$ dpkg -i getlibs-all.deb

# incorrect in adobe's instructions (corrected)
$ getlibs -p gnome-keyring
$ getlibs -p libhal-storage1

Seems like these commands were not needed on Ubuntu 10.04:

$ ln -s /usr/lib32/libnss3.so.1d /usr/lib32/libnss3.so
$ ln -s /usr/lib32/libssl3.so.1d /usr/lib32/libssl3.so
$ ln -s /usr/lib32/libnspr4.so.0d /usr/lib32/libnspr4.so
$ ln -s /usr/lib32/libsmime3.so.1d /usr/lib32/libsmime3.so # missing from adobe's instructions


wget http://airdownload.adobe.com/air/lin/download/2.6/AdobeAIRInstaller.bin
chmod 777 AdobeAIRInstaller.bin
./AdobeAIRInstaller.bin

The final step is to download the Adobe AIR client here:

$ wget http://downloads.hipchat.com/hipchat.air
$ /opt/Adobe\ AIR/Versions/1.0/Adobe\ AIR\ Application\ Installer hipchat.air

Thursday, November 3, 2011

Debugging Facebook's JavaScript code

In reviewing Facebook's JavaScript code, there apparently is a way to enable debugging of the JavaScript. If you set fb_debug=1, then the logging option will be enabled:
if (!options.logging &&        window.location.toString().indexOf('fb_debug=1') < 0) {      FB._logging = false;    }