Monday, December 31, 2012

How LinkedIn's JS API implements JavaScript parameters

If you've looked at LinkedIn's API, you may have noticed that they follow a curious approach to providing script parameters to an external JavaScript file:

https://developer.linkedin.com/documents/configuration-and-compatibility

<script type="text/javascript" src="http://platform.linkedin.com/in.js">
  api_key:    [API_KEY]
  onLoad:     [ONLOAD]
  authorize:  [AUTHORIZE]
</script>

Initial inspection seems to suggest that you are declaring three separate JavaScript variables: api_key, onLoad, and authorize. In actuality, you are just defining text inside the script tag that gets parsed with a bunch of regexeps and removing of whitespaces.

Basically the code below appears to extract out the innerHTML and then set the variables r and K to be the key/value pairs. White spaces are removed with the replace() function.

http://platform.linkedin.com/in.js

        try {
            m = f.innerHTML.replace(A, n)
        } catch (z) {
            try {
                m = f.text.replace(A, n)
            } catch (y) {
            }
        }
    }
    m = m.replace(J, "$1").replace(A, n).replace(F, n);
    aa = C.test(m.replace(j, n));
    for (var T = 0, S = m.split(k), q = S.length; 
    T < q; 
    T++) {
        var s = S[T];
        if (!s || s.replace(j, n).length <= 0) {
            continue
        }
        try {
            V = s.match(g);
            r = V[1].replace(A, n);
            K = V[2].replace(A, n)
        } catch (X) {
            if (!aa) {
                console.warn("script tag contents must be key/value pairs separated by a colon. Source: " + X)
            }
            continue
        }
        N(r, K)
    }

Some of the regexpes are defined at the top of in.js:

var R = {
        "bootstrapInit": +new Date()
    }, p = document,
        l = (/^https?:\/\/.*?linkedin.*?\/in\.js.*?$/),
        b = (/async=true/),
        D = (/^https:\/\//),
        J = (/\/\*((?:.|[\s])*?)\*\//m),
        F = (/\r/g),
        j = (/[\s]/g),
        g = (/^[\s]*(.*?)[\s]*:[\s]*(.*)[\s]*$/),
        x = (/_([a-z])/gi),
        A = (/^[\s]+|[\s]+$/g),
        u = (/^[a-z]{2}(_)[A-Z]{2}$/),
        C = (/suppress(Warnings|_warnings):true/gi),
        d = (/^api(Key|_key)$/gi),

OAuth2: The Road to Hell

http://hueniverse.com/2012/07/oauth-2-0-and-the-road-to-hell/

http://hueniverse.com/2010/09/oauth-2-0-without-signatures-is-bad-for-the-web/


RealtimeConf - "OAuth 2.0 - Looking Back and Moving On" by Eran Hammer from &yet on Vimeo.

Saturday, December 22, 2012

Upgrading NetworkManager and ModemManager with Ubuntu 12.04

Ever since I've upgraded to Ubuntu 12.04, I've noticed that Franklin U600 USB modem disconnects are frequent (especially when moving in a train ride) and prevent you from reconnecting to the USB device unless you reboot. Upgrading to the latest Ubuntu packages doesn't seem to resolve the issue. One of the main issues appears to be that the ModemManager that comes with Ubuntu 12.04 runs on version 0.5, which appears to have some stability issues with random disconnects (possibly because of this issue with modem disconnects, callbacks, and weak references).

To try to understand better, I started to dig into Modem Manager. Modem Manager has a Wiki article about troubleshooting. To prevent modem-manager from being restarted each time it was killed, I simply renamed the /usr/bin program and then started the program manually by using the –debug flag. After numerous tries of watching the modem get disconnected, I noticed that this state change seemed to occur quite a bit. During the disconnection process, the modem would suddenly jump back to the connected state:
Nov 12 08:49:37 my-laptop modem-manager[8862]:   Modem /org/freedesktop/ModemManager/Modems/0: state changed (connected -> disconnecting)
Nov 12 08:49:37 my-laptop NetworkManager[8912]:    SCPlugin-Ifupdown: devices removed (path: /sys/devices/virtual/net/ppp0, iface: ppp0)
Nov 12 08:49:37 my-laptop modem-manager[8862]:   Modem /org/freedesktop/ModemManager/Modems/0: state changed (disconnecting -> connected)
ModemManager 0.6.0 appears to fix these issues, as well as introduce a new approach for writing plugins for the package.

Currently Ubuntu 12.04 don't have the latest packages for ModemManager, and the issue was severe enough to compel me to try to compile the latest version myself (http://ftp.gnome.org/pub/gnome/sources/ModemManager/):
sudo apt-get install libdbus-glib-1-dev
sudo apt-get install libgudev-1.0-dev
wget 
./configure
make
sudo apt-get uninstall modem-manager
make install
I've been running with the latest 0.6.4 version and the problem doesn't happen anymore. If the connection drops, the modem will eventually be made available again in the Network Manager applet.

Another problem I've noticed is that the Network Manager applet often has blank entries for VPN connections, which appears to be a reported bug. I've also encountered issues with the Gigabit Ethernet port not reliably connecting with a longer cable, so finally decided to take the hit to try to install the latest versions of Network Manager and the Network Manager applet. There is also the issue that the Network Manager applet fails to respond to any mouse clicks, though typing "nmcli nm wwan on" enables the Mobile Broadband option.

These issues all seemed to necessitate upgrading Network Manager from the standard Ubuntu 12.04 packages. Be forewarned: if you attempt this process, you could easily disable your ability to access the Internet. Without the magic that NetworkManager does to handle all your wired and wireless connections, you will then have to resort to using ifconfig to setup a static IP address (i.e. ifconfig eth0 192.168.1.xxx), setting up a static default gateway (route add default gw 192.168.1.1), and adding a nameserver entry in /etc/resolv.conf (nameserver x.x.x.x). So I'd strongly suggest you try this process out on a local network unless you're adept at also connecting to wireless LAN's via the command line.

Download the latest version of Network Manager from this address:
http://ftp.gnome.org/pub/GNOME/sources/NetworkManager/0.9/

Unpack the files and make sure you have these dev libraries installed:
sudo apt-get install intltool
sudo apt-get install libnl-dev 
sudo apt-get install uuid-dev
sudo apt-get install libnss3-dev
sudo apt-get install ppp-dev
./configure --sysconfdir=/etc --libexecdir=/usr/lib/NetworkManager --localstatedir=/var
make

At this point, you should avoid doing any make installs. You want to make sure both Network Manager and the Network Manager applet compiles without any issues before removing them. See http://projects.gnome.org/NetworkManager/developers/ since there are also different compile options in the instructions.

Note: The -sysconfdir is set to /etc, which will allow the compiled version of NetworkManager to find your current configuration (in /etc/NetworkManager). This way, you can re-use all your wired, wireless, and mobile broadband connections. The -libexecdir is needed since default Ubuntu installs dump the DHCP/PPTP scripts inside this directory, which are whitelisted by AppArmor. If you start seeing Permission Denied errors, chances are your paths are being blocked by AppArmor. You also need the -localstatedir option for this same reason, since this directory speicifies where the DHCP client will try to write the PID data.

The Network Manager can be downloaded from this location:

http://ftp.gnome.org/pub/GNOME/sources/network-manager-applet/0.9/network-manager-applet-0.9.6.4.tar.xz
tar xvfp network-manager-applet-0.9.6.4.tar.xz
sudo apt-get install libgconf2-dev
sudo apt-get install libgnome-keyring-dev 
sudo apt-get install libnotify-dev 
sudo apt-get install libgtk-3-dev 
./configure --sysconfdir=/etc --libexecdir=/usr/lib/NetworkManager
make
Note: if you try to compile with libgtk2.0-dev, chances are that the compile will fail. Network Manager needs GTK 2.0+ to compile, but on Ubuntu 12.04 it seems that the libgtk-3-dev library is really what is needed.

If the compiles were successful, now is the time to remove Network Manager. Be forewarned that at this point you could easily block your ability to access the Internet. So only proceed to this point if you're confident enough to recover.
sudo apt-get remove network-manager
sudo apt-get purge network-manager-gnome && sudo apt-get install network-manager-gnome

The --libexecdir for the Network Manager is needed for invoking other plugins, such as VPN. Also, if you intend to use VPN, you'll also need to git clone the repo you need to use (in this case, we're cloning the PPTP module).
git clone git://git.gnome.org/network-manager-pptp
cd network-manager-pptp/
./autogen.sh --sysconfdir=/etc --libexecdir=/usr/lib/NetworkManager --localstatedir=/var
make
sudo make install
You can then do make installs in both project directories and attempt to start Network Manager (or logging out and logging back in)

Note: if you don't see the nm-applet at the top, here are a few places to check. The first requires using the dconf-editor, which needs to be apt-get installed:
apt-get install dconf-tools
dconf-editor

The instructions to use dconf-editor are listed here, which is basically to make sure that notifications are not disabled in the nm-applet namespace: http://askubuntu.com/questions/150406/how-do-i-re-enable-disabled-network-notifications-in-gnome-shell

You also want to make sure using dconf-editor that the com.canonical.Unity.Panel systray-whitelist is set toto "['all'] (or at least to include nm-applet). I found that my system tray was not set to this value and therefore prevented nm-applet from being rendered:

http://askubuntu.com/questions/136733/some-system-tray-icons-invisible-in-gnome-classic-12-04

Normally I wouldn't recommend trying to compile and install Modem Manager and Network Manager, but the current packaged versions supplied in Ubuntu 12.04 appear to be so buggy and unstable that this approach appears to help deal with many of the issues that have been reported.

Thursday, December 20, 2012

Uploading videos to SmugMug

If a video upload fails, SmugMug allows you to view your Upload Log in your Settings page to see the error. However, the wrong video resolution causes it to return an "invalid video codec." After trying all different types of formats (WebM, H.264, FLV), I finally succeeded by converting the video from its original format of 400x224 to 320x240, which seemed to resolve the problem.

avconv -i <MP4 input file> -s 320x240 <output file>

There isn't much documentation on SmugMug's site to give you any indication that it's the resolution size that has to be adjusted:

http://help.smugmug.com/customer/portal/articles/84569-how-to-convert-and-format-a-video-for-upload-to-smugmug

Side note: if you want to flip the audo/video streams, here's how you do it.   The first "0:1:0" map parameter means to take file input 0 (since you can have multiple input files) and take stream 1 and map it to stream 0.   You do the same for stream 1 of input file 0 and map to stream 0.

avconv -i <input file 0> -vcodec copy -acodec copy -map 0:1:0 -map 0:0:1 <output file>

Wednesday, December 19, 2012

Why lxml.find() calls return False..

https://mailman-mail5.webfaction.com/pipermail/lxml/2005-August/000332.html


What happens is that an element evaluates to False if it has no 
children, and True if it does. The presence of text content, attributes 
or a tail does not affect the boolean status; if no elements exist it'll 
still be False.

find() has the behavior to return None if the value cannot be found. You 
can change your tests to something like:

if xmldoc.find('child') is not None:
     ...

to check whether you have a child.

Friday, December 14, 2012

Microsoft Internet Explorer 12152 errors..

IE has a 1 min keepalive timeout: http://support.microsoft.com/kb/813827

Apache2 has a default 5 second timeout: http://httpd.apache.org/docs/2.2/mod/core.html#keepalive

Therefore, Internet Explorer assumes the connection will close in a minute but Apache2 attempts to close after 5 seconds.  The problem is that if the connection is closed and the initial POST fails, Internet Explorer will only send the header but not the HTTP body.  Since the Apache2 server is expecting the rest of the data, eventually a timeout occurs and the 12xxx errors.

http://stackoverflow.com/questions/4796305/why-does-internet-explorer-not-send-http-post-body-on-ajax-call-after-failure

These patches exist in IE7-IE9, but they aren't by default activated.

This hotfix is included in Internet Explorer 7. However, the hotfix must still be enabled as described in the "How to enable this hotfix" section.

Internet Explorer 8

This hotfix is included in Internet Explorer 8. However, the hotfix must still be enabled as described in the "How to enable this hotfix" section.

Internet Explorer 9

This hotfix is included in Internet Explorer 9. However, the hotfix must still be enabled as described in the "How to enable this hotfix" section.

http://support.microsoft.com/kb/895954
http://support.microsoft.com/kb/831167/en-us

We could disable keepalive's for all Internet Explorer sessions, but here's one way using SetEnvIf in Apache 2.2 to accomplish it:

  # Remove after upgrading to Apache 2.4 and we can use SetEnvIfExpr instead of
  # this funky logic.
  #
  # Test by:
  # curl -v --user-agent 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.2; WOW64; Trident/6.0; .NET4.0E; .NET4.0C; .NET CLR 3.5.30729; .NET CLR 2.0.50727; .NET CLR 3.0.30729; BRI/2;)' -H "X-Requested-With: XMLHttpRequest" -k 
  #
  # ...and all different combinations.
  #
  #  Based on: http://stephane.lesimple.fr/blog/2010-01-28/apache-logical-or-and-conditions-with-setenvif.html
  #
  # Logic below will disable keepalive's on Ajax-based IE browsers.  We may see a small performance
  # hit but better than random 12xxx Microsoft errors because of http://support.microsoft.com/kb/895954.
  # See http://stackoverflow.com/questions/4796305/why-does-internet-explorer-not-send-http-post-body-on-ajax-call-after-failure for
  # more context.
  #
  SetEnvIf User-Agent "^" disable_keepalive=0
  SetEnvIf User-Agent "MSIE [17-9]" disable_keepalive=1
  # Negative lookahead regexp matching.  If there is no Ajax XmlHttpRequest, we can invert
  # the flag that attempts to disable keepalive's.  Equivalent to performing an AND.
  SetEnvIf X-Requested-With "^(?!XMLHttpRequest).*" !disable_keepalive
  SetEnvIf disable_keepalive 1 nokeepalive downgrade-1.0 force-response-1.0
http://blogs.msdn.com/b/ieinternals/archive/2011/03/26/https-and-connection-close-is-your-apache-modssl-server-configuration-set-to-slow.aspx Also note that Apache2's default-ssl file has this configuration:
BrowserMatch "MSIE [2-6]" \
    nokeepalive ssl-unclean-shutdown \
    downgrade-1.0 force-response-1.0
# MSIE 7 and newer should be able to use keepalive
BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown

(Yes, the MSIE [17-9] regex is correct https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/626728)

 What does ssl-unclean-shutdown do?  It lists this information in /etc/default/sites-available/default-ssl:
#   SSL Protocol Adjustments:
#   The safe and default but still SSL/TLS standard compliant shutdown
#   approach is that mod_ssl sends the close notify alert but doesn't wait fo
#   the close notify alert from client. When you need a different shutdown
#   approach you can use one of the following variables:
#   o ssl-unclean-shutdown:
#     This forces an unclean shutdown when the connection is closed, i.e. no
#     SSL close notify alert is send or allowed to received.  This violates
#     the SSL/TLS standard but is needed for some brain-dead browsers. Use
#     this when you receive I/O errors because of the standard approach where
#     mod_ssl sends the close notify alert.
More details:
http://blogs.msdn.com/b/askie/archive/2009/06/18/change-in-behavior-with-internet-explorer-7-and-later-in-regard-to-connect-requests.aspx
https://groups.google.com/group/microsoft.public.winhttp/tree/browse_frm/month/2004-07/ee64525371504ef0?rnum=21&lnk=ol&pli=1\


Addnedum: The problem can often happen if you have load balancers that have a shorter timeout than the keepalive.  In this case, you may wish to increase your load balancer timeouts to be greater than IE Ajax timeouts.  If you adjust the load balancer timeouts, chances are you will see these 12xxxx errors disappear too.

Thursday, December 13, 2012

Configuring Ubuntu 12.04 to take screenshot areas with shortcuts

MacOSX has Command-Shift-4 to take a screenshot area.

You can do the same with Ubuntu...

http://askubuntu.com/questions/170163/how-do-i-set-a-shortcut-to-screenshot-a-selected-area

The instructions are listed below.  The confusing part is that you add the Name/Command and then assign the keyboard shortcut by clicking on the level after adding the entry.

  1. Open System Settings -> Keyboard settings -> Shortcuts
  2. Select Custom Shortcuts(you can go to Screenshot-s too and it will work)
  3. Click +
  4. Fill fields
    • Name to Take a screenshot of area
    • Command to gnome-screenshot -a or shutter -s(if u prefer shutter)
  5. click OK

Wednesday, December 12, 2012

Difference between max-age and expires cookies

http://mrcoles.com/blog/cookies-max-age-vs-expires/


  • Expires sets an expiry date for when a cookie gets deleted
  • Max-age sets the time in seconds for when a cookie will be deleted
  • Internet Explorer (ie6, ie7, and ie8) does not support “max-age”, while (mostly) all browsers support expires
http://blogs.msdn.com/b/ieinternals/archive/2009/08/20/wininet-ie-cookie-internals-faq.aspx

http://www.adobe.com/devnet/coldfusion/articles/coldfusion-securing-apps.html

Any cookies that you create with the httponly attribute will not be present in JavaScript's document.cookievariable on browsers where HttpOnly is supported. Browsers will still send HttpOnly cookies when making AJAX calls or XMLHttpRequest calls, however their values still cannot be accessed from your JavaScript code.

Saturday, December 1, 2012

Using M2Crypto with virtualenv/pip

The M2Crypto version 0.21.1-1 via pip installs breaks when attempting to use in virtualenv: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=637904 If you do apt-get install python-m2crypto, you can create a symbolic link:
ln -s /usr/lib/python2.7/dist-packages/M2Crypto~ /.virtualenvs/[virtualenv dir] /local/lib/python2.7/site-packages/M2Crypto