Deconstructing Facebook's Flash Cross Domain Handler in IE..

I got tired of trying to understand why Facebook kept breaking on their IE support, so decided to peer directly into how they use Flash to implement the cross-domain receiver (you can see inside the .js file they supply that it fetches the SWF file http://static.ak.fbcdn.net/rsrc.php/v1/yx/r/WFg56j28XFs.swf I now suspect the problems Fri. morning that other people reported had to do with 64-bit Internet Explorer, since the Adobe Flash version is still highly experimental, and rebooting my machine (and my other machines at work had no problems) solved the issue.

Nonetheless, the hard part was to find a decompiler that could convert the SWF file that gets loaded in Internet Explorer back into bytecode.  I tried to use flasm but this file was encoded using Adobe Flex 4, which was released around March 2010 so the open source tools are a bit scant.   I ended up using an unregistered version of Sothink's SWFDecompiler, which wants $70.00 for a license.  An older 6.0 version seems to have some quirky bugs where it actually lets you export the ActionScript files back to the original format.  Regardless, the program preserved all the original variable names, so you can pretty much have the original format, and I found a way to post the decompiled files in their entirety here:
http://hustoknow.blogspot.com/2011/06/facebooks-flash-xdcomm-receiver.html

The way in which Facebook has implemented cross-domain message passing in Internet Explorer mimics the use of the HTML5 postMessage.  Instead of creating an event sender/listener for messages from a different window to pass, Facebook relies on SWF objects to talk with other in Internet Explorer. In Flash, there is the concept of LocalConnection objects, where SWF objects can talk with other even when they are on different domains.   It injects this SWF file into both your site as well as after you login through the window popup (facebook.com).

<object type="application/x-shockwave-flash" id="XdComm" name="XdComm" classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" allowscriptaccess="always"><param name="movie" value="https://s-static.ak.fbcdn.net/rsrc.php/v1/yx/r/WFg56j28XFs.swf"></param><param name="allowscriptaccess" value="always"></param></object>

So when you go to any Facebook Connect-enabled web site, you instantiate a SWF object that creates a XdComm object, along with an extra postMessage object.   The XdComm class holds a PostMessage() object, which in turns instantiates a LocalConnection() object, which is basically becomes a port sender/receiver for messages.  Adobe SWF objects, which are contained within ActiveX can be accessed via JavaScript, so the object tag gets added with the XdComm ID and Facebook's JavaScript code can access the postMessage_init() function, which is initialized in the SWF file.

   public function PostMessage()
        {
            this.connection = new LocalConnection();
            this.connection.client = this;
            this.connection.connect(Math.random().toString());
            ExternalInterface.addCallback("postMessage_init", this.init);
            ExternalInterface.addCallback("postMessage_send", this.send);
            return;
        }// end function

When you do an FB.init() on  your localhost, it instantiates FB.XD._openerOrigin (i.e. http://dev.myhost.com/ + 16-character random string),  which is used to define the connection name that SWF objects will be use to communicate.   The FB.XD_openerOrign was the key to the problems this past week -- Facebook was messing around with stuff for IE8, and broke the connection name that the SWF objects would agree on listening.   They later fixed it based on the reviews of the diff I examined.   When you type your login and password, the URL that is used when you submit goes to http://static.ak.fbcdn.net/connect/xd_proxy.php, along with the origin= query string parameter.    So origin must be the same on the window popup connecting to facebook.com as the original one that was initialized on your own Facebook Connect-enabled site.

  } else if (f.transport == 'flash') {
                    var d = window.location.toString() + Math.random();
                    window.FB_OnFlashXdCommReady = function() {
                        document.XdComm.postMessage_init('dummy', d);
                        document.XdComm.postMessage_send(a, f.origin);
                    };
                    createXdCommSwf();
                }

In addition to setting up the origin= query parameter, the FB.init() functions also sets up 3 separate callback functions: ok_session, cancel_session, and no_user, which are instantiated into a callback dictionary each accessible by  a random 16-character signature.   So if you've ever wondered why the query string on the Facebook login is so long, it's partially because of all the info they encode to handle the results of different authentications.  Regardless, the message that gets passed includes a bunch of different URL-encoded string, but one of them is a cb= parameter, which specifies which callback function to run.

Once that callback function is run, then the FB JavaScript calls FB.Auth.setSession() and fires off events...but the message between facebook.com and your site has been successfully exchanged...all via SWF objects in Internet Explorer!