http://blog.jquery.com/2012/09/20/jquery-1-8-2-released/
...and they added this patch, which means the problems with the Comcast Protection Suite at least is resolved.
#12423: jQuery breaks with Comcast Protection Guard and any anti-keylogging protection software on IE7+
Monday, September 24, 2012
Friday, September 21, 2012
More Unicode strangeness in Python 2.x
$ python -c "print u'Hey there\u2013t'" Hey there–t $ python -c "print u'Hey there\u2013t'" > `tempfile` Traceback (most recent call last): File "The solution is:", line 1, in UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 9: ordinal not in range(128)
PYTHONIOENCODING="utf_8" python -c "print u'Hey there\u2013t'" > `tempfile`
Unicode Quirks in Django
Python 2.x's Unicode implementation really leaves something to be desired. The major issue often arises when you need to go between str() and unicode() types, the former is a 8-bit character while the latter is a 16-bit character. The problem is that doing .encode('utf-8') on a unicode object is idempotent (i.e. u'\u2013t'.encode('utf-8') but doing it on a str() twice will cause Python to trigger ascii codec errors.
Here's a great introduction of troubleshooting Unicode issues:
http://collective-docs.readthedocs.org/en/latest/troubleshooting/unicode.html
There's a great PowerPoint slide about demystifying Unicode in Python, which should be required reviewing. It's more detailed about the complexities of UTF-encoding, but it's worthwhile to review.
http://farmdev.com/talks/unicode/
One of the general rule of thumbs that you'll get from this talk is 1) decode early 2) unicode everywhere and 3) encode late.
In Django, this approach is closely followed when writing data to the database. You usually don't need to convert your unicode objects because it's being handled at the database layer. Assuming your SQL database is configured properly and your Django settings are set correctly, Django's database layer handles the unicode to UTF-8 conversion seamlessly. For example, just look inside the MySQLdb Python wrapper and right before a query is executed, the entire string is encoded into the specified character set:
MySQLdb/cursors.py:
What if you attempt to use logging.info() on Django objects? (i.e. logging.info("%s" % User.objects.all()[0]) If you searched on Stack Overflow, you'd see a recommendation to create a __str__(self) in your Python classes that call unicode() and convert to UTF-8:
http://stackoverflow.com/questions/1307014/python-str-versus-unicode
Normally, Python handles string interpolations automatically by determining whether the string is unicode or str() type. Consider these cases:
Assuming your character set on your database is set to UTF-8, consider this example and how Python deals with string interpolations for class. Normally Python does unicode conversions automatically, but for Python classes, "%s" always means to invoke the str() function.
In this failing case, the problem is that printing the A class results in printing a str() type intermixed with a.tst, which is a unicode type. When this issue happens, you're likely to see the UnicodeDecodeError
The same problem happens when trying to attempt to declare the __unicode__() method in your Django models and attempt to print out Django objects and attributes that have Unicode characters, similar to the issues reported in this Stack Overflow article. Because Python string interpolation will invoke the __str__() method, you have to be careful about intermingling Django objects and Django attributes when printing or logging them.
What's the solution? In your Django models, it actually may be useful to force returning the Unicode type in the __str__() method, assuming you also have a __unicode__() method defined. One of the quirks of Python is that if a Unicode type is returned, the __unicode__() method will be attempted to execute. It's somewhat counter-intuitive, but by adding this section of code, you can avoid the hazards of intermingling Django objects and attributes:
The recommendation is also consistent with this python-dev discussion about how to implement __str__() and __unicode__() methods:
Another alternative is to prefix your logging statements with u', which will force the Python string interpolation to run __unicode__() instead of str(). But it's easy to forget this prefix, so overriding the Django base model with this __str__() has helped to avoid triggering these UnicodeDecodeException errors for me.
Here's a great introduction of troubleshooting Unicode issues:
http://collective-docs.readthedocs.org/en/latest/troubleshooting/unicode.html
There's a great PowerPoint slide about demystifying Unicode in Python, which should be required reviewing. It's more detailed about the complexities of UTF-encoding, but it's worthwhile to review.
http://farmdev.com/talks/unicode/
One of the general rule of thumbs that you'll get from this talk is 1) decode early 2) unicode everywhere and 3) encode late.
In Django, this approach is closely followed when writing data to the database. You usually don't need to convert your unicode objects because it's being handled at the database layer. Assuming your SQL database is configured properly and your Django settings are set correctly, Django's database layer handles the unicode to UTF-8 conversion seamlessly. For example, just look inside the MySQLdb Python wrapper and right before a query is executed, the entire string is encoded into the specified character set:
MySQLdb/cursors.py:
if isinstance(query, unicode):
query = query.encode(charset)
if args is not None:
What if you attempt to use logging.info() on Django objects? (i.e. logging.info("%s" % User.objects.all()[0]) If you searched on Stack Overflow, you'd see a recommendation to create a __str__(self) in your Python classes that call unicode() and convert to UTF-8:
http://stackoverflow.com/questions/1307014/python-str-versus-unicode
def __str__(self):
return unicode(self).encode('utf-8') def __str__(self):
if hasattr(self, '__unicode__'):
return force_unicode(self).encode('utf-8')
return '%s object' % self.__class__.__name__
Normally, Python handles string interpolations automatically by determining whether the string is unicode or str() type. Consider these cases:
>>> print type("%s" % a)
<type 'str'>
print type("%s" % 'hey')
<type 'str'>
print type("%s" % u'hey')
<type 'unicode'>
class A(object):
def __init__(self):
self.tst = u'hello'
def __str__(self):
if hasattr(self, '__unicode__'):
return self.__unicode__().encode('utf-8')
return 'hey'
def __unicode__(self):
return u'hey\u2013t'
>>> a = A()
>>> print "%s" % a
hey-t
>>> print "%s %s" % (a, a.tst)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3: ordinal not in range(128)
In this failing case, the problem is that printing the A class results in printing a str() type intermixed with a.tst, which is a unicode type. When this issue happens, you're likely to see the UnicodeDecodeError
The same problem happens when trying to attempt to declare the __unicode__() method in your Django models and attempt to print out Django objects and attributes that have Unicode characters, similar to the issues reported in this Stack Overflow article. Because Python string interpolation will invoke the __str__() method, you have to be careful about intermingling Django objects and Django attributes when printing or logging them.
What's the solution? In your Django models, it actually may be useful to force returning the Unicode type in the __str__() method, assuming you also have a __unicode__() method defined. One of the quirks of Python is that if a Unicode type is returned, the __unicode__() method will be attempted to execute. It's somewhat counter-intuitive, but by adding this section of code, you can avoid the hazards of intermingling Django objects and attributes:
# http://www.gossamer-threads.com/lists/python/bugs/842076
def __str__(self):
return u'%s object' % self.__class__.__name__
The recommendation is also consistent with this python-dev discussion about how to implement __str__() and __unicode__() methods:
This was added to make the transition to all Unicode in 3k easier: . __str__() may return a string or Unicode object. . __unicode__() must return a Unicode object. There is no restriction on the content of the Unicode string for __str__().
Tuesday, September 18, 2012
Daily Deals/Result Glider Spyware
It has surprised me the extent to which this spyware has made its way into a lot of people's browsers. The sites dropdowndeals.com,app.mysupercheap.com and resultglider.com all seem to be the same company (see DNS trace below). Both Mozilla and Intuit mention this spyware, so it's fairly pervasive.
If anyone finds a site which is prompting these users to download this plug-in, let me know...Mozilla mentions that a lot of travel sites are offering this browser download.
Here's an article that talks about this company:
I also wonder if that's why we were seeing invalid crossdomain.xml requests too:
DNS records below:
Non-authoritative answer:
Name: dropdowndeals.com
Address: 4.30.3.59
Non-authoritative answer:
Name: resultglider.com
Address: 4.30.3.140
Non-authoritative answer:
Name: mysupercheap.com
Address: 4.30.3.177
They also appear to hosting YouTube Best Video Downoader (http://www.bestvideodownloader.com) and are using a mail forwarding address in Michigan:
http://www.bbb.org/western-michigan/business-reviews/internet-services/alactro-in-grandville-mi-38141335
They also appear to hosting YouTube Best Video Downoader (http://www.bestvideodownloader.com) and are using a mail forwarding address in Michigan:
http://www.bbb.org/western-michigan/business-reviews/internet-services/alactro-in-grandville-mi-38141335
Sunday, September 16, 2012
E911 for ObiTalk
This article seems to provide the best instructions about how to activate 911 service dialing through your ObiTalk device:
http://www.obitalk.com/forum/index.php?topic=339.msg1766#msg1766
Basically the ObiTalk devices are by default setup to dial 911 though the 'li' (Line Port) in their default configuration. The OutboundCallRoute provides a series of rules that are followed to determine how the call should be made (see p. 179-180 of the ObiDeviceAdminGuide.pdf document). The default rule is listed below:
If you want route 911 calls to a local 24-hour emergency line by adding a rule that instead of routes to the li port, you need to remove the 911 redirecting to line port and then add {(<911:1xxxxxxxxxx):spX} where spX is the SIP line you're using (i.e. sp1).
One further note: the config changes should be done on your obitalk.com settings, not directly on your ObiTalk firmware. For some reason, when you reboot, the changes will be overridden by those set on ObiTalk.com (unless you uncheck the checkbox).
http://www.obitalk.com/forum/index.php?topic=339.msg1766#msg1766
Basically the ObiTalk devices are by default setup to dial 911 though the 'li' (Line Port) in their default configuration. The OutboundCallRoute provides a series of rules that are followed to determine how the call should be made (see p. 179-180 of the ObiDeviceAdminGuide.pdf document). The default rule is listed below:
How do these rules work? First, each rule is described in {} as OR operations. The first rule {(<#>:911):li}, for instance, describes how dialing # or 911 will route to the line port. The second rule dialing **0 will invoke the automated attendant. The remaining descriptions of how these rules are described on page 180 of the manual:
One further note: the config changes should be done on your obitalk.com settings, not directly on your ObiTalk firmware. For some reason, when you reboot, the changes will be overridden by those set on ObiTalk.com (unless you uncheck the checkbox).
Wednesday, August 29, 2012
The pernicious effects of the Comcast Protection Suite....
This mystery had been eluding me for at least 5-6 months since we started introducing JavaScript exception monitoring, but I finally was better able to understand why the Comcast Protection Suite has been causing so many problems for many our users. No, it isn't injecting its own jQuery, and no it isn't defining its own $, but rather it's trying to do anti keystroke logging....
We knew anecdotally that disabling the Comcast plug-in solved the issue, but I could never explain why the exception occurred (such as the one below). Apparently the Comcast Protection Suite installs a Browser Helper Object DLL into Internet Explorer. The DLL has just as much control over the DOM that JavaScript modules do. It apparently is also adding keyup (and perhaps keydown) event handlers to the DOM, presumably to keep somebody else from capturing your keystrokes.
Not only does it slow down overall performance on browsers, but the DLL also causes conflicts with jQuery because jQuery tries to execute the native events after executing jQuery events (i.e. running 'onclick' events after bind('click') or live('click') events). I guess there are some issues running DLL onclick events in JavaScript, since an exception gets generated in not being able to call apply() on these handlers. If you try...catch them, the problem at least gets mitigated...but this requires a change directly within jQuery 1.7.2. jQuery 1.8.0 is out, but it still has this same problem.
Also, apparently the Comcast Protection Suite software that gets passed along for a lot of new Comcast users, which is why we believe this problem is so pervasive. I'm even more astounded by Norton, which had this response after a user complained about the slower keystroke rates once this software was installed:
http://community.norton.com/t5/Other-Norton-Products/disappearing-keystrokes-in-webform/td-p/613012
This particular issue appears to be isolated to this specific site and is directly related to a JavaScript function the website owners have implemented to test whether the first name or last name text filed is only alpha characters. The method employed at the website to check for alpha characters is not the standard approach, which is to check for the input as it is added onKeyUp. The standard JS best practice is to use a regular expression that checks and validates the entire field input at form submission. Therefore we expect this to be an isolated issue.
The jQuery bug report is here: http://bugs.jquery.com/ticket/12423
We knew anecdotally that disabling the Comcast plug-in solved the issue, but I could never explain why the exception occurred (such as the one below). Apparently the Comcast Protection Suite installs a Browser Helper Object DLL into Internet Explorer. The DLL has just as much control over the DOM that JavaScript modules do. It apparently is also adding keyup (and perhaps keydown) event handlers to the DOM, presumably to keep somebody else from capturing your keystrokes.
Not only does it slow down overall performance on browsers, but the DLL also causes conflicts with jQuery because jQuery tries to execute the native events after executing jQuery events (i.e. running 'onclick' events after bind('click') or live('click') events). I guess there are some issues running DLL onclick events in JavaScript, since an exception gets generated in not being able to call apply() on these handlers. If you try...catch them, the problem at least gets mitigated...but this requires a change directly within jQuery 1.7.2. jQuery 1.8.0 is out, but it still has this same problem.
Also, apparently the Comcast Protection Suite software that gets passed along for a lot of new Comcast users, which is why we believe this problem is so pervasive. I'm even more astounded by Norton, which had this response after a user complained about the slower keystroke rates once this software was installed:
http://community.norton.com/t5/Other-Norton-Products/disappearing-keystrokes-in-webform/td-p/613012
This particular issue appears to be isolated to this specific site and is directly related to a JavaScript function the website owners have implemented to test whether the first name or last name text filed is only alpha characters. The method employed at the website to check for alpha characters is not the standard approach, which is to check for the input as it is added onKeyUp. The standard JS best practice is to use a regular expression that checks and validates the entire field input at form submission. Therefore we expect this to be an isolated issue.
Besides asking every user to disable the Comcast toolbar plug-in, the workaround/fix is actually quite simple. Apparently the native onclick handler checks need to verify that apply() can be run. Normally DLL onclick handlers return 'undefined', which therefore are missing an apply() function. We can enforce this check within the jQuery code, for which we'll be submitting a patch soon.
| 3212 | 3212 | // Note that this |
| 3213 | 3213 | handle = ontype && |
| 3214 | if ( handle && | |
| 3214 | // Workaround for Comcast | |
| 3215 | // Apparently | |
| 3216 | if ( handle && | |
| 3215 | 3217 | event. |
| 3216 | 3218 | } |
| 3217 | 3219 | } |
Using JavaScript line breaking in YUI Compressor
For the past 5 months of introducing JavaScript exceptions logging, there have been "Object doesn't support this property or method" or "Object doesn't support property or method 'apply' that has been elusive in trying to diagnose in Internet Explorer browsers. The error messages never occurred in other browsers but we saw them quite often in different users. While some of these exceptions were generated from actually calling methods on JavaScript objects didn't exist, many of the stack traces seemed to emanate directly from jQuery.
One of our challenges was to try to understand from where these exceptions within jQuery were occurring. jQuery usually comes minified with no line breaks, so we ran our JS minifiers with the YUI Compressor with the --line-break option (i.e. --line-break 150). Before adding this option, Internet Explorer would often report an error on line 2, which pretty much amounted to the entire jQuery code. By breaking the minified code into smaller chunks, the line numbers could then allow further information on pinpointing this exact source of conflicts:
java -jar ../external/yuicompressor-2.4.7.jar jquery-1.7.2.min.js --line-break 150 > jquery-1.7.2_yui.min.js
This stack trace helped us pinpoint the issue to the Comcast Protection Suite, since it indicated the problem was happening directly inside the jQuery Event module. The jQuery Event module is used to attach and trigger jQuery events, as well as to implement the event propagation path described in the W3 standard. By issuing try/except clauses within the dispatch() code, we were able to find exactly where the exception occurred:
In other words, what jQuery seems to do is execute all the jQuery-related events before attempting to call the native JavaScript events (i.e. jQuery 'click' events will first be executed before the native 'onclick' events get called). Somehow the Comcast Protection Suite adds an onclick handler that appears as 'undefined' to jQuery. The if statement passes except fails when attempt to execute the handle.apply() statement.
More on this finding on the Comcast Protection Suite in this next writeup...
One of our challenges was to try to understand from where these exceptions within jQuery were occurring. jQuery usually comes minified with no line breaks, so we ran our JS minifiers with the YUI Compressor with the --line-break option (i.e. --line-break 150). Before adding this option, Internet Explorer would often report an error on line 2, which pretty much amounted to the entire jQuery code. By breaking the minified code into smaller chunks, the line numbers could then allow further information on pinpointing this exact source of conflicts:
java -jar ../external/yuicompressor-2.4.7.jar jquery-1.7.2.min.js --line-break 150 > jquery-1.7.2_yui.min.js
url: https://www.myhost.com/static/js/jquery-1.7.2_yui.min.js
line: 33
context:
(f._data(m,"events")||{})[c.type]&&f._data(m,"handle"),q&&q.apply(m,d),q=o&&m[o],q&&f.acceptData(m)&&q.apply(m,d)===!1&&c.preve
ntDefault()
}c.type=h,!g&&!c.isDefaultPrevented()&&(!p._default||p._default.apply(e.ownerDocument,d)===!1)&&(h!=="click"||!f.nodeName(e,"a"))&&f.acceptData(e)&&o&&e[h]&&(h!=="focus"&&h!=="blur"||c.target.offsetWidth!==0)&&!f.isWindow(e)&&(n=e[o],n&&(e[o]=null),f.event.triggered=h,e[h](),f.event.triggered=b,n&&(e[o]=n));return c.result}},dispatch:function(c){c=f.event.fix(c||a.event);var d=(f._data(this,"events")||{})[c.type]||[],e=d.delegateCount,g=[].slice.call(arguments,0),h=!c.exclusive&&!c.namespace,i=f.event.special[c.type]||{},j=[],k,l,m,n,o,p,q,r,s,t,u;g[0]=c,c.delegateTarget=this;if(!i.preDispatch||i.preDispatch.call(this,c)!==!1){if(e&&(!c.button||c.type!=="click")){n=f(this),n.context=this.ownerDocument||this;for(m=c.target;m!=this;m=m.parentNode||this){if(m.disabled!==!0){p={},r=[],n[0]=m;for(k=0;ke&&j.push({elem:this,matches:d.slice(e)});fo
r(k=0;k
url: https://www.myhost.com/static/js/jquery-1.7.2_yui.min.js
column: None
line: 34
func: filter
url: https://www.myhost.com/static/js/jquery-1.7.2_yui.min.js
column: None
line: 31
func: trigger
for ( i = 0; i < eventPath.length && !event.isPropagationStopped(); i++ ) {
cur = eventPath[i][0];
event.type = eventPath[i][1];
handle = ( jQuery._data( cur, "events" ) || {} )[ event.type ] && jQuery._data( cur, "handle" );
if ( handle ) {
handle.apply( cur, data );
}
// Note that this is a bare JS function and not a jQuery handler
handle = ontype && cur[ ontype ];
if ( handle && jQuery.acceptData( cur ) && handle.apply( cur, data ) === false ) {
event.preventDefault();
}
}
In other words, what jQuery seems to do is execute all the jQuery-related events before attempting to call the native JavaScript events (i.e. jQuery 'click' events will first be executed before the native 'onclick' events get called). Somehow the Comcast Protection Suite adds an onclick handler that appears as 'undefined' to jQuery. The if statement passes except fails when attempt to execute the handle.apply() statement.
More on this finding on the Comcast Protection Suite in this next writeup...
Subscribe to:
Posts (Atom)