Monday, December 30, 2013

Android NinePatch Image Pain

Since 2010, Android has provided 9-patch image support. It enables app developers to specify bitmap areas that can be stretched in order to allow text to fit inside graphics. This format allows custom graphics to be used as buttons with fairly quick ease.  Even iOS appears to have borrowed this concept and third-party libraries are available to leverage this format.

There are a multitude of sites explaining how to use the tool to create your own 9 patch images.  The documentation on Android's site provides a walkthrough about how to use the draw9patch tool to create new ones.  The idea of 9-patch is to define the areas that can be stretched both horizontally and vertically by drawing a line along the outside border.  If you also want to specify where the text should fix, you also need to define the region by creating lines along the right and bottom lines.


The draw9patch tool provides the ability to show how these areas are defined as you change the borders, but it often doesn't show you what the gotchas are.  I encountered some of these hiccups when trying to create custom speech bubbles with the Android Map Utils library.  Here was my first attempt to define the stretchable region:


Here's the result:


The arrow gets stretched too, so we actually want to define two stretchable regions:


The actual result?

Android's documentation says that if the bottom and right lines are not included, it uses the left and top lines.   It doesn't say what happens what happens if two stretchable regions are defined.   To avoid this confusion, we add a black line to the border below:


Now we get:


Also, if you're trying to understand how NinePatch internals works, you actually have to dig into NinePatchImpl.cpp.  The NinePatch implementation is actually a JNI interface to C++ code. The header file ResourceTypes.h is actually the best documented source of how the stretchable regions are defined internally.    The implementation code handles the magic of stretching the regions and proportionately scaling them.  In the example above,  only 1 pixels are defined on both left and right so they will be stretched uniformly.

Underlying all these internals is that Android relies on the Skia 2D library, so many of the esoteric units that start with Sk has to deal with this fact that much of the library deals in SkScalar metrics.   The magic happens with the calculateStretch() function, which figures out how many pixels to scale based on the total boundary size and the remaining number of stretchable regions.  It just turns out that much of Android's graphics library is built with Skia, so trying to understand how any of it works leads you down the path of reviewing the C++ code

Finally, keep in mind that the only way the draw9patch/Android knows that your images are 9-patch drawable is by the file extension.  If you save your custom button with rounded corners with the 9.png file extension, it's likely you'll find your corners cut off since it assumes that the outer 1px borders have been created to define the stretchable and content regions.   There's no magic number type, much less any type of header prefix, so be careful about naming of files.

Furthermore, there's also a source and compiled version of 9.png files.   If you try to retrieve the 9.png file from the .apk file, chances are it will not look like what's in your source code.

Sunday, December 22, 2013

RSA encryption and IntelliJ's 128-bit RSA keys

With the recent press about the NSA's attempts to introduce backdoors into the RSA algorithm and a research paper about how GnuPG encryption keys could be derived from acoustic analysis, I decided to refresh my understanding of how the encryption algorithm works. One of the best intros is located here, and while it doesn't cover the advanced topics of Fermat's Little Theorem, Extended Euclidean Algorithm, or the Chinese Remainder Theorem, it does show it works in basic mathematical terms.

I also noticed that the license keys for IntelliJ's JetBrains products are only using 128-bit RSA encryption keys (updated previously from 256-bits to reflect the right size). The reverse engineering work to figure out how they are generated shows how 128-bit keys can be easily factored and the private key derived from the public key and modulus. Since many of IntelliJ's products are built in Java/Swing, the developers must have known that the bytecode could easily be decompiled. Why would they introduce such weak encryption?   Given that RSA algorithms are license-free, the most plausible explanation to me seems export-related. If they were to use stronger encryption keys, they would be subject to export reviews.

Tuesday, December 10, 2013

DD-WRT, Afraid.org, and truncating base64 hashes

I've noticed that since 2010 that DD-WRT has had problems with correctly implementing Dynamic DNS authentication with Afraid.org.   You enter your username and password, but the authentication fails.   To get around this issue, you need to follow this workaround: http://www.dd-wrt.com/phpBB2/viewtopic.php?p=781615

I decided to poke into the DD-WRT source code to understand why this nuisance continues to persist to this day.  What did I find?  Inside router/httpd/validate.c, DD-WRT makes a call to http://freedns.afraid.org/api/?action=getdyndns&sha=[SHA] where [SHA] represents the SHA1 function of "username|password" (as an example of how this authentication process works, see https://github.com/atdt/afraid/blob/master/afraid/__init__.py#L94)).  The response, assuming you aren't using the Curl command-line, comes in this form:

DYNDNS_HOST|IP_ADDRESS|http://freedns.afraid.org/dynamic/update.php?HASH_VALUE

Afraid.org needs this hash value when DD-WRT reports its DNS location. The problem is that DD-WRT only assumes there are only 36 characters in the hash. The offending section is here: https://github.com/mirror/dd-wrt/blob/master/src/router/httpd/validate/webs.c#L3438

This pull request should fix this issue:

https://github.com/mirror/dd-wrt/pull/5/files

How did I verify?  Well, I saved the response from making an authenticated call to Afraid.org:

>>> open("/tmp/hash", "w").write(urllib.urlopen("http://freedns.afraid.org/api/?action=getdyndns&sha=xxxx").read())

Then I created and compiled this source code to verify:
#include <stdio.h>
#include <malloc.h> char *main() { int i; FILE *in = fopen("hash", "rb"); while (getc(in) != '?' && feof(in) == 0) ; i = 0; char *hash = malloc(64); if (feof(in)) { free(hash); return NULL; } for (i = 0; i < 63 && feof(in) == 0; i++) { hash[i] = getc(in); if (hash[i] == EOF) break; } fclose(in); hash[i] = 0; printf("%s", hash); return hash; }
Note in the bolded text that we have to break out of the loop if the character is EOF, since feof() will only exit after one extra loop.  We could also set the string terminator to be i-1 too (see Stack Overflow article about the challenges feof() presents)

Buffalo WZR-600DHP Firmware Fun

I got this Buffalo WZR-600DHP router thinking that it was a worthy successor to the WRT64GL router, which radio stopped working on me over the weekend. The Buffalo router also had DD-WRT installed, so I thought it meant one less step to install. Turns out to have been a 2-hour pain to configure.

1. The default username/password didn't even work, trying every single combination of admin/password, root/password, buffalo/dd-wrt123, etc. that was possible. I had to hold down the Reset button for 30 seconds to reset everything back to a default state.  (If you hold down shorter than this period, I often had issues resetting the firmware).

2. Soon after tweaking the IP address to something other than the default IP address of 192.168.3.1 and rebooting, the Buffalo router started to create 302 redirects on its web interface to 192.168.11.1. Sure you could use the router so long as you never needed to touch the admin interface. Was there something in the firmware that has some hard-coded setting? Or is it running a DD-WRT version that makes assumptions about its IP address?

 3. The solution for both items was simply to replace Buffalo's stock DD-WRT firmware with the latest one available. Make sure to "reset to default settings" when uploading the new firmware to clear out the old stuff setup by Buffalo.  Buffalo's stock firmware also prevents you from enabling SSH into the router, which makes even harder given problems with #2 to troubleshoot.

4. Don't close any windows until the firmware is flashed.

5. Connect to the router at 192.168.1.1. If you're using DynDNS and setting up with afraid.org, keep in mind that the hostname specified needs to be hostname,hash value (i.e. xxxx.mooo.com,abcde). There's still a bug in DD-WRT with the hash being truncated (http://www.dd-wrt.com/phpBB2/viewtopic.php?p=781615). You can go to http://freedns.afraid.org/dynamic/ to copy/paste the right hash value from the direct URL (i.e. http://freedns.afraid.org/dynamic/update.php?HASHVALUEHERE)

Thursday, December 5, 2013

Running Django's unit tests

1. Create a settngs.py file (or copy from django/conf/project_template/project_name/settings.py).

2. Make sure the databases are setup:
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'test',
        'USER': 'test',
        'PASSWORD': 'test'
    }
}

3. DJANGO_SETTINGS_MODULE=settings PYTHONPATH=. tests/runtests.py

If you get:
from django.utils._os import upath
ImportError: cannot import name upath

Chances are your DJANGO_SETTINGS_MODULE and PYTHONPATH are set.

Sunday, December 1, 2013

Samsung DVR firmware

It turns out that the Samsung DVR firmware is basically a gzip'ed package of an embedded lighttpd Linux kernel built for the ARM processor.   If you're curious about the code that it runs, you can mount the image and inspect.  There's no source code included, but Samsung should probably open source it given there are so many inherent securrity vulnerabilities.

1. Download the firmware http://www.samsungsv.com/Support/DVRFirmwareUpdate

2. Unzip the zip file.

3. Gzip decompress the .img file.

4. If you have Linux, you can mount the jffs2 file.  You'll need to install the packages to mount JFFS2 images.

sudo apt-get install mtd-tools
sudo modprobe -v mtd
sudo modprobe -v jffs2
sudo modprobe -v mtdram total_size=256000 erase_size=256
sudo modprobe -v mtdchar
sudo modprobe -v mtdblock

5. Create a mount directory for the image (i.e. mkdir /mt/tst)

6. Create the JFFS2 image by writing to the /dev/mtdblock0 device:

sudo dd if=hi3520_data.jffs2 of=/dev/mtdblock0

7. Mount the device:
mount -t jffs2 /dev/mtdblock0  /mnt/tst


8. Much of the web server code resides in the /root/www directory.

Friday, November 29, 2013

Seriously Samsung? Home surveillance cameras that exposes usernames and passwords to login.

SDE-4001N
Samsung's home surveillance IP cameras (including the SDE and SDR models) must be controlled with Silverlight Internet Explorer plugins, so I decided over the Thanksgiving holiday to see whether I could study the network traffic and understand how these devices worked.  While studying some of the endpoints, I came upon this security bulletin published 3 months ago.

http://www.securityfocus.com/archive/107/528120/30/0/threaded
Samsung provides a wide range of DVR products, all working with nearly
the same firmware. The firmware it's a Linux embedded system that
expose a web interface through the lighttpd webserver and CGI pages.

The authenticated session is tracked using two cookies, called DATA1
and DATA2, containing respectively the base64 encoded username and
password. So, the first advise for the developers is to don't put the
user credentials into the cookies!

Anyway, the critical vulnerability is that in most of the CGI, the
session check is made in a wrong way, that allows to access protected
pages simply putting an arbitrary cookie into the HTTP request. Yes,
that's all.

This vulnerability allows remote unauthenticated users to:

- Get/set/delete username/password of local users (/cgi-bin/setup_user)

- Get/set DVR/Camera general configuration

- Get info about the device/storage

- Get/set the NTP server

- Get/set many other settings
It turns out that the major Samsung home surveillance DVR have this security flaw. Essentially all the username and passwords are unhashed and can be retrieved by using an arbitary cookie.   The source code that retrieves this info is included here:

http://www.andreafabrizi.it/download.php?file=samsung_dvr.py

If you want to verify whether your device is vulnerable, check out :

http://ismysamsungdvrhacked.appspot.com/ (GitHub code here)

Hoping to find that Samsung fixed the issue, I went to http://www.samsungsv.com/Support/DVRFirmwareUpdate and discovered that the firmware versions have yet to be updated.

Why does this issue matter? For one, Samsung provides a Dynamic DNS service at samsungipolis.com (i.e. http://samsungipolis.com/home1) that enables customers to have their systems devices report the IP location.  This IP address is stored by Samsung and provides a redirect to the owner's hosted cameras.

$ curl http://www.samsungipolis.com/testing
<body>

    <meta http-equiv="refresh" content="0;url=http://98.17.152.5:80"/>


</body>

In other words, all you have to do is find a name that's using Samsung's Dynamic DNS service, point the Python script at this IP, and assuming the owner has enabled remote web interface access for the user account, you can login to their service (and possibly gain admin access to make any changes).  You obviously have to use Internet Explorer to test, since the plug-ins were built in Silverlight.  (If you get a blank white screen, you're probably using a non-IE browser.)

I've sent a note to their customer support and tweeted to them about this unpatched security flaw (now going on 3 months)  If you are running one of these systems, you're best option is to disable the web interface until this flaw is patched.

Update (12/1/2013): Samsung's web site at samsungsv.com does not have the updated firmware images, but you can sign up for a free account at http://developer.samsungtechwin.com and get access to newer versions of the firmware.

For the SDE-5001 DVR device, for instance, the latest version is v1.05.  However, this last update was made on 01/2013, about a year ago, which suggests that the patch has yet to be applied.  Also, by mounting the flash image, we can also see that the binary for the cgi_login program which has this security hole does not appear to have changed since the v1.02 version.   I also checked the lighttpd configuration and did not see any differences in the files. In other words, the security flaw still seems to exist.

Friday, November 22, 2013

Auto populating PDF form data

How easy is it to auto-populate form data in a PDF? Well, it turns out there's a tool called PDFTk (described in http://www.mactech.com/articles/mactech/Vol.20/20.11/FillOnlinePDFFormsUsingHTML/index.html) that provides the ability to dump the interactive fields into a FDF and load it back into the PDF. The download links don't appear on the main site, but you can go to this link to grab the latest versions.

Ubuntu v12.04 has PDFTk version 1.4, which isn't the latest version. To compile the latest, download the source and apt-get install gcj, which is a Java to native bytecode compiler. Link the Makefile.Debian to Makefile, and run make. At the end, you should have a working pdftk binary!

PDFTk seems to provide a wrapper over the iText PDF library. There is one issue whereby if an owner password is set, you are prevented from opening it (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=531529) The workaround is either to patch the source accordingly (see https://github.com/onehub/pdftk/blob/master/debian/patches/drm_fix) or use qpdf --decrypt to remove the encryption before using pdftk with it.

Friday, November 15, 2013

Mac Excel 2011 doesn't support UTF-8?

It turns out the workaround is to install TextWrangler and export documents via UTF-8..

http://legacy.x3ro.de/excel-mac-2011-utf8/

Sunday, September 29, 2013

Android emulator temporary directory

Wondering where your disk space goes when using the Android emulator?   Apparently it goes inside /tmp/android-$USER (where $USER is the current user logged in):

https://github.com/mveytsman/android_emulator_spoofing/commit/c0e96f0ec192eff4b36574f703e62030da028bbf

Saturday, September 28, 2013

Experimenting with a HDMI to VGA converter....

Recently I bought an HDMI to VGA converter in an attempt to feed the output from a laptop to an older LCD projector. When trying to test this converter with a Thinkpad T510 laptop and 25" HP monitor, I noticed that there was no video signal detected.   I tried another Toshiba laptop but could get no signal from the monitor.  At first I thought that this device was defective and need to be returned.

However, after feeding the HDMI signal from the laptop to an HDMI splitter and connecting one of the HDMI outputs with this converter, I could start to see a video signal.    Apparently the HDMI to VGA converter draws power from pin 18 of the HDMI connector, which provides the 5 volt power supply.  The HDMI specs specify that source devices should provide a minimum of 55 mA, but the chipset used by the HDMI to VGA converter must be drawing much more than current than what can be provided by the laptop.

Even more interesting, the video signal could be detected with a 25 foot MediaBridge cable but not with a 25 foot Aurum cable.  The major difference?  The former uses 28 AWG gauge wiring, while the latter uses 30 AWG gauge wiring.

Why does the gauge of the wiring matter?   With the lower gauge wiring, the diameter is usually larger, so the resistance is lower and therefore the current is higher.   Even when using the HDMI splitter that will provide additional power, this current can be reduced by the longer cabling.

The solution seems to be to use an active HDMI to VGA converter, since sufficient current draw is mentioned as a possible issue by the active converter devices sold on Amazon.   However, after this finding, you may also need to consider what HDMI cabling you're using too!

Playing 720p and 1080p videos on Ubuntu 12.04

If you find the playback for MP4 files saved to 720p or 1080p resolution on Ubuntu 12.04 choppy and you're using an Nvidia graphics card, you might want to try using the proprietary Nvidia drivers and make sure to install the VDPAU library installed.

The stock Movie Player does not appear to support the hardware acceleration capabilities, nor does the VLC version v2.0 that comes with 12.04.  You'll need to install VLC 2.2 from the repo.

VLC 2.2 upgrade installations are here:

http://linuxg.net/how-to-install-vlc-2-2-on-ubuntu-13-10-13-04-12-10-12-04-and-linux-mint-15-14-13-via-ppa/



You can also force MPlayer from the command-line to always use the VDPAU extensions:



Sunday, September 15, 2013

Android x86 on Ubuntu

Apparently using the Android virtual emulator is a bit slow on x86 machines, so the recommendation is to use the Android x86 versions.

The images can be downloaded from here:

http://code.google.com/p/android-x86/downloads/list

The instructions are here:

http://www.upubuntu.com/2012/03/how-to-install-android-x86-40-using.html

Though the Android SDK has instructions for downloading this image directly:

http://developer.android.com/tools/devices/emulator.html

Tuesday, August 27, 2013

Jenkins and Google Apps OpenID SSO

Seeing this screen with Jenkins and the OpenID plugin? ("The requested URL /... is too large to process).  


The problem is that the plugin tries to use the OpenID team memberships extension, which isn't part of the Google OpenID implementation.  Google must have recently started limiting the URL that gets passed during the OpenID flow or our list of people in our authentication list got too big.  Either way, the query string is enormous and the way to fix it is to disable this extension.

To fix this issue, you have to download this patch and apply it to the current OpenID v1.7 branch.  I tried commenting out the offending code, but tests within this plugin started to break and prevent you from building the .hpi plugin.

1. git clone https://github.com/jenkinsci/openid-plugin
2. wget https://github.com/jenkinsci/openid-plugin/pull/4.diff
3. patch -p1 < 4.diff
4. mvn package

Recompile the OpenID plugin and copy the target/openid.hpi plugin into your jenkins/plugins dir.  Restart Jenkins and the problem should be gone.

We're hoping this commit gets merged soon!


Update (1/16/2014): Add the disable statement in your /etc/default/jenkins file with the JAVA_ARGS variable. Because Java system properties must be defined before .jar files are included, declaring them with JENKINS_ARGS will not work.
JAVA_ARGS="-Dhudson.plugins.openid.impl.TeamsExtensiondisable=true"
JENKINS_ARGS="--webroot=/var/cache/jenkins/war --httpPort=$HTTP_PORT --ajp13Port=$AJP_PORT"

Wednesday, July 31, 2013

Python data structures in GDB

One of the neat use cases of the gdb-heap project is the ability to inspect the Python data structures within Gdb.  For some reason, the current version adapted for Ubuntu 12.04 doesn't seem to categorize the structures with the C-runtime of Python, so I sought out to find out why.

I've noticed that even after installing the Ubuntu 12.04 python-dbg, the various Python data types with the stock installation do not always resolve even after loading the symbol file from /usr/lib/debug/usr/bin/python2.7:

gdb --args python /tmp/tst.py

(gdb) run
[Ctrl-C]

I noticed that none of the data types can be found:
(gdb) ptype PyObject
No symbol "PyObject" in current context.
(gdb) ptype PyVarObject
No symbol "PyVarObject" in current context.

...but for the debug build, gdb --args python-dbg /tmp/ac.py works:

(gdb) ptype PyObject
type = struct _object {
    Py_ssize_t ob_refcnt;
    struct _typeobject *ob_type;
}
(gdb) ptype PyVarObject
type = struct {
    struct _object *_ob_next;
    struct _object *_ob_prev;
    Py_ssize_t ob_refcnt;
    struct _typeobject *ob_type;
    Py_ssize_t ob_size;
}

When setting verbose mode on within gdb (set verbose mode on), I noticed that gdb was looking up the debug symbols to the various modules with python2.7-dbg, but not with python2.7:

(gdb) set verbose on
(gdb) info types
Reading in symbols for ../Modules/symtablemodule.c...done.
Reading in symbols for ../Modules/zipimport.c...done.
Reading in symbols for ../Modules/_weakref.c...done.
Reading in symbols for ../Modules/_codecsmodule.c...done.
Reading in symbols for ../Modules/_sre.c...done.
Reading in symbols for ../Modules/pwdmodule.c...done.
Reading in symbols for ../Modules/errnomodule.c...done.

In stock Python, you don't see these symbols being referenced:

(gdb) set verbose on
(gdb) info types
Reading in symbols for bsearch.c...done.
Reading in symbols for ../sysdeps/x86_64/multiarch/init-arch.c...done.
Reading in symbols for ../sysdeps/x86_64/multiarch/cacheinfo.c...done.
Reading in symbols for wordcopy.c...done.
Reading in symbols for ../sysdeps/x86_64/multiarch/memmove.c...done.
Reading in symbols for ../sysdeps/x86_64/multiarch/rtld-memcmp.c...done.
Reading in symbols for ../sysdeps/unix/sysv/linux/x86_64/sigaction.c...done.
Reading in symbols for environ.c...done.
Reading in symbols for ../nptl/sysdeps/unix/sysv/linux/getpid.c...done.

After looking further, I noticed that the filenames for the linked modules were changed. The xxx.ltrans.o files seem to suggest that gcc link time optimization (-flto/-fltrans) was enabled to compile with the Ubuntu 12.04 install:


readelf -a /usr/lib/debug/usr/bin/python2.7 | grep FILE
    35: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
    43: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
    48: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ccvMPsIN.ltrans0.o
   166: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ccvMPsIN.ltrans8.o
   196: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ccvMPsIN.ltrans9.o
   210: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ccvMPsIN.ltrans10.o
   326: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ccvMPsIN.ltrans11.o
   369: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ccvMPsIN.ltrans12.o
   400: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS ccvMPsIN.ltrans13.o
.
.
.

$ readelf -a /usr/bin/python2.7-dbg | grep FILE
    35: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
    43: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS crtstuff.c
    48: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS python.c
    49: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS getbuildinfo.c
    52: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS acceler.c
    55: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS grammar1.c
    58: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS listnode.c
    63: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS node.c
    67: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS parser.c
    77: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS parsetok.c
    81: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS bitset.c
    82: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS metagrammar.c
   112: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS firstsets.c
   115: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS grammar.c
   118: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS pgen.c
   148: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS myreadline.c
   151: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS tokenizer.c
   184: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS abstract.c
   217: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS boolobject.c
   229: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS bufferobject.c
   258: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS bytes_methods.c



Also, typing python-config doesn't seem to indicate any of this special flag being used:

python-config --cflags --ldflags
-I/usr/include/python2.7 -I/usr/include/python2.7 -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security
-L/usr/lib/python2.7/config -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic -Wl,-O1 -Wl,-Bsymbolic-functions


I compiled my own version and using -O2/-O3 on an Ubuntu 12.04 install didn't seem to perform this special optimization. If you specifically however specify CFLAGS="-lto" LDFLAGS="-lto" ./configure, then the link-time optimization code will be added and the xxx.ltrans.o files will be used. For some reason using this compilation flag removes debugging symbols. More about link optimization is discussed here.

This finding seems to imply that using Gdb to inspect the Python data structures can only be done with the python-dbg binary or using your own compiled version of Python without using link-time optimizations. The latter seems preferable since the python-dbg changes the internal data strutures of Python and requires recompiling all C extension modules.

Friday, July 19, 2013

How Sentry discovers plugins

You can use the pkg_resources library to query against sentry.plugins. Each plugin has an .egg directory that defines an entry_point.txt file.
>> import pkg_resources
>> list(pkg_resources.iter_entry_points('sentry.plugins'))
EntryPoint.parse('sentry_statsd = sentry_statsd.plugin:StasdPlugin'), EntryPoint.parse('hipchat = sentry_hipchat.models:HipchatMessage')]

Sunday, July 14, 2013

SQLite3

On Ubuntu 12.04, I was trying to see whether SQLite supports multi column inserts:

sqlite> CREATE TABLE test_table
   ...> (col1 INTEGER NOT NULL,
   ...> col2 CHAR(25));
sqlite> insert into test_table (col1, col2) values (1, 'ab');
sqlite> insert into test_table (col1, col2) values (1, 'ab'),(2, 'bc');
Error: near ",": syntax error

As of SQLite 3.7.11 (Ubuntu has 3.7.9 installed), this syntax is possible: http://sqlite.org/releaselog/3_7_11.html 

 Older responses on Stack Overflow state that it doesn't, referring to the diagram from the SQLite documentation. But this diagram has already been updated to reflect this new change:

Tuesday, June 11, 2013

How are sbrk_base and main_arena related in gdb?

Inside the malloc.c code, there is the use of sbrk_base. How is it related to the main heap (declared as main_arena)?

From the glib v2.15+ code:

malloc/malloc.c
/* A contiguous main_arena is consistent with sbrk_base.  */
  if (av == &main_arena && contiguous(av))
    assert((char*)mp_.sbrk_base + av->system_mem ==
       (char*)av->top + chunksize(av->top));

You can gdb to a process and try to verify. You'll need to have the libc6-dev and libc6-dbg packages to get the correct debugging symbols for libc:

(gdb) print (char *)main_arena->top + (main_arena->top->size & ~(0x4 | 0x2 | 0x1))
$1 = 0xc17000
(gdb) print (char *)mp_.sbrk_base + main_arena->system_mem
$2 = 0xc17000

Therefore, the sbrk_base does not really applied for other heap arenas (see http://siddhesh.in/journal/2012/10/24/malloc-per-thread-arenas-in-glibc/ for more context)

Sunday, June 9, 2013

Exploring how your memory gets used in Python...

A PyCon 2011 talk called "Dude, Where's My Ram?" by David Malcolm presented gdb-heap, a debugging tool that takes advantage of GDB v7's new Python API.  The nice part of this tool is that it allows you to inspect your heap core dumps and Python processes without having to add additional instrumentation beforehand.

The code released was originally designed and implemented for Fedora 13 and Fedora 14 back in 2011, but I've managed to figure out how to get it working on Ubuntu v12.04.   The GitHub repo is located here: https://github.com/rogerhu/gdb-heap

The code in its current form did not appear to have updates to support glibc v2.15's multiple allocation arenas (see http://stackoverflow.com/questions/10706466/how-does-malloc-work-in-a-multithreaded-environment), so I've started to try to add more support in this GitHub repo.  The libheap project is a similar implementation but incorporates multiple arenas, but doesn't carry some of the special logic heuristics and query parser for searching for certain heap sizes.


Thursday, June 6, 2013

Why your Python program can't start when using python-dbg...

Recently, I installed python-dbg and started noticing errors when trying to load C-extension modules such as the PyCrypto library. There are errors such as "undefined symbol" called by Py_InitModule4_64:
>>> import Crypto.Cipher.AES
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/dist-packages/Crypto/Cipher/AES.py", line 50, in 
    from Crypto.Cipher import _AES
ImportError: /usr/local/lib/python2.7/dist-packages/Crypto/Cipher/_AES.so: undefined symbol: Py_InitModule4_64
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 1, in 
    from apport.report import Report
  File "/usr/lib/python2.7/dist-packages/apport/report.py", line 20, in 
    import apport.fileutils
  File "/usr/lib/python2.7/dist-packages/apport/fileutils.py", line 22, in 
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python2.7/dist-packages/apport/packaging_impl.py", line 20, in 
    import apt
  File "/usr/lib/python2.7/dist-packages/apt/__init__.py", line 21, in 
    import apt_pkg
ImportError: /usr/lib/python2.7/dist-packages/apt_pkg.so: undefined symbol: Py_InitModule4_64

Original exception was:
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/dist-packages/Crypto/Cipher/AES.py", line 50, in 
    from Crypto.Cipher import _AES
ImportError: /usr/local/lib/python2.7/dist-packages/Crypto/Cipher/_AES.so: undefined symbol: Py_InitModule4_64

What's going on? Apparently the python2.7-dbg binary is configured with the --with-pydebug flag, which turns on the Py_DEBUG ifdef directive, which turns on the Py_TRACE_REFS directive:

From the /usr/share/doc/python2.7-dbg/README.debug file:
python2.7-dbg contains two sets of packages:

 - debugging symbols for the standard python2.7 build. When this package
   is installed, gdb will automatically load up the debugging symbols
   from it when debugging python2.7 or one of the included extension
   modules.

 - a separate python2.7-dbg binary, configured --with-pydebug, enabling the
   additional debugging code to help debug memory management problems.
.
.
   Py_DEBUG implies LLTRACE, Py_REF_DEBUG,
   Py_TRACE_REFS, and PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled).
   In addition, C assert()s are enabled (via the C way: by not defining
   NDEBUG), and some routines do additional sanity checks inside
   "#ifdef Py_DEBUG" blocks.



We can find this definition declared inside the Python source code under Include/modsupport.h file. Normally under 64-bit platforms,
the Py_InitModule4 is renamed to Py_InitModule4_64 as defined in PEP0353.

#ifdef Py_TRACE_REFS
 /* When we are tracing reference counts, rename Py_InitModule4 so
    modules compiled with incompatible settings will generate a
    link-time error. */
 #if SIZEOF_SIZE_T != SIZEOF_INT
 #undef Py_InitModule4
 #define Py_InitModule4 Py_InitModule4TraceRefs_64
 #else
 #define Py_InitModule4 Py_InitModule4TraceRefs
 #endif
#endif


Once the python2.7-dbg package is installed, you can use gdb with the Python debugging symbols but it apparently doesn't provide nearly the rich annotated information supplied with the Py_TRACE_REF flag. The problem though is the python-dbg binary however is incompatible with modules not compiled with them as mentioned in /usr/share/doc/python2.7/SpecialBuilds.txt.gz:
Py_TRACE_REFS
-------------

Turn on heavy reference debugging.  This is major surgery.  Every PyObject grows
two more pointers, to maintain a doubly-linked list of all live heap-allocated
objects.  Most built-in type objects are not in this list, as they're statically
allocated.  Starting in Python 2.3, if COUNT_ALLOCS (see below) is also defined,
a static type object T does appear in this list if at least one object of type T
has been created.

Note that because the fundamental PyObject layout changes, Python modules
compiled with Py_TRACE_REFS are incompatible with modules compiled without it.

Py_TRACE_REFS implies Py_REF_DEBUG.

The solution seems to be to use gdb with the standard debugging symbols and the default Python binary (which appear to have compiler optimizations that prevent gdb from reading Python frame information), or recompiling all your C-extension modules with the --with-pydebug flag.  Ubuntu comes with -dbg packages for many Python libraries, but if you have ones normally installed via pip installs, these may need to be recompiled manually (i.e. via python-dbg setup.py --debug).

Wednesday, June 5, 2013

PyLint E1121 errors

If you've upgraded to Pylint v0.28.0 recently, you may have found that Pylint is starting to report this issue:
import hashlib
hmm = hashlib.sha1('tst')  
hmm.digest() 

E1121: 3,0: Too many positional arguments for function call

Strangely enough, the code below does not report this issue.
import hashlib
hmm = hashlib.sha1('tst')  
hmm.hexdigest() 

Because the hashlib library is implemented as a C-level module (see http://www.logilab.org/78354 for more context), Pylint needs to have these string declarations in place to do proper static code analysis. It turns out that recent changes to Pylint added the digest() method to the hashlib module but forgot to include the self parameter.

Within the Pylint type checker (in /usr/local/lib/python2.7/dist-packages/pylint/checkers/typecheck.py, line 268), the function is inferred to require an additional 'self' argument. However, because the string declaration does not, this error message gets reported.
if isinstance(called, astng.BoundMethod):
    # Bound methods have an extra implicit 'self' argument.
    num_positional_args += 1
The proper fix is included at this link: https://bitbucket.org/rogerjhu/astng/commits/fd99960bc86a26503cb0fc2eb5f7f484c4861ccd


Saturday, June 1, 2013

Debugging Python programs in GDB

The instructions at http://wiki.python.org/moin/DebuggingWithGdb make it seem complicated, but it turns out to be very simple in Ubuntu 12.04 to use the GDB to debug Python code. The downside is that you have to run the Python program themselves with an interpreter compiled with the symbols in order to take advantage of this functionality.

If you're using Ubuntu 12.04, you can use the following apt-get install:
sudo apt-get install python-dbg

Suppose we had a Python program called debug_me.py:

python-dbg debug_me.py &

To attach to the process, you would do:

gdb python-dbg [Python PID]

The commands that you could use are (help py-)

(gdb) py-list
   2    
   3    
   4    def debug_me():
   5        for i in xrange(10000):
   6            print i
  >7            time.sleep(5)
   8    
   9    
  10    debug_me()

(gdb) py-up
#8 Frame 0x2195810, for file /tmp/debug_me.py, line 9, in <module> ()
    debug_me()
</module>

(gdb) py-print i
local 'i' = 0

(gdb) py-bt
#5 Frame 0x2852f20, for file /tmp/debug_me.py, line 7, in debug_me (i=2)
    time.sleep(5)
(gdb) py-print i
local 'i' = 2
(gdb) py-locals 
i = 2

You can also use standard GDB commands but any C extensions must also be compiled with the debugging symbols too! For more info, check out the file installed in /usr/share/doc/python2.7-dbg/README.debug (or alternativly at https://wiki.ubuntu.com/PyDbgBuilds)

Tuesday, May 28, 2013

Ambiguities in the wait_timeout parameter in MySQL

The wait_timeout parameter in MySQL defines the duration of inactive connections to remain open. However, when doing SHOW VARIABLES at a MySQL shell, you may be fooled into thinking that the setting is set to 28800 for all new connections:

mysql> show variables like '%wait_timeout%';
+--------------------------+----------+
| Variable_name            | Value    |
+--------------------------+----------+
| wait_timeout             | 28800    |
+--------------------------+----------+
You have to keep in mind though that the setting is derived from either interactive_timeout or wait_timeout, depending on how you initiated the connection from MySQL. If you're using the MySQL client, chances are it's using the interactive_timeout.  This article helps provide some good explanation of the behavior: http://blog.mozilla.org/it/2012/04/24/when-is-qwait_timeout-not-wait_timeout/

Therefore, it's important to use the GLOBAL modifier to determine what the default settings are currently set: http://dev.mysql.com/doc/refman/5.1/en/show-variables.html
With the GLOBAL modifier, SHOW VARIABLES displays the values that are used for new connections to MySQL. If a variable has no global value, the session value is displayed. With SESSION, SHOW VARIABLES displays the values that are in effect forthe current connection. If no modifier is present, the default is SESSION. LOCAL is a synonym for SESSION.

Tuesday, May 7, 2013

Continued Reverse Engineering of the SDE-5001

After posting my previous writeup, I was sort of puzzled by the intricate nature of the authentication protocol for the Samsung SDE-5001.  There are third-party apps such as IP Cam Viewer that appear to login and authenticate correctly (assuming we designate the device as a Night Owl RTSP camera).  Were they simply implementing this proprietary Samsung protocol?

To better understand this issue, I decided to find a way to capture the network traces from a smartphone or a tablet.  To do so, I put my laptop's wireless card into hotspot mode and plugged the laptop into a wired Ethernet connection.  It turns out that Android (including Jelly Bean) devices can't connect to wireless networks placed in this mode, but at least Apple devices can do so.  I use an IPad to establish a WiFi connection to the laptop, allowing me to capture the network traces from the wireless Ethernet card.

By examining the Wireshark traces, it turns out that even though you specify RTSP port 4520, the IP Camera apps connects to port 4524 (+4 offset), which appears to accept authentication requests.

Capturing the network trace you see the following:
OPTIONS rtsp://myipcamera.example.com:4524/1 RTSP/1.0
CSeq: 1
User-Agent: IPCamViewer

When this request is given, the Samsung DVR responds with a 401 Unauthorized, a realm=, and a nonce= value:

RTSP/1.0 401 Unauthorized
CSeq: 1
Date: Wed May  8 06:39:55 2013 GMT
WWW-Authenticate: Digest realm="NET-i", nonce="0000000000000000000000002E923766"
The next sequence includes the Authorization: Digest header.  The realm value and nonce are replayed, and a response= hash is calculated.
OPTIONS rtsp://myipcamera.example.com:4524/1 RTSP/1.0
CSeq: 2
User-Agent: IPCamViewer
Authorization: Digest username="johnsmith", realm="NET-i", nonce="0000000000000000000000007B719AE4", uri="rtsp://myipcamera.example.com:4524/1", response="758b43dd292609c5b6e80084aa688ac5"

How is the response= value calculated?  Aparently it appears to follow the RFC2069 spec:


The documentation at the bottom of http://tools.ietf.org/html/rfc2617 shows how this digest can be calculated in the C language.  The Python equivalent is shown below.  Note that the OPTIONS: is used  since the previous network packet uses it instead of the more typical GET/POST verb requests.
import hashlib
username = "johnsmith"
realm="NET-i"
password="123456"
uri="rtsp://myipcamera.example.com:4524/1"
nonce="0000000000000000000000002E923766"

ha1 =  hashlib.md5("%s:%s:%s" % (username, realm, password)).hexdigest()
ha2 = hashlib.md5("OPTIONS:%s" % uri).hexdigest()

response = hashlib.md5("%s:%s:%s" % (ha1, nonce, ha2)).hexdigest()
print response
I was able to confirm the hash value matched the one the Samsung device transmitted with this Python code. Therefore, it seems as if the Samsung device has both a proprietary way in its ActiveX control to authenticate and a more standards based to do so.

Monday, May 6, 2013

Reverse Engineering the Samsung SDE-5001's authentication protocol...

One of the major limitations in many of Samsung's IP cameras including the SDE-5001, is that they depend on Internet Explorer and ActiveX plugins to view video streams over the web. You can use the dedicated mobile apps from the
respective iPhone and Android app stores, but what if you wanted to use other browsers or clients? Since the documentation mentions the use of the Real Time Streaming Protocol (RTSP), couldn't any compatible client work?

This question prompted me to verify whether the camera could actually followed the RTP/RTSP specification (http://www.ietf.org/rfc/rfc2326.txt).  According to the RFC documents, RTSP normally uses a similar type of authentication flow as Basic HTTP authentication requests. An Authorization: line is added with the username and password base64-encoded in the request header.

Sifting through documents of different Samsung DVR models publicly available, I tried many different variations of the RTSP protocol to try to elicit a response from the Samsung device:

telnet 192.168.0.25 4520

DESCRIBE rtsp://192.168.0.25/profile3/media.smp RTSP/1.0
Authorization: Basic [base64-encoded username]:[base64-decoded pw])
CSeq:1

OPTIONS rtsp://192.168.0.25/profile3/media.smp RTSP/1.0
CSeq: 1

DESCRIBE rtsp://10.240.56.200/mpeg4/media.smp RTSP/1.0
CSeq: 1
Accept: application/sdp
Bandwidth: 384000
Accept-Language: de-DE
User-Agent: QuickTime/7.4.5 (qtver=7.4.5;os=Windows NT 5.1Service Pack 3)

None of these commands would cause the server to reply. This prompted me to check to see if the Samsung device was also using a proprietary protocol to authenticate. Using Wireshark to capture the network traces while attempting to
login successfully to the DVR through Internet Explorer, I found one packet of interest:



The initial network frames exchanged in particular interested me because you can see the plaintext 'AUTHENTICATE' in the initial data payload.  The packet also contains the ID: and PW: (not shown) and then traces of AUTHENTICATE_OK in the subsequent packet responses.  This protocol is consistent with an online Samsung document called
"DVR Protocol Specification for NET-i (and WebViewer)" -- there are Part I and Part II specs. Part I deals with the authentication aspect.  The packet structure is shown below:


The response packet according to the documentation matches the payload being sent for the SDE-5001 devices.  I could see that S & V were the first two characters and AUTHENTICATE being in the middle of the packet. To help confirm that this packet would elicit a reply, I saved the data payload from Wireshark to a file called "authenticate_packet" and wrote this small Python script to see whether I could replay the authentication sequence:

import telnetlib

tn = telnetlib.Telnet("192.168.0.25", 4521)
data = open('authenticate_packet', 'r').read()
tn.write(data)
sess_op = tn.read_some()
print sess_op

Instead of no response from the Samsung device, the response from the Samsung device
all of a sudden yielded:

00000000 53 56 36 41 55 54 48 45 4e 54 49 43 41 54 45 5f |SV6AUTHENTICATE_|
00000010 4f 4b 01 54 45 58 54 53 55 43 43 45 53 0a |OK.TEXTSUCCES.|
0000001e

This experiment helped confirm that the Samsung SDE-5001 follows the same protocol as the Samsung DVR spec. However, the documentation also seemed to suggest that the username and password (ID/PASSWORD) were sent as cleartext.  Decoding the authentication packet, however, yielded a bunch of encoded text.  For instance, for the username "joesmith" and password "123456", the corresponding data was "GtxTGa3KAcI=" and "7c4a8d09ca3762af61e59520943dc26494f8941b".

Understanding how the ID and PASSWORD strings required some extra reverse engineering with the Samsung web interface and the ActiveX modules that get dynamically loaded in the browser. One thing to notice in the JavaScript
of the Samsung web interface that it embeds an <IFRAME> tag and attempts to log the user in with two functions. The data_parser() JavaScript code appears to encode data into a base64, while the second function hex_func_five() appears
to be the MD5 hash of the password.

document.login_page_submit.data1.value = data_parser(document.login_page.data1.value);
document.login_page_submit.data2.value = hex_func_five(document.login_page.data2.value);

The data_parser() function appears to base64-encode the username, but the string doesn't match
the username string "GtxTGa3KAcI=" sent in the Wireshark network packet.   Therefore, it appeared that the Samsung RTSP port was using some additional type of encoding that could only be studied by examining the ActiveX code.

When watching a successful authentication to occur, I noticed that Samsung's web
server renders a page that injects <object> tag into the DOM. This <object> tag is the ActiveX module.

The name of the file is webviewer.cab:
<script language="javascript">
writeObjCtrl(720, 576, 'obj', 'GTileContainerCtlDetroit', 'CLSID:26E1BEAF-C1A1-482b-8714-08844F1BCF7F', '../webviewer.cab#version=1,0,0,276');
</script>

This CAB file can actually be downloaded from the system and extracted using the cabextract utility in Ubuntu:

wget http://192.0.0.25/webviewer.cab

$ cabextract webviewer.cab
webviewer.cab: WARNING; possible 7208 extra bytes at end of file.
Extracting cabinet: webviewer.cab
  extracting H264GTileContainer.ocx
  extracting H264GCalendarCtrl.ocx
  extracting H264GTimeLineCtrl.ocx
  extracting H264GSetupBrowser.ocx
  extracting H264GMediaPlayer.ocx
  extracting H264GMediaSource.ax
  extracting H264GVideoDecoder.ax
  extracting H264GAudioDecoder.ax
  extracting H264GAudioDecoderADPCM.ax
  extracting H264GVideoRenderer.ax
  extracting XNS_h264.dll
  extracting H264MediaMan.dll
  extracting XRegistry.ocx
  extracting device_shr_h264.dll
  extracting h264decoder.dll
  extracting lib_VoiceEngine_dll.dll
  extracting webviewer.inf

The files that were extracted include OCX and DLL files, which are all Portable Executable (PE) files that can be loaded and run in Microsoft Windows. The ActiveX modules that are loaded by Internet Explorer to provide the GUI and controls for the Samsung SmartViewer software. The additional H.264-related modules appear to perform the audio/video decoding too.

Which of the modules is responsible for handling the initial authentication? Doing a search of the string exports with PEBrowser in the modules or simply a grep for the string for "AUTHENTICATE_OK" reveals that this code is actually contained in the file, device_shr_h264.dll.

grep AUTHENTICATE_OK *
Binary file device_shr_h264.dll matches

The interactive debuggers Olly Dbg or IDA Pro (32-bit mode) can be used to try to understand what's happening. We can attach a process to Internet Explorer, login to the Samsung interface with a known username and password, and view the device_shr_h264.dll loaded in memory (listed as device~1.dll). We view the exported strings in the file and trace where the strings are used inside the code:


We can search for "AUTHENTICATE" and find the matching strings:



We can trace into this function and notice that the sprintf() command for the username and password is being generated:


A breakpoint can be added further upstream until you notice that the username and password are being passed into registers.  OllyDbg will show these values in the registers window, so it's easy enough to see when they are referenced.


The exact location in which the user and password can reloading the web page and setting breakpoints until the appropriate sections can be isolated.  I noticed that the username started to show up as a register location where the use of the ASCII string "VSS_SVP2.0" gets referenced.


In the following section of the assembly code, the byte referencing the address location DS:[ESI+10027BDC] is moved into the EDI register, XOR'd with the previous character (the EDL register is cleared to 0x00 earlier), and repeated until the end of the string is reached.  This latter operation is done by using the SCAS opcode is used to increment the ESI register until what is stored in the AL register, 0x00, which is essentially implementing the strlen() function).


The equivalent code for the above assembly code in Python would essentially be:
constant = "VSS_SVP2.0"

init_val = 0x00

for c in constant:
    init_val ^= ord(c)

There is additional code that attempts to XOR this value with other bytes referenced by 0x69A7BE8, but inspecting the memory contents using OllyDbg's Follow In Dump -> Immediate constant reveals that all the data is 0x00, leaving the result of the previous code to be unchanged.  This section therefore seems to be a placeholder for more XOR mixing, but seems to be unused.

In the next section of the code, we observe that a memory address location 0x069A9880 is referenced.     We notice that this address location is stored in the EDI register and the byte is XOR'd with the username. Essentially we have found the key used to perform the rest of the XOR encryption.

The key at address 0x69A9880 appears to start with the data 0x52, 0x50, 0x70, etc..   The maximum amount of space allocated to the username and password appears to be 32-bytes each, so the XOR operation seems to be limited to a 32-byte operation.

The above section of the assembly code is equivalent to the following Python code. Since many of the assembly instructions are performing byte-level operations, we often need to take the results of the AND/XOR operation and mask
the bits with the 0xFF byte. Assuming the secret key is 64-bytes, the XOR algorithm is therefore listed as follows:

def samsung_xor_encode(username, init_val):
  secret_key = [0x52, 0x50, 0x7, 0x15, 0x23, 0x33, 0x41, 0x49, 0x50, 0x53, 0x61, 0x00, 0x08, 0x16, 0x24, 0x26,
       0x34, 0x42, 0x10, 0x28, 0x36, 0x44, 0x63, 0x56, 0x03, 0x11, 0x19, 0x29, 0x37, 0x45, 0x33, 0x56,
       0x57, 0x04, 0x12, 0x20, 0x30, 0x38, 0x46, 0x58, 0x05, 0x13, 0x21, 0x31, 0x39, 0x47, 0x62, 0x06,
       0x14, 0x22, 0x32, 0x40, 0x48, 0x08, 0x37, 0x18, 0x34, 0x26, 0x12, 0x38, 0x26, 0x58, 0x45, 0x08]

  val = ""
  for n, u in enumerate(username):
     result = (init_val ^ ord(u))
     val += chr(result)
     cl = (~init_val & 0xFF)
     cl = cl >> 7
     init_val = init_val << 1
     cl = (cl + init_val) & 0xFF
     init_val = cl ^ secret_key[n]

  return val

We can then use this function to implement the XOR encryption:

init_val = 0x00

for c in constant:
  init_val ^= ord(c)

encoded = samsung_xor_encode("johnsmith", init_val)
decoded = samsung_xor_encode(encoded, init_val)

print "Encoded username: %s" % (encoded.encode('base64'))
print "Decoded username: %s" % (decoded)

The  XOR algorithm can be verified by using known plaintext usernames and observing the authentication packets between the ActiveX module and the Samsung DVR.   Since the XOR algorithm is used for both decryption and encryption, we can verify both by encoding the plaintext username as well as decoding the encrypted username. Fortunately, the password doesn't require this extensive reverse-engineering, since it only computes a standard MD5 hash against the password for authentication.

One important point that should be noted is that this information in this article details an encryption algorithm only used for authentication and not used for any type of copy protection.   The intention of this writeup is to document my understanding of the authentication protocols for these DVR systems and to enable more browser-friendly plug-ins beyond Internet Explorer to work with these systems, especially since Samsung uses custom but publicly disclosed RTP/RTSP protocols to request video/audio streams once the authentication sequence is completed.  If you're interested in doing so, please let me know!

GitHub's web hooks

Recently GitHub has made changes to their web hooks...

1. Your Webhooks need to be URL-encoded now.  In the past, you could get away with it and GitHub would handle the url-encoding themselves.   No longer.

2. All webhooks are now by default monitoring/tracking only push events.  If you have other events, you need to use the GitHub hooks API.

The best way to make these changes?  Use Chrome's Dev HTTP client: https://chrome.google.com/webstore/detail/dev-http-client/aejoelaoggembcahagimdiliamlcdmfm?hl=en

a. Add a Header with Authorization: token <token>where <token> represents your OAuth token for GitHub.

b. Change to use https://

c. Set to api.github.com/repos/<your GitHub owner>/<repo name>/hooks

d. Grab the webhook ID in the JSON response.

c. Set the JSON payload to be add_events and push like the following:

Friday, April 19, 2013

RequireJS

The RequireJS documentation appears to suggest that you can define configurations such as the following:

requirejs.config({baseUrl: 'http://app.example.com',
                  paths: { underscore : 'static/js/external/underscore.js',
                           backbone : 'static/js/external/backbone.js'})

If you try to use this approach, you may find RequireJS appending an extra .js file.

RequireJS also assumes by default that all dependencies are scripts, so it does not expect to see a trailing ".js" suffix on module IDs. RequireJS will automatically add it when translating the module ID to a path. With the paths config, you can set up locations of a group of scripts. All of these capabilities allow you to use smaller strings for scripts as compared to traditional <script> tags.

There may be times when you do want to reference a script directly and not conform to the "baseUrl + paths" rules for finding it. If a module ID has one of the following characteristics  the ID will not be passed through the "baseUrl + paths" configuration, and just be treated like a regular URL that is relative to the document:
  • Ends in ".js".
  • Starts with a "/".
  • Contains an URL protocol, like "http:" or "https:".
Within the RequireJS code, there is a test for the question mark (?):
url += (ext || (/\?/.test(url) ? '' : '.js'));

...to avoid RequireJS add an extra .js to the end, you can do the following to avoid the addtion.JS:

requirejs.config({baseUrl: 'http://app.example.com',
                  paths: { underscore : 'static/js/external/underscore.js?r=1',
                           backbone : 'static/js/external/backbone.js?r=1'})


This pull-request still seems to fix this issue:

https://github.com/jrburke/requirejs/pull/716

Tuesday, April 9, 2013

mailto: links and plus encoding

Python has urllib.quote() and urllib.quote_plus().  Which one should use for URL-encoding mailto: links?  Answer: use urlib.quote(), since spaces will normally be converted to urllib.quote_plus() and cause confusion in whether it actually is part of a person's email address.

http://tools.ietf.org/html/rfc6068#page-8

When creating 'mailto' URIs, any reserved characters that are used in
the URIs MUST be encoded so that properly written URI interpreters
can read them. Also, client software that reads URIs MUST decode
strings before creating the mail message so that the mail message
appears in a form that the recipient software will understand. These
strings SHOULD be decoded before showing the message to the sending
user.

Software creating 'mailto' URIs likewise has to be careful to encode
any reserved characters that are used. HTML forms are one kind of
software that creates 'mailto' URIs. Current implementations encode
a space as '+', but this creates problems because such a '+' standing
for a space cannot be distinguished from a real '+' in a 'mailto'
URI. When producing 'mailto' URIs, all spaces SHOULD be encoded as
%20, and '+' characters MAY be encoded as %2B. Please note that '+'
characters are frequently used as part of an email address to
indicate a subaddress, as for example in .

The 'mailto' URI scheme is limited in that it does not provide for
substitution of variables. Thus, it is impossible to create a
'mailto' URI that includes a user's email address in the message
body. This limitation also prevents 'mailto' URIs that are signed
with public keys and other such variable information.

Monday, April 8, 2013

Using Jekyll and iFrame's

It turns out the use of your Markdown converter affects whether you can embed YouTube clips (i.e. iFrame's).  For this reason, I switched to using rdiscount instead of the default one included in Jekyll:

https://github.com/mojombo/jekyll/issues/232

More context here:

https://github.com/mojombo/jekyll/wiki/install


MLB.tv's zip code detector tool

It's baseball season....want to know what your IP adresss maps in terms of approximate zip code location?

http://www.mlb.com/mlb/subscriptions/iplocation.jsp

Wednesday, April 3, 2013

Verizon's MiFi 4620L firmware upgrade 3.20.16

The firmware 3.20.16 is now available for the Verizon MiFi 4620L device Verizon has a document http://support.verizonwireless.com/pdf/system_update/mifi4620l.pdf that describes the improvements. It includes a new data meter along with some fixes associated with dormancy mode...let's hope that it solves the issue where the device has sporadic issues simply powering off.

You can upgrade the firmware by logging into the web browser and checking for firmware updates.

MiFi4620_3.20.11_3.20.16_Real
Size: 653.51 KB
Estimated Install Time 1 minute 40 seconds
Description: MiFi4620L Upgrade from 3.20.11 to 3.20.16

Some previous documented shortcomings are here:

https://community.verizonwireless.com/message/534516/

Tuesday, April 2, 2013

Using Django proxy = True..

If you're using Django's proxy=True and see this error message:

  File "/usr/local/lib/python2.7/dist-packages/django/db/models/base.py", line 89, in __new__
    new_class._base_manager = new_class._base_manager._copy_to_model(new_class)
AttributeError: 'NoneType' object has no attribute '_copy_to_model'

... chances are you're assigning proxy = True to the base class instead of the inherited classes.

Sunday, March 31, 2013

Turbo Tax and SEP IRA

For some odd reason, I noticed that Turbo Tax 2012 doesn't have any step-by-step wizards for handling SEP IRA contributions. You can search for the tax form and can find it yourself within the software tool. The main issue that isn't covered is how to calculate how much you're allowed to deduct with self-employed IRA's. Normally you are allowed to deduct up to 20% of self-employment income, but you have to use the IRS' Publication 560 to subtract the employment tax.  You can scroll down the entire contributation rate table and max out at the 25%, which using the reduced rate table, appears to push you down to 20%.

If you factor this amount, divide the amount you're to contribute the SEP IRA by your total self-employment income, you should come up with the 18.587045% mentioned in the Wiki article.

Note: this isn't tax advice but does seem to be a weird oddity in the tax code that can only be understood by correlating multiple sources to understand what's going on.

Friday, March 29, 2013

More Python 2.7 Unicode strangeness

Apparently in Python 2.7 you can't mix Unicode strings with UTF-8 encoded:

>>> x = "sdds"
>>> y = u'\u2013t'.encode('utf-8')
>>> y
'\xe2\x80\x93t'
>>> x = u"sdds"
>>> x + y
Traceback (most recent call last):
  File "", line 1, in 
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

But you can do:

>>> x.encode('utf-8') + y
'sdds\xe2\x80\x93t'

Wednesday, March 27, 2013

What 536871023 means..

Using the Celery framework and the librabbitmq package, we start seeing these errors when we specified an invalid host:

Type: 
Value: Error opening socket: Unknown error 536871023

What does the 536871023 means? To understand what the code means, you can fork the librabbitmq code base and look inside the AMQP-related code.

git clone git://github.com/celery/librabbitmq.git

Within the amqp_open_socket(), the error code is returned:

int amqp_open_socket(char const *hostname,                                                                
             int portnumber)   {

.
.
.

    return -amqp_socket_error();   

}

Where amqp_socket_error() is equal to the errno code masked with a constant:
int                                                                                                     
amqp_socket_error(void)                                                                                 
{                                                                                                      
    return errno | ERROR_CATEGORY_OS;                                                                  
}     

This constant is defined in rabbitmq-c/librabbitmq/amqp_private.h:
#define ERROR_CATEGORY_OS (1 << 29) /* OS-specific error codes */  

>>> int(x)
536871023
>>> hex(int(x))
'0x2000006f'

...so we ignore the higher bit and convert the 0x6f..

>>> int(0x6f)
111
>>>

The 111 amounts to a Connection refused error message, which indicates an issue connecting to the RabbitMQ host.

Saturday, March 23, 2013

Multi-threading in Python

What do you think you will get the from the above result?

import threading

class A(threading.local):
    def __init__(self, *args, **kwargs):
        self.val = 1

bla = A()


class Example(threading.Thread):

    def __init__(self, thread_id, value):
        self.thread_id = value
        self.value = value
        super(Example, self).__init__()

    def run(self):
        bla.val = self.value
        print "Thread: %s, id=%s" % (self.thread_id, bla.val)


example1 = Example(1, "A")
example2 = Example(2, "B")

example1.start()
example2.start()

example1.join()
example2.join()

print "Final value %s" % bla.val


The output is:

Thread: A, id=A
Thread: B, id=B
Final value 1

Thread-local storage (TLS) is actually implemented at the hardware level. This Stack Overflow article provides a link to how TLS is implemented on Linux.
http://people.redhat.com/drepper/tls.pdf

This link provides a good general explanation of the Python thread locals library.

Thursday, March 7, 2013

RSA-OAEP and OpenSSL

The XML Digital Signature specs detail the use of RSA-OAEP padding. The best explanation is available at Wikipedia with the following diagram on the right. Sections 9.1.1 of the PKCS 2.0 standard also explain how this padding algorithm works. http://www.w3.org/TR/xmlenc-core/#sec-RSA-OAEP
5.4.2 RSA-OAEP

Identifier:
http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p (REQUIRED)
The RSAES-OAEP-ENCRYPT algorithm, as specified in RFC 2437 [PKCS1], takes three parameters. The two user specified parameters are a MANDATORY message digest function and an OPTIONAL encoding octet string OAEPparams. The message digest function is indicated by the Algorithm attribute of a child ds:DigestMethod element and the mask generation function, the third parameter, is always MGF1 with SHA1 (mgf1SHA1Identifier). Both the message digest and mask generation functions are used in the EME-OAEP-ENCODE operation as part of RSAES-OAEP-ENCRYPT. The encoding octet string is the base64 decoding of the content of an optional OAEPparams child element . If no OAEPparams child is provided, a null string is used.

Schema Definition:
     
     
     An example of an RSA-OAEP element is:

  
     9lWu3Q== 
    
  
If you want to use OpenSSL to decode RSA-OAEP padded data, however, the OAEPparams option cannot be used. Normally, when a message is initially prepared, it is hashed with an input parameter, which is step 3 in Section 9.1.1 RFC 2437 under "Generate an octet string PS consisting of emLen-||M||-2hLen-1 zero octets. The length of PS may be 0.".

But according to http://www.openssl.org/docs/crypto/RSA_public_encrypt.html, EME-OAEP as defined in PKCS #1 v2.0 with SHA-1, MGF1 and an empty encoding parameter. (EME standards for encoding methods for encryption). Therefore, the octect string PS must be empty.

Friday, March 1, 2013

Why Python, Ruby, and JavaScript are slow

Jinja2, which can be used in lieu of Django's templating engine, recommends in its documentation to use the markupsafe library:

More Speed with MarkupSafe

As of version 2.5.1 Jinja2 will check for an installed MarkupSafe module. If it can find it, it will use the Markup class of that module instead of the one that comes with Jinja2. MarkupSafe replaces the older speedups module that came with Jinja2 and has the advantage that is has a better setup script and will automatically attempt to install the C version and nicely fall back to a pure Python implementation if that is not possible.

The C implementation of MarkupSafe is much faster and recommended when using Jinja2 with autoescaping.
>>> import markupsafe
>>> markupsafe

 
If we don't use the markupsafe package, Jinja has to rely on a Python implementation, which basically means for each character it encounters, it needs to expand the buffer by the extra # of characters, move the data, and repeat the process all over again. The C library extension seems to try to be smarter by using BYOB (bring your own buffer) by expanding the buffer to the total size and block copying the data when escaping is not needed.

This deck helps to sum up why the C implementation is much faster: https://speakerdeck.com/alex/why-python-ruby-and-javascript-are-slow

Tuesday, February 19, 2013

IP Cam Viewer App with the Samsung SDE-5001

The Samsung iPolis mobile app for the Samsung SDE-5001 home security system doesn't work with Android Jelly Bean 4.2.2 (it works fine for Ice Cream Sandwich).   However, you can also use Robert Chou's IP Cam Viewer app for Android and IOS, which provides a Lite version for 6 camera angels and a Pro version for unlimited.  

Here are things to note when using the IP Cam Viewer app with the SDE-5001:

1) Samsung's SDE-5001 system comes with a free www.samsungipolis.com Dynamic DNS service. However, unlike most Dynamic DNS services, it relies on a URL path associated with each dynamic host (i.e. http://www.samsungipolis.com/HOMEDVR) instead of a dedicated subdomain (it appears to use HTTP meta tags to perform the same functionality as traditional Dynamic DNS).  As a result, the IP Cam Viewer currently does not support Samsung's Dynamic DNS approach.

If you try to configure http://www.samsungipolis.com/123 with port 4524, the current app seems to only pick up the samsungipolis.com URL and chooses the original 554 port.    (We're hoping the author will be able to update the IP Cam Viewer app to support Samsung's Dynamic DNS service.  The service relies on HTTP <meta> refresh tags instead of HTTP headers, so there is a bit more work involved.)

You can signup for no-ip.com but there apparently is a requirement to login every 30 days to confirm your account.    You will need to reconfigure the Samsung SDE-5001 to use no-ip.com instead of the IPolis DDNS service.

2) If you tried to select the Samsung SDE DVR option with the IP Cam Viewer app, you may fail to get a camera image. But if you use the Scan functionality with the camera, you may find that many other cameras work with the SDE-5001 including the Night Owl and Night Owl RTSP(TCP) DVR service.  Use the Night Owl Make and Night Owl RTSP(TCP) DVR with RTSP port 4524 and you should be able to connect to the system.

Tuesday, February 12, 2013

Passing variables to awk

Note: the ENVIRON variables only work in bash for those that have been exported:

$ export ABCD=123; 
$ echo | awk '{print ENVIRON["ABCD"]}'
123

http://rosettacode.org/wiki/Environment_variables

The ENVIRON array contains the values of the current environment:
$ awk 'BEGIN{print "HOME:"ENVIRON["HOME"],"USER:"ENVIRON["USER"]}'
HOME:/home/suchrich USER:SuchRich
Environment variables can also be assigned to awk variables before execution, with (-v) options:
$ awk -v h=$HOME -v u=$USER 'BEGIN{prin

Sunday, February 10, 2013

Setting up the Samsung SDE-5001 DVR for online viewing

With all the recent attention of Samsung catching up as a viable competitor to Apple, the trouble I had today getting their SDE-5001 DVR working for online access really made me wonder whether they had invested much time on any part of building an intuitive design for using this device.   It appears that much of their design team has been focused on the mobile phone business, but their expertise hasn't spilled into other areas of their product lines such as their home security systems.

The SDE-5001 device comes with Dynamic DNS support through their http://www.samsungipolis.com service.  When you register your device at this site, you also create a Product ID.  The URL will then become part of the URL (i.e. http://www.samsungipolis.com/abcdef).    Assuming your device is configured for the Dynamic DNS service, this web service will redirect your browser to the IP address with META HTTP refresh tags(<meta http-equiv="refresh" content="0;url=http://1.2.3.4:80"/>).

One trouble that I encountered was that the stock firmware 1.0.0 when using the iPolis DynDNS had a User ID and password prompt, but you couldn't fill out the password prompt through a web browser much less through the infrared control.   I noticed today that Samsung has released the 1.0.2 firmware that probably fixed this bug (http://www.samsungsv.com/Support/DVRFirmwareUpdate), but there are very few instructions about how to go about doing so.    The first thing you need to do is to verify which firmware to use, since some of Samsung's firmware include material codes.

To upgrade the firmware, it turns out that you have to download the .zip file to a USB drive in the root directory and then insert this device into the DVR.  You then have to go into the System Management section of the DVR and wait 20-30 seconds before a screen shows up.  If the DVR recognizes there is a USB device (USB hard drives may not work), it will indicate a "Upgrade Software" button that you can use.  I tried several times and often got "Upgrade Failed", so I eventually decided to unzip the image and place it on the same drive while switching to a smaller 32GB flash drive.   Eventually after trying different USB sticks and unzipping the firmware files, I got the upgrade to succeed.

There also isn't much documentation about how to setup port forwarding.  If you navigate to the Protocol section, you will also noticed that the protocol type is set to TCP, port 4520, webviewer port 80:


You can also see at the bottom of this picture is "TCP Port 4505-4530, Webviewer Port 80". According to the SDE 5001 User manual (http://www.samsungsv.com/Download/SDE-5001_User_Manual.pdf), what this section really means is:

Multi Browser Support : You can use a Silverlight-compliant web browser to connect to the Web Viewer. Multi browsers are enabled only if the TCP port is set to between 4505 and 4530 with the Web Viewer port of 80. This is a security measure in order to enable Silverlight.

Basically it means that the Samsung DVR really only works with Internet Explorer.  You also appear to need to set TCP ports 4505-4530 open, as well as port 80 to make it work.   It appears that you can choose either TCP or UDP connection mode, so the grayed out parameters seem to apply for whichever mode.

For good measure, I setup TCP port 80/4505-4530 and UDP 8000-8160 port forwarding to support both modes.

Also, if you connect with a non-Safari and non Internet Explorer, you get a blank white screen. If you look at the JavaScript for the page, the reason is that this section of the code has specific checks whether to even render the login page if you're not using Safari or IE. You can bypass by adding the same parameters at the end (i.e. http://[YOUR IP]/cgi-bin/webviewer?ip=0&_port=0) but you still need Silverlight/ActiveX plugins and it still appears to render a blank screen.

There is a mobile app, but the Samsung IPolis appears to be broken for Android Jelly Bean 4.2: https://play.google.com/store/apps/details?id=com.samsung.ipolis&hl=en. You can connect and request channels, but no video panels will render. I looked through a bunch of forum comments and people have been complaining that the latest update broke things (https://play.google.com/store/apps/details?id=com.samsung.ipolis&hl=en).  You can use the IP Cam Viewer app for IOS and Android, which is described in this blog post.

Finally, if you're wondering about what RTSP port should be used for the IPolis mobile app, it should be the last port defined in the range (i.e. for port range 4520-4524, use 4524).  


Footnote: there is a major security vulnerability that is still unpatched in the web interface software.  For more details, see http://hustoknow.blogspot.com/2013/11/samsung-dvr-security-vulnerability.html