OpenSSL encryption libraries add a Salted__ header.

http://openssl.6102.n7.nabble.com/Decrypting-with-key-and-initialization-vector-don-t-decrypt-the-first-8-bytes-td10420.html

When you encrypted data with a password using openssl command line, the 
first 16 bytes of the output are actually a header of the form 
'Salted__XXXXXXXX' where the last 8 bytes represent the salt used to 
derive the key and the IV. 

subprocess module

Awesome writeup/explanation of all the use cases of the Python subprocess module..

http://sharats.me/the-ever-useful-and-neat-subprocess-module.html

lxml cannot use absolute path on element

It turns out when you use lxml.fromstring(XML), it returns back the element instead of a document tree (unlike lxml.parse()).  You have to do lxml.fromstring(XML).getroottree().  This way, doing findall() will avoid triggering the "cannot use absolute path on element" error.

http://www.inductiveload.com/solutions-to-errors/

LXML

Error: SyntaxError: cannot use absolute path on element
Problem: I was trying to do:
root = doc.getroot()
tags = root.findall("//tagname")
Solution: You need to use the XML document directly to use an absolute path:
tags = doc.findall("//g")

ngrep

Another useful network debugging tool:

http://ngrep.sourceforge.net/usage.html

It works similar to grep except it scans network ports!

You can use -d to specify the Ethernet interface and -x to dump the data in hex.    It also has primities such as "port 80" to listen to only web service traffic.  The -W byline command helps to expose the line feeds.

 -W normal|byline|single|none
              Specify  an  alternate manner for displaying packets, when not in hexadecimal mode.  The ``byline'' mode honors embedded line‐feeds, wrapping text only when a linefeed is encountered (useful for observing HTTP transactions, for instance).  The ``none'' mode  doesn't  wrap under any circumstance (entire payload is displayed on one line).  The ``single'' mode is conceptually the same as ``none'', except that everything including IP and source/destination header information is all on  one  line.   ``nor‐mal'' is the default mode and is only included for completeness.  This option is incompatible with ``-x''.

Socket Stats

A bunch better tool than netstat...it can even show you memory usage for all your sockets:

http://www.cyberciti.biz/files/ss.html

What do all these memory letters represent?

Recv-Q Send-Q       
   0      0            
 mem:(r0,w0,f0,t0)

You can do man netstat to understand what Recv-Q and Send-Q mean:


 Recv-Q
     The count of bytes not copied by the user program connected to this socket.

 Send-Q
     The count of bytes not acknowledged by the remote host.

The rmem, wmem, fmem, and tmem refers to specific Linux kernel values:

http://timesinker.blogspot.com/2010/03/linux-networking-secrets.html

rmem = sk->sk_rmem_alloc; 

wmem = sk->sk_wmem_queued; 

fmem = sk->sk_forward_alloc; 

tmem = sk->sk_wmem_alloc;

http://useranswer.com/answer/kernel-socket-structure-and-tcp_diag/

sk_forward_alloc is the forward allocated memory which is the total memory currently available in the socket’s quota.

sk_wmem_queued is the amount of memory used by the socket send buffer queued in the transmit queue and are either not yet sent out or not yet acknowledged.

http://www.cse.scu.edu/~dclark/am_256_graph_theory/linux_2_6_stack/structtcpdiag__meminfo.html#m1

For more information, see Chapter 9 of TCP/IP Architecture, Design and Implementation in Linux:
http://books.google.com/books?id=sKoo-cFNPYsC&lpg=PA300&ots=We7wmJfrVj&pg=PA298#v=onepage&q&f=false

The deflate algorithm

The deflate algorithm, the compression approach used in the original Pkzip 2.0 version, is a fascinating read defined in the RFC1951 standard.  In order to understand the spec, it's helpful to work through the "hello" examples discussed in this article.

Section 3.2.2. in the RFC defines the basic approach to creating Huffman trees, but this article provides a solid explanation about how it works and shows how it's implemented.   The key insight is that Huffman codes are generated by counting the lengths of the variable-length codes (For more background/context, see Stanford lectures on algebraic error control codes and/or the textbook used in the class).

Gzip is essentially using the deflate algorithm but adds a header/trailer to the end.  RFC1952 provides a breakdown of how the header is generated, but you can also test for yourself:

$echo "hello" > hello

$gzip hello

$python

>>> data = open("hello.gz", "r").read()

>>> data[0:2]

'\x1f\x8b'

>>> data[3:4]

'\x08'

>>> [hex(ord(a)) for a in data[4:8]]

['0x9e', '0x2e', '0xfa', '0x50']

>>> 0x50fa2e9e
1358573214
>>> datetime.datetime.fromtimestamp(1358573214)
datetime.datetime(2013, 1, 18, 21, 26, 54)

>>> data[10:16]
'hello\x00'

More related articles too:

http://www.zlib.net/manual.html
http://www.adayinthelifeof.nl/2010/06/02/deflating-the-universe/
http://www.w3.org/Graphics/PNG/RFC-1951#codes
http://www.commandlinefanatic.com/cgi-bin/showarticle.cgi?article=art007

RSA block type is not 02 error

Ever seen the error message "block type is not 02" when performing RSA decryption via M2Crypto?  It's a direct result of an RSA decryption not succeding.  Before data is encrypted, it's padded with a header and a block type of 01 or 02, depending on whether the public or private key was used.  If the private key was used, the RFC spec (http://www.ietf.org/rfc/rfc2313.txt) says that the padded header should be 01 (though it also says 00 can be used).  If the public key was used to encrypt the data, the RFC header will be set the BT block to 02:

A block type BT, a padding string PS, and the data D shall be
   formatted into an octet string EB, the encryption block.

              EB = 00 || BT || PS || 00 || D .           (1)

   The block type BT shall be a single octet indicating the structure of
   the encryption block. For this version of the document it shall have
   value 00, 01, or 02. For a private- key operation, the block type
   shall be 00 or 01. For a public-key operation, it shall be 02.

During the decryption phase, the RSA encryption algorithm first converts things back to a block of data (ultimately each block is converted from an integer) and then does one additional verification process against this block type.   If the block type doesn't match 01 or 02, it's likely the wrong key was used to decrypt.   You have the wrong decryption key, may be not using the private key if the public key was used to encrypt, or the public key if the private key was used.  (In the latter two cases, just the other pair has to be used to reverse the original operation.)