FileZilla Fail..

If you get a GnuTLS error -12, a TLS fatal alert has occurred, with VSFTPd, chances are that you have an SSL cipher suite negotiation problem. The problem first appears to be that FileZilla 3.5.3 (and not FileZilla 3.5.2) has changed its supported encryption schemes.

http://trac.filezilla-project.org/ticket/7873

The changes in FileZilla appear to have changed the cipher suites that allowed to be supported. The gnutls_priority_set_direct() function appears to now expect 3DES-CBS as the main cipher. The SECURE256 flag got renamed to SECURE192 (according to the ChangeLog inside GNU TLS 3.0.9) defines these types of suites:

static const int sign_priority_secure192[] = {
  GNUTLS_SIGN_RSA_SHA384,
  GNUTLS_SIGN_ECDSA_SHA384,
  GNUTLS_SIGN_RSA_SHA512,
  GNUTLS_SIGN_ECDSA_SHA512,
  0
};

A more higher security got used in the latest version of FileZilla:
http://svn.filezilla-project.org/filezilla/FileZilla3/trunk/src/engine/tlssocket.cpp?r1=4384&r2=4383&pathrev=4384

The diff got changed from using SECURE256:

gnutls_dh_set_prime_bits(m_session, 512);          
res = gnutls_priority_set_direct(m_session, "SECURE256:+CTYPE-X509:-CTYPE-OPENPGP", 0);

...to 3DESC-CBC:

res = gnutls_priority_set_direct(m_session, "NORMAL:-3DES-CBC:-MD5:-SIGN-RSA-MD5:+CTYPE-X509:-CTYPE-OPENPGP", 0);
gnutls_dh_set_prime_bits(m_session, 2048);

The workaround is to change ciphers=HIGH inside /etc/vsftpd.conf, since the default is DES-CBC3-SHA. This obviously will cause more restrictions for SSL clients and forces them to use 3DES instead of DES encryption.