Friday, January 28, 2011

Integration PyFlakes into Hudson...

There are a few web links out there that discuss how one can add PyFlakes to your Hudson integration. There's also a PyFlakes Hudson plug-in but then you'd have to re-compile the plug-in into the main Hudson code base. An alternative (as mentioned in http://www.topfstedt.de/weblog/ and http://reinout.vanrees.org/weblog/2010/09/22/hudson-technical-social.html) is that you can use the Violations Plug-in and coerce the output of the pylint into a format that the pylint parser expects. You can then provide the path of this output in place of where the pylint violations are usually recorded.

The command line used in the line can be used:
pyflakes [path_to_src] | awk -F\: ‘{printf “%s:%s: [E]%s\n”, $1, $2, $3}’ > pyflakes.txt
How does it work? To best understand, you need to checkout the violations plugin:
svn co https://svn.java.net/svn/hudson~svn/trunk/hudson/plugins/violations/ (see http://wiki.hudson-ci.org/display/HUDSON/Checking+out+existing+plugins)
Within the plugin/violations/types/pylint dir, the regex inside PyLintParser.java is listed as follows:
/**
     * Constructor - create the pattern.
     */
    public PyLintParser() {
        pattern = Pattern.compile("(.*):(\\d+): \\[(\\D\\d*).*\\] (.*)");
    }
Keep in mind that in Java, the regexp's require two backslashes. Therefore, \d is actually expressed as \\d. The regex pattern that needs to be generated is basically:
re.compile("(.*):(\d+): \[(\D\d*)\] (.*)")
We can see why the different message types are allowed. Right now all results can be used to either generate warning or errors.
/**
     * Returns the Severity level as an int from the PyLint message type.
     *
     * The different message types are:
     * (C) convention, for programming standard violation
     * (R) refactor, for bad code smell
     * (W) warning, for python specific problems
     * (E) error, for much probably bugs in the code
     * (F) fatal, if an error occured which prevented pylint from doing
     *     further processing.
     *
     * @param messageType the type of PyLint message
     * @return an int is matched to the message type.
     */
    private void setServerityLevel(Violation violation, String messageType) {


One undocumented feature in the Hudson Violations plug-in is that after each build, it creates a violations.xml file and violations/file directory containing each of the individual files with violations. These XML files are generated from the PyFlakes output and saved into the corresponding subdirectory and file.
File xmlFile = new File(
            build.getRootDir(),
            MagicNames.VIOLATIONS + "/" + MagicNames.VIOLATIONS + ".xml");
        try {
            model = new BuildModel(xmlFile);
            ParseXML.parse(
                xmlFile, new BuildModelParser().buildModel(model));
        } catch (Exception ex) {
            LOG.log(Level.WARNING, "Unable to parse " + xmlFile, ex);
            return null;
        }

        modelReference = new WeakReference(model);
        return model;

For instance, we would have a violations/file/modulename/foobar.py.xml here:
<violations>
  <type name='pylint'>
    <file
      name="modulename/foobar.py"
      count = '2'>
      <severity level="0" count="2"/>
      <source  name="E1" count="2"/>
  </file>
</violations>
If you see "No violations found" when drilling into the file on the Hudson control panel, it's most likely that your search includes a relative path "./" that may be triggering an issue. Since Hudson depends on using URL matching, the "./" could be causing issues for the Violations plug-in to locate the appropriate XML file. The following sed expression may help to remove these "./" paths.
pyflakes <dir> | sed 's/^[./]*//' 

8 comments:

  1. This trick will also work with pep8:

    pep8 $PACKAGE | awk -F\: '{printf "%s:%s: [%s]%s\n", $1, $2, substr($4,2,4), substr($4,6)}' >> pymetrics.txt

    ReplyDelete
  2. Helpful, thanks. (BTW, Microsoft has conveniently converted the first command line you listed into gibberish [the quotes].)

    ReplyDelete
  3. Go-daddy provides a control panel that is capable of creating multi emails with security ensured. Go-daddy also offers the HTTPS which is the normal hypertext tenser protocol but with some form of encryption. assignment expert

    ReplyDelete
  4. Park View Signature Apartments Karachi, Park View Lahore DHA, and Park View Villas Lahore are some of the main prolific projects of this group. Apart from this, the group is headed by senior politicians and philanthropists Mr. Abdul Aleem Khan.

    park view lahore payment plan

    Rudn Enclave payment plan
    Blue world city Islamabad map

    ReplyDelete
  5. I studied Java in IT testing courses, and I didn't really understand the subject at the time, but over time I began to delve into it in more detail. This article is useful and it is written in detail that everyone will understand.
    But I took courses for self-development, I am a writer by profession.
    And I really like to combine my work with different areas.
    I write cheap custom essay writing for students.
    They always leave good reviews about my work and it is more motivating to work

    ReplyDelete
  6. Delta International Recruiting Agency gives gifted and qualified work to the Gulf countries and various bits of the world at all levels in the space of FMCG, Engineering/Construction, Information Technology, Telecommunication, Healthcare, Finance, etc other than a tremendous piece of Services industry regions.

    ReplyDelete
  7. Kingdom Valley Islamabad a new luxury-housing society developing in the core of Islamabad. This project is located in the middle of luxury housing projects on Chakri road. With its serene infrastructure and amenities like hospitals, parks, schools etc. Kingdom Valley takes your lifestyle one grade higher. It offers you different sizes of commercial and residential plots. Kingdom Valley offers you a friendly-budget payment plan and easy installments that make an ease for the investors to invest.

    ReplyDelete
  8. We are here to helping you with your investment plan. As the team of topline marketing, we are the best in consulting and guiding investors in property. Although we are the best known in property consultant, therefore, we can provide the best consulting about investing in the estate.
    Blue World City Payment Plan
    Nova City Peshawar

    ReplyDelete