How This Site Is Built

Introduction

This page is meant to provide a detailed discussion of the techniques that I use to generate this site. The simplified overview is available on the About page. This page was born out of feedback from some of the folks at the Agile Bazaar, who saw the site and wanted to hear more about the nitty-gritty.

Inspiration

The idea for using XML and XSL to build a static website came from a couple of different places. Part of the idea came from a thread on the Test-Driven Development mailing list, wondering how a graphical user interface could be developed in a test-driven manner. Another part came from a discussion about agile Web development at the Agile Bazaar's September 2002 meeting, where Tom Stambaugh expressed a great deal of frustration over trying to find a tool that would let him evolve a Web site without constant manual tweaking of images, scripts, etc.

Test-Driven Development

Test-driven development is an idea that was pioneered by Kent Beck, one of the leading lights in the patterns, Smalltalk and Extreme Programming communities. In "traditional" software development, first you write code for some piece of functionality, then you test it. TDD turns that cycle around - the first thing you do is write a test for the functionality you're working on, and run that test as part of an automated test suite. The test should fail (because you haven't written the code yet); in fact, it might not even compile (and if that's the case, you write a non-functional stub first, so you can compile your test and watch it fail). Only when you have a failing test can you write the code that actually makes the test pass. Once you've gotten your new test to pass, you look through your code, removing duplication, "magic numbers" and similar nastiness through a process called Refactoring. You repeat this process, particularly including the running of all the automated tests, throughout development. (Kent's got a book coming out about TDD, and I had the honor to be part of the review process.)

Obviously, this is a radically different way of developing software. Probably every developer knows how time pressures can force one to put off or skimp on testing. And we all know the consequences: bugs that show up late in development and that are hard or impossible to track down as a result. Under TDD, we're constantly running the test suite, so any bugs that we put into the code by mistake are caught literally within seconds.

On the other hand, the TDD community has discovered that building GUIs test-first is rather difficult. GUI-building tools typically don't provide an easy way to programmatically check what they're doing, and screen-scraping robot programs are usually hard to configure for developer's purposes. I'm trying out the idea of using XML and XSL to develop this site in large part because it's relatively easy to write automated tests (using tools like JUnit) for XML files. As an example, I have one test that loops through all my XML content files and all my XSL stylesheets and makes sure that each stylesheet handles each tag in the content file:

public void testAllXSLHandlesAllXML() throws Exception {
    for (Iterator xsli = Arrays.asList(allXSLFiles).iterator(); xsli.hasNext(); ) {
        String xsl = "xml/" + (String) xsli.next();
        for (Iterator xmli = Arrays.asList(allXMLFiles).iterator(); xmli.hasNext(); ) {
            String xml = "xml/" + (String) xmli.next();
            assertXSLHandlesAllXMLTags(xsl, xml);
        }
    }
}

private void assertXSLHandlesAllXMLTags(String xslFile, String xmlFile)
    throws Exception {
    Document xsl = GetDOMDoc(xslFile);
    Document xml = GetDOMDoc(xmlFile);
    String msg = "xsl: " + xslFile + ", xml:" + xmlFile;
    Element xmlElem = null;
    int nodesChecked = 0;
    // get all element nodes in pagecontent
    NodeIterator xmlni = XPathAPI.selectNodeIterator(xml, "//pagecontents//*");
    while ((xmlElem = (Element)xmlni.nextNode()) != null) {
        /*
         * For some reason, I can't get selectSingleNode to find children
         * in the xsl: namespace; so instead of using this to find templates:
         *     XPathAPI.selectSingleNode(xsl, ".//xsl:template")
         * I'll loop through all the element nodes the hard way. :-(
         */
        nodesChecked++;
        Element xslElem = null;
        NodeIterator xslni = XPathAPI.selectNodeIterator(xsl, "//*");
        boolean found = false;
        while ((xslElem = (Element)xslni.nextNode()) != null) {
            if (xslElem.getNodeName().equals("xsl:template")) {
                if (xslElem.getAttribute("match").indexOf(xmlElem.getNodeName()) > -1) {
                    found = true;
                    break;
                }
            }
        }
        assertTrue(msg + ", checking for xml node "+xmlElem.getNodeName(), found);
    }
    assertTrue(msg + ", # of nodes checked: "+nodesChecked, (nodesChecked > 0));
}

When I first added the <code> tag to this page and ran the test suite, this test caught the fact that I didn't have any templates to properly process <code> tags. It also caught the typos that I made while typing in the code fragments for this test! If I didn't have this test, then I might still be wondering why the pages looked screwy.

(By the way, if anyone has a bright idea as to why XPathAPI.selectSingleNode(xsl, ".//xsl:template") doesn't work the way I expected it to, please tell me!)

Xerces and Xalan

The Apache Xalan XSLT engine, which includes the Xerces XML parser, is an open-source implementation of the World Wide Web Consortium XSL Transformation specification. XSLT specifies ways to convert XML data into other formats; HTML is an obvious (and common) target.

Obviously, a complete how-to on using XSLT is way beyond the scope of a simple page; in fact, there are shelves full of books about XSLT at your local techie bookstore. (See my list of recommended professional books for what I'm using.) Basically, though, XSLT stylesheets (which have to be well-formed XML) consist of template rules. The XSLT transformation engine takes well-formed XML input (from a file, a URL, or whatever) and looks for nodes in the input that match templates in the stylesheet. When the transformation engine finds a match, it generates output nodes based on the contents of the template rule. Here's a sample from the main.xsl sheet used to generate the full-graphics pages:

<xsl:template match="navout">
	<a href="{@dest}"><xsl:value-of select="."/></a>
</xsl:template>

This template rule matches a node in the source XML file which is named "navout". When the XSLT engine finds a node in its input stream that matches this rule, the engine will output an anchor node whose href attribute is the dest attribute of the <navout> tag and whose content is whatever the content of the <navout> tag was. Translated into English, this template takes this:

<navout dest="http://www.w3.org">World Wide Web Consortium</navout>

as input, and produces this:

<a href="http://www.w3.org">World Wide Web Consortium</a>

as output.

Ant

Apache Ant is a remarkably powerful tool for managing build tasks. Unlike C-style makefiles, Ant runs in a Java environment and uses XML files to define the various tasks to be done. Most Ant tasks include some degree of timestamp checking, so that unchanged files aren't processed unnecessarily.

One particularly powerful Ant task is the <xslt> task, which runs an XSLT processor on defined files. (Obviously, I was happy to find this!) Here's an excerpt from the build.xml file that creates the full-graphics pages:

<xslt basedir="${xml.home}" 
      includes="*.xml"
      destdir="${build.home}/g_temp"
      extension=".html" 
      style="${xml.home}/main.xsl"
      processor="trax">
	<param name="MOD_DATE" expression="${build.timestamp}"/>
</xslt>
<copy todir="${build.home}">
	<fileset dir="${build.home}/g_temp" />
</copy>

This <xslt> task takes all the files with an extension of .xml (including this file), applies the main.xsl XSLT stylesheet, and stores the results in the g_temp subdirectory. The <copy> task then copies the resulting HTML files to the main build directory, from which I can transfer them to the web server.

The complete build.xml file includes a number of other, primarily housekeeping tasks (creating directories, copying static content such as images and style sheets, etc.).