The tests currently implemented are:

  1. Document construction time - measure the time taken to construct the representation of a document.
  2. Tree walk time - measure the time taken to walk through all the elements of the tree, in document order, counting elements, attributes, and characters of text.
  3. Text generation time - measure the time taken to output the XML text for a document. This test is included because the results may be relevant to many users, but as implemented it uses only the simplest form of output for each model. Tuning of output options or custom text serialization code can provide considerably higher performance than measured by these tests.
  4. Modification time - performs a transformation on the document. This somewhat arbitrary processing eliminates isolated whitespace character data content, wraps other character data content in a new, added, element, and adds an attribute to existing elements with non-whitespace character data content.
  5. Document memory usage - measure the memory usage of the document representation.
  6. Serialization out time - measure the time taken to output the document representation using Java serialization (not supported by all representations).
  7. Serialization in time - measure the time taken to reconstruct the document representation from the Java serialized form (not supported by all representations).
  8. Serialized size - measure the size of the Java serialized form of the document.

The test framework uses the approach of running one particular test some number of times (10 for the results shown, except in the large document case) on a document or collection of documents, tracking the best and average times for that test. When one test is completed the framework moves on to the next test on the same document. After the full sequence of tests has been completed on one document, it repeats the process with the next document or collection of documents. In order to prevent interactions between the document models, only one model is tested in each execution of the test framework.

Timing benchmarks using HotSpot and similar dynamically optimizing JVMs are notoriously tricky; small changes in the test sequence often cause large variations in timing results. I've found that this is especially true for average times executing a particular piece of code; the best times are much more consistent, and are the values I've shown in these results.

Testing the memory usage of the representations works a little differently, in that the program keeps all the constructed copies of the document and pauses between relevant tests to encourage garbage collection. Memory usage per copy of the representation is found by dividing the total memory used by the number of copies.

All tests involving I/O use memory buffers to avoid any external timing variables. Input and output uses streams (specifically ByteArrayInputStream and ByteArrayOutputStream) to most closely simulate the normal usage. Some of the models support direct input from character arrays or Strings with higher performance than stream input, but using this type of input for testing gives misleading results; in real world applications, text documents are rarely resident in memory to be passed directly to parsers. Validation is turned off in all tests, and the documents used for the test do not specify DTDs.