Test Results

Test Results
Build time
Walk time
Combined time
Modify time
Text serialization
Memory size
Large document
Java serialization

This page provides the full set of test results as charts with commentary. Most of the test results are shown as a pair of charts, the first showing performance with the small document collections and the second the performance with medium sized documents. Test results for the single large document included in the tests are shown in a separate section. The menu table at right provides direct links to each section of the results.

Document build time

The document build time test looks at the time required to parse a text document and construct the document representation. I've included the SAX2 parse time using Crimson and Xerces SAX2 parsers in the chart for comparison purposes, since most of the document models (all except EXML and XPP) use a SAX2 parse-event stream as input to the document building process. Figures 1 and 2 show the test results.

Figure 1. Document build time (ms.) - small document sets

For small documents XPP2 clearly provides the fastest document build times by a substantial margin (Xerces2 SAX does not actually build a document, it only goes through the motions of parsing the text while discarding results).

EXML and Xerces2 DOM are the next best, though taking almost twice as long as XPP2. EXML is not directly comparable, though, because it discards all whitespace separating elements in the input document text (see the discussion of this in the EXML description). Xerces DOM is much slower than Xerces2 DOM, but still faster than the remaining three document models (dom4j, JDOM, and Crimson DOM).

One of the most interesting results from this test is the poor performance of Xerces and Xerces2 DOM with deferred node expansion. This seems likely to be due to some form of initialization or setup overhead associated with the deferred node handling. When using either of these models with smaller documents you should disable the deferred node expansion feature for best performance. This can be done with a  setFeature("http://apache.org/xml/features/dom/defer-node-expansion", false) method call to the DOMParser instance you're using.

Looking only at the SAX2 parsers, Xerces2 does much better than either the original Xerces or Crimson for the small document collections.

Figure 2. Document build time (ms.) - medium documents

XPP2 and both versions of Xerces DOM with deferred node expansion provide the best performance for the medium sized documents. These are not directly comparable, though, since the deferred node expansion version of the DOM support requires additional processing when accessed. That leaves XPP2 as the clear performance leader for the medium sized documents as well as the small documents.

dom4j offers the next best performance, followed by both version of Xerces DOM with deferred node expansion disabled. EXML, JDOM, and Crimson DOM show the poorest performance. Here again, EXML benefits by discarding whitespace sequences separating elements, resulting in a much lower number of objects created. When this is taken into account EXML's performance is considerable worse than the other models.

For the medium sized documents Xerces2 SAX2 parser performance lags that of both the original Xerces and Crimson. Looking at this pair of graphs in combination, Xerces2 appears to offer a much faster startup than either the original Xerces or Crimson, though at the cost of somewhat lower performance on larger documents.

Document walk time

The document walk time test looks at the time required to walk the constructed document representation, going through each element, attribute, and text content segment in document order. It's intended to represent the performance of the document model interfaces, which can be important for applications that repeatedly access information from a document once it's been parsed. Overall, the walk times are much faster than the parse times. For applications that make only a single pass through each parsed document, the parse times are going to be much more significant than the walk times. Figures 3 and 4 show the results.

Figure 3. Document walk time (ms.) - small document sets

XPP2 is the best performer in this test by a large margin, partially because the API provides methods to access all the information used in the test without the need for creating objects such as iterators or lists. Both versions of Xerces DOM (built with deferred node expansion disabled) show the next best performance. EXML comes close to Xerces, but again gains a substantial advantage from its discarding of whitespace. dom4j and JDOM are next, with Crimson DOM showing the worst performance. 

Note that the soaps document collection doesn't register with any of the document models, and only one document collection (webs) registers for XPP2 at all. This is because these test times were less than the clock granularity of 1 ms. In reality, the true total XPP2 time is probably about twice what is shown by the graph.

Figure 4. Document walk time (ms.) - medium documents

Medium sized documents show essentially the same performance characteristics as the small document collections, though dom4j performance suffers by comparison. Here again, XPP2 is fast enough that the clock granularity comes into play. The periodic.xml time for XPP2 is less than the 1 ms. granularity, so it does not register on the chart at all. The other XPP2 times are also truncated. An accurate measurement of the XPP2 time for this test would probably be around 5 ms., still much faster than any of the other alternatives.

Combined build-walk time

Xerces with deferred node creation trades off faster build performance for added overhead when accessing the document representation for the first time. XPP2 has a similar but more extreme approach, where the entire document representation can be built in a deferred manner (the pull node approach documented in the XPP2 API). For an accurate look at the performance of these alternatives it's useful to combine the document build time with the time taken for walking the document representation for the first time. Figures 5 and 6 show the results, along with the corresponding results for the non-deferred approaches.

Figure 5. Build and walk times combined (ms.) - small document sets


For smaller documents Xerces deferred node expansion gives poor performance, mainly due to the build time overhead (as shown in Figure 1). XPP2 does much better with its pull node construction, suffering a relatively small performance penalty from constructing the representation as it is used.

Figure 6. Build and walk times combined (ms.) - medium documents

Xerces deferred node expansion offers much better relative performance for larger documents. Here the combination of build and walk times actually comes out faster with deferred node expansion enabled than with full expansion on build. This surprising result is due to the poor build performance with deferred node expansion disabled for the xml.xml document, which suggests that Xerces has some problems with mixed content handling in this case.

XPP2 shows more of a performance penalty for the pull node construction technique than with smaller documents. The XPP2 setup time for these individual documents (equivalent to the build time for the other models) does not even register in these tests, but the build overhead shows up in the walk time. The pull node building approach still provides a good performance tradeoff in cases where only the first part of the document may need to be processed. It also offers a major advantage when latency is an issue, in which case the ability to start working with the first part of the document immediately is important. If your application needs to go through the entire document it's generally better to just build it up front, though.

Document modify time

This test looks at the time required to systematically modify the constructed document representation. It walks the representation, deleting all isolated whitespace content and wrapping each non-whitespace content string with a new, added, element. It also adds an attribute to each element of the original document that contained non-whitespace content. This test is intended to represent the performance of the document models across a range of modifications to the documents. As with the walk times, the modify times are considerably faster than the parse times. As a result, the parse times are going to be more important for applications that make only a single pass through each parsed document. The results are shown in Figures 7 and 8.

Figure 7. Document modify times (ms.) - small document sets

EXML appears to be the performance leader in this test, but this is deceptive - EXML discards the isolated whitespace sequences on input before the document representation is constructed, so it does not execute the full test. The difference amounts to almost half the work done by the other models. That leaves XPP2 as the clear performance leader, with all the others roughly equivalent. As with the time for walking the document sets (Figure 3), the soaps collection takes too little time to measure with the 1 ms. clock granularity.

Figure 8. Document modify times (ms.) - medium documents

Results for medium sized documents are very similar to those for the small document collections.

Text generation time

This test looks at the time required to output document representations as text XML documents. This step is likely to be an important part of the overall performance for any applications that are not pure consumers of XML documents, especially since the time required to output the document as text is generally close to the parse time on document input. This time is heavily effected by formatting options, though, and this test uses only the simplest form of text output for each model. Because of this the results should be interpreted as showing the default performance for each model, not the best performance. To make the times directly comparable, the test uses the original documents, not the modified versions generated by the previous test.

Figure 9. Text generation time (ms.) - small document sets

Figure 10. Text generation time (ms.) - medium documents

There's not as much of a difference between models as seen on some of the earlier test results. Both versions of Xerces do the best on the medium sized documents by a substantial margin, with the original Xerces showing the best performance overall. EXML performance is again biased by the discarded whitespace content, reducing the number of document components it handles .

Document memory size

This test looks at the memory space used for document representations. This can be a significant concern for developers who work with large documents, or multiple smaller documents concurrently. Figures 11 and 12 show the results of this test.

Figure 11. Document memory size (bytes) - small document sets

With small documents the Xerces deferred node expansion shows very large overhead, so from both time and memory standpoints it looks like this option should be turned off when working with smaller documents. XPP2 with pull node construction also shows a large overhead, due to the construction of a separate pull node parser for each document. Aside from these standouts, the models do not show major differences in memory usage for the small documents.

Figure 12. Document memory size (bytes) - medium documents

The case of medium sized documents is a little more interesting for memory size. These are shown in Figure 12, as both the initial memory usage and the added usage after the document is walked for the first time. dom4j has the most compact representation overall, about on a par with the unexpanded form of the Xerces DOM. EXML is the least compact of the represenations built in fully-expanded form (despite discarding whitespace, resulting in a much lower number of component objects). XPP2 shows little difference in memory size between the pull construction form and the full construction on build.

Both versions of Xerces show a large added memory overhead when the representation is constructed with deferred node expansion and then walked. The Xerces results are a little unclear, though, and it's possible that some of the added memory may eventually be freed. I'll try to check this better in the next version of the tests.

JDOM also shows an increase in memory usage after the representation is walked for the first time. In JDOM's case this is due to the construction of lists when checking for attributes or children of an element; if there are no attributes or children already present, JDOM creates an empty list for this purpose. The created list remains permanently attached to the element involved.

Large document performance

Time performance for the various tests on a single large (2.9 MB) document are shown in Figure 13. The format of this chart differs from the earlier ones in combining a variety of tests within a single chart. The test run also differed from earlier results in using a smaller number of copies of the document (3), because of problems with the JVM running out of memory if larger numbers of copies were used, so these results are not as consistent from one test run to another as the earlier results. SAX2 parser times are included in the chart for build time comparison purposes, as with the document build times shown earlier.

Figure 13. Large document times (ms.)

With this document both versions of Xerces do very well, using either full construction on initial build or deferred node expansion. EXML also does well, but again this is biased by the much lower number of document components used after EXML discards whitespace in the initial build.

The most interesting result here is the JDOM performance on modify. As indicated in the figure, the full magnitude of the JDOM test time is not shown in the chart because it reduces the scale of the other differences too much. This dramatically bad performance on modifying the document is probably due to the use of a linked list within the JDOM representation.

Figure 14. Large document memory size (bytes)

Memory usage for the large document is shown in Figure 14. As in Figure 12, both the base memory usage and the added usage after the representation is walked for the first time are included in the chart. The results are very similar in scale to those shown in Figure 12, though Crimson DOM comes out the best with this document. The difference is probably due to the very restricted nature of the document in this case; the earlier results combine a variety of types of documents, so are probably more representative of general results.

All the models are likely to require too much memory for very large documents since even the most compact take up more than five times the raw document text size in bytes. XPP2 pull and dom4j offer the best support for very large documents by providing means of working with partial document representations. XPP2 pull does this by only building the portions of the representation which are actually accessed, while dom4j includes support for event-based handling to construct or process only a portion of the document at a time.

Java serialization

These tests measure the time for Java serialization of the document representations. This is mainly a concern for applications that transfer a representation between Java programs using Java RMI (Remote Method Invocation), including EJB (Enterprise JavaBean) applications. I included only those models that support Java serialization in these tests. The set of figures below show the results of this test.

Figure 15. Serialization output time (ms.) - small documents

Figure 16. Serialization input time (ms.) - small documents

Figure 17. Serialization output time (ms.) - medium documents

Figure 18. Serialization input time (ms.) - medium documents

EXML shows the fastest serialization performance, but this again is due to the advantage EXML has from discarding whitespace on input. This results in a considerably smaller number of component objects in the representation and correspondingly less work for the serialization processing. With this taken into account dom4j shows the best performance overall, while the two versions of Xerces DOM consistently show the poorest.

All performance - both in time and size (not shown) - would actually be much better if the documents were output as text and parsed back in to reconstruct the documents, instead of using Java serialization. The structure of the XML document representations as large numbers of unique small objects is the problem here. Java serialization does not handle this type of structure efficiently, resulting in high overhead in both time and output size.