Test Details

I ran performance tests in two different configurations, first using a client with a local server, and then a client and server connected by an otherwise unloaded 100 megabit Ethernet. I used Jakarta Tomcat 5.0.25 as the server in all web services tests, running on a Mandrake 9.1 Linux Athlon XP 2200+ system with the Sun 1.4.2 JVM for Linux (with Tomcat's default JVM settings). In the case of a separate client, the client I used was running Mandrake 10.0 Linux on a Pentium IIIm 1GHz system with the Sun 1.4.2 JVM for Linux

The Axis tests all used the daily Axis 1.2 build from July 3, 2004, in order to give the benefit of the most recent performance enhancements to the 1.2 code. The JAX-RPC RI test used the 1.1.2 version included in the latest Java Web Services Developer Pack, 1.4. The JibxSoap test used the Alpha 0.2 release.

I'd originally hoped to include some commercial web services frameworks in the comparison, but I was denied permission to benchmark The Mind Electric's GLUE (now owned by webMethods), and did not receive any response from other vendors I tried to contact.

The test parameters were chosen to provide a range of average response sizes. In each case the actual number of matching quakes varies from one query to the next in the generated sequence, so individual response sizes vary. Here are the actual test parameters used, along with the total number of matching quakes for the sequence and the total number of bytes of request and response XML (as measured for the JibxSoap implementation - the other doc/lit byte counts are about the same, while the rpc/enc version can be 3-4 times as large):

  • Very Low density: 0.01 range, 200 queries (total of 48 matching quakes, 93,424 bytes of request XML, 62,629 of response XML)
  • Low density: 0.05 range, 100 queries (total of 853 matching quakes, 46,708 bytes of request XML, 210,261 bytes of response XML)
  • Medium density: 0.2 range, 20 queries (total of 3,052 matching quakes, 9,354 bytes of request XML, 530,088 bytes of response XML)
  • High density: 0.5 range, 10 queries (total of 6,155 matching quakes, 4,666 bytes of request XML, 1,008,054 bytes of response XML)

Before each actual timing test run I stopped and restarted the Tomcat server, then ran the full sequence of tests once locally to initialize the server code and load all classes required for processing requests. I next ran first the local tests and then the remote tests to get the actual timing results shown. For these timed tests I used a "quiet" option to limit the console listing of results to a simple summary at the end of the test run. Here's the actual local test script (which calls the run.sh script supplied with each version of the test application):

run.sh .01 200 quiet localhost
run.sh .05 100 quiet localhost
run.sh .2 20 quiet localhost
run.sh .5 10 quiet localhost

Test Results

Figure 1 shows the timing results when the client was run on the same system as the server, while Figure 2 shows the results when the client was run on a separate system connected via a local network. Actual transmission time does not appear to be a major factor in either test, though it certainly contributes to the longer times in the Figure 2 results. This is especially true at the lower response data densities. At higher response densities it appears that the slower processing speed of the client is more of an influence than the actual transmission time.

Figure 1. Client to local server times (seconds)
Client to local server times (seconds)

Figure 2. Client to remote (LAN) server times (seconds)
Client to remote (LAN) server times (seconds)

Analysis of Results

RMI delivers better performance than any of the SOAP web services. This is as expected, since RMI is specifically for Java-to-Java communications and uses binary data encoding, while web services are cross-platform and use XML text representations. The most surprising point here is that JibxSoap comes very close to matching RMI performance, matching or surpassing the results shown by Sun's Fast Web Services project built around binary data encodings substituted for XML.

For both configurations and all response densities the JibxSoap implementation gave much better performance than any of the other SOAP alternatives, including the other doc/lit implementations which used the same wire formats for data. The JAX-RPC RI implementation shows the second best results, generally running 3 to 5 times slower than the JibxSoap implementation. The Apache Axis implementations all show weak performance at lower response densities, but in the case of the doc/lit version actually beats JAX-RPC RI performance by a small margin at higher response densities. The version using Castor data binding is considerably slower in all cases, with the performance getting worse with higher response density. The Axis rpc/enc version shows even worse performance as the response density increases.

The performance of the JibxSoap implementation is likely due to two factors. First off, the light-weight nature of the framework and efficient internal operation adds very little per-message overhead. This per-message overhead looks to be especially high with Axis (as shown by the poor performance of all the Axis versions at very low response density, where the number of messages exchanged is considerably larger than all the other test cases combined). Secondly, the JiBX data binding basis of JibxSoap allows very fast conversion between XML and Java objects, minimizing the overhead for larger messages. The net result is that JibxSoap offers performance that's very close to that of RMI (though it starts to suffer by comparison in the remote test with large response messages, where both the longer transmission time of text XML and the relatively slow client have the most impact on performance).

Looking Ahead

JibxSoap could be combined with something like my XBIS XML encoding format to deliver even better performance, probably beating RMI by a substantial margin for the types of data exchanges typically used with web services. The JibxSoap-XBIS combination would also have the advantage of preserving full compatibility with text XML, as opposed to alternatives based on binary data. Alternative schema-based binary representations (similar to those used by the Sun project) generated directly by a modified form of JiBX data binding are likely to be several times faster still, when working with large data volumes. Given this expectation, it seems surprising that the Sun project has not obtained better results than they've reported from their work with binary representations. Until Sun makes code available from their project there's really no way to say why their results are not as good as I'd expect.

I believe JibxSoap shows a very promising direction for the future of doc/lit web services in Java. The older JAX-RPC and Apache Axis frameworks were designed primarily to suit the needs of rpc/enc services, and I feel that this legacy is going to limit their flexibility and performance going forward. JibxSoap demonstrates that alternative approaches can make web services in Java both easier to use and much faster in operation.