Our initial investigations explored a number of available tools for source code conversion. The only one that looked at all promising was a commercial product, Tangible[6]. We bought a license to evaluate its capabilities, and the exercise taught us a lot about where the difficulties were going to arise. It was immediately apparent that we would have considerable difficulties with Java generics, with anonymous inner classes, and with our extensive use of the Java CharSequence interface, which has no direct equivalent in .NET. The exercise also taught us that Tangible, on its own, wasn't up to the job. (Having said that, the conversions performed by Tangible helped us greatly in defining our own rules.)

Our next step was to reduce our dependence on the constructs that were going to prove difficult to convert: especially generics, and the use of CharSequence. I have described in more detail how we achieved this in blog postings: [7] [8].

Generics are difficult because although Java and C# use superficially-similar syntax (for example List<String>) the semantics are very different. In C# instances are annotated at run-time with the full expanded type, and one can therefore write run-time tests such as x is List<String>. Writing x is List will return false: List<String> is not a subtype of List. By contrast, with Java, the type parameters are used only at compile time and are discarded at run time (the process is called Type Erasure). This means that on Java, x instanceof List<String> is not allowed, while x instanceof List returns true.

We decided to reduce the scale of the problem by dropping some of our use of generics from the product. In particular, in Saxon 9.9, two key interfaces, Sequence and SequenceIterator, were defined with type parameters restricting the type of items contained in an XDM sequence, and we dropped this in Saxon 10.0. The use of type parameters here had always been somewhat unsatisfactory, for two reasons:

Most of the time, the code has to deal with sequences-of-anything: we don't know statically, when we write the Saxon code, what type of input it is going to be dealing with (that depends on the user-written stylesheet). So providing the type parameter (Sequence<Item>) simply doesn't add any value.

The XDM model for sequences has the property that an item is a sequence of length one. So Item implements Sequence<Item>. Which means that a subclass of Item, such as DateTimeValue, implements Sequence<DateTimeValue>. Which followed to its logical conclusion mans that a DateTimeValue is an Item<DateTimeValue>, and a generic item is therefore an Item<Item> (or is it an Item<Item<Item<...>>>?). Modelling the XDM structure accurately using Java generics proved very difficult, and in the end, it introduced a whole load of complexity without adding much value. Getting rid of it was welcome.

As far as the CharSequence interface is concerned, we used this extensively in interfaces where strings are passed around, to enable us to use implementations of strings other than the Java String class. For example, the whitespace that often occurs between elements in an XML document is compressed using run-length encoding as a CompressedWhitespace object, which implements the CharSequence interface, and can therefore be substituted in many cases for a Java String.

The use of CharSequence isn't perfect for this purpose, however. Firstly, it has the same problem as a Java String in that it models a string as a sequence of 16-bit UTF-16 char values, using a surrogate pair to represent Unicode astral codepoints. In XPath, strings need to be codepoint-addressible (at least for the purposes of functions such as substring() and translate()), and neither String nor CharSequence meets this requirement. There are also issues concerning comparison across different implementations of the CharSequence interface, plus the fact that many commonly used methods in the standard Java class library require the CharSequence to be converted to a String, which generally involves copying the content. In addition, the CharSequence interface doesn't guarantee immutability. For these reasons, we had already introduced another string representation, the UnicodeString, which we were using in many corners of the code, notably when processing regular expressions.

C# has no direct equivalent of CharSequence: that is, an interface which is implemented by the standard String class, but which also allows for other implementations. The interface IEnumerable<Char> comes close, but that doesn't allow for direct addressing to get the Nth character in a string.

So we decided to scrap our extensive use of CharSequence throughout the product, and replace it with our own UnicodeString interface – which allows for direct codepoint addressing, rather than char addressing with surrogate pairs. There is a performance hit in doing this, because there's a lot of conversion between String and UnicodeString when data crosses the boundary between Saxon and third-party software (notably the XML parser, but also library routines such as upperCase() and lowerCase()). However, it's sufficiently small that most users won't notice the difference, and we can mitigate it – for example we have our own UTF-8 Writer used by the Saxon serializer, and it was easy to extend the UTF-8 Writer to accept a UnicodeString as input, bypassing the conversion of UnicodeString to String prior to UTF-8 encoding.