XmlDocs: Linking to the Overloads Page

XML documentation has a natural "link to code" element: the <see> tag. When a function is overloaded, the resulting help file contains an "overloads" page like this, but getting the see element to link to the overloads page is not exactly straightforward.

The <see> tag is one of the tags that is verified by the compiler, so it's not possible to just stick anything in there. The see.cref attribute must be a resolvable code element. The compiler doesn't allow you to resolve to a method group; it wants a single, unambiguous member reference.

Example warning/error message when attempting to link to an overload group:
Warning as Error: Ambiguous reference in cref attribute: 'FindFiles'. Assuming 'Nito.KitchenSink.WinInet.FtpHandle.FindFiles(string, Nito.KitchenSink.WinInet.FtpHandle.FindFilesFlags)', but could have also matched other overloads including 'Nito.KitchenSink.WinInet.FtpHandle.FindFiles()'.

Here's a little-known fact about the <see> tag: it will not verify any see.cref values that start with a single character followed by a colon. This enables specifying full DocumentationId links such as "T:Nito.Async.ActionDispatcher".

There is a standard extension of the DocumentationId format for overloads that is understood by Sandcastle: it uses the "Overload:" prefix as such: "Overload:System.Windows.Threading.Dispatcher.Invoke". Unfortunately, Visual Studio (as of 2008) will attempt to resolve a link like this, and will fail.

The workaround is to use the "O:" prefix for such links (this prefix is unused by the DocumentationId format), and modify the XML documentation file before it is passed to Sandcastle. The "O:" prefix bypasses Visual Studio's verification, and the "Overload:" prefix is correctly understood by Sandcastle.

In my projects, I use the following XSLT transformation to automatically translate see.cref references starting with "O:" to have a prefix of "Overload:" instead:

<?xml version='1.0'?>
<xsl:stylesheet xmlns:xsl='http://www.w3.org/1999/XSL/Transform' version='1.0'>
 <xsl:output method="xml" indent="yes"/>

   <!-- Copy all documentation as-is except for what matches other rules -->
   <xsl:template match="@* | node()">
    <xsl:copy>
     <xsl:apply-templates select="@* | node()"/>
    </xsl:copy>
   </xsl:template>

   <!-- Convert "cref" references that start with "O:" to starting with "Overload:". -->
   <xsl:template match="@cref[starts-with(., 'O:')]">
    <xsl:attribute name="cref">
     <xsl:value-of select="concat('Overload:', substring-after(., 'O:'))"/>
    </xsl:attribute>
   </xsl:template>
</xsl:stylesheet>

By the way, it's not difficult to include an XSLT transformation as part of an MSBuild project file (with the MSBuild Extension Pack). It's beyond the scope of this blog post, but you can check out the Nito.Async main project build file for an example.

XML Documentation: The Basics

This is the first in a new topic of blog posts: dealing with XML documentation. XML docs are great, but suffer from a bit of a lack of tooling. I put most of my focus on using XML documentation for IntelliSense and CHM output, since there isn't an end-user distribution story for Help v2. (As of this writing, there isn't an end-user distribution story for Help v3 a.k.a. "Microsoft Help 1" either...)

IntelliSense help is easy; just tell Visual Studio to generate an XML documentation file along with the binary, and it will be detected and used as long as it stays in the same directory as the binary.

CHM help files are a bit more complex; you'll need a suite of tools to transform the XML into HTML, which is then compressed into a CHM file. Historically, there has been a lot of "project churn" as solutions were developed and abandoned. However, the current leader appears to have a lot of staying power, and even has concessions of former-leader compatibility.

This current leader is SandCastle. My preferred toolset includes:

• Sandcastle - the core "compiler" for xml documentation.
• Sandcastle Styles - bug fixes and style updates for Sandcastle.
• Sandcastle Help File Builder - a nice GUI (now within Visual Studio) for working with Help project files.
• HTML Help Workshop - the same, sad old buggy software we've been dependent on for years...

Once you've got all those installed, then you're ready to start authoring XML docs that can be used for CHM output as well as IntelliSense.

Inheritance in General, and Deriving from HashAlgorithm in Particular

In the early days of OOP, the general consensus was that inheritance would become the key to software reusability. After experience showed that multiple implementation inheritance was too easily misused (e.g., C++), modern OOP languages adjusted to only allow multiple interface inheritance (e.g., C#).

Furthermore, in recent years the true difficulty of designing for inheritance has become known. The main problem is that when designing a base class, instead of one API there are two: the public API and the protected API. If an "API" was just a set of methods, then this would not be too much; however, one must consider invariants (for each API) as well as how the invariants behave when both APIs are used simultaneously.

This quickly becomes complex for all but the simplest classes, which has led to the modern design guideline "prefer composition over inheritance." This guideline seeks to simplify the inheritance situation by only using one API. Such renowned C# gurus as Jon Skeet have stated that classes should be sealed (non-inheritable) by default, and we may yet see the next OOP language following that advice.

Most classes are not designed for inheritance. Even for classes that are designed for inheritance, a common problem surfaces: only the public API is sufficiently documented. Since this is the API used by the vast majority of developers, the protected API is too often neglected.

I ran into an example of this today, when writing up a general CRC-32 implementation. Naturally, I wanted to derive from HashAlgorithm, but the MSDN documentation is completely lacking. After surfing around a few other implementations, I kept seeing a lot of the same mistakes.

Plunging into Reflector, I dissected HashAlgorithm once and for all, and here's what should be on MSDN under "Notes to Inheritors":

• You should invoke Initialize in your constructor. The "all-in-one" methods like ComputeHash do not start by calling Initialize (but do call it at the end).
• You should set HashSizeValue in your constructor to set the return value for HashSize. Overriding HashSize is unnecessary.
• The State value is set to nonzero after TransformBlock and reset to 0 after TransformFinalBlock. This enables derived classes to restrict their set of legal operations when in the middle of calculating a hash.

After performing this exercise, I pondered previous similar encounters. To my surprise, I cannot recall one time when I needed to derive from a class and was actually able to do it supported only by the documentation. Every time I've had to implement a derived type, I've always had to peek into the implementation of the base because the protected API was insufficiently documented.

Hence the guideline, "prefer composition over inheritance." I go one step futher: every class I write is sealed unless it truly needs to be a base class (and even then, usually every base class is abstract and the non-abstract classes are sealed). This is a design guideline that I've followed since my C++ days, and it serves me equally well in C#.

I/O Limitation in Windows

Earlier today I was stress-testing a SerialPort component for Nito.Async when I ran into an unusual error: ERROR_NO_SYSTEM_RESOURCES (1450).

This error can be caused by exhausting any of several OS resources, though all the examples I've found deal with exhausing memory-related resources. In my particular example, I was trying to shove a 600 MB file across a serial port all at once.

There's a limit to how big of a user-mode buffer one can send to a device driver (so this comes into play if you're talking to a device, such as a serial port or named pipe; it also affects I/O to regular files if FILE_FLAG_NO_BUFFERING was used). According to Dan Moseley of Microsoft, the basis of this limitation is in how the I/O Manager creates its memory descriptor list (MDL).

I'm in a position where I will need to transfer large amounts of data over serial ports, so I wanted to know how much data can be transferred in a single call. Dan Moseley's original description updated with the IoAllocateMdl MSDN docs, along with the page size information from the latest revision of Windows Internals was enough information to calculate the answer, which I've summarized below.

Maximum I/O Buffer Size for Individual Unbuffered Read/Write Operations

Operating System	Architecture	Page Size	Calculation	Maximum I/O Buffer Size
2K/XP/2K3	x86	4096	PAGE_SIZE * (65535 - sizeof(MDL)) / sizeof(ULONG_PTR)	67076096 bytes (63.97 MB)
XP/2K3	x64	4096	PAGE_SIZE * (65535 - sizeof(MDL)) / sizeof(ULONG_PTR)	33525760 bytes (31.97 MB)
2K/XP/2K3	IA-64	8192	PAGE_SIZE * (65535 - sizeof(MDL)) / sizeof(ULONG_PTR)	67051520 bytes (63.95 MB)
Vista/2K8	x86 & x64	4096	(2 GB - PAGE_SIZE)	2147479552 bytes (1.999996 GB)
Vista/2K8	IA-64	8192	(2 GB - PAGE_SIZE)	2147479552 bytes (1.999992 GB)
Win7/2K8R2	x86 & x64	4096	(4 GB - PAGE_SIZE)	4294963200 bytes (3.999996 GB)
Win7/2K8R2	IA-64	8192	(4 GB - PAGE_SIZE)	4294959104 bytes (3.999992 GB)

The lowest entry here is for XP/2K3 running on x64. So, if 64-bit XP is important, then you should not use I/O buffers over ~31 MB. If you ignore 64-bit XP, then you can use I/O buffers up to ~63 MB. Newer operating systems take great strides towards removing this limitation completely.

Note that this table only applies to the buffer passed to a single API call. There are other I/O-related restrictions; in particular, I cannot simply split up my 600 MB file into 16 MB chunks and still send them all at once; the serial port will not be able to keep up with the requests and will eventually run into another limitation (with the same error code, ERROR_NO_SYSTEM_RESOURCES (1450)). The solution is to implement buffering in the application.

Sharp Corner: Value Types Are Never Reference-Equal

There are two types of equality testing in the .NET framework: reference equality and value equality (if the type being tested supports value equality). There are numerous ways to test for equality (object.Equals, operator ==, IEqualityComparer<T>, etc), but at the end every one of them resolves to either value equality or reference equality.

Conceptually, two objects are "reference equal" iff they are actually the same object. For example, two strings may have the same value (and thus be "value equal"), but they may be two different objects (and thus not "reference equal").

Eventually, one hits a corner:

[TestMethod]
public void ValueTypes_AreNeverReferenceEqual()
{
    var num = 13;
    
    Assert.IsFalse(object.ReferenceEquals(num, num));
}

Of course, people rarely wish to test value types for reference equality; this corner is more likely to be found while testing instances of a generic type for reference equality. This result is often surprising; if everything in C# is an object (including a value of type Int32, which derives from ValueType, which derives from object), then why can't they be compared for reference equality?

The reason that this does not work is because unboxed value types are not objects. They are a "special case" in the C#/.NET world, given special treatment for efficiency reasons. They are convertible to an object (via a boxing conversion), but they are not actually objects themselves. C# really goes far to pretend that they are objects (e.g., "7.ToString()"), but it can't cover every corner.

In the example above, the value instance is implicitly converted to an object - twice - and then these objects are compared. Naturally, they refer to different objects, so they are not reference-equal.

Boxed value types are real objects (though they lose their compile-time type information). They may be compared for reference equality:

[TestMethod]
public void BoxedValueTypes_CanBeReferenceEqual()
{
    var num = (object)13;

    Assert.IsTrue(object.ReferenceEquals(num, num));
}

Conclusion: contrary to popular opinion, not everything in C# is an object.

Introducing Sharp Corners

Every language, no matter how well-designed, has its "corners." These are the Dark Places that every programmer eventually runs into. I define such a "corner" as a situation where the language has surprising results. Corners are discovered as a programmer moves from one phase to the next in language mastery; e.g., as a programmer who has an "intermediate" skill in the language becomes "advanced," he will discover corners during that transition.

Most modern languages are easy to learn and contain few if any surprises at the beginner and intermediate level. It is when one begins to push the boundaries of the language that one finds the corners. Corners are a valuable learning experience. Languages are based on logical rules, defined by the language specification; when a programmer begins learning a language, he seldomly begins with the spec. As time goes on, the programmer develops a "feel" for the language based on experience rather than the language spec; internally, he develops an assumed set of rules for the language.

Eventually the programmer will extend their assumptions and realize that his assumed rules are not the same as the language specification (in many cases, in fact, there is a drastic difference). At this point, the good developer will explore this newly-uncovered corner and discover the actual rule being used by the language. When the developer discards their previously-held assumptions and embraces the language rule, he takes another step towards language mastery.

"Corners" are slightly different than "warts." A "wart" is usually a holdover from an earlier revision of the language that is mainained only for backwards compatibility. The "recommended way" of programming a given language avoids the warts (since they have been replaced with a new and better way of accomplishing the objective). However, corners are the result of the language's underlying logic and cannot be avoided.

There are a few programmers who do read (and understand) language specifications. Most, however, follow a natural process of discovering and exploring corners.

On my blog, I hope to build up a small collection of corners for the C# language (hence the name "Sharp Corners"). The intention is to help educate any C# programmer on their way to language mastery.

Of course I must end with a reference Jon Skeet's famous C# Brainteasers page, which is like a collection of guru-level C# corners. Mine will probably not be that complex. :)