Today I passed 70-505 and 70-563, officially becoming a Microsoft Certified Professional Developer (Windows Developer 3.5 on Visual Studio 2008). Then I got to return home and feed our baby for the first time! :)
Baby has been sick this week, so once again my preparation for these exams was lacking. By the grace of God, however, I did pass.
Now I'm thinking, why not go for the Enterprise Developer certification before the end of the year? It's only four more exams. Hmmm... Maybe.
Today marks a good day: at long last, Nito.Async Version 1.4 has been released!
One thing I'm proud of in this release is my unit test coverage. Those were some very adventerous unit tests to write, too!
The major additions this time around are the ActionThread (an ActionDispatcher with a dedicated Thread) and the SynchronizationContextRegister (which the Nito.Async classes now use to check if their SynchronizationContexts satisfy their requirements).
There were a slew of other "cleanup" kinds of changes as well: publisher policy files were added, the help system updated, source-indexed pdbs were included, samples were added to documentation, etc.
There was a good post today by the Parallel Programming .NET Team where they clarify how the upcoming Task class can make use of SynchronizationContext. In reading that post, I was reminded of Yet Another SynchronizationContext Gotcha.
Given a SynchronizationContext of unknown type, you cannot tell if you are on a thread that is associated with that SynchronizationContext. This is the old "Do I need to invoke?" question that is answered by ISynchronizeInvoke.InvokeRequired on Windows Forms and Dispatcher.CheckAccess on Windows Presentation Foundation. However, use of these methods is usually an earmark of poor design; most methods should not be designed to be called "from any thread" (unless they are part of a class used for thread synchronization).
Back to the original statement: SynchronizationContext does not provide any way to determine if the current thread is associated with that SynchronizationContext. Note the vague term "associated with": normally, when people think of a SynchronizationContext, they assume that there is a single, specific associated thread with that SynchronizationContext, or that SynchronizationContext instances may be compared. However, that is not necessarily the case. In fact, there is not a 1:1 correspondence between a SynchronizationContext and a Thread. Consider the various types of SynchronizationContexts provided in the 3.5 framework:
By now, it should be clear that there is not a 1:1 correspondence between SynchronizationContext instances and threads. Furthermore, implementations of SynchronizationContext may have a specific associated thread, a set of associated threads, or no associated threads at all.
SynchronizationContext implementations also do not override equality to indicate whether they are in the same "associated thread set". While it is possible to do this with the current implementations, it is easy to imagine an implementation where this would be much more difficult (a custom thread pool, for instance, comprised of foreground threads).
This situation is problematic for the asynchronous component designer. Starting with Version 1.4, the Nito.Async library contains a SynchronizationContextRegister that understands which guarantees are provided by which implementations of SynchronizationContext. The previous gotchas I mentioned are included (e.g., whether Post is non-reentrant), and the "single specific associated thread" guarantee covered in this post is also included. This is done in a generic way, allowing programs to override the built-in default values and also provide their own if they have their own implementations of SynchronizationContext.
However, the fact remains that there is no generic way to determine if a SynchronizationContext is "associated" with the current thread. That is one "gotcha" that remains.
From time to time, I find myself having to design a new computer language. The smallest of these languages only contain expressions, but some have branched out into statements. The design of languages is a fascinating topic, and one which I believe is very benificial to any programmer.
Of course, the best way to go about designing a language is to find out what worked and didn't work for other languages. For example, the official Gawk manual starts its section on string concatenation with the quote "It seemed like a good idea at the time," so it's probably best not to use a space character as a string concatenation operator... Likewise, Bjarne Stroustrup started out in favor of resumption-based exception handling, but "over the next four years, I learned otherwise..." (he tells the fascinating story about Cedar/Mesa in section 16.6 of The Design and Evolution of C++).
One of the best sources for language design is the Python mailing list archives; not a single change is made to the language without being carefully scrutinized by some of the brightest minds in language design. Python, as a result, is one of the finest languages. I personally recall being positively electrified watching generator syntax evolve; and this same idea several years later was brought into C# (where they're called "iterator blocks").
However, many new languages do continue to make the mistakes of the old languages, simply because "it's always been done that way." In particular, there are three syntactical elements that I always put in my languages now, and I think they're underused in modern language design. Two of them are stolen straight from Python; one is my own invention. The next time you need to design a language, please consider these.
It seems like everyone has a different way of writing "if (x > 3 && x < 7)". If you're a strict follower of Code Complete, that would be "if (3 < x && x < 7)". I like to allow in my languages the "math conditionals", which look like "if (3 < x < 7)". It doesn't have a big impact on your EBNF to support this. More complex math conditionals are also not difficult: "if (3 < x < y <= z < 7)".
One of Python's trademark features is the use of indentation to define scope. It seems like people either love it or hate it, and I'm in the former camp. Furthermore, it's not that hard to get working once you discover the trick of using indentation as a lexical symbol instead of whitespace.
OK, anyone who's been programming any reasonable amount of time has been bitten by this one at least once. The logical operators (&&, ||) and bitwise operators (&, |) have had strictly defined precedence since the beginning days of C. But why? Whoever writes "(x | y == 0 && !z && y & x == 1)" should be dragged out into the street and shot.
That's why my languages define all the logical operators on the same precedence level, and all the bitwise operators on the same precedence level. It does allow chaining, but only for the same operator. E.g., "(x && y && z)" is legal, but "(x && y || z)" is not; likewise, "(x | y | z)" is accepted, but "(x | y & z)" is rejected. This restriction enforces the use of parenthesis whenever the interpretation is not immediately obvious. I usually also combine the precedence levels of bitwise and logical operators, so that expressions like "(x | y && y & z)" require additional parenthesis.
Unlike the other two recommendations above, this one does have a bigger impact on the EBNF grammar, "partitioning" the set of allowed expressions. I do believe the increased complexity is totally worth it, though.
In a post not too long ago, I mentioned that PulseEvent was broken. That got me to thinking: Monitor has Pulse/PulseAll methods; are they broken, too?
Many years ago, when Cygwin was young, I recall reading several articles and mailing list discussions about the difficulty of implementing a monitor (a.k.a. condition variable) on Windows platforms. It turns out that the built-in manual-reset and auto-reset events are very different than a monitor, though they appear similar at first glance. Many wrong monitor implementations have been written around a simple mutex/event pairing (sometimes with a second event), using PulseEvent.
Unfortunately, a true monitor implementation cannot be implemented simply on Windows. It is actually necessary to manage the wait queues manually in order to implement it correctly.
Fortunately, Microsoft did implement their monitor correctly. This is good, because it means I won't have to do it. ;)
I did Google around for details of Microsoft's monitor implementation, but wasn't able to find anything specifically addressing the PulseEvent problem. So, I downloaded Rotor and verified it myself. I did not verify the entire Monitor implementation - that would take a lot of time that I don't have at the moment. However, I did verify that it is not based on PulseEvent and manintains its own wait queues.
The other day, I was working on a project that had several layers of abstraction, each one exposing a purely asynchronous API (using IAsyncResult) to the next higher layer. At one point, I wanted to add an "additional handler" to an IAsyncResult.AsyncWaitHandle from a lower layer. The regular handler would take care of getting the results (calling End*), and the additional handler would perform an additional action when the operation completed.
Well, that's what ThreadPool.RegisterWaitForSingleObject is for, right? I coded it up, but there was a little nagging feeling in the back of my mind...
The catch is that IAsyncResult-based APIs use Begin* methods to construct IAsyncResult objects, and End* methods are used in place of IDisposable. So, when the asynchronous operation completes, the normal handler passes it to End*, which may include the semantics of IDisposable for that IAsyncResult object.
The problem: What happens if the IAsyncResult object is disposed while the thread pool is still waiting on its AsyncWaitHandle? More generally, could the handle be set to an unsignalled state, causing the thread pool to wait forever?
The handle in question was to a ManualResetEvent, which is a common implementation of IAsyncResult.AsyncWaitHandle (but not the only possible implementation). In my particular case, though, I thought that multiple threads waiting on a single ManualResetEvent should all get triggered if the event is set. Still, it seemed like this wasn't quite correct.
Indeed, it turns out to be wrong in the general case. Concurrent Programming on Windows (pg 231) clarified the issue nicely: when a thread is waiting on a handle, it may be interrupted by a kernel-mode asynchronous procedure call. [Many programmers are surprised that device drivers may "steal" their threads, but it happens all the time; most device drivers do not have a thread of their own, and just borrow whatever user-mode thread happens to be running when they need to do some processing]. This implies, among other things:
The situation where multiple threads are waiting on a manual reset event and one of the waiters resets the event is fundamentally broken. It suffers from the same problem as PulseEvent.
On the other hand, it turns out that ThreadPool.RegisterWaitForSingleObject does actually have a workaround for the situation where multiple threads are waiting on a manual reset event and one of the waiters closes the event. Internally, it increments the reference count for the WaitHandle's SafeWaitHandle. Then, the decrementing of the reference count is treated as an unmanaged resource wrapped in the RegisteredWaitHandle object.
This is why we do need to try to Dispose all RegisteredWaitHandle objects: to decrement the reference count of the waitable handles passed to ThreadPool.RegisterWaitForSingleObject. I had always wondered why that was recommended "even if you specify true for executeOnlyOnce".
Note: there is one other caveat for passing IAsyncResult.AsyncWaitHandle to ThreadPool.RegisterWaitForSingleObject: the same handle value can't be passed twice to that method. My particular situation never exposed the IAsyncResult to the end-user, but if it did, I'd have to DuplicateHandle the waitable handle first.
It's interesting to consider that once you bring DuplicateHandle into the mix, you actually end up with reference counted reference counts...
A bit earlier this summer, I was loading some Compact Framework code when this interesting error popped up. The error message is very detailed but completely meaningless to me.
The most fun part of this error is that it was an "infinite error" - that is, no matter how many times you would click the "OK" button, the same exact error would pop up again. Eventually, I had to kill Visual Studio to reclaim my machine.
I normally don't write a post just to link to a post on another blog, because I want this blog to be mostly original content. However, today I read an excellent article/blog post on the B# blog: Taming Your Sequence's Side-Effects Through IEnumerable.Let.
The author does a great job describing how side effects in sequences can be problematic, and how to create a "MemoizeEnumerable" to prevent multiple evaluations of a source sequence. It's a great application of dynamic programming (see The Algorithm Design Manual).
Side effects when dealing with enumerables are rare; few people have needed an "IEnumerable.Let" operator. Side effects are much more common when dealing with events, e.g., the up-and-coming Rx framework. I highly recommend reading the B# blog entry to understand the rationale behind "Let" for enumerables; it will prove good background information to understand "Let" for observables.
I haven't been blogging a great deal the last week or so. I wish I could say it was because I was studying for the 70-536, but the truth is that it's because I have a 6-week-old at home. :)
Saturday morning, I was a little nervous to be taking my first MS certification exam without adequate preparation, but I said a short prayer and hopped in the car (the closest exam site is about 3 hours away). The exam oversight guy (I'm sure he has a more proper title...) stated that I passed "with flying colors" (and seemed almost surprised!). So, I give thanks to God for that one; it sure wasn't because I was prepared for it!
I wasn't able to get the MCTS Self-Paced Training Kit, which would have been nice to have, but I did find the book Professional C# (which I already owned) to be very helpful. I was able to shore up some weak areas (e.g., caspol), but didn't have time for others (e.g., globalization).
So, now I may be an MCP. Or maybe not. The web page is pretty ambiguous on the topic. If I understand it correctly, I'm not an MCP but I'm allowed to use the MCP logo. Go figure on that one. It doesn't matter a great deal, because I've signed up for 70-505 and 70-563 in two weeks. After 70-505, I'll be an MCP/MCTS, and by the end of the day, I'll be an MCP/MCTS/MCPD. If I pass. :)
Hopefully the baby will let me study for these ones. :) Or maybe I'll finally learn that I can't do everything. :)
As someone who does a lot of asynchronous programming on .NET, I had heard about the Rx ("reactive") framework some time ago. The details online have been a little sketchy, so I finally took some time to watch Erik Meijer's talk. Below are my notes and initial thoughts. I haven't had time to play with the actual framework itself, but it's something that is definitely coming to .NET eventually (it's rumored to be in 4.0).
This talk is titled "Live Labs Reactive Framework", given by Erik Meijer at the 2009 Lang.NET Symposium.
interface IEnumerable<out T> { IEnumerator<T> GetEnumerator(); }
interface IEnumerator<out T> : IDisposable { bool MoveNext(); T Current { get; } /* Implicit throw */ }
interface IObservable<in T> { IDisposable Register(IObserver<T> o); }
interface IObserver<in T> { void OnCompleted(bool done); T OnUpdate { set; } void OnError(Exception e); }
interface IObservable<in T> { IDisposable Attach(IObserver<T> o); }
interface IObserver<in T> { void OnCompleted(); void OnUpdate(T value); void OnError(Exception e); }
class Control { event Action<T> KeyUp; }
IObservable<T> GetKeyUp(this Control w)
{
return new IObservible<T>
{
IDisposable Attach(IObserver<T> h)
{
var d = new Action<T>(h.Yield); // Create delegate to call observer
w.KeyUp += d; // Attach observer to event directly
return new IDisposable
{
void Dispose() { w.KeyUp -= d; } // Disconnect on dispose
};
}
};
}
static IDisposable Attach<T>(this IObservable<T> src, Action<T> yield); static IDisposable Attach<T>(this IObservable<T> src, Action<T> yield, Action<Exception> throw);
Action<T> handler = ...; txtbox.KeyUp += handler; txtbox.KeyUp -= handler;
var keyup = txtbox.GetKeyUp(); var detacher = keyup.Attach(...); detacher.Dispose();
var W = ... control to be dragged ...;
// Set up mouse down/up detector
var mouseDowns = from md in W.GetMouseDown() select true; // Pushes "true" every time the mouse button goes down
var mouseUps = from md in W.GetMouseUp() select false; // Pushes "false" every time the mouse button goes up
var mouseClicks = mouseDowns.Merge(mouseUps); // A simple combination; pushes "true" when down, "false" when up
// Set up mouse movement detector and measurer
var mouseMoves = from mm in W.GetMouseMove() select new { mm.X, mm.Y }; // Pushes the X,Y location of the mouse every time it moves
// (This next line has a bug)
var mouseDiffs = from diff in mouseMoves.Skip(1).Zip(mouseMoves) // This part pushes the current and last location of the mouse every time it moves
select new { dX = diff.First.X - diff.Second.X,
dY = diff.First.Y - diff.Second.Y }; // Pushes the difference in location each time the mouse moves
// Set up mouse drag & drop detector
var mouseDrag = from mousedown in mouseClicks
from delta in mouseDiffs where leftdown
select delta; // Pushes the difference in location each time the mouse moves while a button is down
mouseDrag.Attach(delta => { ... move W according to delta ... });
var mouseDiffs = from diff in mouseMoves.Skip(1).Zip(mouseMoves) // This part pushes the current and last location of the mouse every time it moves
select new { dX = diff.First.X - diff.Second.X,
dY = diff.First.Y - diff.Second.Y }; // Pushes the difference in location each time the mouse moves
should be:
var mouseDiffs = mouseMoves.Let(_mouseMoves => // This part avoids side effects on mouseMoves
from diff in _mouseMoves.Skip(1).Zip(_mouseMoves) // This part pushes the current and last location of the mouse every time it moves
select new { dX = diff.First.X - diff.Second.X,
dY = diff.First.Y - diff.Second.Y }; // Pushes the difference in location each time the mouse moves
Asynchronous programming has always been difficult. One of the hardest parts is the correct handling of "asynchronous state", objects whose sole purpose in life is to track the state of input as events come in. I expect the Rx framework will really shine at removing the need for explicitly tracking state, especially noticeable in situations where keeping "asynchronous state" is complex.
The documentation will have to be good regarding side effects, so that programmers can more easily determine when Let/Until operators are required. Currently, I'm not aware of any documentation, even on the .NET 4.0 beta MSDN. This is just a preemptive warning: the docs will have to be really, really good; better than what was done for LINQ (where the docs are often unclear which operators cause buffering).
There is still a missing piece in the "pipeline story". Historically, every language has gradually developed the same ideas: consumers, then producers, then a full pipeline. Push algorithms are usually the response to the "tee problem" in a pull-based framework. I did the same thing in C++ about 6 years ago, developing what I called "pipe algorithms", named after the Unix pipe. Currently, we have Enumerable (pull) and Observable (push) models, but there are no easy translators between the two (these "translators" require dedicated threads).
It's also important to note that one is not "better" than the other. Most algorithms are more naturally defined in an Enumerable model, so that is the one that is more intuitive. Other algorithms are a more natural fit to the Observable model. Once we have translators back and forth between the two models, then we'll have a more mature language.
One person stated that the Rx framework will remove the need for EBAP components. I have not yet been convinced of this; I see EBAP as still being a useful technique for synchronizing events. The synchronized events can then be used as input to an Rx query. In this sense, I see the EBAP and Rx approaches as complementing each other.
I spent some of this last weekend writing my first real unit tests, and I noticed by the end of the weekend that they were not exactly maintainable (a common problem for first-time unit testers). I understood the theory of unit tests, so I didn't make the mistakes of having dependent tests or testing multiple failures in a single unit test. However, my unit tests are on the long side (up to ~20 lines), mainly because I'm testing a complex threading and synchronization library (Nito.Async)
My biggest problem was naming. There are many different situations that need to be tested when working with these low-level synchronization objects (indeed, I've been enlightened that code coverage is a nearly useless metric for the Nito.Async library). As a result, there are more than 100 unit tests already, and I still have one of the more complex components to test. Even though the unit tests are grouped by component, the individual test names are still inadequate.
Take one example unit test method for the ActionDispatcher, named "TestCurrentPropertyInsideAction". Not a very good name; it does describe the situation (the Current property is accessed from inside an action), but it doesn't describe the expected behavior (that the Current property should be that action's ActionDispatcher). For that matter, the situation could read better, too. I wanted to fix this test name, but also develop a consistent naming scheme.
So, I turned to my trusty book (which I have not finished reading), The Art of Unit Testing and skipped to the tips on maintainable unit tests. Long story short: he had some recommendations I didn't particularly care for regarding naming unit tests. Unit testing is still relatively new, and "best practices" are still in development, so I turned to Google, and there are a couple of independently discovered alternatives.
Here's my opinion on three different approaches I've found. Each one is tested with my problem case "TestCurrentPropertyInsideAction" as an example.
Each of the naming conventions have a "method", "stimulus", or "subject" that they are testing. They also have a "state", "context" or "scenario" defining the situation in which the test takes place. Finally, each convention has a "behavior" or "result" that is expected. These naming conventions were developed independently, from different perspectives, and they still show a remarkable commonality. The obvious conclusion is that these three elements are criticial components in any unit test name.
This is the method recommended by The Art of Unit Testing; if you're following along at home, this pattern is also described on the author's blog.
The "Method" is the name of the method/property that is being tested. The "State" is the state of the object and parameters passed to that method. The "Behavior" is the expected behavior of that method or expected value of that property.
Original example: "Sum_NegativeNumberAs1stParam_ExceptionThrown()"
Applied example: "Current_FromInsideAction_ActionDispatcherForThatAction()"
The main reason I dislike this approach is because it has an emphasis on method and property testing. In general, I think this would lead to more "procedural tests" rather than "object-oriented tests"; I'm not talking about the unit test methods themselves (which are of course procedural), but rather about how one conceives of the component under test. This would have the side effect of writing unit tests to achieve code coverage, rather than testing state as well as behaviour.
Another reason I'm not too fond is that it does suffer from some readability problems. More readable examples would be "Sum_WhenFirstParamIsNegative_ThrowsException()" or "Current_WhenReadFromInsideAction_IsTheActionDispatcherForThatAction()".
The Stimulus/Result/Context approach is described on this blog post. The "Stimulus" is what the object is requested to do. The "Result" is the expected behavior. The "Context" is the relevant state of the object.
The original example was a bit unreadable for my taste: "CalculatePayIncludingSalesBonusAfterSalesBonusGrantedToEmployee()".
Splitting the components with underscores makes it better: "CalculatePay_IncludingSalesBonus_AfterSalesBonusGrantedToEmployee()".
Applied example: "CurrentProperty_IsSameActionDispatcherAsScopingAction_WithinActionScope()".
Still not too readable. Better examples would be "CalculatedPay_IncludesSalesBonus_AfterSalesBonusGrantedToEmployee()" and "CurrentProperty_IsSameActionDispatcherAsScopingAction_WhenWithinActionScope()".
The main reason I don't like this approach is that it places the result in the middle and the context last. I think it's more natural to think of context in the middle and the result last. For example, I think it's more natural to state "if the car is exceeding the speed limit, then it becomes eligible to receive a ticket" rather than "a car becomes eligible to receive a ticket if it exceeds the speed limit". This is personal opinion (like the rest of this post!); I just prefer "if/then" rather than "then/if".
Another reason is that it does seem to emphasize testing behavior rather than state. It is, however, more object-oriented than option 1.
Described here, this approach seems to combine a data-oriented mindset with placing the result last.
The "Subject" is the item under test. The "Scenario" is the context of the test. The "Result" is the expected result of the test.
Original example: "ProductPurchaseAction_IfStockIsZero_RendersOutOfStockView()".
Applied example: "Current_FromInsideAction_IsActionDispatcherForThatAction()".
This naming convention is more data-centered (note the "ProductPurchaseAction" subject), so it should work well with MVVM-based designs. If "subject" could be interpreted as "object method" as well, then this approach could be called more object-oriented.
In short, this option is a more readable and more object-oriented version of the first option. I've decided to go forward with this one for my project; your mileage may vary.
_1371_1372_1260_1258_1259.png)
Posts
All Comments
