2011-06-30

Option Parsing: Option Names

An option may have a long name, a short name, or both. "Short names" are just single characters, while "long names" are strings. Option names may not contain the special characters : or =.

Commonly-used options should have both a long name and a short name. The short name enables faster typing on the command line, while the long name enables self-documenting command lines (for use in script and batch files). Normally, the short name is the first character of the long name, but this is not required.

Less-common options should have just a long name; this avoids polluting the short name namespace.

using System;
using Nito.KitchenSink.OptionParsing;

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("level", 'l')]
    public int Level { get; set; }

    [Option("priority")]
    public int Priority { get; set; }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();
      
      Console.WriteLine("Level: " + options.Level);
      Console.WriteLine("Priority: " + options.Priority);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}
> CommandLineParsingTest.exe
Level: 0
Priority: 0

> CommandLineParsingTest.exe /l 3
Level: 3
Priority: 0

> CommandLineParsingTest.exe /level 3
Level: 3
Priority: 0

> CommandLineParsingTest.exe /priority 1
Level: 0
Priority: 1

> CommandLineParsingTest.exe /p 1
Unknown option  p  in parameter  /p

Normally, options do not have just a short name without a long name, but you can do it if you want do.

Multiple Long and Short Names

Options may have "aliases" (multiple long and/or short names). The easiest way to add aliases is to have separate properties on your Option Arguments class that refer to the same underlying field.

The following example shows one alias that is used to change an old option "level" into a more descriptive option "frob-level", marking the old option as obsolete. Another alias "frobbing-level" is also added, which is just a regular alias (without any options marked obsolete).

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    // The old "level" option, now obsolete and made into an alias for "frob-level".
    [Option("level", 'l')]
    [Obsolete]
    public int Level
    {
      get { return FrobLevel; }
      set
      {
        Console.Error.WriteLine("Warning: The --level option is obsolete; use --frob-level instead.");
        FrobLevel = value;
      }
    }

    [Option("frob-level")]
    public int FrobLevel { get; set; }

    // Another alias for "frob-level"; this one is not obsolete.
    [Option("frobbing-level")]
    public int FrobbingLevel
    {
      get { return FrobLevel; }
      set { FrobLevel = value; }
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();

      Console.WriteLine(options.FrobLevel);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return 2;
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}
> CommandLineParsingTest.exe /level 4
Warning: The --level option is obsolete; use --frob-level instead.
4

> CommandLineParsingTest.exe /frob-level 4
4

> CommandLineParsingTest.exe /frobbing-level 6
6

Abbreviated Option Names

Some programs support abbreviated option names; for example, the option "pack" may be abbreviated as "pa" or "p" (assuming there is no other option that starts with "pa" or "p", respectively). However, this causes backwards compatibility issues; for example, an updated version of the program may introduce an option named "push", and any scripts that used the abbreviated option "p" then become ambiguous. For this reason, Nito.KitchenSink.OptionParsing does not include automatic support for abbreviated option names. If you need abbreviated option names, you may use explicit aliases to achieve the same effect.

> CommandLineParsingTest.exe /frob 6
Unknown option  frob  in parameter  /frob

2011-06-23

Option Parsing: Error Handling

The Nito.KitchenSink Option Parsing Library wraps all option parsing errors into an exception derived from Nito.KitchenSink.OptionParsing.OptionParsingException. There are three more specific exception types (InvalidParameterException, OptionArgumentException, and UnknownOptionException), but they are seldomly needed.

All steps of the option parsing pipeline should only throw exceptions derived from OptionParsingException. In particular, this is true for custom validation (which will be described in detail in a future post).

The following example program shows how option parsing errors should be handled in a console application:

using System;
using Nito.KitchenSink.OptionParsing;

class Program
{
  private sealed class Options : OptionArgumentsBase
  {
    [Option("level", 'l')]
    public int Level { get; set; }

    public static int Usage()
    {
      // Standard console size:
      //                      [                                                                                ]
      Console.Error.WriteLine("Usage: CommandLineOptionTest <arguments>");
      Console.Error.WriteLine("  -l, --level=LEVEL        level at which to operate");
      return 2;
    }
  }

  static int Main()
  {
    try
    {
      var options = OptionParser.Parse<Options>();
      
      // Program logic
      Console.WriteLine(options.Level);

      return 0;
    }
    catch (OptionParsingException ex)
    {
      Console.Error.WriteLine(ex.Message);
      return Options.Usage();
    }
    catch (Exception ex)
    {
      Console.Error.WriteLine(ex);
      return 1;
    }
  }
}

First, the Options class is declared, which defines the options our program takes. It also exposes a Usage method, which displays command-line usage information. Usage writes its information to Console.Error and returns an error code.

The Nito.KitchenSink.OptionParsing library does not attempt to write the Usage method for you automatically. Other option parsing libraries have attempted this, but the results are (IMHO) less than ideal.

The program's Main method returns an int, and contains a top-level try/catch. The try block parses the options, performs its requested task (in this case, the program just writes the Level option to the console), and then returns 0 (meaning "success").

If there is an option parsing exception, then the exception message is written to Console.Error, usage information is displayed, and an error code is returned.

If there is some other (unexpected) exception (during option parsing or program logic), then the entire exception (including the call stack) is written to Console.Error and an error code is returned.

Notes

For console programs, a return value of 0 indicates success and any other return value usually indicates an error. I used two different error codes in the example above, but they could just as easily be a single error code because distinguishing usage errors is not normally useful.

An options class does not have to include a Usage method; I just usually put it there so it's along with the class that defines the options. In future blog posts, I'll post example code that skips the Usage method to avoid distractions, but it should be included in real-world code.

2011-06-16

Option Parsing: Lexing

The first step in parsing a command line is lexing, which converts a single string (the command line) into a sequence of strings (individual options and/or arguments). Actually, the very first step takes place before the program even runs: the command shell has its own simple lexer.

Command Shell Escaping and Quoting

The information in this section is derived from the TechNet articles Command shell overview (webcite) and The Windows NT Command Shell (webcite).

The command shell has these special characters: &, |, (, ), <, >, and ^. There are two ways to pass these special characters on the command line: escaping and quoting.

The ^ character is the shell escape character. You may prefix any of the special shell characters with that escape character, and the special shell character will be passed to the program (without the escape character).

The command shell also supports quoting; special characters may be passed within a pair of double-quotes. In this case, the special characters are passed to the program along with the surrounding quotes.

The shell escaping and quoting appears to be a simple algorithm: escaped characters (including normal characters) are passed through directly, and each (non-escaped) double-quote either starts or ends a quoted string. Consider the outputs from this example program:

static void Main(string[] args)
{
  Console.WriteLine(Environment.CommandLine);
}
> CommandLineParsingTest.exe ^^ "^"
CommandLineParsingTest.exe ^ "^"

> CommandLineParsingTest.exe ^"^"
CommandLineParsingTest.exe ""

> CommandLineParsingTest.exe ^""
CommandLineParsingTest.exe ""

> CommandLineParsingTest.exe "^"^"
CommandLineParsingTest.exe "^""

> CommandLineParsingTest.exe "^"^^"
CommandLineParsingTest.exe "^"^"

> CommandLineParsingTest.exe "^^
CommandLineParsingTest.exe "^^

Shell escaping and quoting are applied to every process by the Command Shell; there is no way to opt out of this behavior. After the command shell does its own escaping and quoting, the command line is passed to the program.

Default .NET Lexing

The command line is split up into a list of process arguments by the .NET runtime. The algorithm is described in the documentation for Environment.GetCommandLineArgs. The same results (except for the process name) are also passed as the single argument to the Main method, if present.

The .NET lexing also uses a combination of escaping and quoting, but it has some surprising results because escaping is allowed inside quoting. The escape character is \, and the quote character is the double-quote.

Each non-escaped double-quote starts or ends a quoted string, just like command shell quoting. However, unlike command shell quoting, escaping is allowed within quoted strings. The .NET lexing also allows two consecutive double-quotes inside a quoted string to represent a single double-quote. Consider the outputs from this example program:

static void Main(string[] args)
{
  foreach (var arg in args)
    Console.WriteLine(arg);
}
> CommandLineParsingTest.exe "a"
a

> CommandLineParsingTest.exe \"a"
"a

> CommandLineParsingTest.exe \"a
"a

> CommandLineParsingTest.exe "a\"
a"

> CommandLineParsingTest.exe "a\\"
a\

> CommandLineParsingTest.exe a \"
a
"

> CommandLineParsingTest.exe "a \\"
a \

> CommandLineParsingTest.exe "a\"b"
a"b

> CommandLineParsingTest.exe "a""b"
a"b

> CommandLineParsingTest.exe a "" """"
a

"

This lexing behavior is particularly problematic when passing directories. Since directories may contain spaces, they should be wrapped with quotes. However, if the directory ends with a backslash, the closing quote will be escaped:

> CommandLineParsingTest.exe "c:\my path"
c:\my path

> CommandLineParsingTest.exe "c:\my path\"
c:\my path"

This is a rather serious limitation of the default .NET lexer. It is possible to write your own replacement lexer using a different algorithm. This lexer would take the process command line as input and produce a sequence of strings.

The Nito.KitchenSink.OptionParser library does not have a lexer of its own, but it will accept a sequence of strings as input into its parsing methods. If no sequence of strings is passed to a parsing method, then the method will use the process' command line lexed with the default .NET lexer.

2011-06-09

Option Parsing: The Option Parsing Pipeline

There are three main phases during option parsing:

  1. Lexing
  2. Parsing
  3. Validation

The Lexing phase deals with the escaping and quoting of special characters and splitting the command line string into a sequence of strings. The Parsing phase evaluates the sequence of strings from the lexing phase, and interprets them as options and arguments; this includes parsing arguments as necessary, e.g., converting a string argument "3" into the numeric argument value 3. The Validation phase determines if the options and arguments represent a valid command for the program to perform.

The Nito.KitchenSink.OptionParser library does not have a lexer, but does have a parser and hooks for validation. The easiest way to use the library is by calling a single method:

var options = OptionParser.Parse<MyOptionArguments>();

This single method wraps all the phases of the option parsing pipeline:

  1. The command line for the process is lexed using the default .NET lexing.
  2. The option and argument definitions are inferred from properties and attributes on the MyOptionArguments type.
  3. These definitions are used to parse the lexed command line, saving the results into properties on a default-constructed MyOptionsArguments object.
  4. Validation is performed on the MyOptionsArguments object, which is then returned.

Future posts will show how each of these steps may be configured (or replaced).

2011-06-02

How to Run Processes Remotely

Today I'm going to delve deeply into something I discovered many years ago (c. 2003). It's an interesting little trick that hopefully no one will ever have to use.

When a process running on one computer needs to perform some operation on another computer, the common solution is to actually have two processes that use interprocess communication. The one process sends its commands to the other process, which executes them on behalf of the first process. Normally, one must install a server on one computer and a client on the other. So, if someone needs to perform an operation on another computer, then that computer must already have the software installed.

However, there is a way to send a program to a remote computer and run it, without having any special existing software on the target machine. This approach doesn't work in every situation, but it's useful to know. The command line programs in the famous PSTools suite use the approach documented here to "inject" copies of themselves onto remote computers; this allows a simple form of remote administration. The white paper PsExec Internals (webcite) includes the specific details for PsExec.

Step 1: Establish an Authenticated Connection

About Connections

A user session on one computer may have network connections to other computers. One common example is network drives; each network drive is a connection to another computer. Network connections may also exist without mapping a drive letter.

Network connections may be examined and modified using the Windows Networking (WNet) API or the net command. Unfortunately, there are no .NET wrappers for this API in the BCL.
About Authentication

Each network connection has to be authenticated, but there are situations where this happens automatically. When you map a network drive using Explorer, by default Windows will use your local logon to attempt to log onto the remote machine, and if it's accepted, you won't actually get prompted for credentials. This is particularly common in Domain environments.

The net use command allows you to display current connections to other computers, and add or remove those connections.
Authentication Quirks

Microsoft made the design decision that any number of network connections may exist between two different computers, but that the same credentials must be used for all those connections. You may use different credentials for connections to two different servers, but all connections to the same server must use the same credentials. According to a rather dated KB106211 (webcite), this is done "for security purposes." The newer KB183366 (webcite) documents the limitation in more detail, but does not give a reason.

If you do attempt to use different credentials for different connections to the same server, you'll get a 1219 error: "Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed. Disconnect all previous connections to the server or shared resource and try again." I've also seen this error when Explorer tries to auto-reconnect its mapped drives and it gets confused; it appears to happen more commonly on wireless networks when resuming from a low-power state.

There's a "greybeard" trick used to get around this limitation: connect to the IP address instead of the hostname (or, if you want more work, set up multiple hostnames for that server). The logic behind "the same server" appears to be just a string comparison. This workaround has been documented in KB938120 (webcite).

There are some notable situations where it's not possible to establish an authenticated connection:

  • If the target (server) machine is running a client OS "Home" edition (e.g., XP Home, Vista Home Basic, Vista Home Premium, Windows 7 Home Basic, Windows 7 Home Premium), then no authenticated connections are possible.
  • If the target (server) machine is running a client OS "Professional" edition (e.g., XP Professional, Vista Business/Enterprise/Ultimate, Windows 7 Professional/Enterprise/Ultimate), then that machine must either be a member of a domain or turn off "simple file sharing" to support authenticated connections.

Note that if you're working in a domain enviroment, Everything Just Works. For the rest of us, we have to turn off "simple file sharing."

If the server is running a Home edition, or if it is not connected to a domain and is using simple file sharing, then it does not support authenticated connections. Instead, every incoming network connection is authenticated with the Guest account; see KB300489 (webcite).

Another non-authenticated approach is to use null sessions, which are truly anonymous. This means they work even if the Guest account is disabled. Null sessions are disabled by default and considered a security risk.

To send a program to a remote computer, you'll need an authenticated connection. A Guest authentication (or null session) is insufficient.

Common Shares

There are some hidden network shares for Windows systems. They are recreated automatically on reboot if they've been deleted.

Hidden shares are not shown in the normal GUI, but they can be displayed by the command net share.

The standard hidden share names that are important to us are:

  • IPC$ - An share that is used only for authentication.
  • ADMIN$ - The equivalent of %SYSTEMROOT% (usually "C:\Windows").
You can create your own hidden shares: KB314984 (webcite). You can also prevent the automatic creation of the standard hidden shares: KB954422 (webcite), but this may cause lots of problems: KB842715 (webcite).

With all of that background information, our first step is to actually establish the authenticated connection to \\computer\IPC$. The other steps are quite simple in comparison!

Step 2: Copy the Program to the Target

Just copy the program to \\computer\ADMIN$, right into the Windows directory. I recommend renaming the file during the copy to a unique name, to avoid conflicts. You don't need to explicitly establish a network connection to \\computer\ADMIN$; the existing connection to \\computer\IPC$ will be your authentication.

Step 3: Register and Execute the Program

This step makes use of the little-known fact that Win32 services may be installed remotely. The service configuration API can be used to install the service on the remote computer and then start it.

The .NET ServiceController class does expose remote control of services (starting, stopping, etc), but it does not expose remote installation of services.

Step 4: Securely Communicate

Once the service is running on the remote computer, it is simple matter to communicate with the original process and carry out its instructions. It's not quite as simple to do so in a secure manner, though; strongly consider encrypting all network communication and using impersonation in the service.

Also remember that - as a service - you are limited in what you can do.

Enjoy!

There aren't too many good use cases for this technique. Remote administration is one, as demonstrated by the PsTools suite from Microsoft TechNet Systems Internals.

Another possible application is to inject an installer for remote control software, such as VNC or pcAnywhere. This could be useful in the rare case where a computer is physically inaccessible.