Tuesday, November 29, 2016

SourceTree Authenticate prompt

September 2021 note: SourceTree is a clumsy bug-ridden mess that has only marginally been improved in the four years since I made this post. The great news is that you can uninstall and delete SourceTree from your mind because Visual Studio now has all the Git source control functionality you need integrated into it.


Sometime in the last few weeks SourceTree 1.9.x for Windows started popping-up the Authenticate dialog prompt, over and over and over and over... You could you enter your password until hell froze but it just kept asking.

I noticed that Tools > Options > Authentication had an empty list of saved passwords, whereas for the last couple of years I remember seeing my user name in there.

Countless web searches on this problem revealed lots of complaints about it, but all the discussions and suggestions were utterly irrelevant or useless. Going through the tedious steps of installing SSH keys did not help.

Out of frustration and disgust I decided to uninstall and purge SourceTree completely from my PC. I noticed that there were 3 icons with different versions in the Programs and Features list and they did not uninstall cleanly. Then I searched my whole C: drive for anything with the word 'SourceTree' in the name and deleted it. I then ran regedit as Admin and similarly deleted any keys or values containing 'SourceTree'.

After reinstalling the most recent copy of SourceTree and entering my fresh credentials the problem went away and the program now seems to be running trouble free.

SourceTree issues an irritatingly frequent number of incremental updates, which I usually accept, so I'm guessing that an accumulation of these updates has corrupted something. I'm angry that I had to use such a ham-fisted fix for this, but I document it here in case it helps others who suffer this problem.

Saturday, November 19, 2016

SQLite and Dapper ORM

This morning I tried to read a SQLite database for the first time in at least 3 years. I remember the last time I tried to integrate SQLite into a C# project with Entity Framework 6 and designer support I just about went mad. It took hours and hours of searching and bumbling around to eventually get the right combination of references and config values. I even restored a backup of an old project that used SQLite, and I couldn't get it working again, probably due to missing dependent installs.

This time I went looking for a simpler way, with minimal dependencies and simpler code. Remembering a hint from someone several weeks ago in the ozdotnet forum I searched for "SQLite Dapper" and came upon StackExchange/dapper-dot-net. This lightweight ORM has a tiny footprint of a single DLL and one class that adds extension methods to IDbConnection, so it's mostly database agnostic. The readme.md on the main page contains a neat summary of how to perform basic database and query operations. The code is so terse that some of it seems "magical" and you need to read the samples carefully to see the pattern.

As a sanity check I attempted to read the Songs table in a MediaMonkey db3 database file. I made a POCO class which is a subset of the dozens of table columns.

public class Song
{
  public long ID { get; set; }
  public string Album{ get; set; }
  public string SongTitle { get; set; }
}

Add Nuget packages Dapper and System.Data.SQLite.Core to your project. The following code enumerates all the Songs table rows.

using (var connect = new SQLiteConnection(@"Data Source=M:\Monkey.db3"))
{
  connect.Open();
  foreach (var song in connect.Query<Song>("Select ID,SongTitle,Album from Songs", buffered: false))
  {
    Info($"{song.ID} {song.SongTitle} - {song.Album}");
  }
}

Dapper is a long way from being a full ORM, but you can still easily perform all of the typically needed database operations in a strongly typed way. You can even do eager loads over joins, a bit like the Include extension method used in Entity Framework queries (using a curious syntax).

Best of all, you only need two small Nuget packages, there is no need to put anything in a config file, you can learn to use it in minutes and the code is brief.

Monday, October 31, 2016

Wordpress install on Windows

WordPress claims to have the famous 5-minute installation, but where is it? Attempting to install WordPress in IIS on a modern Windows server is a ridiculously confusing process for the uninitiated. If you go to wordpress.com you're invited to create an account and a hosted site. If you go to wordpress.org you may eventually stumble upon the Installing WordPress page which tells you to unzip the raw files, create a database, rename a file and run the famous installation process. However, the Minimum Requirements specify PHP and MySQL, and without those you can't even get started.

If you go searching for installers for PHP and MySQL you will find many versions with conflicting or confusing information. The PHP installer I found was simply a zip with no clues about how to integrate PHP into IIS. The MySQL installer wizard offers dozens of optional components and you have no idea which ones are required to support WordPress.

After hours of searching and stumbling around I eventually found this Install WordPress for Windows page that suggests you install Microsoft Web Platform Installer 5.0, go to the Applications tab, Add WordPress, click Install and follow the reasonably sensible instructions. The installation adds a PHP handler to IIS and configures a MySQL database. The first time you visit the index.php page the famous 5-minute installation begins and you will be asked some more questions, after which you login to the site as an administrator where you can manage the site and its contents from the Dashboard page.

So overall, the trick is to use the Web Platform Installer to bootstrap the whole process. I suspect there are different ways of achieving the same result.

Wednesday, July 6, 2016

Windows Universal String.GetHashCode

Since late 2021 you can use the XxHash classes in the System.IO.hashing namespace for generating 32-bit and 64-bit hashes. There is no need to use the custom hashing code below. The namespace also includes classes for the CRC-32 and CRC-64 algorithms.

I thought I was going barking mad because a user settings value I persisted with a hashed key kept coming back empty every time I restarted my App. After wasting half and hour on traces I could see that the same string produced different GetHashCode values every time I ran the App. Impossible?!

Now I understand perfectly that the implementation of String.GetHashCode might change on different platforms or different .NET Framework versions, but across different runs of the same App ... it's inconceivable. It's true. Read the documentation or run a web search and be bewildered.

I'm utterly gobsmacked by this change of behaviour. The vast change of scope of the predictability of the hash codes in Windows 10 is a devious trap for the unprepared.

Technically it's true, you shouldn't persist string hash codes, but I always thought it was okay to stretch the rules a little bit and use them in something with a relatively short lifetime like user settings which might live for days or weeks. Now the rules are locked down tight ... don't persist hash codes outside the lifetime of an AppDomain.

So what do you use instead of the really convenient String.GetHashCode? I immediately needed some short, simple and fast code to hash my string keys. Luckily I have hobby researched this subject before and published the results here: Hashing Short Strings.

Following my own advice, I picked the following code which combines the FNV-1a hash with a "quick and dirty" pseudo-random sequence.


static int StableHash(string value)
{
  uint hash = 2166136261u;
  for (int i = 0; i < value.Length; i++)
  {
    hash ^= value[i];
    hash = hash * 16777619;
    hash = (1664525 * hash) + 1013904223;
  }
  return (int)hash;
}

Put this method in a handy utility library. The string hash codes it produces are actually more collision resistant and more evenly distributed than String GetHashCode.

For a 64-bit version of stable hashing use this.

static long StableHash64(string value)
{
  ulong hash = 14695981039346656037UL;
  for (int i = 0; i < value.Length; i++)
  {
    hash ^= value[i];
    hash = hash * 1099511628211UL;
    hash = (2770643476691UL * hash) + 4354685564936844689UL;
  }
  return (long)hash;
}


Thursday, June 23, 2016

Parsing Color strings

This is a reminder to myself about how to parse strings like "MistyRose" and "#ffe4e1" into a strongly typed Color. I haven't performed extensive tests, but all of the methods here seem to support the x11 Color Names and #RGB format.

WPF

Color c = (Color)ColorConverter.ConvertFromString(string value);
You have to cast the result because the return type of this static method is object. There are overloaded instance methods.

System.Drawing

Color c = ColorTranslator.FromHtml(string htmlColor);

Wednesday, June 22, 2016

DataContractSerializer ReadObject

I recently authored a WebApi REST style service in Visual Studio 2015 which is being consumed by various client apps without any problem. Out-of-the-box WebApi services provide JSON and XML formatters, with JSON being the default one. The first clients apps for my service were .NET desktop apps and one written in Xamarin, and they all used the ubiquitous NewtonSoft library to exchange JSON serialized objects in the requests and responses from the service. This usage of JSON to serialize messages between REST services and client apps is a popular and convenient pattern in recent times.

However, I did have an older client app that needed to call my new service, but its coding style was all based on XML and I felt it would be inelegant or impure to add new code that used JSON and the NewtonSoft libraries. Since a WebApi REST service has implicit support for both XML and JSON, this was a good chance to prove that the service was fully functional for XML based clients. By putting ?format=xml in GET urls and adding Content-Type: application/xml to POST headers the client could override the JSON default serializer and demand XML formatting instead.

Of course it works. I could issues a GET request and the response contains a complex .NET class object serialized as XML. A typical response might look like this:

<AuthenticationData
        xmlns:i="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://schemas.datacontract.org/2004/07/RCS.Licensing.WebApi.Portable">
  <Jobs>
    <Job>
      <CaseLimit>0</CaseLimit>
      <Cases>10000</Cases>
      <Client i:nil="true" />
  --cut--

Note the formatting of the XML which contains interesting namespaces and the use of i:nil to represent null values (which is actually a W3 standard that developers seem to forget). I absent-mindedly tried to use the XmlSerializer class to deserialize this XML, and after adding the namespace it seemed to be working well ... until I noticed all the i:nil values where coming back as zero-length strings. Clearly I was using the wrong technique.

The I noticed (and remembered, see highlight text above) that the WebApi uses the data contract serializer, not the XML serializer. Simply use DataContractSerialize ReadObject and the XML coming back in WebApi responses will unwind the way you want.

Tuesday, March 15, 2016

Nuget and AssemblyInformationalAttribute

After uploading some updated Nuget packages I noticed they did not appear in the Visual Studio 2015 nuget.org browse results or in any package lists from the command line tools. A day later I was quite worried, so after running a bulk file comparison with the versions that were last visible in Nuget I noticed only one suspicious difference: I had added an AssemblyInformationalAttribute to the projects. I had placed a build time string in this attribute, something I thought would be handy when looking at the properties.

Web searches soon revealed that this attribute has special meaning to Nuget (see Nuget Versioning). My value did not match any conventional format mentioned in the article, so there's no way of guessing how it was interpreted. The unfortunate result was that the packages were perhaps regarded as some kind of pre-release, but they didn't show up even when that browse option was ticked.

I always assumed that this attribute was simply an additional comment for annotating executable images and had no special meaning to Windows or Nuget. It's easy to miss the Nuget "fine print" regarding this attribute. Don't use it unless you've read the documentation.

Sunday, February 21, 2016

Hide Windows 10 useless folders

I just migrated over to a fresh development PC running Windows 10. It took many hours of work to remove most of the "junk" that clutters the desktop, processes and apps. I removed all tiles from the Start menu. I uninstalled every modern app that's possible (news, weather, sport, etc). I disabled services that will be unused. I disabled the firewall and Defender. Using Sysinternals tools I found startup items and running processes that were useless. And so on ... for hours. I still haven't figured out how to disable or remove Adobe Flash yet, but I swear I will, despite the fact they've baked it right into the operating system.

One challenge was to shrink the number of folders in Windows Explorer. You get a Quick access node at the top containing Desktop, Downloads, Documents, etc, but these folders are duplicated under This PC, which a dreadful waste of space. Web searches on this revealed lots of stupid and dangerous ideas, but finally I found a chap who correctly assumed that the appearance of the Windows Explorer tree was controlled by the Registry, and he found the correct keys to adjust and hide the useless folders. Look here with admin permissions:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft
  \Windows\CurrentVersion\Explorer\FolderDescriptions

Find the child Guid nodes containing a REG_SZ Name of the folders you want to hide. There will be a child key PropertyBag\ThisPCPolicy which you change from Show to Hide. One entry didn't have the ThisPCPolicy key and I had to add it manually.

Setting the values to Hide takes affect as soon as you relaunch Windows Explorer. You can see in this picture how I had removed a few folders and was about to remove the last two, leaving a much cleaner looking tree.


One overarching question needs to be asked here ... Why am I doing this? I'm not the only person who is "cleaning up" Windows 10 to remove all the clutter and junk. It hints that Microsoft's marketing and design departments are losing touch with reality. Overall, Windows is getting dumber and dumber, progressively looking and feeling more like an iMac (maybe that's what they want!).

There are so many subtle changes to Windows 10 that make life harder for power users and developers. Many actions and commands in Windows 7 require more mouse clicks, scrolling and navigation in Windows 10. I've been forced to pin many apps to the start bar or create desktop icons to allow quick Run as Administrator.

Addendum #1

While continuing to disinfect Windows 10 I stumbled across this fabulous article by Chris Hoffman which tells you how to use Powershell to uninstall the pesky modern apps like Camera, Groove Music, Get Skype, Music & TV, People, Weather, etc, that can't simply be uninstalled. The effect of running his commands is beautiful: all the built-in modern app garbage is erased. Once again ... why am I doing this?!

Addendum #2

I boot my PC the following morning and Windows 10 told me that updates had been installed and all my files were exactly where they used to be. The screen background had gone blank, and absolutely everything I did in the steps above had been undone. All folder icons and all Windows 10 apps had been restored.

Addendum #3

Someone in the forum has hinted that the reversal of my changes was an unfortunate coincidence, possibly caused by a major 10.1 automatic update. The update history has no entries to confirm this. My reapplied changes have been preserved for a couple of weeks now.

Sunday, February 14, 2016

Web API secret plumbing

August 2021 Note: Most of this post is no longer meaningful after the arrival of .NET Core with significant changes to the ASP.NET libraries. The note below about dropping WebForms and preferring MVC to write web apps is now completely outdated and redundant. The arrival of Blazor Webassembly has driven a stake into the heart of server-side ASP.NET apps. I can write a Blazor app in about one fifth the time and with one fifth the code compared to a similar ASP.NET app. Unfortunately, we are still stuck with ASP.NET Web API for RESTful data services (although Azure Function apps are a good alternative).


February 2017 Note : When I wrote this post a year ago I stated that I would not use ASP.NET MVC to write web applications, only WebAPI services. I have changed my mind and in future will drop WebForms and switch to MVC. I eventually got sick of the complicated pipeline of WebForms events and the lack of control over the rendered HTML, which is often an incomprehensible mess. I was also sick of getting hundreds of KB of ViewState for simple grids, and you can't disable the grid's ViewState or commands no longer fire.
However, if you create a skeleton MVC app from a Visual Studio 2015 template you get about 30 references to libraries and NuGet packages, most of which are JavaScript framework garbage. For a simple web app that needs little to no JavaScript you have to tediously remove packages and their dependencies in the correct order, then delete useless references and files. You can eventually strip the project down to a sensible size and start developing without all the clutter.

A few years ago I became fed-up with how complicated ASP.NET WebForms programming was. The pipeline of events is really long and delicate. If you do something in the wrong order you may get events that don't fire, controls that don't update, missing postback data, and so on. I've spent many tearful hours of my life trying to find out why a row click in a grid doesn't work to eventually discover I was doing something in the event handler that should have waited until PreRender, or vice versa. Another problem is the lack of control over formatting, exemplified by a tiny single-page app I wrote few months ago containing just a grid and a few buttons which renders an incomprehensible mess of html.

A few years ago, ASP.NET MVC was getting a lot of good comments and publicity so I ran the tutorials and bought some books in the hope that I would find the whole web programming model had been simplified with a tiny pipeline and you had more control over the rendered output. I found this to be generally true, but there was still "secret plumbing" all over the place. Magic values and naming conventions had simply replaced the difficulty of WebForms programming with a new type of mysterious difficulty. Everything I wanted to do in ASP.NET MVC required searching for a magical piece of plumbing that had to be named or registered.

After a few months I gave up in disgust at the prospect of using ASP.NET MVC. I just can't imagine what twisted minds can take something as simple as the http request-response model and wrap it in so many layers of complexity. I have been tempted at times to return to the early 1990s and write a web app with a single ashx handler file that takes the raw request and converts it into a raw response, thereby skipping all plumbing and frameworks.

Although I refuse to write web applications using ASP.NET MVC, I do write web services of the ASP.NET Web API variety which share a lot of coding style with MVC using routes and controllers. Even through the Web API is a stripped down MVC, it still unfortunately shares a fair amount of "secret plumbing" that you have to discover and use correctly. I have stumbled upon some important "secrets" which I'm putting in this post as a reminder to myself and anyone else who cares.

ActionFilterAttribute

Write a class that derives from ActionFilterAttribute and override OnActionExecuting and OnActionExecuted so you can automatically intercept the request and response of every API call. You can, for example, validate requests with code in a single place instead of spreading it all over the controller methods. You can extract request information like an auth token and place it in the request's property collection so it's handy at all times. Anything you want to intercept or do automatically on all API calls can be placed in your action filter class. Register the filter on controller methods like this example:

[HttpDelete]
[Route("")]
[MyActionFilter(...)]
public IHttpActionResult Delete()
{
  ...
}


You can pass parameters to the filter's constructor if it's useful.

ExceptionFilterAttribute

Write a class that derives from ExceptionFilterAttribute and override OnException to automatically intercept all errors without the need to spread try-catch code all over the controllers. You can log errors or transform them into different types of responses that are perhaps more friendly for callers of your API. You register the exception filter the same way as the action filter.

HttpParameterBinding

I uncovered this devious and tricky class while trying to automatically convert different types of query values into a controller method's bool parameter. It's quite sensible to allow API callers to specify a true/false value as Y, y, N, n, false, TRUE, 0, 1, and so on. I guessed there was a way of automatically coercing query values like &blah=1 or &blah=y into a bool value, but I had to find the secret plumbing that did this. An obscure hint in a web search pointed me to the HttpParameterBinding class, but further searches revealed ambiguous and confusing use of the class and the context parameters available to it. By trapping a call in the debugger I eventually managed to figure out just which context values were required to perform the transform I required. Here is the stripped down code that does what I want.

public override Task ExecuteBindingAsync(...)
{
  var dict = HttpUtility.ParseQueryString(actionContext.Request.RequestUri.Query);
  string raw = dict[Descriptor.ParameterName];
  bool? value = null;
  if (Regex.IsMatch(raw ?? "", $"^(?:y|true|1)$", RegexOptions.IgnoreCase))
  {
    value = true;
  }
  else if (Regex.IsMatch(raw ?? "", $"^(?:n|false|0)$", RegexOptions.IgnoreCase))
  {
    value = false;
  }
  actionContext.ActionArguments.Add(Descriptor.ParameterName, value ?? Descriptor.DefaultValue);
  return Task.FromResult(0);
}


The raw string value named in Descriptor.ParameterName is parsed out of the request query string, inspected and transformed into the required bool value and placed in the actionContext.actionArguments collection for passing into the controller method. You can use this sort of parameter binding code to transform any incoming value into another type. You will need to create a small custom Attribute to apply your binding to a parameter, then use it like this:

internal class BoolBindingAttribute : ParameterBindingAttribute
{
  public override HttpParameterBinding GetBinding(HttpParameterDescriptor parameter)
  {
    return new BoolParameterBinding(parameter);
  }
}

public IHttpActionResult Foo([MyBoolBinding] bool blah)
{
  ...
}


So if a request contains &blah=1 it will be silently transformed into true.

AddQueryStringMapping

It is a web service convention that the response body format obey the Accept request header MIME type value. So for example sending the header Accept: application/xml will result in an XML response body. For the convenience of callers, and for easier testing, it's possible to map a query parameter to a MIME type. In the static WebApiConfig Register method, add lines like these popular examples:

config.Formatters.JsonFormatter.AddQueryStringMapping("format", "json", "application/json");
config.Formatters.XmlFormatter.AddQueryStringMapping("format", "xml", "application/xml");

You can then simply add ?format=xml to a request query to get the same effect as adding the corresponding request header.

WPF Binding Recommendations

I just audited the binding code in a reasonably large WPF application I wrote in July 2015 and was embarrassed to find lots of subtle bugs. Some controls were not bound to the specific properties they depended upon, some controls were bound to irrelevant or incorrect properties, and some binding properties were orphaned and unused.

The Visual Studio designer and the compiler can't fully bridge the gap between XAML and code, so human error is bound to creep in. You may accidentally cut-and-paste the same binding property into two different places, you may misspell a property, and so on. Many errors like this will not show up in the designer and you may only detect them by looking for binding error messages in the Visual Studio output window. Despite the danger of coding errors, I have set the following vague rules for myself to help binding work more reliably:
  1. Bind whenever it's possible.
  2. Put all binding logic in the converters.
  3. Don't forget to use the powerful MultiBinding.

When Possible

Although the syntax of binding is verbose and it's implementation is clumsy and incomplete, it's a great advance in the separation of business logic from UI behaviour. In the 1990s you could use MFC/C++ Data Exchange macros and classes in an attempt to move data in and out of the UI, but it was tedious and unnatural. In 2005 I participated in writing a huge Windows Forms app where we simulated WPF style binding in code. It generally worked, but the amount of code (and hacks) was staggering. A year later, Framework 3.0 and WPF were released, which made most of our WinForms code redundant. So modern binding is a great facility and you should use whenever possible.

Sometimes you'll find properties that can't be bound, a good example is the DataGrid's SelectedItems property. Some people use clever tricks like custom attached properties and behaviors to bind the unbindable, but I think this is overkill and I prefer to use a little bit of code-behind to fill the gap. With the DataGrid example I would do this:

private void Grid_SelectionChanged(object sender, SelectionChangedEventArgs e)
{
  var selects = ((DataGrid)sender).SelectedItems.Cast<Foo>().ToArray();
  controller.SelectedFoos = selects;
}

Purists don't like any code-behind at all, but I think this is quite acceptable and easy to read. The SelectedFoos property on the controller participates in INotifyPropertyChanged so controls can bind to the property.

Use Converters

Always put the binding logic code inside value converter classes, don't be lazy and leave it in controllers. I used to do the following, if for example I wanted a control to be visible if Foo is not null:

public Foo SelectedFoo
{
  get { return _selfoo; }
  set
  {
    _selfoo = value;
    SendChange("SelectedFoo");
    SendChange("IsThingVisible");
  }
}

public Visibility IsThingVisible
{
  get { return _selfoo != null ? Visibility.Visible : Visibility.Collapsed; }
}

What I've done is make a fake binding property that converts null to a Visibility enum. This works and looks easy, but as the number of these fake properties grows, the relationships between them all gets confusing and errors can creep in. You must put this logic in a value converter where it's reusable and isolated, like this:

if (convertArg == "SomeToVisible")
{
  return value != null ? Visibility.Visible : Visibility.Collapsed;
}

Then the XAML to use this is something like:

Visibility="{Binding SelectedFoo,
    Converter={StaticResource MyConverter},
    ConverterParameter=SomeToVisible}"

You will probably find that a lot of binding logic simply converts between different types, so it can be packaged into neat groups in the type converter and reused throughout the XAML. The important thing is that all binding logic and conversion takes place inside your value converter classes.

Use MultiConverter

Like the sample above where I was lazy and put binding logic in the controller, I also fell into the bad habit of using fake properties to allow binding to multiple conditions. If for example I wanted a control to be visible if something is not null and a flag is false, I used to do this in the controller.

public List Foos
{
  ...
  set
  {
    _foos = value;
    SendChange("Foos");
    SendChange("IsThingVisible");
  }
}
public bool IsFoosLoading
{
  ...
  set
  {
    _foosloading = value;
    SendChange("IsFoosLoading");
    SendChange("IsThingVisible");
  }
}
public Visibility IsThingVisible
{
  get { return _foos != null && !_isFoosLoading ? Visibility.Visible : Visibility.Collapsed; }
}

This code manually duplicates exactly what MultiBinding was invented for. In this case, I should use MultiBinding with XAML like this example:

<Button ...>
<Button.Visibility>
  <MulitBinding Converter="{StaticResource MyMultiConverter}"
         ConverterParameter="SomeAndFalse">
    <Binding Path="Foos"/>
    <Binding Path="IsFoosLoading"/>
  </MultiBinding>
</Button>

The logic is now moved into the multi value converter class where it belongs.

if (convertArg == "SomeAndFalse")
{
  bool? flag = values[1] as bool?
  return values[0] != null && flag != true ? Visibility.Visible : Visibility.Collapsed;
}

Note that I didn't just cast values[1] directly to bool, as the value is unpredictably the value UnsetValue. I haven't determined exactly when or why this "try cast" necessary, so I have fallen into the habit of doing it all non-trivial cases to avoid runtime cast crashes.

Summary

Basically everything I've said in this post is a simple restatement of how binding works, but it's a reminder of how important it is to do everything "by the book" and use every binding feature exactly the way it was intended to be used.

Sunday, January 31, 2016

Preserving Encoding and BOM

I was running some files through Regex.Replace and a diff showed the first line of some files had changed, but the text looked the same. It turns out that by using StreamReader to read the input files and StreamWriter to write the output files I had removed the BOMs (byte order markers). For example, some of the input XML files started with the UTF-8 BOM bytes 0xEFBBBF but they were stripped off the output files.

After an hour of searching and fiddling around I came to the conclusion that the none of the System.IO classes correctly report the encoding type, which means you can't simply and automatically round-trip a file's encoding when it's processed as a text file. Other web comments seems to support this, but if anyone knows otherwise, please let me know.

I reluctantly used the following code to calculate a file's text encoding.

private static Encoding CalcEncoding(string filename)
{
  var prencs = Encoding.GetEncodings()
      .Select(e => new { Enc = e.GetEncoding(), Pre = e.GetEncoding().GetPreamble() })
      .Where(e => e.Pre.Length > 0).ToArray();
  using (var reader = File.OpenRead(filename))
  {
    var lead = new byte[prencs.Max(p => p.Pre.Length)];
    reader.Read(lead, 0, lead.Length);
    var match = prencs.FirstOrDefault(p => Enumerable.Range(0, p.Pre.Length).All(i => p.Pre[i] == lead[i]));
    return match == null ? null : match.Enc;
  }
}

This method 'sniffs' the file and finds any encoding with preamble bytes that match the start of the file. It's clumsy to have to do this. If you get null back then you have to chose a suitable default encoding, and a new UTF8Encoding(false) class is a good choice on Windows where UTF8 encoding without a BOM is the default for most text file processing.

Once you have the original encoding (or a suitable default), pass it into the StreamWriter's constructor and you can be sure that the original encoding and BOM will be preserved.

Wednesday, January 27, 2016

IISExpress Trace/Log Files

I noticed by accident while using TreeSize Free that this folder:

MyDocuments\IISExpress\TraceLogFiles

Contained hundreds of log files taking over 120MB. I'd never noticed them before and they looked useless, even to a software developer. Some web articles also hinted the log files weren't of much use and suggested that delete the contents of both the Logs and TraceLogFiles folders. To disable the logging it was suggested that you edit:

MyDocuments\IISExpress\config\applicationhost.config

And comment out the line containing HttpLoggingModule in the globalModules section. I found it in two places and commented out both of them. After rebooting and compiling and running some web apps it looks like no more logging or trace files are generated.

Thursday, January 21, 2016

Inter-Process Locking

Many years ago I had to share a file between apps running in different Windows processes, and I had to guarantee that only one process would update the file at any time. Now I can't find my old C# code or remember exactly how I created an inter-process lock. Web searches reveal lots of different code samples, and they're much longer and more complicated that what I remember of my old code. So after bumbling around for half an hour I finally rediscovered the simplest possible code, which I publish here so it won't be forgotten again.

Console.WriteLine("Before lock");
using (var m = new Mutex(false, "TheMutexName"))
{
  m.WaitOne();
  Console.WriteLine("Pretend work now. Press enter to continue...");
  Console.ReadLine();
  m.ReleaseMutex();
}
Console.WriteLine("After lock. Press enter to exit");
Console.ReadLine();

The code between the WaitOne and ReleaseMutex is guarded against entry by multiple threads in any processes on the machine.

Saturday, January 16, 2016

Azure Table Combined Conditions

If you have multiple conditions that must be combined to build a complete Azure Table query, the code can get verbose, increasingly so if there are 3 or more conditions. Extra complexity arises if not all of the conditions are needed. You might finish up with a cluttered mess of code that uses GenerateFilterCondition and CombineFilter methods sprinkled with if statements. The CombineFilter method only takes two arguments, so to combine more conditions you have to nest calls to it.

A LINQ Aggregate trick can be used to simplify building query strings. Here is a sample that has four imaginary conditions which must be combined if the argument to each one is not null.

public string BuildWhere(string pk, string rk, string user, int? rating)
{
  var conditions = new string[]
  {
    pk == null ? null : TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, pk),
    rk == null ? null : TableQuery.GenerateFilterCondition("RowKey", QueryComparisons.Equal, rk),
    user == null ? null : TableQuery.GenerateFilterCondition("User", QueryComparisons.Equal, user),
    rating == null ? null : TableQuery.GenerateFilterConditionForInt("Rating", QueryComparisons.Equal, rating.Value)
    // There may be more conditions...
  }
  return conditions
      .Where(c => c != null)
      .DefaultIfEmpty()
      .Aggregate((a, b) => TableQuery.CombineFilters(a, TableOperators.And, b));
}

1. Build a string array of all possible conditions, leaving null entries for those not needed. The GenerateFilterXXX methods safely generate strings with the correct syntax required by Azure Table queries.

2. Run LINQ Aggregate over the conditions to multi-nest the non-null ones inside CombineFilter calls. The sample code works correctly no matter how many conditions are non-null, even one or zero. The resulting string has this structure:

Combine(Combine(Combine(pk,rk),user),rating)

The Aggregate processing creates a effect similar to the way Mathematica's Nest function works. It's a neat and elegant example of the power of LINQ and functional programming style.

P.S. The CombineFilters in my sample is just a fancy and verbose way of putting the word 'and' between two string conditions. Knowing this, the technically correct Aggregate code above can be simplified to this:

return string.Join(" and ", conditions.Where(c => c != null));

Azure Table Key Encoding

I was getting BadRequest errors inserting rows into an Azure Table. I eventually noticed that some of the PartitionKey strings had \ (backslash) inside them. I then found various online arguments about which characters were forbidden and various silly workarounds, like base64 encoding all the bytes in the key string (which works of course, but is a completely ham-fisted approach).

The MSDN documentation clearly states which characters are forbidden in the key strings, but it doesn't mention % (percent) which some people have reported as troublesome. I have added % to the forbidden list.

My preferred way of encoding and decoding arbitrary strings as row keys is to use Regex Replace, which produces safe encoded strings where forbidden characters are replaced by +xx hex escape sequences. The +xx escape characters have no special meaning, I just made it up because it looks vaguely readable. You can invent your own preferred escaping system. The resulting encoded strings are still reasonably readable. The + escape prefix character itself also has to be considered forbidden and be encoded.

To encode:

string encodedKey = Regex.Replace(sourceKey,
  "[\x00-\x1f\x7f-\x9f\\\\/#?%+]",
  m => $"+{((int)m.Value[0]).ToString("X2")}");

To decode:

string decodedKey = Regex.Replace(value,
  "(\\+[0-9A-F]{2})",
  m => ((char)(Convert.ToByte(m.Groups[0].Value.Substring(1), 16))).ToString());

The string that originally caused my BadRequest error encodes like this:

Plain*RCS\Demo
Encoded*RCS+5CDemo

Running a worst case string through the code produces:

Plain*Back\\Slash \t\vSlash/Hash#Q?Pct%Dot\x95Plus+ΑΒΓΔ
Encoded*Back+5CSlash +09+0BSlash+2FHash+23Q+3FPct+25Dot+95Plus+2BΑΒΓΔ

Some web articles suggest that you use the Uri and HttpUtility classes to perform the encoding, but unfortunately they do not have the correct behaviour. The documentation makes no comment about Unicode characters outside of the explicitly forbidden set, so I ran a quick experiment to roundtrip a row with a PartitionKey containing the Greek characters ΑΒΓΔ and it worked okay.