Investigating a Silverlight crash in Firefox

When Firefox 3.6.4 introduced a new plug-in isolation feature (basically they are running plug-in content like Silverlight or Flash now in a separate process), one of my business applications suddenly stopped working. That is, it returned random errors when I was communicating to a server, and often the Silverlight plug-in in Firefox crashed. In Firefox 3.6.3 and all other browsers I tested, everything was fine. It took me a while to find the reason for this, and then I posted a bug report to Bugzilla, Firefox's issue tracking system. Later, I was contacted by someone from Microsoft and sent in a demo project too. When Tim Heuer was discussing the upcoming service release for Silverlight, he also mentioned bug fixes specific to Firefox, so my hopes were high that these issues would be fixed, but now that the service release is here, I see they're not. Again I was asked by someone else at Microsoft to send in a demo project, and that when I thought of also writing this post :). You can download the complete project at the end of the post.

The problem

In Silverlight, you almost always have a service you're communicating with, when you're working with data. What I wanted to do is add some meta information to the service calls the client makes. Since that information is not really part of my domain, I didn't want to clutter all my service calls with it (as arguments), but looked into a way how to attach the information to the technical call, for example just like authentication cookies are sent behind the scenes when you make that calls. A good solution seemed to use custom http headers to transport this information I needed. These headers are additional, small data fragments sent along with a request you make to the server.

And a solution

In that project, I was using RIA Services, which internally makes use of WCF. The server-side solution looks as follows:

• Use a message inspector that looks at the incoming messages from clients and extracts the information.
• Create an operation behavior so the service operations can make use of the message inspector.
• Attribute the service calls with the operation behavior.

First, the message inspector. All you have to do for this is implement the IDispatchMessageInspector interface. For my demo project, it looks like this:

public class ClientCustomHeadersDispatchMessageInspector : IDispatchMessageInspector
{
    // for demonstration purpose only!
    public static string LastReceivedHeader;

    public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext)
    {
        LastReceivedHeader = HttpContext.Current.Request.Headers["X-MyCustomHeader"];

        return null;
    }

    public void BeforeSendReply(ref Message reply, object correlationState)
    {
    }
}

All it does is inspect the http headers of the incoming request and extract and store my custom header (will be null if it isn't there).

The operation behavior is also an attribute, so I can attach it to the service operations easily:

public class ClientCustomHeadersOperationBehaviorAttribute : Attribute, IOperationBehavior
{
    public void AddBindingParameters(OperationDescription operationDescription, BindingParameterCollection bindingParameters)
    {
    }

    public void ApplyClientBehavior(OperationDescription operationDescription, ClientOperation clientOperation)
    {
    }

    public void ApplyDispatchBehavior(OperationDescription operationDescription, DispatchOperation dispatchOperation)
    {
        ClientCustomHeadersDispatchMessageInspector inspector = new ClientCustomHeadersDispatchMessageInspector();
        dispatchOperation.Parent.MessageInspectors.Add(inspector);
    }

    public void Validate(OperationDescription operationDescription)
    {
    }
}

All it does is inject my custom message inspector into the pipeline so it will be used when an incoming message needs to be dispatched. From this point, I can decorate any service call in my RIA Service with the attribute, and it will make use of the custom message inspector, and therefore extract my custom http header:

[EnableClientAccess()]
public class FirefoxTestDomainService : DomainService
{		
    [ClientCustomHeadersOperationBehavior]
    public SampleEntity GetLastCustomHeader()
    {
        SampleEntity result = new SampleEntity();
        result.CustomHeader = ClientCustomHeadersDispatchMessageInspector.LastReceivedHeader;
        result.SomeDate = DateTime.Now;

        Random rnd = new Random();

        byte[] junk = new byte[1024];

        for (int i = 0; i < 1024; i++)
        {
            junk[i] = (byte)rnd.Next(65, 91);
        }

        string randomString = System.Text.UTF8Encoding.UTF8.GetString(junk);
        result.RandomData = randomString;			

        return result;
    }
}

What I do here needs explanation. First of all, the only service method is decorated with my attribute, which means that the custom http header is extracted and accessible in the (not so multi-threading-safe) static field. That simple demo method does nothing else but bounce the custom header back to the client. In addition however, it also generates a chunk of random data and sends it along. In my testing I found that it's much easier to provoke errors and crashes in Firefox when the data sent across the wire has a certain size. With small amounts of data you may have 20 successful calls until the first error, but with that kilobyte of random data I never managed to get one single successful call. The used "SampleEntity" class is just a simple container with properties for all the data I want to transfer.

That's pretty much it for the server. The client needs to do similar steps to inject custom http headers into the service calls. First of all, again we need to implement a message inspector, this time implementing IClientMessageInspector.

public class AttachingCustomHeadersClientMessageInspector : IClientMessageInspector
{
    public void AfterReceiveReply(ref Message reply, object correlationState)
    {			
    }

    public object BeforeSendRequest(ref Message request, IClientChannel channel)
    {
        HttpRequestMessageProperty property = (HttpRequestMessageProperty)request.Properties[HttpRequestMessageProperty.Name];
        property.Headers["X-MyCustomHeader"] = "Firefox SL crash test!";

        return null;
    }
}

What we do here is inject the custom header just before the message is sent. Again, to make use of this message inspector, we need to somehow make the service client use it. To this end, we:

• Implement a custom endpoint behavior and
• hook into the creation process of the domain context to use that behavior.

The endpoint behavior looks like this:

public class AttachingCustomHeadersEndpointBehavior : IEndpointBehavior
{
    public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters)
    {
    }

    public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
    {
        AttachingCustomHeadersClientMessageInspector inspector = new AttachingCustomHeadersClientMessageInspector();
        clientRuntime.MessageInspectors.Add(inspector);
    }

    public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher endpointDispatcher)
    {
    }

    public void Validate(ServiceEndpoint endpoint)
    {
    }
}

And to actually use it, we create a partial class for the domain context and use the behavior in the creation process:

public partial class FirefoxTestDomainContext
{
    partial void OnCreated()
    {
        WebDomainClient<FirefoxTestDomainContext.IFirefoxTestDomainServiceContract> webDomainClient = 
            (WebDomainClient<FirefoxTestDomainContext.IFirefoxTestDomainServiceContract>)DomainClient;
        AttachingCustomHeadersEndpointBehavior endpointBehaviour = new AttachingCustomHeadersEndpointBehavior();
        webDomainClient.ChannelFactory.Endpoint.Behaviors.Add(endpointBehaviour);
    }
}

Now, whenever we use the domain context, the custom endpoint behavior and in turn the custom message inspector that attaches the http header, is used automatically. The rest of the client is just some UI and logic to actually make the service call and look for the results. I've used a simple button in one of the views for that, and display the results of the service call in a text block:

private void SendCustomHeaderButton_Click(object sender, System.Windows.RoutedEventArgs e)
{
    FirefoxTestDomainContext dc = new FirefoxTestDomainContext();
    dc.Load(dc.GetLastCustomHeaderQuery(), GetLastCustomHeader_Completed, null);
}

private void GetLastCustomHeader_Completed(LoadOperation<SampleEntity> operation)
{
    if (operation.HasError)
    {
        ResultMessage.Text = "Operation failed! " + operation.Error;
        operation.MarkErrorAsHandled();
    }
    else
    {
        SampleEntity entity = operation.Entities.First();
        ResultMessage.Text = "Succeeded!" + 
            Environment.NewLine + 
            "Custom header: " + entity.CustomHeader + 
            Environment.NewLine + 
            "Random data: " + entity.RandomData;
    }
}

The bug

Ok, let's run that and see what happens. For the test, I'm using the current Firefox version 3.6.8. But as I said before, the bug was introduced with 3.6.4. The first attempt to call the service gave me a simple error message:

As you can see it's complaining about elements that are not closed in the response. However, when you look at the resonse in Fiddler (and decode it with the WCF binary message inspector), you'll see that everything's ok here:

<GetLastCustomHeaderResponse xmlns="http://tempuri.org/">
    <GetLastCustomHeaderResult xmlns:a="DomainServices" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
        <a:TotalCount>-1</a:TotalCount>
        <a:RootResults xmlns:b="http://schemas.datacontract.org/2004/07/FirefoxTest.Web.Models">
            <b:SampleEntity>
                <b:CustomHeader>Firefox SL crash test!</b:CustomHeader>
                <b:Key>dcdaac87-d383-41d7-a2e6-716d29e88d1c</b:Key>
                <b:RandomData>ZKDITDSJCOEMEMXBEUMYEGUSQDZHSN[truncated]</b:RandomData>
                <b:SomeDate>2010-09-04T10:19:22.5796352+02:00</b:SomeDate>
                <b:SomeNumber i:nil="true">
                </b:SomeNumber>
            </b:SampleEntity>
        </a:RootResults>
    </GetLastCustomHeaderResult>
</GetLastCustomHeaderResponse>

It took me three more clicks on that button to invoke the service call to make the Firefox Silverlight plug-in crash completely:

When you remove the custom http header from the message, the calls succeed every time.

Silverlight/WCF or RIA Services - the cause?

When I first came across that bug, it seemed clear that it's a Silverlight problem, because after all, everything on the server-side was OK, and inspecting the traffic with Fiddler showed that the server also sends correct responses. However, someone made a very valid point and said that this is not necessarily a problem of the Silverlight core runtime (e.g. the WCF client library), but it could also be a RIA Services problem. So what I did was implement the demo project again, but this time without using RIA services. In vanilla WCF, some things are slightly different. For example, on the server side you don't have a HttpContext, but use the OperationContext. But since RIA Services makes use of WCF behind the scenes, eventually it would come down to the same. And also the general architecture, using message inspectors and behaviors, is exactly the same.

The same project implemented with WCF only does not show any problems. At least not after clicking on that button for like a hundred times :). Seriously... this problem seems to be an error specific to the RIA Services Silverlight libraries.

For your convenience, I'll attach both projects here:

FirefoxTest_RiaServices.zip (495.74 kb)

FirefoxTest_Wcf.zip (38.36 kb)