Determine screen resolution in Silverlight OOB Firefox Debug Helper (Visual Studio Add-in)

Pitfalls in Visual Studio Add-in programming

Published on Wednesday, September 22, 2010 12:42:00 PM UTC in Programming

As I promised in the post about my Firefox Debug Helper Visual Studio Add-in, I present you the top three pitfalls I stumbled across when I was trying to develop that add-in. Some of the problems I came across (and their solutions) really weren't obvious, so I hope this post will help others in some way. The topics I'm going to cover are:

  1. Problems when capturing Visual Studio events
  2. Debugger events/operations being non-reentrant
  3. Features "hidden" in interface implementations

Hooking Visual Studio events

To write the debug helper, I needed to hook some debugger events so my add-in is notified when a new debugging session starts. So what I tried was to add an event handler to the respective event in the "OnStartupComplete" override of my add-in, like that:

_applicationObject.Events.DebuggerEvents.OnEnterRunMode += 
    new _dispDebuggerEvents_OnEnterRunModeEventHandler(DebuggerEvents_OnEnterRunMode);

Looks like a completely valid line of code to a "normal" .NET developer. The problem with this was that my event never was raised. I could debug through the code and see the event handler was attached correctly, but I really never received any notifications. I tried to attach to other events, but it was always the same: none of my code in the event handlers ever got called. I took me quite a while to find the answer in this article. Apparently, when you access the "DebuggerEvents" property (or any other events property), a wrapper is created internally for these events. The crucial part of this is that Visual Studio itself does not hold a reference to this wrapper. This means that the events wrapper went out of scope once the "OnStartupComplete" was finished, and the garbage collector cleaned it up, which was the reason for my add-in never receiving any notifications.

The solution to this is to simply hold a reference to the wrapper yourself, to keep it alive. I've added a field for that, and then attached my event handler like this:

private DebuggerEvents _debuggerEvents;

...

_debuggerEvents = _applicationObject.Events.DebuggerEvents;
_debuggerEvents.OnEnterRunMode +=
    new _dispDebuggerEvents_OnEnterRunModeEventHandler(DebuggerEvents_OnEnterRunMode);  

And guess what? It worked :). In my opinion, that implementation of the "DebuggerEvents" property violates Microsoft's own guidelines for class library programming in various ways. Acquiring these wrappers really should have been designed as a method, which would have made perfectly clear what happens here.

Non-reentrant operations

What I wanted to do with the add-in was: when a new debugging session starts, attach to the plugin-container.exe process of Firefox. When I tried to do that, I received the following error message:

"A macro called a debugger action which is not allowed while responding to an event or while being run because a breakpoint was hit."

Erm... okaaay. This basically says you cannot invoke any debugger actions in response to an event, which I would consider a pretty common requirement for someone who writes a Visual Studio add-in. However, I can also imagine some of the problems that might occur if this was allowed, and why that restriction has been introduced in the first place. The problem here really was to find a way to solve this. After a lot of searching, I found this post in an MSDN blog that suggested to turn whatever operation you want to do into an asynchronous call using the .NET thread pool. I was a bit surprised by this approach and thought about whether this really is a safe thing to do; but of course I don't know the internals of Visual Studio and whether these operations are thread-safe, and since this really seems the only option to solve the problem, I went for it.

Note: Later in the development of my add-in other problems forced me to move the attaching code to a separate thread anyway, so you won't find an explicit implementation of the code in the linked post in my source code anymore. But the first attempts really used exactly that.

"Hidden" features

The central feature of my add-in was that it should attach to a local process. To this end, the "Debugger" property of the application object offers an enumerable of local processes that can be used.

foreach (Process localProcess in _applicationObject.Debugger.LocalProcesses)
{
    ...
}

The problem I ran into is that the "Process" interface has a method "Attach" to attach the debugger, but there is no way of telling Visual Studio what debug engine to use. For my add-in, I needed to attach the Silverlight debug engine, but depending on the last selected debug engine in the "Attach to process" dialog of Visual Studio it was possible that an incorrect debugger was chosen. Even when I left that setting on "Auto", sometimes the Silverlight debug engine was not chosen, probably because the call to attach the debugger happened too quickly (so the Silverlight runtime wasn't fully loaded and could not be detected by Visual Studio).

That was an annoying problem, but after some searching I found the solution. Apparently there is a successor to the "Process" interface which is simply named "Process2" (a tradition with Microsoft to simply increase a number), and that interface implements a different "Attach" method that allows to specify the engine to use. My new code then looked like this:

foreach (Process localProcess in _applicationObject.Debugger.LocalProcesses)
{   
    ...

    Process2 localProcess2 = localProcess as Process2;
    if (localProcess2 != null)
    {
        localProcess2.Attach2("Silverlight");
    }

    ...
}

The problem here is that Microsoft evolved the "Process" interface into the new "Process2" interface. But for obvious reasons they could not change the enumerator implementation of the "LocalProcesses" property (as that would be a breaking change). So as a developer you really have to *know *that there's another interface and check for that. I do not see an easy solution Microsoft could have provided for this, so I don't blame anyone. However, it's a good idea for any add-in developer to keep in mind that whenever you're missing a certain feature, you should check with the documentation if a new interface has been introduced at some point, with a new Visual Studio version.

Tags: .NET · Visual Studio Add-in