Steven Edouard

Developer Advocate. Tech Enthusiast. Life Enthusiast.

NAVIGATION - SEARCH

What's it like being an SDET at Microsoft?

You really have to admit that the role of a software 'tester' doesn't sound as sexy as other developer jobs. In fact that's probably why the official title of 'testers' at Microsoft is 'Software DEVELOPMENT ENGINEER in Test' to convey that these folks are just as capable of developers as your standard software developer. Microsoft has truly innovated on this role in a way that most other software firms haven't. So much so, that in the past few years the 'SDET' role has just started spreading out to other reputable organizations such as Google and Amazon.

I spent a little over 2 years as a tester at Microsoft within the .NET Runtime (Common Language Runtime for you .NET fanboys out there) right after I got my Computer Engineering undergraduate degree. During that time Microsoft has churned the organization heavily as you may have seen in the tech blogs. With that churn, the role of the test position has shifted as well.

As a Microsoft SDET my job really wasn't to ensure that a particular API behaved in a certain way or that the unit test coverage of a certain feature was implemented. Our developers were responsible for doing that work. Instead the SDET work focused on test tooling such as static validation tools and sophisticated, reliable infrastructure that would run end-to-end and unit level tests that drove developer productivity. Developers were responsible for ensuring that the quality of the code they checked in was up to snuff. Testers like myself ensured that the holistic quality of the product was sound. This could be anything from harnessing apps to creating sophisticated semantic validation tools that could determine if two pieces of generated code were the 'same'.

Now, don't get me wrong my SDET experience may be differ from teams with products with faster cycles like online services but the theme remains the same. Testers at Microsoft are not simply 'point-n-click' testers. They do a lot of really sophisticated things and focus on ensuring the quality of the customer scenarios in the product. This is a lot more fun than what the title of 'tester' conveys.

What are the skills required? In general I would say it's nearly the same as any highly talent developer with an increased emphasis on creativity. You gotta have that creative edge to think about really cool ways to validate the product and even validate things like the usage of a feature through things like telemetry systems.

 

Testing in Isolation: Mocking an Http service

If you’re a web developer, you probably know about the awesome web debugging tool Fiddler. It was a project originally created by a Microsoft employee that eventually brought the project over to Telerik. Any good tester knows the awesome value of testing in isolation and this is one problem that I had with connected apps.

 

Fiddler is the next-best-thing to sliced bread for testers of these types of applications. Fiddler actually exposes a core api set that allows you to quickly write a system proxy server that can programagically manipulate http and even https traffic on your box. 

 

I had the pleasure of tinkering around with this for a quick little project. Using a Mock web service (or client) can address two important issues:

 

  1. Test reliability can be questionable because a malfunction in one piece (server or client) causes the other piece to misbehave.

 

  1. Noise in network traffic causes noise in performance testing.

 

Now, Fiddler's out-of-the box UI has an "AutoResponder" feature:

 

 

 

Unfortunately its way too tedious to use on any application s you have to add these rules manually.

 

 

What if we could simply "record" all the requests from a particular target process, and then just respond later with a saved version of that message? Or even respond with a slightly modified version if we recognize a pattern?

 

Fiddler actually exposes an API which makes this really easy to do. In fact I have an example that does in a pretty small amount of code. You can find my sample here.

 

The Fiddler api feels a bit dated compared to modern .NET apis (no async/await for example). Essentially the entire api is exposed as a static class FiddlerApplication. Which is probably done this way since there can only be 1 proxy for the system at a time.

 

The api has two very important events to register to:

 

 

//This event is fired after the client sends the HTTP/S request but before the message actually arrives to the server
Fiddler.FiddlerApplication.BeforeRequest += delegate(Fiddler.Session oS)

//This event is fired after the server sends the HTTP/S request but before the message actually arrives to the client
Fiddler.FiddlerApplication.BeforeResponse += delegate(Fiddler.Session oS)





These API's allow you be the "middle man" between the client and the server. When your "middle man" has understood all the questions that would be asked by the client or server, you can start to respond to the questions with previously recorded responses.

 

In my example, the BeforeRequest handler filters for a process of interest, (so that you don't block traffic for the entire system) and decides to either let the request go to the server, or to bypass the server with its own "Mocked" response:

 

Fiddler.FiddlerApplication.BeforeRequest += delegate(Fiddler.Session oS)
            {
                Console.WriteLine("Before request for:\t" + oS.fullUrl);
                // In order to enable response tampering, buffering mode must
                // be enabled; this allows FiddlerCore to permit modification of
                // the response in the BeforeResponse handler rather than streaming
                // the response to the client as the response comes in.
                oS.bBufferResponse = true;
                bool isProcessOfInterest = false;
                string processName = null;
                foreach (Process p in Process.GetProcesses())
                {
                    if (p.Id == oS.LocalProcessID)
                    {
                        var results = from x in processNames where Path.GetFileNameWithoutExtension(x).Equals(p.ProcessName, StringComparison.CurrentCultureIgnoreCase) select x;
                        int count = 0;
                        foreach(string s in results)
                        {
                            count++;
                            processName = s;
                            break;
                        }
                        if(count > 0)
                        {
                            isProcessOfInterest = true;
                            break;
                        }
                    }
                }

                if (isProcessOfInterest && !record && mSnoops.ContainsKey((KeyString(oS)).GetHashCode()))
                {
                    oS.utilCreateResponseAndBypassServer();
                    //UpdateTimeStamp(Path.Combine(storagePath, processName) + (KeyString(oS)).GetHashCode().ToString() + ".txt");
                    oS.LoadResponseFromFile(Path.Combine(storagePath, processName) + (KeyString(oS)).GetHashCode().ToString() + ".txt");
                    Console.WriteLine("MOCKED RESPONSE FOR: " + KeyString(oS) + " with " + Path.Combine(storagePath, processName) + (KeyString(oS)).GetHashCode().ToString() + ".txt");
                    return;
                }

                int questionMark = oS.fullUrl.IndexOf("?");
                if (!record && questionMark > 0 && isProcessOfInterest)
                {
                    var result = from HttpSnoop x in mSnoops.Values where x.FileName.StartsWith(oS.RequestMethod+"."+oS.fullUrl.Substring(0, questionMark).Replace("/",".")) select x;
                    foreach (var snoop in result)
                    {
                        oS.utilCreateResponseAndBypassServer();
                        oS.LoadResponseFromFile(Path.Combine(storagePath, processName) + (snoop.FileName).GetHashCode().ToString() + ".txt");
                        Console.WriteLine("MOCKED !!!SIMILAR!!! RESPONSE FOR: " + KeyString(oS) + " with " + Path.Combine(storagePath, processName) + (snoop.FileName).GetHashCode().ToString() + ".txt");
                        oS.SaveResponse(Path.Combine(storagePath, processName) + (snoop.FileName).GetHashCode().ToString() + ".txt", false);
                        return;
                    }
                }
                if(isProcessOfInterest && !record)
                {
                    Console.WriteLine("Error, no response for request: " + KeyString(oS));
                    oS.utilCreateResponseAndBypassServer();
                    return;
                }
            };

 

 

If you notice, I'm actually doing something funky here; in the "replay" leg of this handler, if I don't have an exact matching recorded request URL I will respond to the request with a "similar" url, where in this case the "similar" url is simply anything with the same url root (everything before the where "?" character). This worked well for me in the scenario of an app doing a random query on an endpoint. Obviously this may not work in all scenarios.

 

Another really handy API to know is:

 

//tells Fiddler that this message's response will originate from the proxy server.
Fiddler.Session.utilCreateResponseAndBypassServer();

Calling this keeps your application of interest from ever hitting the target server, guaranteeing that that app never talks to the server. This is very handy since you don't want to block up traffic on the entire machine.

 

Using Fiddler.Session.LoadResponseFromFile/Fiddler.Session.SaveResponseToFile makes it really easy to manage the storage of http message bodies. In my example I used a Dictionary to map the request method and URL to the the file that contained the recorded response for the request.

The BeforeResponse event handler is similar, but opposite of the BeforeRequestHandler. Note that you have to lock around shared resources as many different threads will be invoking the events. Here we simply save the response if it is something we haven't seen before:

 

 
Fiddler.FiddlerApplication.BeforeResponse += delegate(Fiddler.Session oS)
            {
                
                bool isProcessOfInterest = false;
                string processName = null;
                foreach (Process p in Process.GetProcesses())
                {
                    if (p.Id == oS.LocalProcessID)
                    {
                        var results = from x in processNames where Path.GetFileNameWithoutExtension(x).Equals(p.ProcessName, StringComparison.CurrentCultureIgnoreCase) select x;
                        int count = 0;
                        foreach (string s in results)
                        {
                            count++;
                            processName = s;
                            break;
                        }
                        if (count > 0)
                        {
                            isProcessOfInterest = true;
                        }
                    }
                }
                
                if (!record || !isProcessOfInterest)
                {
                    return;
                }

                
                lock (mapLock)
                {
                    if (!mSnoops.ContainsKey((KeyString(oS)).GetHashCode()))
                    {
                        try
                        {
                            //Save the response to a text file
                            oS.SaveResponse(Path.Combine(storagePath, processName) + (KeyString(oS)).GetHashCode().ToString() + ".txt", false);
                            Console.WriteLine("CACHED RESPONSE FOR: " + KeyString(oS));
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine("ERROR: Could not save response - {0}", processName + (KeyString(oS)).GetHashCode().ToString() + ".txt -" + e.ToString());
                        }
                        //Add the mapping of Request.Uri -> file name
                        mSnoops.Add((KeyString(oS)).GetHashCode(), new HttpSnoop()
                            {
                                FileName = KeyString(oS)
                            });
                    }
                }
};




After these two event handlers are implemented you have a very basic but functional "Mock" web service where you can test your client or your server in isolation.

The full Gist post of the implementaiton is here.

Happy testing!

How to test Asp.NET Web Api's with Visual Studio Test Explorer

So myself and my buddy Jamie decided to quickly start working on a web app powered by Asp.NET MVC4 and Web API. I'm working on most of the back end and I really wanted to have a good Test-Driven-Development environment in Visual Studio (2013). Naturally I wanted to use the VS Test Explorer window to mimick the client calls to each API to validate my dev work.

After searching around a bit I found a few handy blog posts like this one from one of my colleagues at Microsoft. This post is great because it shows you how to test your WebApi completley in-process without having to use the networking stack. However I like to use the in-box Visual Studio tools rather than xUnit that this blog had.

To give you a background on my setup I have a web app called GiftMe. Here's my Visual Studio solution setup:

Giftme.sln

     >GiftMe.csproj (The ASP.NET web api)

     >GIftMe.Tests (A unit test framework library)

 

So I have my Web API controller in the GIftMe.csproj project which looks something like this:

public class AuthenticationController : ApiController
{
     // POST /api/authentication/authenticate
     [ActionName("Authenticate")]
     public async Task<IHttpActionResult> PostAuthentication(User user)
     {
          //Actual implementation hidden
          return Ok();
     }
}

 

And then afterwards I naturally added a new unit test library, GiftMe.Tests.csproj. Afterwards you have to add the required dependency libraries:

 

GiftMe <solution project reference>

System.Web

System.Web.Http

System.Web.Http.WebHost

(You probably want to add Json.NET while you're at it from the Nuget package manager if you are sending JSON to your web api).

 

Then I added the test:

[TestMethod]
        public void Authenticate()
        {

            HttpConfiguration config = new HttpConfiguration();
            config.Routes.MapHttpRoute(
                name: "DefaultApi",
                routeTemplate: "api/{controller}/{action}/{id}",
                defaults: new { id = RouteParameter.Optional }
            );
            HttpServer server = new HttpServer(config);
            
            
            using (HttpMessageInvoker client = new HttpMessageInvoker(server))
            {
                using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, "http://localhost/api/authentication/authenticate"))
                {
                    string user = JsonConvert.SerializeObject(new User()
                    {
                        FacebookId = "112512525",
                        AccessToken = SecureCloudConfigurationManager.GetSetting("GiftMe.Facebook.AccessToken"),
                        Email = "someemailaddress@outlook.com"
                    });
                    request.Content = new StringContent(user);
                    request.Content.Headers.ContentType.MediaType = "application/json";

                    Trace.WriteLine("Sending: " + user + " to api/authentication/authenticate");
                    using (HttpResponseMessage response = client.SendAsync(request, CancellationToken.None).Result)
                    {
                        //verify status code
                        Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
                    }
                    
                }
            };
        }



To my surprise, I kept getting HTTP404 errors back from the request. After a while it I was wondering how Web API 'magically' fond your controller to route http messages. It turns out that it looks at all loaded modules in the process for anything that extends the ApiController type. It also turned out that if you make no reference to the actual controller, the .NET runtime loader will never load the DLL with the Web API controllers (even though you reference GiftMe.dll) unless you actually use something in that DLL.

How do you fix this? Well for this example if you just reference the type AuthenticationController anywhere you'll force the runtime to load GiftMe.dll allowing the in-proc sever to find your Web API. I recommend putting this in the class initializer for the test library like so:

public sealed class ControllerTests
    {
        AuthenticationController m_AuthController;
 
        [ClassInitialize]
        public static void Initialize(TestContext context)
        {     
               //force runtime to load GiftMe.dll for http handlers
               m_AuthController = new AuthenticationController();
        }
 
//...begin test methods


And viola! Your controller should be able to handle the request from the in-proc server and you have a nice, integrated test-driven-development environment for all of your web apis organized in a sane way.