Steven Edouard

Developer Advocate. Tech Enthusiast. Life Enthusiast.

NAVIGATION - SEARCH

Using Services Buses with Cloud Storage to Transfer Files Anywhere

One thing I notice is the lack of awareness of cloud tools that can save your team a lot of time in infrastructure development for very little $$$.

There's all this hype of the cloud and how its the way of the future and this is true. But the cloud isn't a binary thing, you can use parts of it, all of it or just for certain applications.

Recently I had a small problem that I needed to solve. I needed to get data from customer and partner servers living a data center. I needed those files from the servers to get to our sever for diagnostics analysis. The data also needed to be sent securely.

Problem is, there was no (really) easy way to get their files from their server to mine. A few options were considered from using an FTP server (which is horribly unsecure) to SkyDrive and similar web apis. These options although feasible were clunky and lack some simplicity that I wanted.

Then I tried using Azure Blob Storage and Service bus. Azure blob storage is a place you can store any binary file data you want and the service bus allows you to send tiny messages between any computers connected to the internet (usually some serializable info in xml or JSON). These services have excellent .NET client libraries via NuGet (and for other langugages, too!) which takes a lot of work off your hands.

Fast forward about 4 hours of work later I have a sample solution to share. It uses 2 console apps:

 

Uploader.exe (of which there may be N instances running around the world.)

 

Downloader.exe. (In my situation I have just 1 running on my server)

 

I use these apps in batch automation to periodically send data files to my server.

 

https://github.com/sedouard/UploadAnywhere

Create your service bus queue and storage account. Once you do that you can use the console apps in this solution by changing the app configs:

-Downloader

  -App.config

and

-Uploader

  -App.config

<!-- Storage account connection to  service blob storage-->
    <add key="Uploader.Storage.ConnectionString" value="[YourStorageAccountConnectionString]" />
    <!--Service bus info to use to notify of file upload to downloader-->
    <add key="Uploader.ServiceBus.ConnectionString" value="[YourQueueConnectionString]" />
    <add key="Uploader.UploadQueueName" value="[queuename]" />

After changing the configs to your queue names you can use the uploader as such:

Uploader.exe [continername] [localfilename] [cloudfilename]

Its important to note that your container name must not have punctuations or capital letters. Although the app will auto-lowercase the name. The cloud file name should be something like folder1/folder2/myfile.txt

What will happen is that Uploader will first upload the file to a location in cloud storage and then send a JSON message to the download queue (Specified by Uploader.UploadQueueName) pointing to that file. When you run Downloader.exe check the app.config to confirm where you want the app downloaded file root directory to be specified by Downloader.DownloadRootDir:

<add key="Downloader.DownloadRootDir" value="C:\temp\Downloads" />




The cool thing here is that when your run Downloader.exe, it will just hang around and just wait for messages from the download queue. The message is a simple .NET object that is serialized via good ol' JSON.NET:


public class FileUploadedMessage
    {
        public string FileName { get; set; }
        public DateTime UploadTime { get; set; }

        public string ContainerName { get; set; }
    }

 

In uploader.exe this is sent to the service bus like so:

static void SendFileUploadedMessage(string fileName, string containerName)
        {
            FileUploadedMessage message = new FileUploadedMessage()
            {
                FileName = fileName,
                UploadTime = System.DateTime.UtcNow,
                ContainerName = containerName
            };
            //Send the message up to the queue to tell the downloader to pull this file.
            //not async this will block but not a big deal.
            s_QueueClient.Send(new          BrokeredMessage(JsonConvert.SerializeObject(message)));
        }



The message obviously needs to happen after you place the file in the blob storage. Afterwards the Downloader.exe app can read the message and pull the data off blob storage and save it to a coresponding file:

var message = s_QueueClient.Receive();

                    //no new messages go back and wait again for more
                    if(message == null)
                    {
                        continue;
                    }

                    FileUploadedMessage uploadedMessage = JsonConvert.DeserializeObject<FileUploadedMessage>(message.GetBody<string>());

                    Console.WriteLine("Got uploaded file notification for " +uploadedMessage.FileName);

                    // Retrieve a reference to a container. 
                    CloudBlobContainer container = blobClient.GetContainerReference(uploadedMessage.ContainerName.ToLower());

                    //Get the cloud blob which represents the uploaded file
                    var blockBlob = container.GetBlockBlobReference(uploadedMessage.FileName);

                    //build the local file path based on the download root folder specified in .config file
                    var localFilePath = Path.Combine(s_DownloadRootFolder, uploadedMessage.ContainerName.ToLower());
                    var filePath = uploadedMessage.FileName.Replace("/", "\\");
                    localFilePath = Path.Combine(localFilePath, filePath);

                    if (!Directory.Exists(Path.GetDirectoryName(localFilePath)))
                    {
                        Directory.CreateDirectory(Path.GetDirectoryName(localFilePath));
                    }
                    //Replace the file on disk if the cloud uploaded file is in the same mapped location in the downloads directory
                    if (File.Exists(localFilePath))
                    {
                        File.Delete(localFilePath);
                    }

                    //Get the file
                    using (Stream cloudStoredBits = blockBlob.OpenRead())
                    using (FileStream fs = new FileStream(localFilePath, FileMode.CreateNew, FileAccess.ReadWrite))
                    {
                        Console.WriteLine("Downloading Cloud file [" + uploadedMessage.ContainerName + "]" + uploadedMessage.FileName
                            + " to " + localFilePath);
                        cloudStoredBits.CopyTo(fs);
                    }

                    //Delete it from blob storage. Cloud storage isn't cheap :-)
                    Console.WriteLine("Deleting Cloud file [" + uploadedMessage.ContainerName + "]" + uploadedMessage.FileName);
                    blockBlob.Delete(DeleteSnapshotsOption.IncludeSnapshots);


And just like that, anywhere in the world that uploader.exe was running, it can send files directly to your local server directory as if it was part of your local network.

Here is why this is incredibly powerful:

-No virtual machines here are used. Virtual machines are inherently expensive and will run you at least $15/mo on azure for the smallest instance.
-(Almost) no storage is really used. Since cloud redundant storage isn't exactly cheap, if you don't need to keep files up there, remove them.
-Service bus messages are NEARLY free (well, depending on how many message you send). They run at about $1.00/1 million messages.

Running this entire infrastructure should be nearly free at even modest amounts of load (assuming you delete files from cloud storage as I am doing).

It turns out that since we aren't really using much space since we delete files as we get them, and service bus messages are so dirt cheap you end up having a cloud service bill that would be a tiny fraction of doing something such as running a server. It is also just as reliable as a server and has the scalability of Azure.

I found this to be incredibly useful for on-premesis infrastructure (especially test infrastructure). Imagine you have a lot of on-premesis machines and you would like to test your product. You can distribute the work amongst those machines using messaging queues and package up the tests in temporary staging storage int the cloud. Tests can be unpacked with messages representing the test package and ran on each machine. Implementing such infrastructure in the past would have costed weeks of work just to get right.

You can also imagine how this could be useful for custom telemetry systems.

Testing in Isolation: Mocking an Http service

If you’re a web developer, you probably know about the awesome web debugging tool Fiddler. It was a project originally created by a Microsoft employee that eventually brought the project over to Telerik. Any good tester knows the awesome value of testing in isolation and this is one problem that I had with connected apps.

 

Fiddler is the next-best-thing to sliced bread for testers of these types of applications. Fiddler actually exposes a core api set that allows you to quickly write a system proxy server that can programagically manipulate http and even https traffic on your box. 

 

I had the pleasure of tinkering around with this for a quick little project. Using a Mock web service (or client) can address two important issues:

 

  1. Test reliability can be questionable because a malfunction in one piece (server or client) causes the other piece to misbehave.

 

  1. Noise in network traffic causes noise in performance testing.

 

Now, Fiddler's out-of-the box UI has an "AutoResponder" feature:

 

 

 

Unfortunately its way too tedious to use on any application s you have to add these rules manually.

 

 

What if we could simply "record" all the requests from a particular target process, and then just respond later with a saved version of that message? Or even respond with a slightly modified version if we recognize a pattern?

 

Fiddler actually exposes an API which makes this really easy to do. In fact I have an example that does in a pretty small amount of code. You can find my sample here.

 

The Fiddler api feels a bit dated compared to modern .NET apis (no async/await for example). Essentially the entire api is exposed as a static class FiddlerApplication. Which is probably done this way since there can only be 1 proxy for the system at a time.

 

The api has two very important events to register to:

 

 

//This event is fired after the client sends the HTTP/S request but before the message actually arrives to the server
Fiddler.FiddlerApplication.BeforeRequest += delegate(Fiddler.Session oS)

//This event is fired after the server sends the HTTP/S request but before the message actually arrives to the client
Fiddler.FiddlerApplication.BeforeResponse += delegate(Fiddler.Session oS)





These API's allow you be the "middle man" between the client and the server. When your "middle man" has understood all the questions that would be asked by the client or server, you can start to respond to the questions with previously recorded responses.

 

In my example, the BeforeRequest handler filters for a process of interest, (so that you don't block traffic for the entire system) and decides to either let the request go to the server, or to bypass the server with its own "Mocked" response:

 

Fiddler.FiddlerApplication.BeforeRequest += delegate(Fiddler.Session oS)
            {
                Console.WriteLine("Before request for:\t" + oS.fullUrl);
                // In order to enable response tampering, buffering mode must
                // be enabled; this allows FiddlerCore to permit modification of
                // the response in the BeforeResponse handler rather than streaming
                // the response to the client as the response comes in.
                oS.bBufferResponse = true;
                bool isProcessOfInterest = false;
                string processName = null;
                foreach (Process p in Process.GetProcesses())
                {
                    if (p.Id == oS.LocalProcessID)
                    {
                        var results = from x in processNames where Path.GetFileNameWithoutExtension(x).Equals(p.ProcessName, StringComparison.CurrentCultureIgnoreCase) select x;
                        int count = 0;
                        foreach(string s in results)
                        {
                            count++;
                            processName = s;
                            break;
                        }
                        if(count > 0)
                        {
                            isProcessOfInterest = true;
                            break;
                        }
                    }
                }

                if (isProcessOfInterest && !record && mSnoops.ContainsKey((KeyString(oS)).GetHashCode()))
                {
                    oS.utilCreateResponseAndBypassServer();
                    //UpdateTimeStamp(Path.Combine(storagePath, processName) + (KeyString(oS)).GetHashCode().ToString() + ".txt");
                    oS.LoadResponseFromFile(Path.Combine(storagePath, processName) + (KeyString(oS)).GetHashCode().ToString() + ".txt");
                    Console.WriteLine("MOCKED RESPONSE FOR: " + KeyString(oS) + " with " + Path.Combine(storagePath, processName) + (KeyString(oS)).GetHashCode().ToString() + ".txt");
                    return;
                }

                int questionMark = oS.fullUrl.IndexOf("?");
                if (!record && questionMark > 0 && isProcessOfInterest)
                {
                    var result = from HttpSnoop x in mSnoops.Values where x.FileName.StartsWith(oS.RequestMethod+"."+oS.fullUrl.Substring(0, questionMark).Replace("/",".")) select x;
                    foreach (var snoop in result)
                    {
                        oS.utilCreateResponseAndBypassServer();
                        oS.LoadResponseFromFile(Path.Combine(storagePath, processName) + (snoop.FileName).GetHashCode().ToString() + ".txt");
                        Console.WriteLine("MOCKED !!!SIMILAR!!! RESPONSE FOR: " + KeyString(oS) + " with " + Path.Combine(storagePath, processName) + (snoop.FileName).GetHashCode().ToString() + ".txt");
                        oS.SaveResponse(Path.Combine(storagePath, processName) + (snoop.FileName).GetHashCode().ToString() + ".txt", false);
                        return;
                    }
                }
                if(isProcessOfInterest && !record)
                {
                    Console.WriteLine("Error, no response for request: " + KeyString(oS));
                    oS.utilCreateResponseAndBypassServer();
                    return;
                }
            };

 

 

If you notice, I'm actually doing something funky here; in the "replay" leg of this handler, if I don't have an exact matching recorded request URL I will respond to the request with a "similar" url, where in this case the "similar" url is simply anything with the same url root (everything before the where "?" character). This worked well for me in the scenario of an app doing a random query on an endpoint. Obviously this may not work in all scenarios.

 

Another really handy API to know is:

 

//tells Fiddler that this message's response will originate from the proxy server.
Fiddler.Session.utilCreateResponseAndBypassServer();

Calling this keeps your application of interest from ever hitting the target server, guaranteeing that that app never talks to the server. This is very handy since you don't want to block up traffic on the entire machine.

 

Using Fiddler.Session.LoadResponseFromFile/Fiddler.Session.SaveResponseToFile makes it really easy to manage the storage of http message bodies. In my example I used a Dictionary to map the request method and URL to the the file that contained the recorded response for the request.

The BeforeResponse event handler is similar, but opposite of the BeforeRequestHandler. Note that you have to lock around shared resources as many different threads will be invoking the events. Here we simply save the response if it is something we haven't seen before:

 

 
Fiddler.FiddlerApplication.BeforeResponse += delegate(Fiddler.Session oS)
            {
                
                bool isProcessOfInterest = false;
                string processName = null;
                foreach (Process p in Process.GetProcesses())
                {
                    if (p.Id == oS.LocalProcessID)
                    {
                        var results = from x in processNames where Path.GetFileNameWithoutExtension(x).Equals(p.ProcessName, StringComparison.CurrentCultureIgnoreCase) select x;
                        int count = 0;
                        foreach (string s in results)
                        {
                            count++;
                            processName = s;
                            break;
                        }
                        if (count > 0)
                        {
                            isProcessOfInterest = true;
                        }
                    }
                }
                
                if (!record || !isProcessOfInterest)
                {
                    return;
                }

                
                lock (mapLock)
                {
                    if (!mSnoops.ContainsKey((KeyString(oS)).GetHashCode()))
                    {
                        try
                        {
                            //Save the response to a text file
                            oS.SaveResponse(Path.Combine(storagePath, processName) + (KeyString(oS)).GetHashCode().ToString() + ".txt", false);
                            Console.WriteLine("CACHED RESPONSE FOR: " + KeyString(oS));
                        }
                        catch (Exception e)
                        {
                            Console.WriteLine("ERROR: Could not save response - {0}", processName + (KeyString(oS)).GetHashCode().ToString() + ".txt -" + e.ToString());
                        }
                        //Add the mapping of Request.Uri -> file name
                        mSnoops.Add((KeyString(oS)).GetHashCode(), new HttpSnoop()
                            {
                                FileName = KeyString(oS)
                            });
                    }
                }
};




After these two event handlers are implemented you have a very basic but functional "Mock" web service where you can test your client or your server in isolation.

The full Gist post of the implementaiton is here.

Happy testing!