Sankofa » Software Development

The Importance of a Dedicated Build Machine

Ian Uy — Sun, 07 Jun 2009 14:36:59 +0000

When I was an Intern at a large Global 500 company, my major responsibility is to deliver a Software Delivery process that uses the software engineering practice called Continuous Integration. In simple terms, I am responsible for a project to create and setup a Continuous Integration Build Server that will be used to “build” the company’s products.

During those days, when I hear the word “build” or “compile”, the first thing that comes into mind is to press the F5 button. Ever since I started programming, when someone asks me for a copy of my program or if I need to distribute the “binaries” of my program, all I do is press the F5 button to create an EXE (sometimes with a DLL), then browse to the output folder(Bin) and copy the binaries. Suffice to say, that is my “Software Delivery” process however crude it is.

In College, I got a little bit more sophisticated. My major projects are all carefully packaged in a very neat MSI installer — but that is just for the major projects. Most of my standalone tools are still made using the “Press F5 button to build” method.

So you can just imagine my horror when the Software Architect presented me my task on my first day at work. I don’t know anything about Software Delivery, more so Continuous Integration. What is more scary is that I am expected to deliver my output in just 240 hours (my Internship duration). But thru the continuous support of the said Software Architect, I somehow managed to cram the concept of Software Delivery and Continuous Integration in just 2 weeks. Also, this book helped a lot, too!

Expert .NET Delivery Using NAnt and CruiseControl.NET

The build process that I created is quite simple and somewhat standard. However, there was a special requirement that the test harness be run on a different machine and that made the build process a little bit more complicated. This are the different stages of my build process:

Clean-up: Before starting a new build, delete all the previous directories (Source codes, Artifacts, etc.)
Fetch: Get the latest copy from the source code repository
Build: Compile the different projects
Archive: Tag the build then archive it
Test: Run the unit tests, measure code coverage and perform static code analysis
Feedback: Provide feedback by generating reports (with beautiful graphs!)

CruiseControl.NET Build History

The build process will automatically check for any changes in the Source Code Repository every 15 minutes, if it detects any change in the code, it will automatically start the build process. That is one of the advantage of using Continuous Integration — there is no need for a costly “Integration” phase.

In my previous blog posts, I’ve mentioned that I am working on a certain product that will be shipped later this month. Well, that project is nearing its completion. We now have a “Beta” build for our product and it won’t be long before an official Release Candidate is available.

During our weekly status meeting, I brought up a concern regarding how we are building our Beta builds that are being sent to the Test Department. Our current Build Machine (which is physically located locally) also acts as our Development Machine. All of the development activities like bug fixing are being done on the Build Machine. If you are REALLY REALLY careful, this is fine but still, a little bit crazy. But since we are Software Developers, we are never too careful. We are programmers, we cause accidents.

So why is it important to have a dedicated build machine? Here’s why.

Pristine – You want to keep the build machine as clean as possible to make sure that only the needed resources are being included in the build. If your build machine also acts as your development machine, keeping a pristine state is a very tough task. When I inspected our latest Beta build, I was shocked to see that there are *.pdb (debug) files included in the install image.
Control - It is very important to control your Build Environment. All of your builds must be coming from a standard setup from a standard state. The keyword here is standard. This means that the “exact state” of the machine used in creating Build 001 should also be used in creating Build 002. When you use your Build Machine as a Development Machine, it is almost impossible to maintain this “exact state” every time for every build. When we were migrating our product from SQL 2000 to SQL 2005, we added the SQL2005 installer to our local build machine but somehow, someone forgot to add the SQL 2005 installer to the source code repository. When we tried building from a new build machine, the installer would no longer work.
Reliability - All builds coming from the build machine must be Reliable. If you are using your Build Machine as a Development Machine, most often the not, your builds are coming from your local copy and not from the Source Code Repository. In most cases, this is okay. But in our case where the final build will be built using the official build machine (located in the US), this is a big problem. One of the problem that we encountered is that the local copy is not in sync with the repository copy. We are Humans, sometimes we forget to check-in our changes.

To solve the above problems (and since we don’t have any spare machines ), I used a Virtual Machine! Yes, I used a Virtual Machine as a Build Machine. After 3 days of fixing the build scripts and updating the unsync’d files, I was able to create a clean and working build from my VM Build Machine.

I'm a big VirtualBox fan!

Virtual Machines are perfect as Build Machines. You can always maintain a pristine state by creating a snapshot of a freshly installed setup build environment and always building from there. Everytime that you need to create a build, you can just rollback to that pristine state. This feature of build machines gives you control over your build environment. Since you are always rolling back to a pristine state, you will then be forced to always fetch the latest copy from your source code repository! This means that you can be confident the builds coming out from your build machine is always reliable.

Before, I have no faith in our source code repository copy. As a result, I am always too anxious in making sure that I don’t break the local copy. There’s always that constant fear that the official copy from the Source Code Repository is broken or the official copy does not reflect the local “tested” copy. Now, I am always confident that anytime, anywhere in the company, we can always create a clean and reliable build that will work straigth from the repository. That feeling is very liberating!

Share and Enjoy:

Migration: Why Developers Hate Microsoft or Why Developers Hate Vista or My First Encounter how Microsoft can Break your Software

Ian Uy — Sun, 10 May 2009 14:13:29 +0000

Do you know how much Programmer's Blood it cost to put this tiny sticker in a Software Box?

In my previous blog post, I’ve mentioned that I’m working on a project to migrate an entire product line to Vista. What I meant by that is that I am responsible for making sure that our current product will run properly on Windows Vista. To guarantee that it is indeed Running Properly it should behave *exactly* like it behaves in Windows XP. In simple terms, my entire job security is riding on making our products compatible with Microsoft’s failed attempt at a MAC-ish Operating System and with Microsoft’s successful attempt (a first in their history) to make their operating system NOT backward compatible with older applications. So much for Raymond Chen’s camp undying obsession with backward compatibility.

This products that I’m talking about are a set of Web Applications written in ASP.NET version 1.1. This web applications are as old as Windows XP. I checked its File Modification History in our TFS repository and the last time that anyone have touched its code base is in 2005 — and that’s just for a minor code modification to fix a bug. I believe this apps were written way way back in 2004 — when I was in Third Year High School!

To make this products compatible with Vista, there are a couple of steps that had to be done. I’ll enumerate each steps and what it takes to accomplish them.

.NET Upgrade (.NET 1.1 to .NET 2.0)

Windows Vista comes pre-installed with .NET 2.0 (and .NET 3.0). Our products are running on .NET 1.1. One approach that most people will suggest is to just install .NET 1.1 on Windows Vista and everything will be perfect. Right? WRONG. Firstly, .NET 1.1 has already reached its end-of-life. Microsoft pulled support for .NET 1.1 last October 14, 2008. Although it was extended, its a sign that this technology is old enough to be replaced. Secondly, installing .NET 1.1 on a Vista machine is not as easy as you think. No, we are not lazy programmers, we just don’t want to introduce unnecessary complications to the system. We are, by the way, Software Developers.

Those Lazy Programmers

So the approach we took is to upgrade our Visual Studio 2003 solutions to Visual Studio 2005 solutions. This, in turn, will “force” the web application to use .NET 2.0. You might say, “How hard could it be? Visual Studio 2005 can convert old solutions automatically using its Upgrade Wizard. All you have to do is click next!, You lazy programmer!”. That’s correct. Upgrading the solution is no problem. Converting every file to .NET 2.0 compliant code is.

We encountered two conversion problems. The first one is relatively simple to fix. It’s a problem where the old .NET 1.1 Resource files (*.resx) were not converted to the .NET 2.0 resource files. The two resource file formats are different (the .NET 2.0 have a *.designer.cs component) and in very rare cases, the .NET 1.1 *resx files won’t work in .NET 2.0. This problem is very hard to catch since the incompatible *.resx files won’t throw a compile error. The strings just won’t be detected. But once found it was relatively easy to fix. After a day of investigation, we found out that the XML format for the resource files changed in .NET 2.0. As a simple hack (fix), we manually changed the parameters of each .NET 1.1 strings to match the .NET 2.0 parameters.

The second problem is harder to find and much much harder to fix. The problem is related to ASP.NET’s datetime picker control.

Our Calendar Control before upgrading to .NET 2.0

On our previous version running .NET 1.1, there is no problem with the datetime picker. But when we installed the .NET 2.0 version, the calendar objects are all over the place.

The Migrated .NET 2.0 Calendar Control

Two days of investigation later, we found out that the previous developer made this custom Calendar control by inheriting from .NET 1.1’s System.Web.Controls. During pre-render, he reads the rendered Calendar Control’s HTML and then does something funny with it. What he wanted to achieve is to replace the “Month” from the Calendar header (gray part) with a drop down box that contains a list of the months. So what he did is he “catch” the HTML stream from the Calendar Control and perform a String.Replace on it!

His Design Goal

The actual String.Replace command is something like this:

//szHTML = Get current HTML stream
//szCurrentMonth = Get current month from HTML
szHTML.Replace(szCurrentMonth, "

 
");

It worked fine before the migration, but somehow we broke it during migration to .NET 2.0. A further investigation of the “emitted” HTML stream from the .NET 2.0 control shows this:


.....






<

May 2009
>





Sun
Mon
Tue
Wed
Thu
Fri
Sat


26
27
28
29
30

1
2

A comparison with the “emitted” .NET 1.1 control shows this:


.....






<

May 2009
>





Sun
Mon
Tue
Wed
Thu
Fri
Sat


26
27
28
29
30
1
2

Executing the String.Replace command in the .NET 1.1 code will produce the proper output. However, Microsoft decided to upgrade the Calendar control in .NET 2.0 by adding a tooltip for each date!

Tool Tip in .NET 2.0

Since the String.Replace has no checking (this is not REGEX afterall), it also replaced the “Month” in the tooltip with the drop down box (