When I was an Intern at a large Global 500 company, my major responsibility is to deliver a Software Delivery process that uses the software engineering practice called Continuous Integration. In simple terms, I am responsible for a project to create and setup a Continuous Integration Build Server that will be used to “build” the company’s products.

During those days, when I hear the word “build” or “compile”, the first thing that comes into mind is to press the F5 button. Ever since I started programming, when someone asks me for a copy of my program or if I need to distribute the “binaries” of my program, all I do is press the F5 button to create an EXE (sometimes with a DLL), then browse to the output folder(Bin) and copy the binaries. Suffice to say, that is my “Software Delivery” process however crude it is.

In College, I got a little bit more sophisticated. My major projects are all carefully packaged in a very neat MSI installer — but that is just for the major projects. Most of my standalone tools are still made using the “Press F5 button to build” method.

So you can just imagine my horror when the Software Architect presented me my task on my first day at work. I don’t know anything about Software Delivery, more so Continuous Integration. What is more scary is that I am expected to deliver my output in just 240 hours (my Internship duration). But thru the continuous support of the said Software Architect, I somehow managed to cram the concept of Software Delivery and Continuous Integration in just 2 weeks. Also, this book helped a lot, too!

Expert .NET Delivery Using NAnt and CruiseControl.NET

The build process that I created is quite simple and somewhat standard. However, there was a special requirement that the test harness be run on a different machine and that made the build process a little bit more complicated. This are the different stages of my build process:

1. Clean-up: Before starting a new build, delete all the previous directories (Source codes, Artifacts, etc.)
2. Fetch: Get the latest copy from the source code repository
3. Build: Compile the different projects
4. Archive: Tag the build then archive it
5. Test: Run the unit tests, measure code coverage and perform static code analysis
6. Feedback: Provide feedback by generating reports (with beautiful graphs!)

CruiseControl.NET Build History

The build process will automatically check for any changes in the Source Code Repository every 15 minutes, if it detects any change in the code, it will automatically start the build process. That is one of the advantage of using Continuous Integration — there is no need for a costly “Integration” phase.

In my previous blog posts, I’ve mentioned that I am working on a certain product that will be shipped later this month. Well, that project is nearing its completion. We now have a “Beta” build for our product and it won’t be long before an official Release Candidate is available.

During our weekly status meeting, I brought up a concern regarding how we are building our Beta builds that are being sent to the Test Department. Our current Build Machine (which is physically located locally) also acts as our Development Machine. All of the development activities like bug fixing are being done on the Build Machine. If you are REALLY REALLY careful, this is fine but still, a little bit crazy. But since we are Software Developers, we are never too careful. We are programmers, we cause accidents.

So why is it important to have a dedicated build machine? Here’s why.

1. Pristine – You want to keep the build machine as clean as possible to make sure that only the needed resources are being included in the build. If your build machine also acts as your development machine, keeping a pristine state is a very tough task. When I inspected our latest Beta build, I was shocked to see that there are *.pdb (debug) files included in the install image.
2. Control - It is very important to control your Build Environment. All of your builds must be coming from a standard setup from a standard state. The keyword here is standard. This means that the “exact state” of the machine used in creating Build 001 should also be used in creating Build 002. When you use your Build Machine as a Development Machine, it is almost impossible to maintain this “exact state” every time for every build. When we were migrating our product from SQL 2000 to SQL 2005, we added the SQL2005 installer to our local build machine but somehow, someone forgot to add the SQL 2005 installer to the source code repository. When we tried building from a new build machine, the installer would no longer work.
3. Reliability - All builds coming from the build machine must be Reliable. If you are using your Build Machine as a Development Machine, most often the not, your builds are coming from your local copy and not from the Source Code Repository. In most cases, this is okay. But in our case where the final build will be built using the official build machine (located in the US), this is a big problem. One of the problem that we encountered is that the local copy is not in sync with the repository copy. We are Humans, sometimes we forget to check-in our changes.

To solve the above problems (and since we don’t have any spare machines ), I used a Virtual Machine! Yes, I used a Virtual Machine as a Build Machine. After 3 days of fixing the build scripts and updating the unsync’d files, I was able to create a clean and working build from my VM Build Machine.

I'm a big VirtualBox fan!

Virtual Machines are perfect as Build Machines. You can always maintain a pristine state by creating a snapshot of a freshly installed setup build environment and always building from there. Everytime that you need to create a build, you can just rollback to that pristine state. This feature of build machines gives you control over your build environment. Since you are always rolling back to a pristine state, you will then be forced to always fetch the latest copy from your source code repository! This means that you can be confident the builds coming out from your build machine is always reliable.

Before, I have no faith in our source code repository copy. As a result, I am always too anxious in making sure that I don’t break the local copy. There’s always that constant fear that the official copy from the Source Code Repository is broken or the official copy does not reflect the local “tested” copy. Now, I am always confident that anytime, anywhere in the company, we can always create a clean and reliable build that will work straigth from the repository. That feeling is very liberating!

My First Official "Coding Horror" Encounter

Back in College, I have a friend who’s programming prowess is so godly that it can surpass or match mine . He is a C++ Junkie and belongs to the Java Cult. His mental agility is so great that he can beat me in writing complex algorithms anytime of the day. In fact, I chose him to be the “Architect” for our project in Compiler Theory Class (Kompayler Studio 2008: The First Tagalog Kompayler, Programming Language and IDE). He is THAT great of a programmer… BUT

Being a Great Programmer is not the same as being a Great Software Developer

“Aren’t programming and software development the same?” Uhhmm, NO. Programming is just a single part of Software Development. Software Development, on the other hand, is an entirely different song and dance.

In one of our converstations, I told him that instead of reading books about programming he should try to read books about software development. I gave him that advice not because I want him to suck in programming (I’m not that envious). I gave him that advice because I want him to realize that programming alone won’t make a great software product and that programming prowess alone won’t give him the best programming job around… at least here in the Philippines.

One of our main differences is reflected on how we write code (which eventually shows in our software). For example, say that we have a project that requires us to deliver a Simple Number Sorting Application that will ask the user to input 50 numbers and sort it. This project is resource constrained in such a way that we need to deliver it in 2 days.

What I would do is use a pre-existing bubble sort algorithm available in the class library (in .NET, I suppose), then I would add additional functionality, then create a test harness (unit test) to test my functions and then I’m done with the “coding” part. Then I would use the rest of my time to design an intuitive user-interface, provide a user manual and work on the packaging.

My Simple Sorting Application

On the other hand, my friend would create a complex multi-threaded sorting algorithm that will have a runtime efficiency of  O(Log N). Not only that, it will be written in Haskel (or any other hardcore programming language) and it will also be invoked via command-line! So hackerish.

His Complex Sorting "Tool"

My point is simple… In Software Development, there is a thing called Tradeoffs. What do I mean by Tradeoff? Take the above example. I traded-off the uber-cool complex multi-threaded sorting algorithm to USABILITY AND FUNCTIONALITY which, by the way, has more practical and positive impact to my users. You could argue that I suck for using bubble sort and not the complex multi-threaded sorting algorithm and you could also argue that my bubble sort has a runtime complexity of O(N^2). But being a Software Developer, I know that using that “complex multi-threaded sorting algorithm” would only be faster by 0.000000000000001312 seconds (remember, we are only sorting 50 elements) and that “improvement” will not be of any significant use to my end-users and I might as well use my time in creating extra  features, such as Print and Copy to Clipboard, that will surely make my product more functional and easy to use.

At work, I am working on a project to “migrate” our products to Vista. To Migrate meaning “to make it compatible with Vista”. One of our product has an Autorun executable which will decide which “components” (SQL Server, VC++ 8.0 Redistributable, .NET Framework) to install. During a routine test run at the Test Deparment, they found out that this certain product is not working on Vista and filled a Track against it. The said Track was assigned to me and I’m expected to fix it. So I pulled the Visual Basic 6.0 source code of the Autorun executable from our TFS and performed an investigation. While investigating, I saw a very weird block of code…

Private dim VCREDIST as long
Private dim NETFRAMEWORK as long
Private dim SQLSERVER as long
Private dim COMPONENT1 as long
Private dim COMPONENT2 as long
Private dim COMPONENT3 as long
Private dim COMPONENT16 as long
 
VCREDIST = 1
NETFRAMEWORK = 2
SQLSERVER = 4
COMPONENT1 = 8
COMPONENT2 = 16
COMPONENT3 = 32
COMPONENT16 = 65536

If you see a 1 or a 0 in the code, its pretty normal because you know that it is a flag for something . But if you see a 7 or 16 or 32, you know something is wrong. So I ignored that and went straight to the actual function that does the shell execution to install the component and I saw something like this…

While ComponentInstalled != 13
 
	installNum = GetNextComponent	
 
	if installNum AND 1 then
		InstallComponent(installNum)
	else
		installNum = installNum * 2
	end if
 
End While

I was dumbfounded and stupified, why is there an “AND 1″ in that code? and what is the purpose of the * 2? So I asked my manager for his input. And after staring at the code for 30 minutes, my brilliant manager shouted “That code is using BIT PATTERNS!“. Instantly, I understood what the code does.

VCREDIST = 1                 '0000 0000 0001
NETFRAMEWORK = 2        '0000 0000 0011
SQLSERVER = 4              '0000 0000 0111
COMPONENT1 = 8           '0000 0000 1111
COMPONENT2 = 16          '0000 0001 1111
COMPONENT3 = 32          '0000 0011 1111
COMPONENT16 = 65536    '1111 1111 1111

That block of code is actually a binary pattern used for flagging the components. Each placeholder in that binary pattern is a FLAG (1 or 0) which actually tells the function GetNextComponent if that component should be installed or not. The *2 is used for shifthing the binary pattern so that the active placeholder for a certain flag will be at the rightmost (last bit) of the pattern! Brilliant!

To solve the issue, I just needed to force the Autorun to Install VCREDIST in Vista by adding this line of code in the Vista Condition of the GetNextComponent function:

VCREDIST = VCREDIST OR 1

So it will always install the Visual C++ 8.0 Redistributable in Vista. The actual problem is that the Autorun Executable thinks that Vista does not need the redistributable and will not install it.  That’s why our product that needs the redistributable will not work.

I have great admiration for the brilliance of the programmer who wrote this. The method is so efficient that instead of declaring 12 Integers for Flag, he would only need to use a single long! My manager told me that the one who wrote this application is actually a “CONTROLLER” programmer. A type of programmer in our organization that writes the firmwares for our hardware controllers.

When writing firmwares that needs to be tight and needs to run in realtime, time complexity is your biggest enemy and you need to code against that. But in an Autorun application for installation, you DONT! In this example, Readability and Maintability was traded-off to efficiency that was not even needed! In turn, a simple issue that can be fixed in a matter of seconds was fixed in half a day.

As Software Developers, we should have the right mindset to decide on what trade offs are necessary. We should not only be Great Programmers who can write great codes but we should transcend to being Great Software Developers who can “take a concept, build a team, set up state of the art development processes, design a software product, the right software product, and produce it. Not just any software product: a high quality software product that solves a problem and delights your users. With documentation. A web page. A setup program. Test cases. Norwegian versions. Bokmål and Nynorsk. Appetizers, dessert, and twenty seven eight-by-ten color glossy photographs with circles and arrows and a paragraph on the back of each one explaining what each one was.”

As for my Friend, he is now working at Oracle. My only wish is that he would be able to work on their database management system where efficiency has higher value than code-readability.