Load-Balancing the Build Farm with CruiseControl.NET

Image may be NSFW.
Clik here to view.Our CI system isn't terribly complicated, thankfully.  It evolved from a batch file on a single machine to a farm of VMs running CruiseControl.NET and a single Subversion repository.  The triggering mechanism hasn't changed during this transition though: it's a custom beast, consisting of a post-commit hook on our repository logs revisions into a database, and a custom sourcecontrol task is used to poll the database for revisions that haven't been built yet.  It works fine and naturally creates a balanced build farm: several VMs can be configured to build the same project, and the database sourcecontrol task prevents more than one VM from building the same revision.

As well as it works, it has some problems.  First, the post-commit hook relies heavily on SVN properties, which are a pain to maintain and make it impossible to simply "add a project to the build" without going through a lot of unnecessary configuration.  Moreover, the hook is starting to seriously hold up commits, sometimes as long as 20 seconds.  

Second, and more irritating, the system builds every individual commit.  By that, I mean it builds each and every revision committed to the repository - which may not sound like a bad thing, except that it includes every revision, even those that occurred before the project was added to the CI farm - all the way back to the revision that created the project or branch.  I have to manually fudge the database, adding fake build results to prevent the builds from occurring.  It's not hard, but it is a pain in the ass.  And with the team's overuse of branching I'm finding myself having to fudge more and more. 

I'm really trying to move the system towards what "everyone else does," by which I mean trigger builds by polling source control for changes.  No more database, no more post-commit hook, no more SVN property configuration, just Subversion and CruiseControl.NET.  It would be easy enough to do - simply change our CC.NET project configurations to use the standard SVN source control task.  The problem is that without the database, the farm is no longer automagically load-balancing - every VM in the farm would end up building the same source, which defeats the purpose of the farm.

I figured that I could recoup the load-balancing if I had an "edge" server between Subversion and the build farm.  This server could monitor source control and when necessary trigger a build on one of the VMs in the build farm.  So instead of each farm server determining when a build should occur, there is a single edge server making that decision.

Image may be NSFW.
Clik here to view.

CC.NET ships with the ability to split the build across machines - that is, for a build on one machine (like the edge server) to trigger a build on another machine (like a farm server); however, there is no load-balancing logic available.  So I made some of my own...

Edge Server CC.NET Plugin

The edge server plugin operates on a very simple algorithm:

1. Get a list of farm servers from configuration;
2. Determine which of farm servers are not currently building the project;
3. Fail the build if no farm server is available for the project;
4. Force a build of the project on the first available farm server.
   

If all you want is the project source code, here it is: ccnet.edgeserver.zip (607.87 kb)

Take a look at the configuration of the plugin; I think it will make the code easier to digest.

Edge Server Configuration

The edge server consists of little more than a source control trigger and a list of farm servers:

<cruisecontrol>  <project name="MyProject">    <triggers>      <intervalTrigger seconds="10" />    </triggers>    <sourcecontrol type="svn">      <trunkUrl>svn://sourcecontrol/Trunk</trunkUrl>      <autoGetSource>false</autoGetSource>    </sourcecontrol>    <labeller type="lastChangeLabeller" prefix="MyProject_"/>    <tasks>      <farmBuild>        <farmServers>          <farmServer priority="1" uri="tcp://build-vm-1:21234/CruiseManager.rem" />          <farmServer priority="2" uri="tcp://build-vm-2:21234/CruiseManager.rem" />          <farmServer priority="3" uri="tcp://build-vm-3:21234/CruiseManager.rem" />        </farmServers>      </farmBuild>    </tasks>    <publishers>      <nullTask />    </publishers>  </project></cruisecontrol>

When CC.NET is run with this configuration, it will monitor the subversion repository for changes to the "MyProject" trunk (lines 6-9); note that since autoGetSource is false, no checkout will occur.  The edge server will never have a working copy of the source.

The load-balancing is configured in lines 12-18; in this example, three farm servers are configured in the farm for "MyProject", with build-vm-1 having the highest priority for the build (meaning it will be used first when all three servers are available).  When a change is committed to the repository, the edge server will choose one of these servers based on its availability and priority, and then force it to build the project.

Farm Server Configuration

The farm server is configured just as a normal CC.NET build, except for two key differences: first, it is configured with no trigger; second, a remoteProjectLabeller is used to label the build.  Here's a sample configuration, with mundane build tasks omitted for brevity:

<cruisecontrol>  <project name="MyProject">        <triggers/>    <sourcecontrol type="svn">      <trunkUrl>svn://sourcecontrol/MyProject/Trunk</trunkUrl>      <autoGetSource>true</autoGetSource>    </sourcecontrol>    <labeller type="remoteProjectLabeller">      <project>MyProject</project>      <serverUri>tcp://edgeServer:21234/CruiseManager.rem</serverUri>    </labeller>    <tasks>      <!--              ...       -->    </tasks>    <publishers>      <!--              ...       -->    </publishers>  </project></cruisecontrol> 

Details to note here are:

• the labeller points to the edge server to obtain the build label; this is necessary because labels are generated during the build trigger, which on the farm server is always forced and won't include any source revision information;
• the project name on the farm server matches exactly the project name on the edge server; this is a convention assumed by the plugin.

Source Code Overview

I need a FarmServer type to support the CC.NET configuration layer:

using System;
using System.Collections.Generic;
using System.Text;
using ThoughtWorks.CruiseControl.Core;
using ThoughtWorks.CruiseControl.Remote;
using Exortech.NetReflector;
using ThoughtWorks.CruiseControl.Core.Publishers;
using System.Collections;
using ThoughtWorks.CruiseControl.Core.Util;
namespace CCNET.EdgeServer
{    [ReflectorType( "farmServer" )]    public class FarmServer    {        [ReflectorProperty( "uri" )]        public string Uri;         [ReflectorProperty( "priority" )]        public int Priority;    }
}

No real surprises here.  Each FarmServer instance holds a URI to a CC.NET farm server and it's priority in the balance algorithm. 

The real meat is in the FarmPublisher class:

using System;
using System.Collections.Generic;
using System.Text;
using ThoughtWorks.CruiseControl.Core;
using ThoughtWorks.CruiseControl.Remote;
using Exortech.NetReflector;
using ThoughtWorks.CruiseControl.Core.Publishers;
using System.Collections;
using ThoughtWorks.CruiseControl.Core.Util;
namespace CCNET.EdgeServer
{    [ReflectorType( "farmBuild" )]    public class FarmPublisher : ITask    {        ICruiseManagerFactory factory;         [ReflectorProperty( "Name", Required = false )]        public string EnforcerName;         [ReflectorHash( "farmServers", "uri", Required = true )]        public Hashtable FarmServers;         public FarmPublisher() : this( new RemoteCruiseManagerFactory() ) { }         public FarmPublisher( ICruiseManagerFactory factory )        {            this.factory = factory;            this.EnforcerName = Environment.MachineName;        }         public void Run( IIntegrationResult result )        {            // build a list of available farm servers            //  based off of the plugin configuration            Dictionary<int, ICruiseManager> servers = new Dictionary<int, ICruiseManager>();            FindAvailableFarmServers( result, servers );             if( 0 == servers.Count )            {                Log.Info( "No servers are available for this project at this time" );                result.Status = IntegrationStatus.Failure;                return;            }             // sort the available servers by priority            List<int> keys = new List<int>( servers.Keys );            keys.Sort();             // force a build on the server with the highest             //  priority            ICruiseManager availableServer = servers[ keys[ 0 ] ];            Log.Info( "forcing build on server ..." );            availableServer.ForceBuild( result.ProjectName, EnforcerName );        }        ...

FarmPublisher is configured with a list of FarmServer objects (lines 20-21).  The Run method (starting on line 31) implements the simple load-balancing algorithm:

1. a list of farm servers which are available to build the project is constructed (lines 33-36);
2. if no server is available to build the project, the edge server reports a build failure (line 38-43);
3. the list of available farm servers is sorted by priority (lines 45-47);
4. the project build is started on the farm server configured with the highest priority (line 53).

Determining a list of available farm servers is pretty straightforward:

void FindAvailableFarmServers( IIntegrationResult result, IDictionary<int, ICruiseManager> servers )
{    // predicate to locate a server that isn't actively building     // the current project    Predicate<ProjectStatus> predicate = delegate( ProjectStatus prj )    {        return IsServerAvailableToBuildProject( result, prj );    };     // check the status of each configured farm server    foreach( FarmServer server in FarmServers.Values )    {        ICruiseManager manager = null;        try        {            manager = ( ICruiseManager )factory.GetCruiseManager( server.Uri );             // get a local copy of server's current project status snapshot            List<ProjectStatus> projects = new List<ProjectStatus>( manager.GetProjectStatus() );            if( null != projects.Find( predicate ) )            {                // add the farm server to the list of available servers,                 //  keyed by its configured priority                servers[ server.Priority ] = manager;            }        }        catch( Exception e )        {            Log.Warning( e );        }    }
}

Available servers are saved in the servers dictionary, keyed by their configured priority.  The availability of each farm server listed in the task configuration is checked by obtaining the status of the farm server's projects, and passing them to the IsServerAvaialbleToBuildProject method:

bool IsServerAvailableToBuildProject( IIntegrationResult result, ProjectStatus prj )
{    if( null == prj || null == result )    {        return false;    }    bool status = (                  // project name must match        StringComparer.InvariantCultureIgnoreCase.Equals( result.ProjectName, prj.Name ) &&         // integrator status must be "running"        prj.Status == ProjectIntegratorState.Running &&                         // build activity must be "sleeping"        prj.Activity.IsSleeping()    );    return status;
} 

which simply returns true when:

• the farm server configuration contains the project,
  
• the project is currently running, and
  
• the project isn't currently building.

Download

This code is basically spike-quality at this point.  I fully expect to throw this away in favor of something better (or get my manager to splurge for TeamCity).  There's still a lot of stuff to do.  E.g., the algorithm assumes that a farm server is capable of building more than one project at a time - that is, if a farm server is busy building one project, it can still be available to build another concurrently.  My assumption is that I'll manage this with the farm server priority configuration.  I'd like to leverage the queuing features available in CC.NET; however, I see no way of querying the queue status of a farm server in the CC.NET API.  But at least I can start disabling the post-commit hook on our repository.

The project contains the code, a test/demo, and just a few basic unit tests; I'll update the download if/when the project matures.  If you use it, or have any suggestions, please let me know in the comments of this post.  Enjoy!

ccnet.edgeserver.zip (607.87 kb)