Executing MSBuild Targets in Parallel – Part 1


[ In this series – Part 1, Part 2 ]

Coming in the next release of the MSBuild Extension Pack is a new task which allows you to easily run Targets in parallel. This will be limited to the .NET 4.0 releases, so next up that’s 4.0.5.0. (you can try the task now by getting changeset 74609).

Typically in MSBuild we execute a sequence of targets and MSBuild will do just that; execute sequentially. If you have simple scripts that run quickly then this may not be a problem. If however you have complex scripts that take considerable time to execute, the new parallel functionality may be useful to reduce execution times.

In the following samples I’ll work through how the tasks can be used and various things to be aware of.

Below I have a sample which executes three targets. In each target there is a Sleep task so we can provide some metrics as we extend the sample.

<Project ToolsVersion="4.0" DefaultTargets="Default" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Import Project="$(MSBuildExtensionsPath)\ExtensionPack\4.0\MSBuild.ExtensionPack.tasks"/>
    <Target Name="Default" DependsOnTargets="Target1;Target2;Target3"/>
    <Target Name="Target1">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="1000"/>
    </Target>
    <Target Name="Target2">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="4000"/>
    </Target>
    <Target Name="Target3">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="2000"/>
    </Target>
</Project>

Executing the above sample we get the expected 7 second or so build time.

SNAGHTML11137c7

Assuming there are no dependencies between these targets, the fastest time we we can execute this is around 4 seconds, the time of our slowest target, plus the initialization cost.

Rather than having to run these targets sequentially, what I really want to do is run non-dependent targets in parallel and reduce the build time. In the next sample I’ve removed the DependsOnTargets and added the parallel task to execute BuildTargetsInParallel

<Project ToolsVersion="4.0" DefaultTargets="Default" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Import Project="$(MSBuildExtensionsPath)\ExtensionPack\4.0\MSBuild.ExtensionPack.tasks"/>
    <Target Name="Default">
        <MSBuild.ExtensionPack.Framework.Parallel TaskAction="BuildTargetsInParallel" Targets="Target1;Target2;Target3"/>
    </Target>
    <Target Name="Target1">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="1000"/>
    </Target>
    <Target Name="Target2">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="4000"/>
    </Target>
    <Target Name="Target3">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="2000"/>
    </Target>
</Project>

Executing the above sample we see our build time drop to 4.60 seconds.

SNAGHTML1818dc5

This is a great reduction in build time, but not as close to the four second mark as we may have expected. The reason that we have a slightly higher build time is that the parallel task needs to start MSBuild and initialize a new instance of your build file before it executes each target you have requested to execute in parallel. This is an important point to remember if you have a large build file and a high initialization cost. The aim is not to simply run every target in parallel, but rather to find an efficient way to run your build in parallel.

In this first parallel sample I ran all targets in parallel, initiating the build file four times. It may be more efficient to run sets of targets in parallel. The parallel task allows us to specify BuildTargetSetsInParallel.

<Project ToolsVersion="4.0" DefaultTargets="Default" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <Import Project="$(MSBuildExtensionsPath)\ExtensionPack\4.0\MSBuild.ExtensionPack.tasks"/>
    <ItemGroup>
        <MyTargetSets Include="1">
            <Targets>Target2</Targets>
        </MyTargetSets>
        <MyTargetSets Include="2">
            <Targets>Target1;Target3</Targets>
        </MyTargetSets>
    </ItemGroup>
    <Target Name="Default">
        <MSBuild.ExtensionPack.Framework.Parallel TaskAction="BuildTargetSetsInParallel" Targets="@(MyTargetSets)"/>
    </Target>
    <Target Name="Target1">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="1000"/>
    </Target>
    <Target Name="Target2">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="4000"/>
    </Target>
    <Target Name="Target3">
        <MSBuild.ExtensionPack.Framework.Thread TaskAction="Sleep" Timeout="2000"/>
    </Target>
</Project>

Executing this sample we get a build time of 4.18 seconds.

SNAGHTML1b84aae

In this case we have initiated 3 rather than 4 instances of our build file and chosen to execute two parallel sets of targets. Note that I have the longest running target running by itself and have the remaining targets running sequentially on another instance. Given that the execution time of the remaining targets was less than Target2, there is no need to run more than two sets at once.

In this simple example we’ve managed to reduce our build time by over 42%.

In Part 2 I’ll cover how we maintain state when running in parallel as well as some limitations and tips.

Mike

About these ads

13 thoughts on “Executing MSBuild Targets in Parallel – Part 1

  1. I am having an issue with changing the verbosity of the individual log files usning the extensionpack task with MultiLogVerbosity=”normal”or MultiLogVerbosity=”diagnostic”

    I get the following error for the above 2 examples:

    C:\Builds\Default Agent\TSS\ParallellBuildTest\BuildType\TSSBuild.proj (795): The “MSBuild.ExtensionPack.Framework.Parallel” task failed unexpectedly.System.ArgumentException: Requested value ‘diagnostic’ was not found. at System.Enum.TryParseEnum(Type enumType, String value, Boolean ignoreCase,
    The doc/help for the ext pack says:
    MultiLogVerbosity – Specifies the verbosity to log to the individual files with. Default is diagnostic

  2. Pingback: Executing MSBuild Targets in Parallel – Part 1 « Mike Fourie « Hornet Dear Bernard

  3. Hi,

    IS there a way of running the same target in parallel more than once. What I have is an ItemGroup, and the items in the group are passed to an MSBuild task. I’d like the execution of the MSBuild task to be run in parallel with each item in the ItemGroup…
    e.g.

    ItemA
    SomeA

    ItemB
    SomeB

    ItemC
    SomeC

  4. Pingback: Willy's Cave

  5. Pingback: Executing MSBuild Targets in Parallel – Part 2 « Mike Fourie

  6. Mike,

    Thanks for adding the parallel target task.
    I was giving it a try but am running into issues with dependencies and as a result Msbuild throws an exception.

    Example:

    error:

    c:\project\Main.proj(23,5): error : U
    nhandled Exception: Microsoft.Build.Shared.InternalErrorException: MSB00
    01: Internal MSBuild Error: The process cannot access the file ‘c:\project\a.sln.metaproj.tmp’ becau
    se it is being used by another process.
    c:\project\a.proj(23,5): error : =
    ============

    Inside a.proj;b.proj;c.proj I call sln files among other actions.
    How do I deal with with the dependency issues in parallel target execution?

    Using the suggested target sets would solve the issue but it would not be the most efficient way of running a complex build, since you would have to hard code dependencies between sets. That would mean set 2 would have to wait until set 1 completes, even if there’s only one project in set 1 we are waiting for and some projects in set 2 could already run.

  7. Pingback: MSBuild Extension Pack – April 2012 « Mike Fourie

  8. Hi Mike,

    I am using fxcop activity given in MSBuildExtensionPack, but I have a issue setting up fxcop path for the below task. By default fxcop is looking at C:\program files\Microsoft fxcop 1.36\.
    On my server it is located at C:\program files (x86)\Microsoft fxcop 1.36\ and I am getting error FxCopCmd.exe was not found in the default location. Use FxCopPath to specify it. Searched at: C:\Program Files\Microsoft FxCop 1.36 and \Microsoft FxCop 10.0

    Whne i used Fxcoppath I am getting Access denied /r error.

    Can you please guide me how to use fxcop task here.

    Thanks & Regards,
    Chandra Vuppala.

  9. Pingback: Running Targets in Parallel in MSBuild « Mike Fourie

  10. Pingback: Interesting .NET Links - February 29 , 2012 | Tech Blog

  11. Pingback: Dew Drop – February 28, 2012 (#1,276) | Alvin Ashcraft's Morning Dew

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s