Github A methodical approach to looking at F# compile times · Discussion #11134...

Looking to improve your F# compile times? Read on!

Some context

The F# compiler has steadily been getting faster over time, and that's going to continue. But there's plenty of work to be done.

At the time of writing, the F# compiler could benefit from several items of work:

1. More detailed work where we simply improve compile times with the existing system we have. As the link above shows, this can yield substantial results.
2. Have the F# compiler work with a standardized compiler server host. A compiler server keeps a compiler "warm" and caches metadata references so that there's simply less work to be done for successive compile runs.
3. Implement support for emitting reference assemblies. MSBuild will just copy the output of a build into referenced project directories if the signatures don't change, which should reduce the number of rebuilds in scenarios where a highly depended-upon project is referenced by several other projects.
4. Implement incremental compilation within projects, so that only files you edit and the files they depend on are re-checked by the compiler. Depending on what you edit, this can significantly reduce typechecking for a specific project in a solution.
5. Combine the above pieces and plug into a "hot reload"-like mechanism.

Aside from (1), these are all incredibly challenging and expensive things to do. That doesn't mean they won't get done, but it does mean that it may take time because the team has to balance other priorities and can't be caught in a world where nothing of value is delivered for users of a long period of time.

In the interim, there are some things you can do as a user to help everyone (including the F# team) understand your compile times.

Understand your overall build times

Most F# projects that run on .NET use the .NET build system, MSBuild. MSBuild has non-negligible overhead, and as a first step you should get a handle on how much time is spent "in MSBuild" as opposed to actually compiling code.

I recommend running these four commands at your solution level:

dotnet clean
dotnet msbuild /clp:PerformanceSummary > normal-summary.txt
dotnet clean
dotnet msbuild /clp:PerformanceSummary /m:1  > serial-summary.txt

The first two commands clean your solution and produce an MSBuild performance summary with MSBuild building your solution how it normally would. The second two clean and force MSBuild to run with only one process.

Here is an example of a perf summary for a brand new dotnet new mvc -lang F# project:

Microsoft (R) Build Engine version 16.8.0+126527ff1 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  yeet -> /Users/phillip/scratch/yeet/bin/Debug/net5.0/yeet.dll

Project Evaluation Performance Summary:
      538 ms  /Users/phillip/scratch/yeet/yeet.fsproj    1 calls

Project Performance Summary:
    10454 ms  /Users/phillip/scratch/yeet/yeet.fsproj    1 calls
    
Target Performance Summary:
    ...
    (many small, negligible calls elided)
    ...
      131 ms  FindAssembliesWithReferencesTo             1 calls
      174 ms  ResolvePackageAssets                       1 calls
      201 ms  ResolveTargetingPackAssets                 1 calls
      212 ms  GenerateDepsFile                           1 calls
      338 ms  ResolvePackageFileConflicts                1 calls
      740 ms  Copy                                       5 calls
     3906 ms  Fsc                                        1 calls
     3953 ms  ResolveAssemblyReference                   1 calls

As you can see, MSBuild overhead was actually higher than the time spend in the F# compiler!

For a larger project, here's a summary of the XPlot solution that I maintain:

Microsoft (R) Build Engine version 16.8.3+39993bd9d for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  XPlot.D3 -> /Users/phillip/repos/XPlot/src/XPlot.D3/bin/Debug/netstandard2.0/XPlot.D3.dll
  XPlot.Plotly.Interactive -> /Users/phillip/repos/XPlot/src/XPlot.Plotly.Interactive/bin/Debug/net5.0/XPlot.Plotly.Interactive.dll
  XPlot.GoogleCharts -> /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts/bin/Debug/netstandard2.0/XPlot.GoogleCharts.dll
  XPlot.Plotly -> /Users/phillip/repos/XPlot/src/XPlot.Plotly/bin/Debug/netstandard2.0/XPlot.Plotly.dll
  XPlot.GoogleCharts.Deedle -> /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts.Deedle/bin/Debug/netstandard2.0/XPlot.GoogleCharts.Deedle.dll
  BaselineGenerator -> /Users/phillip/repos/XPlot/tools/BaselineGenerator/bin/Debug/netcoreapp3.1/BaselineGenerator.dll
  XPlot.Plotly.Tests -> /Users/phillip/repos/XPlot/tests/XPlot.Plotly.Tests/bin/Debug/net5.0/XPlot.Plotly.Tests.dll

Project Evaluation Performance Summary:
       95 ms  /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts.Deedle/XPlot.GoogleCharts.Deedle.fsproj   1 calls
      114 ms  /Users/phillip/repos/XPlot/tools/BaselineGenerator/BaselineGenerator.fsproj   1 calls
      114 ms  /Users/phillip/repos/XPlot/tests/XPlot.Plotly.Tests/XPlot.Plotly.Tests.fsproj   1 calls
      150 ms  /Users/phillip/repos/XPlot/src/XPlot.D3/XPlot.D3.fsproj   1 calls
      163 ms  /Users/phillip/repos/XPlot/src/XPlot.Plotly/XPlot.Plotly.fsproj   1 calls
      242 ms  /Users/phillip/repos/XPlot/src/XPlot.Plotly.Interactive/XPlot.Plotly.Interactive.fsproj   1 calls
      372 ms  /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts/XPlot.GoogleCharts.fsproj   1 calls

Project Performance Summary:
     3665 ms  /Users/phillip/repos/XPlot/src/XPlot.D3/XPlot.D3.fsproj   1 calls
     5537 ms  /Users/phillip/repos/XPlot/src/XPlot.Plotly.Interactive/XPlot.Plotly.Interactive.fsproj   1 calls
     5965 ms  /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts/XPlot.GoogleCharts.fsproj   9 calls
                  2 ms  GetTargetFrameworks                        2 calls
                  1 ms  GetNativeManifest                          2 calls
                  0 ms  GetCopyToOutputDirectoryItems              2 calls
     7131 ms  /Users/phillip/repos/XPlot/src/XPlot.Plotly/XPlot.Plotly.fsproj   9 calls
                  1 ms  GetTargetFrameworks                        2 calls
                  0 ms  GetNativeManifest                          2 calls
                  0 ms  GetCopyToOutputDirectoryItems              2 calls
     9494 ms  /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts.Deedle/XPlot.GoogleCharts.Deedle.fsproj   1 calls
    12220 ms  /Users/phillip/repos/XPlot/tools/BaselineGenerator/BaselineGenerator.fsproj   1 calls
    12520 ms  /Users/phillip/repos/XPlot/tests/XPlot.Plotly.Tests/XPlot.Plotly.Tests.fsproj   1 calls
    12876 ms  /Users/phillip/repos/XPlot/XPlot.sln       1 calls

Target Performance Summary:
    ...
    (eliding everything under 100ms)
    ...
      138 ms  _CopyOutOfDateSourceItemsToOutputDirectory   1 calls
      158 ms  _CopyFilesMarkedCopyLocal                  4 calls
      176 ms  FindReferenceAssembliesForReferences       7 calls
      178 ms  PaketRestore                               6 calls
      192 ms  _HandlePackageFileConflicts                7 calls
      378 ms  ResolvePackageAssets                       7 calls
      396 ms  ResolveAssemblyReferences                  7 calls
      686 ms  GenerateBuildDependencyFile                7 calls
     1796 ms  _GetProjectReferenceTargetFrameworkProperties   7 calls
    12853 ms  Build                                      8 calls
    18460 ms  ResolveProjectReferences                   7 calls
    32853 ms  CoreCompile                                7 calls
    
Task Performance Summary:
    ...
    (eliding all tasks under 100ms)
    ...
      178 ms  ResolvePackageFileConflicts                7 calls
      357 ms  Copy                                      24 calls
      366 ms  ResolvePackageAssets                       7 calls
      389 ms  ResolveAssemblyReference                   7 calls
      676 ms  GenerateDepsFile                           7 calls
    32842 ms  Fsc                                        7 calls
    33040 ms  MSBuild                                   17 calls

We have two things to look at, Target and Task performance. Looking at Target data, we can see that MSBuild overhead is quite high (ResolveProjectReferences has me raising my eyebrows), but the CoreCompile target is what's actually the most expenssive. That can be confirmed by looking at the specific Fsc task, where you'll see the timings as nearly identical.

Note that in these reports I did not restrict MSBuild to one process for building.

Here's what it looks like with /m:1:

Microsoft (R) Build Engine version 16.8.3+39993bd9d for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  XPlot.GoogleCharts -> /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts/bin/Debug/netstandard2.0/XPlot.GoogleCharts.dll
  XPlot.GoogleCharts.Deedle -> /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts.Deedle/bin/Debug/netstandard2.0/XPlot.GoogleCharts.Deedle.dll
  XPlot.Plotly -> /Users/phillip/repos/XPlot/src/XPlot.Plotly/bin/Debug/netstandard2.0/XPlot.Plotly.dll
  XPlot.D3 -> /Users/phillip/repos/XPlot/src/XPlot.D3/bin/Debug/netstandard2.0/XPlot.D3.dll
  BaselineGenerator -> /Users/phillip/repos/XPlot/tools/BaselineGenerator/bin/Debug/netcoreapp3.1/BaselineGenerator.dll
  XPlot.Plotly.Tests -> /Users/phillip/repos/XPlot/tests/XPlot.Plotly.Tests/bin/Debug/net5.0/XPlot.Plotly.Tests.dll
  XPlot.Plotly.Interactive -> /Users/phillip/repos/XPlot/src/XPlot.Plotly.Interactive/bin/Debug/net5.0/XPlot.Plotly.Interactive.dll

Project Evaluation Performance Summary:
       42 ms  /Users/phillip/repos/XPlot/tests/XPlot.Plotly.Tests/XPlot.Plotly.Tests.fsproj   1 calls
       45 ms  /Users/phillip/repos/XPlot/tools/BaselineGenerator/BaselineGenerator.fsproj   1 calls
       48 ms  /Users/phillip/repos/XPlot/src/XPlot.Plotly/XPlot.Plotly.fsproj   1 calls
       49 ms  /Users/phillip/repos/XPlot/src/XPlot.D3/XPlot.D3.fsproj   1 calls
       58 ms  /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts.Deedle/XPlot.GoogleCharts.Deedle.fsproj   1 calls
       66 ms  /Users/phillip/repos/XPlot/src/XPlot.Plotly.Interactive/XPlot.Plotly.Interactive.fsproj   1 calls
      234 ms  /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts/XPlot.GoogleCharts.fsproj   1 calls

Project Performance Summary:
     2788 ms  /Users/phillip/repos/XPlot/src/XPlot.D3/XPlot.D3.fsproj   1 calls
     2843 ms  /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts.Deedle/XPlot.GoogleCharts.Deedle.fsproj   1 calls
     4578 ms  /Users/phillip/repos/XPlot/src/XPlot.GoogleCharts/XPlot.GoogleCharts.fsproj   9 calls
                  2 ms  GetTargetFrameworks                        2 calls
                  1 ms  GetNativeManifest                          2 calls
                  0 ms  GetCopyToOutputDirectoryItems              2 calls
     4645 ms  /Users/phillip/repos/XPlot/src/XPlot.Plotly.Interactive/XPlot.Plotly.Interactive.fsproj   1 calls
     4669 ms  /Users/phillip/repos/XPlot/tools/BaselineGenerator/BaselineGenerator.fsproj   1 calls
     4934 ms  /Users/phillip/repos/XPlot/tests/XPlot.Plotly.Tests/XPlot.Plotly.Tests.fsproj   1 calls
     5463 ms  /Users/phillip/repos/XPlot/src/XPlot.Plotly/XPlot.Plotly.fsproj   9 calls
                  1 ms  GetTargetFrameworks                        2 calls
                  1 ms  GetNativeManifest                          2 calls
                  0 ms  GetCopyToOutputDirectoryItems              2 calls
    30524 ms  /Users/phillip/repos/XPlot/XPlot.sln       1 calls

Target Performance Summary:
    (more elided small calls)
      114 ms  FindReferenceAssembliesForReferences       7 calls
      118 ms  _CopyFilesMarkedCopyLocal                  4 calls
      155 ms  _HandlePackageFileConflicts                7 calls
      216 ms  ResolvePackageAssets                       7 calls
      574 ms  GenerateBuildDependencyFile                7 calls
      682 ms  ResolveAssemblyReferences                  7 calls
    27106 ms  CoreCompile                                7 calls
    30504 ms  Build                                      8 calls

Task Performance Summary:
    (more elided small calls)
      146 ms  ResolvePackageFileConflicts                7 calls
      213 ms  ResolvePackageAssets                       7 calls
      243 ms  Copy                                      24 calls
      566 ms  GenerateDepsFile                           7 calls
      678 ms  ResolveAssemblyReference                   7 calls
    27102 ms  Fsc                                        7 calls
    30531 ms  MSBuild                                   17 calls

As you can see, MSBuild overhead is reduced a lot, but my overall build time more than doubled since nothing was built in parallel.

Takeaways from this data

So, what can we learn from this? Several things:

1. MSBuild overhead is real, but it's still worth it for multi-project solutions
2. The F# compiler is indeed the main source of time spent building XPloit, no matter which way you swing it
3. Timings given are CPU time, so a parallel run may take less but report more total CPU time than a serial run. That's because it's using multiple CPU cores!

This is good, since it's what you'd actually want to observe: more time spent compiling than not.

If you do not see data like this, and MSBuild targets/tasks are indeed your main source of timings, then something might be wrong with your build. Most of your build time should be in CoreCompile and Fsc. If it's not, then it's time to dig deeper into why MSBuild is spending so much time doing stuff in your solution.

Binlogs

Binlogs are the best tool for figuring out exactly what your build is doing. If you have a lot of MSBuild overhead in your build times, this will tell you every single thing that MSBuild is doing when you build your codebase.

The next tool you can use is an MSBuild binary log. These are not mean to be used for performance analysis, because producing a binlog will incur overhead that can mess with your build. However, if you find that the timings in a binlog are proportional to your timings in a performance summary, it may be helpful to look at the reported timings.

To produce a binlog, and to avoid confusion over multiple processes, do this:

dotnet clean
dotnet msbuild /bl /m:1

This will produce produce a binary log file that you can view online or install a tool to view on your own machine

Some specific things it will show that are relevant to building F# projects:

• Exactly which references are being passsed to the compiler for each project. Are you referencing enormous .dlls across your entire codebase? Are you referencing way too many .dlls per project? Is that needed? Recall that the F# compiler must crack metadata references during compilation since that's one of the inputs to typechecking. The more .dlls and the bigger they are, the more time is spent doing that.
• Which targets are being called by which project? Is that expected? Could you adjust your build to call an expensive target only once, rather than each time a project is called?

I'm not personally an MSBuild expert so I can't tell you everything about what the binlog would say about F# projects. But I've found value in the above two points in analyzing builds in the past, and have been able to adjust a build to shave off several seconds just be updating dependencies and project references.

ETL traces

The big one. If you're on Windows, you can use PerfView for this. If not, you'll need to install the dotnet-trace global tool.

An ETL trace of a build will give an extremely detailed view of your build from start to finish. You need specific tools to analyze them (PerfView, DotTrace, converting to chrome or SpeedScope). But their information is rich. It will show:

• CPU time overview information
• Where in the compiler CPU time is being spent. For example, eactly which percentage of the total CPU time of the sample is spent in the TcExprThen function in the typechecker, or how much is spent there and in all other functions that it calls.
• Overall memory allocation information, how much time is spent in GC, etc.
• Where in the compiler memory is being allocated and what the call chain for that looks like.

This is the meat and potatoes of detailed performance work in the F# compiler. It is not easy to do, and at this point it's often times best to just submit this data to the F# team and ask them if they notice anything that seems off.

Often times, even though build times are slow, the trace data indicates that everything looks normal. That would mean, unfortunately, that you've run into a case of "the F# compiler should be faster".

But sometimes, way more CPU or memory is being used by something than we'd expect. That's usually a signal that there's some low-hanging fruit for us to go pick. Sometimes the "low-hanging fruit" is actually very difficult to resolve, but it's isolated from any architectural changes and would thus be a low-risk kind of change to make. We'll either fix these outright or explain what the issue is and what a motivated contributor could consider doing to resolve the issue.

There is an existing catalogue of issues that usually have ETL trace data with them tagged here: https://github.com/dotnet/fsharp/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3ATenet-Performance

Hot tips

What some hot tips that will almost certainly make your life better? Here they are:

Make sure you're using the latest F# compiler

This one should be obvious, but I've ran into enough people who use older compilers that it has to come first.

The only way to get performance improvements is to update your compiler and toolset.

If you're still on an older .NET Core LTS or something, you will likely see a significant jump in compile times just by updating your compiler. Please do it!

Split up big projects into smaller ones

Do you have "I have one giant project with everything in it" syndrome like we do in dotnet/fsharp? Well I've got a quick fix for you!

You can proabbly improve your build times significantly if you split up a huge project in your solution. This is for several reasons:

1. More ability to benefit from MSBuild parallelism
2. If you only ever touch one part of that project and not the rest, if the rest was in a different project and didn't change, it wouldn't be recompiled every time

Splitting things into different projects is, in a way, a form of incremental building. There are several circumstances that can lead to a rebuild of things unexpectedly, but you'll find that builds go much quicker if you can split things up more.

Just don't go overboard with this. Putting every little thing into its own project will just make MSBuild overhead too high, so try to group things logically.

Use F# signature files (tooling performance)

Do you have an unavoidably large project? If so, consider using F# signature files (.fsi) for each implementation file (.fs).

The F# compiler will now actively not typecheck several things in a tooling scenario if you have explicit signature files:

• If you're editing a source file but you don't change the matching signature file, then only that file will get typechecked in tools. Nothing else that could depend on it will be typechecked while editing code.
• For modules/types/etc. you depend on when you are editing code (that are in your project), if the signature files for those constructs remain unchanged while editing, their signature data will be re-used and nothing "above" where you're at (from a dependency perspective) will be re-typechecked.

What this means when editing code is that all tooling should be significantly snappier for a large project with a lot of code.

Don't include type providers in your build transitive build graph

Type Providers are a great tool for a variety of applications, but have a non-negligible, unavoidable performance cost when you merely reference one: #7907

The reason is that the F# compiler needs to instantiate a type provider at build time and inspect provided types. This can add upwards of 500ms to each project that depends on one.

Look at your binlog. If you have projects that are being passed type provider references and you aren't actually using them in that project, fix your build!

This hot tip can easily shave several seconds off of your build. It's easy for dependencies to sneak into your build unexpectedly.

Don't get too fancy with type constraints and typelevel programming

Constraint programming in F# can be fun and expressive and useful, but it's possible to go overboard. There is a tendency for some people to get a little "type happy" and start to write typelevel-style programs using the F# constraint system.

In general, try not to write code that over-abstracts with constraints just for the sake of not repeating yourself a few times. Repeating yourself 2 or 3 times really isn't a big deal.

If you really don't want to repeat yourself, or you must make heavy use of typelevel abstractions and constraint-based programming, I suggest the following:

Use FSharpPlus instead instead of going all-in on your own. FSharpPlus is extremely well-written and likely has almost every abstraction you care about anyways. Just use that.

If you really want to do heavy constraint-based programming yourself, you're on your own. The authors of FSharpPlus have ran into and found ways to work around most problems (w.r.t performance) that the F# compiler has, so that's why I recommend using that. If you're set on doing things yourself, just know that you're giving up all of that knowledge as well. Constraint solving in the F# compiler can sometimes lead to exponential compile-time paths and you really don't want to be dealing with that. Just use FSharpPlus if you're going to make heavy use of this kind of programming.

It's also worth noting that heavy use of constraint-based and/or typelevel programming can make your code very difficult for others to understand. Most people really do struggle to understand these kinds of abstractions and you can often be better off by avoiding their use across a wider team or group of contributors.

Try to avoid generating enormous types and match expressions

Lastly, depending on what you do with nested types and matchin on their structures, you can quickly get exploding compile-times too. There's just way too many things the compiler has to solve when you have large, nested types that you're pattern matching on to a different degree (think a union with 100 cases, each made up of optional records with optional data, each of which you expand into sepcific patterns).

We'd love for this not to be a performance issue, but there's really no way around it. Extremely complex nested data with complex patterns torture the compiler to death.