Skip to content

Add benchmark to test compiler performance#16083

Merged
adityapatwardhan merged 8 commits intoPowerShell:masterfrom
daxian-dbw:benchmark
Sep 14, 2021
Merged

Add benchmark to test compiler performance#16083
adityapatwardhan merged 8 commits intoPowerShell:masterfrom
daxian-dbw:benchmark

Conversation

@daxian-dbw
Copy link
Copy Markdown
Member

@daxian-dbw daxian-dbw commented Sep 10, 2021

PR Summary

Add benchmark to test compiler performance.
The benchmark uses the compiler.test.ps1 file in our repo, and try compiling it for every call of CompileFunction.

Instead of compile the compiler.test.ps1 as a whole, the benchmark extracts all top-level functions from the file and compile those functions individually. This is because:

  1. when compiling the whole file, all script in functions or scripts blocks are not really compiled because they only get compiled when they get to run.
  2. compiling individual functions from the file make it easy to update the benchmarks. For example, adding functions that targets a specific compiler scenario.

PR Checklist

@ghost ghost assigned anmenaga Sep 10, 2021
@daxian-dbw daxian-dbw requested review from iSazonov and vexx32 and removed request for TravisEz13 and anmenaga September 10, 2021 23:43
Comment thread test/perf/benchmarks/Engine.Compiler.cs Outdated
@iSazonov iSazonov added the CL-Test Indicates that a PR should be marked as a test change in the Change Log label Sep 13, 2021
@daxian-dbw
Copy link
Copy Markdown
Member Author

daxian-dbw commented Sep 13, 2021

Instead of compiling the compiler.test.ps1 as a whole, the benchmark extracts all top-level functions from the file and compile those functions individually. This is because:

  1. when compiling the whole file, all script in nested functions or scripts blocks are not really compiled because they only get compiled when they get to run.
  2. compiling individual functions from the file make it easy to update the benchmarks. For example, adding functions that targets a specific compiler scenario.

@iSazonov
Copy link
Copy Markdown
Collaborator

I don't see value from one large synthetic perf test (exclusion is startup scenario).
I believe it is more useful to create many tests for simple specific blocks (which can also be used for parser measurements). This way we can check all language constructs and all code paths in the compiler, and easily find bottlenecks and regressions.

@daxian-dbw
Copy link
Copy Markdown
Member Author

daxian-dbw commented Sep 14, 2021

I believe it is more useful to create many tests for simple specific blocks (which can also be used for parser measurements). This way we can check all language constructs and all code paths in the compiler, and easily find bottlenecks and regressions.

My last change made exact that possible. All you need is to add or replace a function in the compiler.test.ps1. The test will read all functions from the test file and run benchmark using each individual function (see the results below). The Use-PSClass is one example that is sort of targeting PowerShell classes.

I'd love to have functions that target specific constructs that we are especially interested in, but I also think having some large-size real-world scripts to track the compilation perf as a whole is helpful.

I'm sure there are better testing script than some of the functions I kept in compiler.test.ps1. I simply chose some big-size functions from build.psm1 and packaging.psm1. You are very welcome to replace some of them and add new ones.

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.19043.1165 (21H1/May2021Update)
Intel Core i7-1065G7 CPU 1.30GHz, 1 CPU, 8 logical and 4 physical cores
.NET SDK=6.0.100-preview.6.21355.2
  [Host]     : .NET 6.0.0 (6.0.21.35212), X64 RyuJIT
  Job-NSAOMR : .NET 6.0.0 (6.0.21.43011), X64 RyuJIT

EnvironmentVariables=POWERSHELL_TELEMETRY_OPTOUT=1  PowerPlanMode=00000000-0000-0000-0000-000000000000  Arguments=/p:DebugType=portable
IterationTime=250.0000 ms  MaxIterationCount=20  MinIterationCount=15
WarmupCount=1

|          Method | FunctionsToCompile |     Mean |    Error |   StdDev |   Median |      Min |      Max |    Gen 0 |   Gen 1 | Allocated |
|---------------- |------------------- |---------:|---------:|---------:|---------:|---------:|---------:|---------:|--------:|----------:|
| CompileFunction | Get-EnvInformation | 15.20 ms | 0.646 ms | 0.744 ms | 15.16 ms | 14.15 ms | 16.87 ms |  66.6667 |       - |    547 KB |
| CompileFunction |     Install-Dotnet | 14.96 ms | 0.694 ms | 0.799 ms | 15.19 ms | 13.43 ms | 16.33 ms |  62.5000 |       - |    270 KB |
| CompileFunction |      New-PSOptions | 14.70 ms | 0.828 ms | 0.953 ms | 14.55 ms | 13.32 ms | 16.62 ms |  62.5000 |       - |    315 KB |
| CompileFunction |    New-UnixPackage | 16.11 ms | 1.239 ms | 1.427 ms | 15.71 ms | 14.49 ms | 19.38 ms | 166.6667 |       - |    918 KB |
| CompileFunction |     Start-CrossGen | 13.84 ms | 1.377 ms | 1.586 ms | 13.20 ms | 12.37 ms | 17.49 ms |  66.6667 |       - |    349 KB |
| CompileFunction |  Start-PSBootstrap | 14.82 ms | 0.741 ms | 0.823 ms | 14.85 ms | 13.13 ms | 16.66 ms |  62.5000 |       - |    557 KB |
| CompileFunction |      Start-PSBuild | 18.11 ms | 0.766 ms | 0.820 ms | 18.02 ms | 16.91 ms | 20.17 ms | 166.6667 |       - |  1,165 KB |
| CompileFunction |    Start-PSPackage | 19.52 ms | 0.916 ms | 1.055 ms | 19.73 ms | 17.96 ms | 22.02 ms | 272.7273 | 90.9091 |  1,580 KB |
| CompileFunction |     Start-PSPester | 16.55 ms | 0.978 ms | 1.126 ms | 16.41 ms | 14.81 ms | 19.12 ms | 142.8571 | 71.4286 |    835 KB |
| CompileFunction |        Use-PSClass | 13.59 ms | 0.744 ms | 0.857 ms | 13.39 ms | 12.70 ms | 15.53 ms |  62.5000 |       - |    296 KB |

throw "Failed to create $createdPackage"
}
}
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using our build and packaging code as a test case 🤣

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I simply chose some big-size functions from build.psm1 and packaging.psm1.
Not sure if that's a good idea though, because the scripting pattern of those functions are more or less the same -- a lot of if statements, a lot switch statements, and a lot command invocations.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is good real life scenario.

// Copyright (c) Microsoft Corporation.
// Licensed under the MIT License.

#if NET6_0
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why just NET6, it would be good if we can compile and run against NET to compare perf between 7.1 and 7.2 right?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test uses the internal API directly. 7.1 and 7.2 doesn't have the internal visible attribute defined for powershell-perf.dll

Comment thread test/perf/benchmarks/assets/compiler.test.ps1
@ghost ghost added Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept and removed Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept labels Sep 14, 2021
@adityapatwardhan adityapatwardhan merged commit 1704ca6 into PowerShell:master Sep 14, 2021
@daxian-dbw daxian-dbw deleted the benchmark branch September 14, 2021 21:50
@ghost
Copy link
Copy Markdown

ghost commented Dec 16, 2021

🎉v7.3.0-preview.1 has been released which incorporates this pull request.:tada:

Handy links:

TrapGodBrim pushed a commit to TrapGodBrim/PowerShell that referenced this pull request Jan 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-Test Indicates that a PR should be marked as a test change in the Change Log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants