Skip to content

Stop pre-emptively creating directories#34521

Merged
amcasey merged 7 commits intomicrosoft:masterfrom
amcasey:LazyDirectoryCreation
Oct 22, 2019
Merged

Stop pre-emptively creating directories#34521
amcasey merged 7 commits intomicrosoft:masterfrom
amcasey:LazyDirectoryCreation

Conversation

@amcasey
Copy link
Copy Markdown
Member

@amcasey amcasey commented Oct 16, 2019

Checking for directory existence is expensive and frequently indicates success. Instead of pre-emptively creating the directory containing a file to be written, attempt to create the file and only do the directory scaffolding if the write fails.

Appears to reduce file write time by 10-20% for a file-I/O heavy partner build.

Thanks to @rbuckton for the suggestion!

@amcasey
Stop pre-emptively creating directories
4ed80b6 
Checking for directory existence is expensive and frequently indicates
success.  Instead of pre-emptively creating the directory containing a
file to be written, attempt to create the file and only do the directory
scaffolding if the write fails.

Appears to reduce file write time by 10-20% for a file-I/O heavy partner
build.

Thanks to @rbuckton for the suggestion!
@amcasey
Inline function to tidy up control flow
b6659e5
@amcasey
Copy link
Copy Markdown
Member Author

amcasey commented Oct 16, 2019

@typescript-bot perf test

@typescript-bot
Copy link
Copy Markdown
Collaborator

typescript-bot commented Oct 16, 2019 •
edited
Loading

Heya @amcasey, I've started to run the perf test suite on this PR at b6659e5. You can monitor the build here. It should now contribute to this PR's status checks.

Update: The results are in!

@amcasey amcasey requested a review from rbuckton October 16, 2019 18:38
@typescript-bot
Copy link
Copy Markdown
Collaborator

typescript-bot commented Oct 16, 2019

@amcasey
The results of the perf run you requested are in!

Here they are:

Comparison Report - master..34521

Metric master 34521 Delta Best Worst
Angular - node (v10.16.3, x64)
Memory used 356,067k (± 0.03%) 356,345k (± 0.02%) +278k (+ 0.08%) 356,151k 356,480k
Parse Time 1.61s (± 0.62%) 1.61s (± 0.58%) +0.00s (+ 0.19%) 1.59s 1.63s
Bind Time 0.85s (± 0.76%) 0.85s (± 0.68%) +0.00s (+ 0.12%) 0.84s 0.87s
Check Time 4.49s (± 0.26%) 4.50s (± 0.38%) +0.01s (+ 0.20%) 4.46s 4.55s
Emit Time 5.28s (± 1.09%) 5.24s (± 0.84%) -0.05s (- 0.89%) 5.17s 5.37s
Total Time 12.23s (± 0.51%) 12.20s (± 0.39%) -0.03s (- 0.26%) 12.12s 12.34s
Monaco - node (v10.16.3, x64)
Memory used 366,969k (± 0.02%) 366,922k (± 0.02%) -46k (- 0.01%) 366,838k 367,120k
Parse Time 1.25s (± 0.56%) 1.25s (± 0.39%) +0.00s (+ 0.24%) 1.24s 1.26s
Bind Time 0.75s (± 0.60%) 0.75s (± 0.74%) 0.00s ( 0.00%) 0.74s 0.76s
Check Time 4.62s (± 0.60%) 4.62s (± 0.41%) -0.00s (- 0.00%) 4.59s 4.67s
Emit Time 2.94s (± 0.49%) 2.94s (± 0.92%) +0.00s (+ 0.07%) 2.89s 3.01s
Total Time 9.56s (± 0.30%) 9.57s (± 0.47%) +0.01s (+ 0.05%) 9.48s 9.67s
TFS - node (v10.16.3, x64)
Memory used 322,541k (± 0.01%) 322,482k (± 0.02%) -59k (- 0.02%) 322,311k 322,658k
Parse Time 0.95s (± 0.58%) 0.95s (± 0.79%) +0.00s (+ 0.21%) 0.94s 0.97s
Bind Time 0.71s (± 1.67%) 0.72s (± 1.24%) +0.00s (+ 0.56%) 0.69s 0.73s
Check Time 4.12s (± 0.46%) 4.12s (± 0.59%) +0.00s (+ 0.07%) 4.06s 4.16s
Emit Time 3.04s (± 0.84%) 3.03s (± 0.67%) -0.00s (- 0.10%) 2.99s 3.08s
Total Time 8.82s (± 0.40%) 8.82s (± 0.44%) +0.01s (+ 0.06%) 8.73s 8.91s
Angular - node (v12.1.0, x64)
Memory used 331,622k (± 0.04%) 331,917k (± 0.04%) +295k (+ 0.09%) 331,391k 332,072k
Parse Time 1.56s (± 0.49%) 1.56s (± 0.85%) +0.01s (+ 0.45%) 1.54s 1.60s
Bind Time 0.84s (± 0.86%) 0.84s (± 0.95%) 0.00s ( 0.00%) 0.83s 0.86s
Check Time 4.42s (± 0.47%) 4.43s (± 0.45%) +0.01s (+ 0.34%) 4.40s 4.49s
Emit Time 5.44s (± 1.27%) 5.45s (± 0.99%) +0.01s (+ 0.18%) 5.34s 5.58s
Total Time 12.25s (± 0.67%) 12.29s (± 0.65%) +0.03s (+ 0.26%) 12.18s 12.51s
Monaco - node (v12.1.0, x64)
Memory used 346,737k (± 0.01%) 346,714k (± 0.02%) -22k (- 0.01%) 346,508k 346,864k
Parse Time 1.21s (± 0.62%) 1.22s (± 0.77%) +0.01s (+ 0.74%) 1.21s 1.25s
Bind Time 0.73s (± 1.72%) 0.72s (± 0.83%) -0.00s (- 0.14%) 0.72s 0.74s
Check Time 4.46s (± 0.30%) 4.49s (± 0.56%) +0.03s (+ 0.74%) 4.44s 4.54s
Emit Time 2.99s (± 0.66%) 3.00s (± 0.61%) +0.01s (+ 0.23%) 2.97s 3.04s
Total Time 9.39s (± 0.34%) 9.44s (± 0.37%) +0.04s (+ 0.48%) 9.37s 9.52s
TFS - node (v12.1.0, x64)
Memory used 304,894k (± 0.02%) 304,877k (± 0.02%) -17k (- 0.01%) 304,734k 305,043k
Parse Time 0.94s (± 0.36%) 0.95s (± 0.74%) +0.01s (+ 0.85%) 0.94s 0.97s
Bind Time 0.68s (± 0.87%) 0.69s (± 1.35%) +0.00s (+ 0.44%) 0.68s 0.72s
Check Time 4.04s (± 0.36%) 4.06s (± 0.70%) +0.03s (+ 0.62%) 4.03s 4.17s
Emit Time 3.08s (± 0.86%) 3.10s (± 1.04%) +0.02s (+ 0.49%) 3.06s 3.21s
Total Time 8.75s (± 0.39%) 8.80s (± 0.73%) +0.05s (+ 0.58%) 8.73s 9.04s
Angular - node (v8.9.0, x64)
Memory used 350,851k (± 0.02%) 351,027k (± 0.02%) +176k (+ 0.05%) 350,895k 351,207k
Parse Time 2.10s (± 0.63%) 2.10s (± 0.36%) -0.01s (- 0.29%) 2.08s 2.12s
Bind Time 0.90s (± 0.56%) 0.89s (± 0.65%) -0.00s (- 0.22%) 0.88s 0.90s
Check Time 5.25s (± 0.41%) 5.26s (± 0.62%) +0.00s (+ 0.08%) 5.21s 5.36s
Emit Time 6.25s (± 1.23%) 6.27s (± 1.29%) +0.02s (+ 0.29%) 6.16s 6.57s
Total Time 14.50s (± 0.61%) 14.51s (± 0.74%) +0.01s (+ 0.06%) 14.33s 14.90s
Monaco - node (v8.9.0, x64)
Memory used 364,682k (± 0.01%) 364,675k (± 0.02%) -7k (- 0.00%) 364,575k 364,837k
Parse Time 1.56s (± 0.48%) 1.56s (± 0.48%) +0.01s (+ 0.45%) 1.55s 1.59s
Bind Time 0.92s (± 0.52%) 0.92s (± 0.84%) +0.00s (+ 0.22%) 0.91s 0.94s
Check Time 5.52s (± 0.51%) 5.54s (± 0.53%) +0.03s (+ 0.45%) 5.48s 5.61s
Emit Time 3.06s (± 0.94%) 3.04s (± 0.58%) -0.02s (- 0.78%) 2.98s 3.07s
Total Time 11.06s (± 0.45%) 11.07s (± 0.36%) +0.01s (+ 0.09%) 10.96s 11.16s
TFS - node (v8.9.0, x64)
Memory used 321,384k (± 0.01%) 321,379k (± 0.02%) -5k (- 0.00%) 321,258k 321,482k
Parse Time 1.27s (± 0.57%) 1.26s (± 0.60%) -0.00s (- 0.24%) 1.24s 1.28s
Bind Time 0.74s (± 0.51%) 0.73s (± 0.88%) -0.00s (- 0.41%) 0.72s 0.74s
Check Time 4.68s (± 0.60%) 4.71s (± 0.68%) +0.03s (+ 0.73%) 4.65s 4.78s
Emit Time 3.22s (± 0.72%) 3.21s (± 0.54%) -0.00s (- 0.12%) 3.18s 3.25s
Total Time 9.91s (± 0.33%) 9.92s (± 0.43%) +0.02s (+ 0.19%) 9.85s 10.03s
Angular - node (v8.9.0, x86)
Memory used 198,953k (± 0.02%) 199,146k (± 0.02%) +192k (+ 0.10%) 199,072k 199,255k
Parse Time 2.02s (± 0.57%) 2.02s (± 0.70%) 0.00s ( 0.00%) 1.99s 2.06s
Bind Time 1.01s (± 0.59%) 1.03s (± 0.90%) +0.01s (+ 1.28%) 1.01s 1.05s
Check Time 4.79s (± 0.61%) 4.78s (± 0.30%) -0.01s (- 0.17%) 4.76s 4.82s
Emit Time 6.14s (± 1.76%) 6.12s (± 2.46%) -0.02s (- 0.41%) 5.83s 6.55s
Total Time 13.97s (± 0.83%) 13.95s (± 1.03%) -0.02s (- 0.13%) 13.63s 14.34s
Monaco - node (v8.9.0, x86)
Memory used 204,190k (± 0.01%) 204,176k (± 0.02%) -15k (- 0.01%) 204,094k 204,270k
Parse Time 1.61s (± 0.51%) 1.61s (± 0.59%) +0.00s (+ 0.06%) 1.59s 1.64s
Bind Time 0.75s (± 1.56%) 0.74s (± 1.42%) -0.01s (- 0.80%) 0.73s 0.78s
Check Time 5.34s (± 1.13%) 5.35s (± 0.75%) +0.01s (+ 0.13%) 5.20s 5.40s
Emit Time 2.89s (± 2.16%) 2.88s (± 2.27%) -0.01s (- 0.35%) 2.82s 3.13s
Total Time 10.59s (± 0.43%) 10.58s (± 0.33%) -0.01s (- 0.08%) 10.52s 10.68s
TFS - node (v8.9.0, x86)
Memory used 180,903k (± 0.02%) 180,926k (± 0.02%) +22k (+ 0.01%) 180,803k 180,982k
Parse Time 1.30s (± 0.57%) 1.32s (± 1.09%) +0.01s (+ 1.15%) 1.29s 1.36s
Bind Time 0.69s (± 0.94%) 0.69s (± 1.09%) +0.00s (+ 0.29%) 0.68s 0.71s
Check Time 4.47s (± 0.65%) 4.50s (± 0.74%) +0.03s (+ 0.65%) 4.43s 4.57s
Emit Time 2.97s (± 0.92%) 2.96s (± 0.32%) -0.01s (- 0.37%) 2.94s 2.98s
Total Time 9.44s (± 0.43%) 9.47s (± 0.48%) +0.04s (+ 0.38%) 9.37s 9.59s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-161-generic
Architecturex64
Available Memory16 GB
Available Memory6 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v10.16.3, x64)
  • node (v12.1.0, x64)
  • node (v8.9.0, x64)
  • node (v8.9.0, x86)
Scenarios
  • Angular - node (v10.16.3, x64)
  • Angular - node (v12.1.0, x64)
  • Angular - node (v8.9.0, x64)
  • Angular - node (v8.9.0, x86)
  • Monaco - node (v10.16.3, x64)
  • Monaco - node (v12.1.0, x64)
  • Monaco - node (v8.9.0, x64)
  • Monaco - node (v8.9.0, x86)
  • TFS - node (v10.16.3, x64)
  • TFS - node (v12.1.0, x64)
  • TFS - node (v8.9.0, x64)
  • TFS - node (v8.9.0, x86)
Benchmark Name Iterations
Current 34521 10
Baseline master 10

@amcasey amcasey added the Domain: Performance Reports of unusually slow behavior label Oct 16, 2019
DanielRosenwasser
DanielRosenwasser reviewed Oct 17, 2019
View reviewed changes
Comment thread src/compiler/program.ts Outdated
@DanielRosenwasser
Copy link
Copy Markdown
Member

DanielRosenwasser commented Oct 17, 2019 •
edited
Loading

👍 but get sign off from others.

sheetalkamat
sheetalkamat requested changes Oct 17, 2019
View reviewed changes
Comment thread src/compiler/program.ts
Comment thread src/compiler/sys.ts Outdated
sheetalkamat
sheetalkamat reviewed Oct 21, 2019
View reviewed changes
Comment thread src/compiler/program.ts Outdated
fileName,
data,
writeByteOrderMark,
(f, d, w) => writeFileWorker(f, d, w),
Copy link
Copy Markdown
Member

@sheetalkamat sheetalkamat Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parameter names seem not very friendly

Copy link
Copy Markdown
Member Author

@amcasey amcasey Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to convey that they're just threaded through to writeFileWorker. If full names are more conventional, I'm happy to expand them.

@amcasey amcasey merged commit 8cf1324 into microsoft:master Oct 22, 2019
@amcasey amcasey deleted the LazyDirectoryCreation branch October 22, 2019 00:41
@microsoft microsoft locked as resolved and limited conversation to collaborators Oct 21, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@DanielRosenwasser DanielRosenwasser DanielRosenwasser left review comments

@sheetalkamat sheetalkamat sheetalkamat approved these changes

@rbuckton rbuckton Awaiting requested review from rbuckton

Assignees

No one assigned

Labels

Domain: Performance Reports of unusually slow behavior

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@amcasey @typescript-bot @DanielRosenwasser @sheetalkamat