Skip to content

Conversation

@sandersn
Copy link
Member

Redo #28564

I haven't looked at the baseline changes; maybe they're horrible.

@sandersn
Copy link
MemberAuthor

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Feb 7, 2020

Heya @sandersn, I've started to run the perf test suite on this PR at 4e3c2c5. You can monitor the build here. It should now contribute to this PR's status checks.

Update: The results are in!

@sandersnsandersn added the For Milestone Bug PRs that fix a bug with a specific milestone label Feb 7, 2020
@typescript-bot
Copy link
Collaborator

@sandersn
The results of the perf run you requested are in!

Here they are:

Comparison Report - master..36665

Metricmaster36665DeltaBestWorst
Angular - node (v10.16.3, x64)
Memory used356,858k (± 0.01%)358,602k (± 0.03%)+1,745k (+ 0.49%)358,432k358,954k
Parse Time1.61s (± 0.50%)1.62s (± 0.58%)+0.01s (+ 0.68%)1.60s1.64s
Bind Time0.89s (± 0.95%)0.88s (± 1.01%)-0.01s (- 0.56%)0.86s0.90s
Check Time4.67s (± 0.44%)4.69s (± 0.68%)+0.01s (+ 0.28%)4.62s4.77s
Emit Time5.22s (± 0.25%)5.20s (± 0.36%)-0.02s (- 0.40%)5.15s5.23s
Total Time12.40s (± 0.23%)12.39s (± 0.35%)-0.00s (- 0.02%)12.31s12.52s
Monaco - node (v10.16.3, x64)
Memory used364,682k (± 0.02%)364,676k (± 0.02%)-6k (- 0.00%)364,503k364,824k
Parse Time1.25s (± 0.66%)1.25s (± 0.27%)-0.00s (- 0.08%)1.25s1.26s
Bind Time0.78s (± 0.48%)0.77s (± 0.58%)-0.00s (- 0.39%)0.76s0.78s
Check Time4.68s (± 0.70%)4.68s (± 0.46%)-0.00s (- 0.04%)4.64s4.74s
Emit Time2.90s (± 0.77%)2.90s (± 0.69%)-0.00s (- 0.14%)2.86s2.95s
Total Time9.61s (± 0.45%)9.61s (± 0.35%)-0.01s (- 0.06%)9.54s9.69s
TFS - node (v10.16.3, x64)
Memory used324,299k (± 0.07%)324,182k (± 0.02%)-117k (- 0.04%)324,068k324,468k
Parse Time0.94s (± 0.52%)0.95s (± 0.50%)+0.00s (+ 0.32%)0.94s0.96s
Bind Time0.75s (± 1.54%)0.75s (± 0.64%)+0.00s (+ 0.00%)0.74s0.76s
Check Time4.23s (± 0.38%)4.22s (± 0.65%)-0.01s (- 0.21%)4.16s4.28s
Emit Time3.01s (± 0.76%)3.00s (± 1.58%)-0.01s (- 0.46%)2.90s3.10s
Total Time8.93s (± 0.43%)8.91s (± 0.68%)-0.02s (- 0.22%)8.75s9.03s
Angular - node (v12.1.0, x64)
Memory used332,632k (± 0.07%)334,315k (± 0.02%)+1,683k (+ 0.51%)334,123k334,442k
Parse Time1.57s (± 0.54%)1.57s (± 0.45%)-0.00s (- 0.00%)1.55s1.58s
Bind Time0.86s (± 1.10%)0.87s (± 1.04%)+0.01s (+ 0.69%)0.85s0.89s
Check Time4.62s (± 0.80%)4.56s (± 0.54%)-0.06s (- 1.26%)4.49s4.61s
Emit Time5.41s (± 0.57%)5.38s (± 0.49%)-0.03s (- 0.48%)5.32s5.44s
Total Time12.46s (± 0.47%)12.38s (± 0.31%)-0.08s (- 0.63%)12.33s12.50s
Monaco - node (v12.1.0, x64)
Memory used344,551k (± 0.02%)344,532k (± 0.02%)-19k (- 0.01%)344,450k344,693k
Parse Time1.22s (± 0.67%)1.21s (± 0.80%)-0.01s (- 0.90%)1.19s1.23s
Bind Time0.75s (± 0.80%)0.75s (± 0.94%)+0.00s (+ 0.54%)0.74s0.77s
Check Time4.54s (± 0.35%)4.54s (± 0.56%)+0.00s (+ 0.07%)4.48s4.59s
Emit Time2.94s (± 0.61%)2.96s (± 0.84%)+0.02s (+ 0.71%)2.93s3.04s
Total Time9.44s (± 0.31%)9.46s (± 0.38%)+0.02s (+ 0.18%)9.35s9.51s
TFS - node (v12.1.0, x64)
Memory used306,384k (± 0.02%)306,445k (± 0.02%)+62k (+ 0.02%)306,357k306,565k
Parse Time0.94s (± 0.66%)0.94s (± 0.51%)-0.00s (- 0.21%)0.93s0.95s
Bind Time0.70s (± 0.53%)0.71s (± 1.06%)+0.00s (+ 0.28%)0.69s0.72s
Check Time4.17s (± 0.44%)4.17s (± 0.62%)+0.01s (+ 0.14%)4.12s4.22s
Emit Time3.07s (± 0.63%)3.08s (± 0.52%)+0.02s (+ 0.55%)3.05s3.12s
Total Time8.88s (± 0.38%)8.91s (± 0.25%)+0.03s (+ 0.30%)8.85s8.95s
Angular - node (v8.9.0, x64)
Memory used351,823k (± 0.01%)353,454k (± 0.01%)+1,630k (+ 0.46%)353,329k353,549k
Parse Time2.10s (± 0.54%)2.13s (± 0.83%)+0.03s (+ 1.29%)2.10s2.19s
Bind Time0.92s (± 0.64%)0.93s (± 1.01%)+0.01s (+ 0.65%)0.91s0.95s
Check Time5.46s (± 0.66%)5.45s (± 0.57%)-0.01s (- 0.11%)5.36s5.50s
Emit Time6.24s (± 0.36%)6.25s (± 0.86%)+0.01s (+ 0.24%)6.14s6.38s
Total Time14.71s (± 0.25%)14.76s (± 0.52%)+0.05s (+ 0.33%)14.56s14.91s
Monaco - node (v8.9.0, x64)
Memory used362,939k (± 0.01%)362,914k (± 0.01%)-25k (- 0.01%)362,768k363,015k
Parse Time1.56s (± 0.43%)1.57s (± 0.68%)+0.01s (+ 0.38%)1.55s1.59s
Bind Time0.95s (± 0.49%)0.94s (± 0.63%)-0.00s (- 0.53%)0.93s0.96s
Check Time5.39s (± 1.25%)5.39s (± 1.77%)-0.00s (- 0.07%)5.27s5.61s
Emit Time3.30s (± 3.17%)3.31s (± 4.02%)+0.00s (+ 0.12%)3.03s3.47s
Total Time11.21s (± 0.55%)11.21s (± 0.40%)+0.00s (+ 0.00%)11.11s11.32s
TFS - node (v8.9.0, x64)
Memory used323,541k (± 0.02%)323,479k (± 0.01%)-62k (- 0.02%)323,386k323,563k
Parse Time1.26s (± 0.44%)1.27s (± 0.65%)+0.00s (+ 0.32%)1.25s1.29s
Bind Time0.75s (± 0.65%)0.75s (± 0.69%)-0.00s (- 0.40%)0.74s0.76s
Check Time4.82s (± 0.56%)4.80s (± 0.59%)-0.02s (- 0.44%)4.76s4.89s
Emit Time3.20s (± 0.73%)3.19s (± 0.58%)-0.01s (- 0.22%)3.14s3.22s
Total Time10.04s (± 0.41%)10.01s (± 0.47%)-0.03s (- 0.26%)9.94s10.16s
Angular - node (v8.9.0, x86)
Memory used200,058k (± 0.02%)200,951k (± 0.02%)+893k (+ 0.45%)200,864k201,062k
Parse Time2.04s (± 1.23%)2.04s (± 0.40%)+0.00s (+ 0.20%)2.02s2.06s
Bind Time1.05s (± 0.85%)1.05s (± 0.59%)+0.00s (+ 0.29%)1.04s1.06s
Check Time4.97s (± 0.70%)4.94s (± 0.59%)-0.03s (- 0.58%)4.86s4.97s
Emit Time6.06s (± 1.14%)6.14s (± 1.39%)+0.08s (+ 1.35%)6.00s6.33s
Total Time14.11s (± 0.61%)14.17s (± 0.64%)+0.06s (+ 0.43%)13.95s14.37s
Monaco - node (v8.9.0, x86)
Memory used203,746k (± 0.02%)203,796k (± 0.02%)+50k (+ 0.02%)203,702k203,876k
Parse Time1.60s (± 0.55%)1.60s (± 0.55%)-0.00s (- 0.19%)1.59s1.63s
Bind Time0.77s (± 1.35%)0.77s (± 0.52%)-0.00s (- 0.39%)0.76s0.78s
Check Time5.19s (± 1.98%)5.18s (± 1.83%)-0.00s (- 0.04%)5.04s5.50s
Emit Time3.13s (± 2.72%)3.15s (± 2.33%)+0.02s (+ 0.51%)2.86s3.23s
Total Time10.69s (± 0.42%)10.70s (± 0.47%)+0.00s (+ 0.04%)10.56s10.79s
TFS - node (v8.9.0, x86)
Memory used182,635k (± 0.03%)182,671k (± 0.02%)+36k (+ 0.02%)182,625k182,723k
Parse Time1.30s (± 0.77%)1.30s (± 0.80%)-0.00s (- 0.15%)1.29s1.34s
Bind Time0.71s (± 0.87%)0.71s (± 0.87%)0.00s ( 0.00%)0.70s0.73s
Check Time4.58s (± 0.77%)4.59s (± 0.53%)+0.00s (+ 0.11%)4.55s4.66s
Emit Time2.96s (± 1.44%)2.95s (± 0.48%)-0.01s (- 0.47%)2.92s2.98s
Total Time9.57s (± 0.68%)9.55s (± 0.27%)-0.01s (- 0.15%)9.51s9.64s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-166-generic
Architecturex64
Available Memory16 GB
Available Memory4 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v10.16.3, x64)
  • node (v12.1.0, x64)
  • node (v8.9.0, x64)
  • node (v8.9.0, x86)
Scenarios
  • Angular - node (v10.16.3, x64)
  • Angular - node (v12.1.0, x64)
  • Angular - node (v8.9.0, x64)
  • Angular - node (v8.9.0, x86)
  • Monaco - node (v10.16.3, x64)
  • Monaco - node (v12.1.0, x64)
  • Monaco - node (v8.9.0, x64)
  • Monaco - node (v8.9.0, x86)
  • TFS - node (v10.16.3, x64)
  • TFS - node (v12.1.0, x64)
  • TFS - node (v8.9.0, x64)
  • TFS - node (v8.9.0, x86)
BenchmarkNameIterations
Current3666510
Baselinemaster10

@sandersn
Copy link
MemberAuthor

No impact to check time, but it doesn't mean much without knowing how many overload errors are in the perf test suite. That's the next thing I need to look at.

@weswigham
Copy link
Member

Hey, it at least means that the hypothesis of "this should only affect compilations with errors" is probably true~

@sandersn
Copy link
MemberAuthor

So, uh, here are the numbers of getCandidateForOverloadFailure calls. They are small. The number in parenthesis is the total number of resolveErrorCalls; that function is called a number of places.

BeforeAfter
Angular0 (4415)8 (4407)
TFS0 (25)0 (25)
Monaco0 (17)2 (14)

I'm still leaning toward yes since it appears (1) to be pay-to-use (2) angular, the project that uses it a little, on the whole checks at the same speed.

@weswigham if you agree, sign off and you or I can merge this.

@sandersn
Copy link
MemberAuthor

Actually, I want to look over the baseline changes before I sign off. So far the look benign, but there are a lot of them.

Copy link
Member

@weswighamweswigham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty much always fine with slight degredation to error-case performance in exchange for either better performance elsewhere or better errors, since "constant and extensive errors" is not the steady state for a codebase, and this should allow us to have much better API and real-world services performance (since we'll be able to reuse the hot diagnostics-producing checker from the constantly made err event for quick info). As for the baselines, it looked to me like a lot were pretty expected - many type baselines got return types when calls had errors, many error baselines got another entry or two as a result of downstream errors exposed by this.

@sandersnsandersn merged commit 71c1da0 into masterFeb 7, 2020
@sandersnsandersn deleted the getCandidateForOverloadFailure-redux branch February 7, 2020 17:55
@sandersn
Copy link
MemberAuthor

Note: breaks jest types in the following call:

constspy3Mock=spy3.mockImplementation(()=>'').mockImplementation()// $ExpectError.mockImplementation((arg: {})=>arg).mockImplementation((...args: string[])=>args.join('')).mockImplementationOnce(()=>'').mockName('name').mockReturnThis().mockReturnValue('value').mockReturnValueOnce('value').mockResolvedValue('value').mockResolvedValueOnce('value').mockRejectedValue('value').mockRejectedValueOnce('value');

The last four calls expect an argument of type never now instead of string. This seems wrong, or at least unintended.

@sandersn
Copy link
MemberAuthor

sandersn commented Feb 10, 2020

Oh, the tests were ALWAYS an error, it's just the we now report downstream errors after the expected error on mockImplemention((arg:{}) => arg).
If you move the mockResolveValue calls above that line, you get the error even before this change.

The test should be making those calls on a mock of a promise, so that's probably the right fix.

@weswigham
Copy link
Member

weswigham commented Feb 11, 2020

@sandersnSomehow this has caused some concerning regressions in rwc (lol we didn't run rwc) to do with elided imports (namely we're failing to mark some imports as used and they're mistakenly elided). I don't think it's a fundamental flaw with this, but it's definitely something we need to look into. The affected projects include Azure_FrameworkTests and immutable. (You can see the diffs on the rwc diff PRs)

The immutable case is something like

//@filename: Traversable.tsexportinterfaceITraversable<T>{}exportfunctionisTraversable(a: any): boolean{returnisList(a)||isOption(a);}exportfunctionisList(a: any){return(ainstanceof_list.Cons)||(ainstanceof_list.Nil);}exportfunctionisOption(a: any){return(ainstanceof_option.Some)||(ainstanceof_option.None);}// @filename: List.tsimport_tr= require('./Traversable');exportclassCons<T>extendsIList<T>{flatten<U>(): IList<U>{returnthis.foldLeft<IList<U>>(newNil<U>(),(acc,t)=>{if(_tr.isList(t)){// usage herevarl=<IList<U>><any>t;returnacc.append(l);}elseif(_tr.isOption(t)){// and herevaro=<_option.IOption<U>><any>t;if(o.isDefined()){returnacc.appendOne(o.get());}elsereturnacc;}else{returnacc.appendOne(<U><any>t);}});}}

I think we're failing to check a child function body or something?

@sandersn
Copy link
MemberAuthor

Probably when there is an error in the parent call expression. I observed this in the user tests too.

@sandersn
Copy link
MemberAuthor

Makes sense that we wouldn't have noticed this since our test coverage of the language service is so much worse, and getCandidateForOverloadFailure was only called from there.

@sandersn
Copy link
MemberAuthor

After discussion with @weswigham, the fix is to make getCandidateForOverloadFailure call resolveUntypedCall the way resolveErrorCall does. resolveUntypedCall is just a barebones set of check* calls.

@microsoftmicrosoft locked as resolved and limited conversation to collaborators Oct 21, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Author: TeamFor Milestone BugPRs that fix a bug with a specific milestone

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

4 participants

@sandersn@typescript-bot@weswigham