test: fix flaky test-policy-integrity

[Trott]

Split the test into seven tests so that it doesn't time out.

Fixes: #40694
Fixes: #38088

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40789/

[Trott]

Welp, now we have two tests that timeout. I'll split things up further to see if it helps at all. Hopefully it's not a process that never exits somewhere.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40790/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40794/

[tniessen]

Do we know how much longer it takes to run the test than we expect? If a legitimately slow test keeps running into timeouts, maybe we should adjust the timeout instead of the test?

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40796/

[Trott]

Do we know how much longer it takes to run the test than we expect? If a legitimately slow test keeps running into timeouts, maybe we should adjust the timeout instead of the test?

We did that in 2883992. This is the one test that still times out. I think it's reasonable that launching 1032 processes might take a while on some systems, so I'm OK breaking this test up like this.

[Trott]

Looks like it works now. PTAL, folks!

[targos]

Can we make a stress test?

[Trott]

Can we make a stress test?

Probably. It was failing about 100% of the time previously though, so I'd like to think anything is an improvement.

I haven't used the win10-2019 bots in a stress test recently (ever?) but if they're working with that job, then these should do it:

Baseline test against master: https://ci.nodejs.org/job/node-stress-single-test/303/
Test against this PR: https://ci.nodejs.org/job/node-stress-single-test/304/

[targos]

There's quite a lot of duplication between the new files. It would be nice to have a TODO somewhere to refactor them.

[Trott]

There's quite a lot of duplication between the new files. It would be nice to have a TODO somewhere to refactor them.

I'm not opposed to a little bit of that here, but I tend to prefer the tests be self-contained and don't mind duplication in tests the way I might in code in other parts of the codebase.

[targos]

It seems like the stress tests don't want to start (both say that "A build with matching parameters is already running").

[Trott]

It seems like the stress tests don't want to start (both say that "A build with matching parameters is already running").

They've started. The second one will wait for the first one to finish. As of this writing, the first one has 8 failures and no passes. (Since the timeout is so long and the failure rate is ~100% on master, I'm only running it 20 times in each stress test.)

Here's the links directly to the console output for the master branch test which should fail 20 out of 20 or something close to that:
https://ci.nodejs.org/job/node-stress-single-test/303/nodes=win10-vs2019/console

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40798/

[Trott]

They've started. The second one will wait for the first one to finish. As of this writing, the first one has 8 failures and no passes. (Since the timeout is so long and the failure rate is ~100% on master, I'm only running it 20 times in each stress test.)

Here's the links directly to the console output for the master branch test which should fail 20 out of 20 or something close to that: https://ci.nodejs.org/job/node-stress-single-test/303/nodes=win10-vs2019/console

Because the timeout is 12 minutes, that first stress test will take 4 hours just to run the tests (and some additional time was needed for compilation), so it will be a while before the other stress test kicks off.

[targos]

It seems like the stress tests don't want to start (both say that "A build with matching parameters is already running").

They've started. The second one will wait for the first one to finish. As of this writing, the first one has 8 failures and no passes. (Since the timeout is so long and the failure rate is ~100% on master, I'm only running it 20 times in each stress test.)

Here's the links directly to the console output for the master branch test which should fail 20 out of 20 or something close to that: https://ci.nodejs.org/job/node-stress-single-test/303/nodes=win10-vs2019/console

Ah, ok. I was looking at https://ci.nodejs.org/job/node-stress-single-test/303/console

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/40802/

[Trott]

Stress test is looking good so far. 100% failure rate on the master branch and a 100%-so-far success rate on this PR branch. This could use another review. @nodejs/testing

[github-actions]

Fast-track has been requested by @Trott. Please 👍 to approve.

[Trott]

Stress test finished with 100% success. Master branch stress test had 100% failure.

[Trott]

Landed in fc4a792