CI failures: 20180925

[joyeecheung]

Failures in node-test-pull-request/17334 to node-test-pull-request/17430 that failed more than 2 PRs
(Generated using ncu-ci walk pr --stats --copy)

JSTest Failure

Reason	parallel/test-bootstrap-modules
Type	JS_TEST_FAILURE
Failed PR	7 (nodejs/node#22845, nodejs/node#22951, nodejs/node#23014, nodejs/node#23010, nodejs/node#23042, nodejs/node#23044, nodejs/node#22998)
Appeared	undefined
First CI	https://ci.nodejs.org/job/node-test-pull-request/17339/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17399/

Example

not ok 154 parallel/test-bootstrap-modules
 ---
 duration_ms: 0.336
 severity: fail
 exitcode: 1
 stack: |-
 
 events.js:167
 throw er; // Unhandled 'error' event
 ^
 AssertionError [ERR_ASSERTION]: Internal Binding contextify,Internal Binding worker,NativeModule events,Internal Binding trace_events,NativeModule internal/safe_globals,NativeModule internal/async_hooks,NativeModule internal/errors,Internal Binding uv,Binding buffer,Internal Binding async_wrap,Binding config,Binding icu,NativeModule util,NativeModule internal/validators,NativeModule internal/encoding,NativeModule internal/util,Binding constants,Internal Binding util,NativeModule internal/util/types,Internal Binding types,NativeModule buffer,NativeModule internal/buffer,NativeModule internal/process/per_thread,NativeModule internal/process/worker_thread_only,NativeModule internal/process/stdio,NativeModule internal/worker,NativeModule assert,NativeModule internal/assert,NativeModule fs,NativeModule path,NativeModul...

---

Reason	parallel/test-gc-net-timeout
Type	JS_TEST_FAILURE
Failed PR	7 (nodejs/node#22331, nodejs/node#23058, nodejs/node#22846, nodejs/node#22974, nodejs/node#22947, nodejs/node#23056, nodejs/node#23047)
Appeared	test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-rackspace-win2012r2-x64-5, test-joyent-smartos16-x64-1, test-osuosl-aix61-ppc64_be-1, test-digitalocean-freebsd11-x64-2, test-digitalocean-ubuntu1804_container-x64-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/17402/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17428/

Example

not ok 1366 parallel/test-gc-net-timeout
 ---
 duration_ms: 120.88
 severity: fail
 exitcode: -15
 stack: |-
 timeout
 We should do 500 requests
 Done: 10/500
 Collected: 0/22
 Done: 33/500
 Collected: 10/44
 Done: 55/500
 Collected: 33/66
 Done: 64/500
 Collected: 55/77
 Done: 77/500
 Collected: 64/99
 Done: 110/500
 Collected: 77/121
 Done: 132/500
 Collected: 110/143
 Done: 154/500
 Collected: 132/165
 Done: 187/500
 Collected: 154/187
 Done: 211/500
 Collected: 187/231
 Done: 231/500
 Collected: 211/253
 Done: 253/500
 Collected: 231/275
 Done: 275/500
 Collected: 253/286
 Done: 289/500
 Collected: 275/308
 Done: 321/500
 Collected: 289/341
 Done: 352/500
 Collected: 321/363
 Done: 375/500
 Collected: 352/396
 Done: 407/500
 Collected: 375/418
 Done: 431/500
 Collected: 407/440
 Done: 451/500
 Collected: 431/462
 Done: 455/500
 Collected: 451/484
 Done: 484/500
 Co...

---

Reason	sequential/test-http2-session-timeout
Type	JS_TEST_FAILURE
Failed PR	7 (nodejs/node#22953, nodejs/node#22951, nodejs/node#23010, nodejs/node#23048, nodejs/node#22331, nodejs/node#22413, nodejs/node#22914)
Appeared	test-requireio_mhdawson-debian9-armv6l_pi1p-1, test-requireio_chrislea-debian9-armv6l_pi1p-1, test-requireio_securogroup-debian9-armv6l_pi1p-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17379/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17409/

Example

not ok 322 sequential/test-http2-session-timeout
 ---
 duration_ms: 6.310
 severity: fail
 exitcode: 1
 stack: |-
 assert.js:126
 throw err;
 ^
 
 AssertionError [ERR_ASSERTION]: function should not have been called at /home/iojs/build/workspace/node-test-binary-arm/test/sequential/test-http2-session-timeout.js:9
 at Http2Server.mustNotCall (/home/iojs/build/workspace/node-test-binary-arm/test/common/index.js:407:12)
 at Http2Server.emit (events.js:182:13)
 at ServerHttp2Session.sessionOnTimeout (internal/http2/core.js:2631:15)
 at Object.onceWrapper (events.js:273:13)
 at ServerHttp2Session.emit (events.js:182:13)
 at ServerHttp2Session._onTimeout (internal/http2/core.js:1306:10)
 at ontimeout (timers.js:454:11)
 at tryOnTimeout (timers.js:326:5)
 at listOnTimeout (timers.js:300:5)
 at processTimers (timers.js:257:5)
 ...

---

Reason	parallel/test-gc-http-client-connaborted
Type	JS_TEST_FAILURE
Failed PR	5 (nodejs/node#22846, nodejs/node#23016, nodejs/node#23075, nodejs/node#23048, nodejs/node#23056)
Appeared	test-rackspace-win2012r2-x64-5, test-digitalocean-ubuntu1604_sharedlibs_container-x64-10, test-digitalocean-alpine37_container-x64-2, test-rackspace-win2008r2-x64-6
First CI	https://ci.nodejs.org/job/node-test-pull-request/17407/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17430/

Example

not ok 184 parallel/test-gc-http-client-connaborted
 ---
 duration_ms: 120.66
 severity: fail
 exitcode: 1
 stack: |-
 timeout
 We should do 500 requests
 Done: 44/500
 Collected: 0/99
 Done: 131/500
 Collected: 44/176
 Done: 219/500
 Collected: 131/264
 Done: 271/500
 Collected: 219/407
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 Done: 271/500
 Collected: 271/500
 ...

---

Reason	parallel/test-gc-http-client-onerror
Type	JS_TEST_FAILURE
Failed PR	4 (nodejs/node#22846, nodejs/node#23016, nodejs/node#23075, nodejs/node#23056)
Appeared	undefined, test-packetnet-centos7-arm64-1, test-digitalocean-alpine37_container-x64-2, test-digitalocean-alpine37_container-x64-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17407/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17426/

Example

not ok 1395 parallel/test-gc-http-client-onerror
 ---
 duration_ms: 120.26
 severity: fail
 exitcode: -15
 stack: |-
 timeout
 We should do 500 requests
 Done: 0/500
 Collected: 0/40
 Done: 50/500
 Collected: 0/90
 Done: 90/500
 Collected: 40/130
 Done: 140/500
 Collected: 80/180
 Done: 190/500
 Collected: 130/230
 Done: 243/500
 Collected: 180/290
 Done: 290/500
 Collected: 240/330
 Done: 335/500
 Collected: 280/380
 Done: 390/500
 Collected: 330/430
 Done: 417/500
 Collected: 380/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 417/500
 Collected: 417/500
 Done: 4...

---

Reason	parallel/test-gc-http-client-timeout
Type	JS_TEST_FAILURE
Failed PR	4 (nodejs/node#23058, nodejs/node#22906, nodejs/node#22947, nodejs/node#23075)
Appeared	test-digitalocean-alpine37_container-x64-2, test-osuosl-aix61-ppc64_be-2, test-digitalocean-ubuntu1804_container-x64-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/17403/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17420/

Example

not ok 1186 parallel/test-gc-http-client-timeout
 ---
 duration_ms: 120.91
 severity: fail
 exitcode: -15
 stack: |-
 timeout
 We should do 550 requests
 Done: 0/550
 Collected: 0/33
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 Done: 0/550
 Collected: 0/44
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 timeout (expected)
 ti...

---

Reason	parallel/test-gc-http-client
Type	JS_TEST_FAILURE
Failed PR	3 (nodejs/node#23072, nodejs/node#23048, nodejs/node#23056)
Appeared	test-rackspace-win2012r2-x64-4, test-digitalocean-alpine37_container-x64-2, test-osuosl-aix61-ppc64_be-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/17417/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17425/

Example

not ok 184 parallel/test-gc-http-client
 ---
 duration_ms: 120.81
 severity: fail
 exitcode: 1
 stack: |-
 timeout
 We should do 300 requests
 Done: 29/300
 Collected: 0/90
 Done: 59/300
 Collected: 29/120
 Done: 90/300
 Collected: 59/150
 Done: 104/300
 Collected: 90/180
 Done: 104/300
 Collected: 90/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 Collected: 104/300
 Done: 104/300
 ...

---

Reason	parallel/test-crypto-keygen
Type	JS_TEST_FAILURE
Failed PR	2 (nodejs/node#22845, nodejs/node#22969)
Appeared	test-packetnet-centos7-arm64-1, test-linuxonecc-rhel72-s390x-3, test-azure_msft-win10-x64-3
First CI	https://ci.nodejs.org/job/node-test-pull-request/17341/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17343/

Example

not ok 2388 parallel/test-crypto-keygen
 ---
 duration_ms: 240.71
 severity: fail
 exitcode: -15
 stack: |-
 timeout
 assert.js:662
 throw actual;
 ^
 
 Error: error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long
 at Sign.sign (internal/crypto/sig.js:83:27)
 at testSignVerify (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:43:42)
 at assert.throws (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:171:7)
 at getActual (assert.js:574:5)
 at Function.throws (assert.js:692:24)
 at AsyncWrap.common.mustCall (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/parallel/test-crypto-keygen.js:170:12)
 at AsyncWrap.<anonymous> (/home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/common/index.js:350:15)
 at AsyncWrap.wrap.ondone (internal/crypto/...

---

Reason	parallel/test-net-connect-options-port
Type	JS_TEST_FAILURE
Failed PR	2 (nodejs/node#22331, nodejs/node#22846)
Appeared	test-packetnet-ubuntu1604-arm64-2, test-digitalocean-alpine37_container-x64-2
First CI	https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17407/

Example

not ok 2361 parallel/test-net-connect-options-port
 ---
 duration_ms: 240.140
 severity: fail
 exitcode: -15
 stack: |-
 timeout
 ...

---

Build Failure

Reason	sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory
Type	BUILD_FAILURE
Failed PR	3 (nodejs/node#22891, nodejs/node#22969, nodejs/node#20713)
Appeared	test-linuxonecc-rhel72-s390x-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17334/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17336/

Example

sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory

---

Reason	ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
Type	BUILD_FAILURE
Failed PR	2 (nodejs/node#22331, nodejs/node#22914)
Appeared	test-requireio_davglass-debian9-arm64_pi3-1, test-requireio_securogroup-debian9-arm64_pi3-2, test-requireio_pivotalagency-debian9-arm64_pi3-2, test-requireio_williamkapke-debian9-arm64_pi3-3, test-requireio_kahwee-debian9-arm64_pi3-1, test-requireio_notthetup_sayanee-debian9-arm64_pi3-1, test-requireio_securogroup-debian9-armv6l_pi1p-2, test-requireio_securogroup-debian9-arm64_pi3-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17427/

Example

ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run? 
For example, /home/iojs/build/workspace/node-test-binary-arm/out/junit/test.xml is 29 min old

Notifying upstream projects of job completion
Finished: FAILURE

---

Reason	error: pathspec ... did not match any file(s) known to git.
Type	BUILD_FAILURE
Failed PR	2 (nodejs/node#22969, nodejs/node#22418)
Appeared	test-azure_msft-win10-x64-3, test-azure_msft-win10-x64-4, test-rackspace-win2008r2-x64-2, test-rackspace-win2008r2-x64-1, test-azure_msft-win2008r2-x64-1, test-azure_msft-win2008r2-x64-2, test-rackspace-win2012r2-x64-9, test-azure_msft-win2012r2-x64-2, test-rackspace-win2012r2-x64-4, test-rackspace-win2012r2-x64-2, test-rackspace-win2012r2-x64-5, test-azure_msft-win2012r2-x64-1, test-rackspace-win2012r2-x64-3, test-rackspace-win2012r2-x64-8, test-azure_msft-win2016-x64-2, test-azure_msft-win2016-x64-5, test-azure_msft-win2016-x64-3, test-azure_msft-win2016-x64-4, test-azure_msft-win10-x64-2, test-rackspace-win2008r2-x64-5, test-rackspace-win2012r2-x64-7, test-rackspace-win2012r2-x64-6
First CI	https://ci.nodejs.org/job/node-test-pull-request/17347/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17359/

Example

error: pathspec 'refs/remotes/jenkins_tmp/jenkins-node-test-commit-windows-fanned-94402b043f54b82c706b86f02d05b5290e48c872-bin-win-vcbt2015' did not match any file(s) known to git.

c:\workspace\node-test-binary-windows>if 1 NEQ 0 exit /b 1 
Build step 'Execute Windows batch command' marked build as failure
Run condition [Always] enabling perform for step [[Execute Windows batch command, Execute shell]]
[node-test-binary-windows] $ cmd /c call C:\Users\ADMINI~1\AppData\Local\Temp\jenkins984036347200566261.bat

---

Reason	fatal: Not a git repository (or any of the parent directories): .git
Type	BUILD_FAILURE
Failed PR	2 (nodejs/node#22950, nodejs/node#23004)
Appeared	test-rackspace-win2008r2-x64-4
First CI	https://ci.nodejs.org/job/node-test-pull-request/17357/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17358/

Example

fatal: Not a git repository (or any of the parent directories): .git

---

Reason	sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
Type	BUILD_FAILURE
Failed PR	2 (nodejs/node#22331, nodejs/node#22914)
Appeared	test-requireio_chrislea-debian9-armv6l_pi1p-1, test-requireio_svincent-debian9-armv7l_pi2-1, test-requireio_mininodes-debian9-armv6l_pi1p-1, test-requireio_mcollina-debian9-armv7l_pi2-1, test-requireio_securogroup-debian9-armv6l_pi1p-1, test-requireio_rvagg-debian9-armv7l_pi2-1, test-requireio_bengl-debian9-armv6l_pi1p-2, test-requireio_joeyvandijk-debian9-armv7l_pi2-2, test-requireio_indieisaconcept-debian9-armv6l_pi1p-1, test-requireio_joeyvandijk-debian9-armv7l_pi2-1, test-requireio_mhdawson-debian9-armv6l_pi1p-1, test-requireio_jasnell-debian9-armv7l_pi2-1, test-requireio_ceejbot-debian9-armv7l_pi2-1
First CI	https://ci.nodejs.org/job/node-test-pull-request/17406/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17427/

Example

sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory

---

Git Failure

Jenkins Failure

Reason	java.io.IOException: Backing channel 'JNLP4-connect connection from ... is disconnected.
Type	JENKINS_FAILURE
Failed PR	2 (nodejs/node#22997, nodejs/node#22998)
Appeared	test-macstadium-macos10.11-x64-2, test-rackspace-win2008r2-x64-4
First CI	https://ci.nodejs.org/job/node-test-pull-request/17356/
Last CI	https://ci.nodejs.org/job/node-test-pull-request/17399/

Example

java.io.IOException: Backing channel 'JNLP4-connect connection from 207.254.58.162/207.254.58.162:62697' is disconnected.
	at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:214)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283)
	at com.sun.proxy.$Proxy85.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1137)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1129)

[joyeecheung]

Progress

• java.io.IOException: Backing channel 'JNLP4-connect connection from ... is disconnected.
• parallel/test-net-connect-options-port

In progress

• sequential/test-http2-session-timeout: trying to improve in Minor improvements for test-http2-session-timeout node#23058
• parallel/test-gc-net-timeout: Investigate flaky test-gc-net-timeout node#23067 , trying to improve in test: improve reliability of test-gc-net-timeout node#23086
• parallel/test-gc-http-client-connaborted: trying to improve in test: improve test-gc-http-client-connaborted node#23091
• parallel/test-gc-http-client-onerror: Investigate flaky test-gc-http-client-onerror node#23089 trying to improve in test: improve test-gc-http-client-onerror node#23090
• parallel/test-gc-http-client-timeout: Investigate flaky test-gc-http-client-timeout node#23066 trying to improve in test: improve test-gc-http-client-timeout node#23088
• parallel/test-gc-http-client: Investigate flaky parallel/test-gc-http-client node#22336 trying to improve in test: improve reliability of test-gc-http-client node#23087

Fixed

• parallel/test-crypto-keygen: Investigate flaky parallel/test-crypto-keygen node#22978 fixed in test: fix flaky key pair generation test node#22980
• parallel/test-bootstrap-modules: lib: lazy load internal/queue_microtask node#23046
• sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory: node-test-commit-linuxone cannot find ./build/jenkins/scripts/select-compiler.sh build#1501

Resume failure

Tracking in nodejs/build#1496

• fatal: Not a git repository (or any of the parent directories): .git
• error: pathspec ... did not match any file(s) known to git.
• ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
• sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory

[richardlau]

sh: line 6: ./build/jenkins/scripts/select-compiler.sh: No such file or directory was addressed by nodejs/build#1501.

[refack]

parallel/test-gc-http-client - nodejs/node#22336
parallel/test-gc-http-client-timeout - nodejs/node#23066
parallel/test-gc-net-timeout - nodejs/node#23067

@joyeecheung could you update the pattern matching to ignore
sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
and
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
Those are errors that don't cause build FAILURE. In those cases the error is probably something else. In the case of node-test-binary-arm it usually means that a resume build failed because the temporary branch passing the binaries was deleted during the original build.

[joyeecheung]

@refack I can add an ad-hoc pattern for now (maybe replace/append that line with (possible resume failures)), yes, but I am not sure how long that association will hold? (Maybe I can just add a TODO in the tool and lift it when there is a new association)

[refack]

@joyeecheung What I meant is that in the current configuration of CI
sh: line 41: /home/iojs/.ssh/known_hosts: No such file or directory
and
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did leafNodes run?
should not cause the build to fail.

You are right about associating them with a real cause, "failed resume" is just a guess.

[joyeecheung]

@refack I see. The reasons here are not accurate reasons, they are more like keys to group the errors by.
Just think of another solution: I can check if the PR build comes from another PR build (i.e. a resumed build) and check if the failed job is a fanned one. If so I'll tentatively mark them as RESUME_FAILURE (a new category). That's how my brain currently works anyway.

[refack]

From a big picture POV, I would like to tip my 🎩 to @joyeecheung @Trott @joaocgreis and rest of @nodejs/build that (AFAIR for the first time ) made it so infra failures do not dominate this list. One small step...

[Trott]

@refack Right back at you times 💯