node --test does not show skipped tests in TAP result

[Dzelix]

Version

v18.12.1

Platform

Darwin Kernel Version 22.1.0 root:xnu-8792.41.9~2/RELEASE_ARM64_T8103 arm64

Subsystem

node:test

What steps will reproduce the bug?

In my project I have a folder named 'test' with two files inside.

The first file (Unit one) contains describe and test with different variations of skip. All the asserts are intentionally failed.

import { describe, it, test } from 'node:test';
import { strict as assert } from 'node:assert';

describe('Unit one: group one', { skip: true }, () => {
 it('test one', () => {
 assert.strictEqual(1, 10);
 })
})

describe.skip('Unit one: group two', () => {
 it('test two', () => {
 assert.strictEqual(1, 10);
 })
})

test('Unit one: test three', { skip: true }, () => {
 assert.strictEqual(1, 10);
})

The second file contains describe and test with correct asserts (Unit two).

import { describe, it, test } from 'node:test';
import { strict as assert } from 'node:assert';

describe('Unit two: group one', () => {
 it('test one', () => {
 assert.strictEqual(1, 1);
 })
})

test('Unit two: test two', () => {
 assert.strictEqual(1, 1);
})

Additionally, if I have the only file with combination of both skipped and ordinary describe and test (Unit three). This file was run separately.

import { describe, it, test } from 'node:test';
import { strict as assert } from 'node:assert';

describe('Unit three: group one', () => {
 it('test one', () => {
 assert.strictEqual(1, 1);
 })
})

test('Unit three: test two', { skip: true }, () => {
 assert.strictEqual(1, 1);
})

describe.skip('Unit three: group three', () => {
 it('test one', () => {
 assert.strictEqual(1, 1);
 })
})

test('Unit three: test three', () => {
 assert.strictEqual(1, 1);
})

How often does it reproduce? Is there a required condition?

Every time when I use node --test

What is the expected behavior?

When I run node --test, I expect to see information about skipped tests in the final TAP result.

What do you see instead?

When I run node --test with 'test' folder, contains of the first and the second files, there are no skipped tests.

TAP version 13
# Subtest: /Users/deni/Desktop/web-tools/test/one.mjs
ok 1 - /Users/deni/Desktop/web-tools/test/one.mjs
 ---
 duration_ms: 47.325042
 ...
# Subtest: /Users/deni/Desktop/web-tools/test/two.mjs
ok 2 - /Users/deni/Desktop/web-tools/test/two.mjs
 ---
 duration_ms: 44.268208
 ...
1..2
# tests 2
# pass 2
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 55.691167

When I run the third file alone I have the same result without skipped tests as well

ok 1 - /Users/deni/Desktop/web-tools/test/three.mjs
 ---
 duration_ms: 47.394208
 ...
1..1
# tests 1
# pass 1
# fail 0
# cancelled 0
# skipped 0
# todo 0
# duration_ms 54.821583

Additional information

On the contrary, if I run node test/one.mjs I have correct TAP result with information about skipped tests.

# Subtest: Unit one: group one
ok 1 - Unit one: group one # SKIP
 ---
 duration_ms: 0.565083
 ...
# Subtest: Unit one: group two
ok 2 - Unit one: group two # SKIP
 ---
 duration_ms: 0.044209
 ...
# Subtest: Unit one: test three
ok 3 - Unit one: test three # SKIP
 ---
 duration_ms: 0.052666
 ...
1..3
# tests 3
# pass 0
# fail 0
# cancelled 0
# skipped 3
# todo 0
# duration_ms 4.347417

As well as I run node test/three.mjs

TAP version 13
# Subtest: Unit three: group one
 # Subtest: test one
 ok 1 - test one
 ---
 duration_ms: 0.1995
 ...
 1..1
ok 1 - Unit three: group one
 ---
 duration_ms: 1.209541
 ...
# Subtest: Unit three: test two
ok 2 - Unit three: test two # SKIP
 ---
 duration_ms: 0.05375
 ...
# Subtest: Unit three: group three
ok 3 - Unit three: group three # SKIP
 ---
 duration_ms: 0.042459
 ...
# Subtest: Unit three: test three
ok 4 - Unit three: test three
 ---
 duration_ms: 0.043833
 ...
1..4
# tests 4
# pass 2
# fail 0
# cancelled 0
# skipped 2
# todo 0
# duration_ms 4.859

[cjihrig]

When you run with --test, each test file is run separately and their outputs are combined into a single TAP document. That's why the test count in the summary matches the number of test files run.

Since Node v19.2.0, the output is combined more intelligently. Now that we have the other pieces in place, I think we should go a step further now and remove the separation by test file in the output. In other words, if file 1 had 3 top level tests, and file 2 had 4 top level tests, then the output from --test would have 7 top level tests instead of 2 as it currently does.

[Dzelix]

But what about the case when I have only one file? The result is the same - all tests passed and no information about skipped tests. Does it mean that node --test combine result into TAP even if there is only one test file?

[cjihrig]

What I'm proposing in #45833 (comment) would apply to the case where there is a single file. The summary printed at the end of a run only applies to top level tests. If you run without --test, the top level is all of the top level test() calls in the file. If you run with --test, the top level is currently the collection of files (even if there is only one). I'm proposing we change the behavior with --test.

I think you already realize this based on your original bug report, but if you only have one file, then as a workaround for now, you can run without --test.

[cjihrig]

cc @manekinekko in case you have any interest in working on this ^ - I think the changes would be confined to the CLI runner and parser.