Skip to content

Conversation

@zcbenz
Copy link
Contributor

@zcbenzzcbenz commented Nov 15, 2023

This PR refactors the install and build_addons scripts so they can be reused by node-ci and electron projects to build the addons tests.

In detail, this PR does following things:

  1. Minor modifications of the GN configurations to fix loading native modules.
  2. Add some flags to tools/install.py script to allow customize locations of config.gypi, v8, etc. With some refactoring to remove global variables.
  3. Rewrite the tools/build_addons.py script:
    1. Rewrite it from js to python, because implementing cmd args handling with vanilla Node.js would take much longer time than a simple rewrite.
    2. Add a few more flags to make it easier to use for embedders.
  4. Modify the Makefile to generate headers in out dir first, and then calls build_addons.py --headers-dir=out_dir/headers (which calls node-gyp --nodedir=out_dir/headers) to build addons. Previously the build system was just calling node-gyp --nodedir=., which relied on quirky behavior of node-gyp and did not really test whether the generated headers work.

Users that rely on make or vcbuild.bat to work on Node are not affected.

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/tsc

@nodejs-github-botnodejs-github-bot added build Issues and PRs related to build files or the CI. dependencies Pull requests that update a dependency file. libuv Issues and PRs related to the libuv dependency or the uv binding. needs-ci PRs that need a full CI run. openssl Issues and PRs related to the OpenSSL dependency. python PRs and issues that require attention from people who are familiar with Python. tools Issues and PRs related to the tools directory. windows Issues and PRs related to the Windows platform. labels Nov 15, 2023
@zcbenzzcbenzforce-pushed the build-addons-gn branch 2 times, most recently from 507f7ab to 353afc9CompareNovember 15, 2023 11:06
@zcbenzzcbenzforce-pushed the build-addons-gn branch 2 times, most recently from 0940864 to 13f87fcCompareNovember 16, 2023 07:32
@zcbenz
Copy link
ContributorAuthor

I have separated some changes out of this PR so review should be easier.

/cc @nodejs/gyp @joyeecheung@richardlau for reviewing changes on tools.

@gengjiawengengjiawen added the request-ci Add this label to start a Jenkins CI on a PR. label Nov 17, 2023
@github-actionsgithub-actionsbot removed the request-ci Add this label to start a Jenkins CI on a PR. label Nov 17, 2023
@nodejs-github-bot
Copy link
Collaborator

@zcbenz
Copy link
ContributorAuthor

Ping @nodejs/gyp @joyeecheung@anonrig for review after the holiday.

Copy link
Member

@joyeecheungjoyeecheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % a correction but I think ideally we should have someone from @nodejs/build to confirm that it works with our releases.

@gengjiawengengjiawen added the request-ci Add this label to start a Jenkins CI on a PR. label Dec 4, 2023
@github-actionsgithub-actionsbot removed the request-ci Add this label to start a Jenkins CI on a PR. label Dec 4, 2023
@nodejs-github-bot
Copy link
Collaborator

@zcbenzzcbenzforce-pushed the build-addons-gn branch 3 times, most recently from 974401b to 3be1111CompareJanuary 4, 2024 02:28
@zcbenzzcbenz deleted the build-addons-gn branch February 23, 2024 07:17
marco-ippolito pushed a commit that referenced this pull request Feb 26, 2024
PR-URL: #50737 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Feb 26, 2024
PR-URL: #50737 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Feb 27, 2024
PR-URL: #50737 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]>
@batrla
Copy link
Contributor

batrla commented Feb 28, 2024

Sorry to report it, but this PR caused regression, build in --shared mode now fails:

11:02:41: stdout installing /some_dir/build/prototype/i386/some_bindir/node 11:02:41: stderr Traceback (most recent call last): 11:02:41: stderr File "/some_dir/build/amd64/tools/install.py", line 419, in <module> 11:02:41: stderr run(parse_options(sys.argv[1:])) 11:02:41: stderr File "/some_dir/build/amd64/tools/install.py", line 372, in run 11:02:41: stderr files(options, install) 11:02:41: stderr File "/some_dir/build/amd64/tools/install.py", line 179, in files 11:02:41: stderr action(options, [os.path.join(output_prefix, output_lib)], 11:02:41: stderr ^^^^^^^^^^^^^ 11:02:42: stderr UnboundLocalError: cannot access local variable 'output_prefix' where it is not associated with a value 

The problem in source code is obvious, the variable 'output_prefix' is only defined in some branches.
Later in code the variable is used in a branch where it was not defined before.

@zcbenz
Copy link
ContributorAuthor

Sorry for breaking it! I'll create a fix soon.

zcbenz added a commit to zcbenz/node that referenced this pull request Feb 28, 2024
Fix a bug caused by nodejs#50737 that, make install fails for --shared mode.
@marco-ippolitomarco-ippolito mentioned this pull request Mar 1, 2024
nodejs-github-bot pushed a commit that referenced this pull request Mar 1, 2024
Fix a bug caused by #50737 that, make install fails for --shared mode. PR-URL: #51910 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Mar 1, 2024
Fix a bug caused by #50737 that, make install fails for --shared mode. PR-URL: #51910 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
marco-ippolito pushed a commit that referenced this pull request Mar 1, 2024
Fix a bug caused by #50737 that, make install fails for --shared mode. PR-URL: #51910 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
zcbenz pushed a commit to photoionization/node_ci that referenced this pull request Mar 6, 2024
- Use node in-tree gn configs when possible. Some in-tree configs (deps/openssl) were modified in our mirror until they are fixed in the upstream node repo [1]. - Trigger landmine to clean out directory (required due to changed targets with new build files) - Update to node-ci-2024-01-17 - Add fp16 to update_deps.py [1] nodejs/node#50737. Bug: v8:14572 Change-Id: I86cc19d23151b2a64d7173cf8f0aa6adc417c091 Reviewed-on: https://chromium-review.googlesource.com/c/v8/node-ci/+/5203209 Commit-Queue: Patrick Thier <[email protected]> Reviewed-by: Victor Gomes <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
PR-URL: #50737 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
Fix a bug caused by #50737 that, make install fails for --shared mode. PR-URL: #51910 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
PR-URL: #50737 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]>
richardlau pushed a commit that referenced this pull request Mar 25, 2024
Fix a bug caused by #50737 that, make install fails for --shared mode. PR-URL: #51910 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
@richardlaurichardlau mentioned this pull request Mar 25, 2024
rdw-msft pushed a commit to rdw-msft/node that referenced this pull request Mar 26, 2024
PR-URL: nodejs#50737 Reviewed-By: Joyee Cheung <[email protected]> Reviewed-By: Benjamin Gruenbaum <[email protected]> Reviewed-By: Yagiz Nizipli <[email protected]>
rdw-msft pushed a commit to rdw-msft/node that referenced this pull request Mar 26, 2024
Fix a bug caused by nodejs#50737 that, make install fails for --shared mode. PR-URL: nodejs#51910 Reviewed-By: Richard Lau <[email protected]> Reviewed-By: Luigi Pinca <[email protected]>
codebytere added a commit to electron/electron that referenced this pull request Oct 16, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 21, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 22, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 28, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 29, 2024
codebytere added a commit to electron/electron that referenced this pull request Oct 31, 2024
jkleinsc pushed a commit to electron/electron that referenced this pull request Nov 4, 2024
* chore: bump Node.js to v22.9.0 * build: drop base64 dep in GN build nodejs/node#52856 * build,tools: make addons tests work with GN nodejs/node#50737 * fs: add fast api for InternalModuleStat nodejs/node#51344 * src: move package_json_reader cache to c++ nodejs/node#50322 * crypto: disable PKCS#1 padding for privateDecrypt nodejs-private/node-private#525 * src: move more crypto code to ncrypto nodejs/node#54320 * crypto: ensure valid point on elliptic curve in SubtleCrypto.importKey nodejs/node#50234 * src: shift more crypto impl details to ncrypto nodejs/node#54028 * src: switch crypto APIs to use Maybe<void> nodejs/node#54775 * crypto: remove DEFAULT_ENCODING nodejs/node#47182 * deps: update libuv to 1.47.0 nodejs/node#50650 * build: fix conflict gyp configs nodejs/node#53605 * lib,src: drop --experimental-network-imports nodejs/node#53822 * esm: align sync and async load implementations nodejs/node#49152 * esm: remove unnecessary toNamespacedPath calls nodejs/node#53656 * module: detect ESM syntax by trying to recompile as SourceTextModule nodejs/node#52413 * test: adapt debugger tests to V8 11.4 nodejs/node#49639 * lib: update usage of always on Atomics API nodejs/node#49639 * test: adapt test-fs-write to V8 internal changes nodejs/node#49639 * test: adapt to new V8 trusted memory spaces nodejs/node#50115 * deps: update libuv to 1.47.0 nodejs/node#50650 * src: use non-deprecated v8::Uint8Array::kMaxLength nodejs/node#50115 * src: update default V8 platform to override functions with location nodejs/node#51362 * src: add missing TryCatch nodejs/node#51362 * lib,test: handle new Iterator global nodejs/node#51362 * src: use non-deprecated version of CreateSyntheticModule nodejs/node#50115 * src: remove calls to recently deprecated V8 APIs nodejs/node#52996 * src: use new V8 API to define stream accessor nodejs/node#53084 * src: do not use deprecated V8 API nodejs/node#53084 * src: do not use soon-to-be-deprecated V8 API nodejs/node#53174 * src: migrate to new V8 interceptors API nodejs/node#52745 * src: use supported API to get stalled TLA messages nodejs/node#51362 * module: print location of unsettled top-level await in entry points nodejs/node#51999 * test: make snapshot comparison more flexible nodejs/node#54375 * test: do not set concurrency on parallelized runs nodejs/node#52177 * src: move FromNamespacedPath to path.cc nodejs/node#53540 * test: adapt to new V8 trusted memory spaces nodejs/node#50115 * build: add option to enable clang-cl on Windows nodejs/node#52870 * chore: fixup patch indices * chore: add/remove changed files * esm: drop support for import assertions nodejs/node#54890 * build: compile with C++20 support nodejs/node#52838 * deps: update nghttp2 to 1.62.1 nodejs/node#52966 * src: parse inspector profiles with simdjson nodejs/node#51783 * build: add GN build files nodejs/node#47637 * deps,lib,src: add experimental web storage nodejs/node#52435 * build: add missing BoringSSL dep * src: rewrite task runner in c++ nodejs/node#52609 * fixup! build: add GN build files * src: stop using deprecated fields of v8::FastApiCallbackOptions nodejs/node#54077 * fix: shadow variable * build: add back incorrectly removed SetAccessor patch * fixup! fixup! build: add GN build files * crypto: fix integer comparison in crypto for BoringSSL * src,lib: reducing C++ calls of esm legacy main resolvenodejs/node#48325 * src: move more crypto_dh.cc code to ncrypto nodejs/node#54459 * chore: fixup GN files for previous commit * src: move more crypto code to ncrypto nodejs/node#54320 * Fixup Perfetto ifdef guards * fix: missing electron_natives dep * fix: node_use_node_platform = false * fix: include src/node_snapshot_stub.cc in libnode * 5507047: [import-attributes] Remove support for import assertions https://chromium-review.googlesource.com/c/v8/v8/+/5507047 * fix: restore v8-sandbox.h in filenames.json * fix: re-add original-fs generation logic * fix: ngtcp2 openssl dep * test: try removing NAPI_VERSION undef * chore(deps): bump @types/node * src: move more crypto_dh.cc code to ncrypto nodejs/node#54459 * esm: remove unnecessary toNamespacedPath calls nodejs/node#53656 * buffer: fix out of range for toString nodejs/node#54553 * lib: rewrite AsyncLocalStorage without async_hooks nodejs/node#48528 * module: print amount of load time of a cjs module nodejs/node#52213 * test: skip reproducible snapshot test on 32-bit nodejs/node#53592 * fixup! src: move more crypto_dh.cc code to ncrypto * test: adjust emittedUntil return type * chore: remove redundant wpt streams patch * fixup! chore(deps): bump @types/node * fix: gn executable name on Windows * fix: build on Windows * fix: rename conflicting win32 symbols in //third_party/sqlite On Windows otherwise we get: lld-link: error: duplicate symbol: sqlite3_win32_write_debug >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:47987 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_sleep >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48042 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_is_nt >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48113 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_unicode >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48470 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_unicode_to_utf8 >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48486 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8 >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48502 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_mbcs_to_utf8_v2 >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48518 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48534 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj lld-link: error: duplicate symbol: sqlite3_win32_utf8_to_mbcs_v2 >>> defined at .\..\..\third_party\electron_node\deps\sqlite\sqlite3.c:48550 >>> obj/third_party/electron_node/deps/sqlite/sqlite/sqlite3.obj >>> defined at obj/third_party/sqlite\chromium_sqlite3/sqlite3_shim.obj * docs: remove unnecessary ts-expect-error after types bump * src: move package resolver to c++ nodejs/node#50322 * build: set ASAN detect_container_overflow=0 nodejs/node#55584 * chore: fixup rebase * test: disable failing ASAN test * win: almost fix race detecting ESRCH in uv_kill libuv/libuv#4341
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

buildIssues and PRs related to build files or the CI.commit-queue-failedAn error occurred while landing this pull request using GitHub Actions.commit-queue-squashAdd this label to instruct the Commit Queue to squash all the PR commits into the first one.dependenciesPull requests that update a dependency file.libuvIssues and PRs related to the libuv dependency or the uv binding.needs-ciPRs that need a full CI run.opensslIssues and PRs related to the OpenSSL dependency.pythonPRs and issues that require attention from people who are familiar with Python.toolsIssues and PRs related to the tools directory.windowsIssues and PRs related to the Windows platform.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants

@zcbenz@nodejs-github-bot@joyeecheung@batrla@benjamingr@anonrig@gengjiawen