reanalyze: add parallel processing for CMT file analysis#8089
Conversation
Remove global ModulePath mutable state used during per-file AST traversal by threading ModulePath.t explicitly through traversal code. This is preparation for parallel per-file processing and reduces shared global state during MAP. Signed-Off-By: Cristiano Calcagno <[email protected]>
Record type dependency edges immediately instead of buffering them in a global ref and flushing at end-of-file. Note: this reorders debug output in reanalyze logs: addTypeReference lines now appear next to extendTypeDependencies, rather than being emitted in a later end-of-file flush. Signed-Off-By: Cristiano Calcagno <[email protected]>
Stop using a global TypeLabels hashtable during AST processing. Instead, compute type-label dependencies in a post-merge pass from merged Declarations and add the corresponding type-reference edges before solving. Fix: use raw decl positions (not declGetLoc/posAdjustment) when building cross-file label indices, since reference graph keys are raw positions. Note: debug output in deadcode expected logs is reordered due to moving label linking from per-file traversal to the post-merge pass. Signed-Off-By: Cristiano Calcagno <[email protected]>
Stop using global mutable exception declaration state during AST processing. Instead, derive a pure exception lookup from merged Declarations and use it when processing CrossFileItems exception refs. Signed-Off-By: Cristiano Calcagno <[email protected]>
- Replace global Values.valueBindingsTable with per-file values_builder - Replace global Checks.checks with per-file checks_builder - processCmt now returns file_result with per-file data - Add runChecks to process all checks after merging values tables - Update Reanalyze.ml to collect exception results and run checks at end - This enables future parallelization of AST processing
Add support for parallel processing of CMT files during dead code and exception analysis using OCaml 5 Domains. Features: - New -parallel N flag: process files using N domains (0 = sequential) - -parallel -1: auto-detect number of CPU cores - New -timing flag: report internal timing breakdown by phase - New benchmark infrastructure for performance testing The parallelization targets the CMT file processing phase, which is the main bottleneck (typically 75-80% of analysis time). The analysis and reporting phases remain sequential as they require merged cross-file data. Signed-Off-By: Cursor AI <[email protected]>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Uh oh!
There was an error while loading. Please reload this page.
Break down timing into sub-phases: - CMT processing: - File loading: time spent reading and traversing CMT files - Result collection: time spent merging results from domains - Analysis: - Merging: combining per-file data into global structures - Solving: running the liveness solver Signed-Off-By: Cursor AI <[email protected]>
cristianoc commented Dec 13, 2025
Updated Timing Breakdown (50 copies, ~4900 files, 12 cores)Added granular timing that separates sub-phases: Sequential ModeParallel Mode (auto-detect = 12 cores)Key Observations
|
@cknitt to test, and eventually enable, parallelism, we need to build with ocaml 5. |
cknitt commented Dec 13, 2025
I kept the older versions in CI to make sure that we don't break compatibility inadvertently, so that we can support a broad version range when we eventually publish the compiler libs on OPAM. But enabling parallelism is a good reason to move to OCaml 5 only, so this is fine with me. |
cristianoc commented Dec 13, 2025
What's the earliest version we should keep? |
cknitt commented Dec 13, 2025
I would say the earliest we can based on what features/lib you require for parallelism. |
- Update CI compatibility test to use OCaml 5.0.1 - Update all package dependencies in dune-project to require OCaml >= 5.0.1 - Update rescript.opam.template to require OCaml >= 5.0.1 - Remove redundant OCaml version check from syntax_benchmarks/dune
- Add 'set -e' to exit immediately on errors - Use 'opam exec --' to match checkformat.sh behavior - Now make format will fail visibly instead of silently continuing when formatting fails
cristianocforce-pushed the reanalyze-reduce-global-state-ast branch from a55cdab to 452a4b1Compare
rescript@rescript/darwin-arm64@rescript/darwin-x64@rescript/linux-arm64@rescript/linux-x64@rescript/runtime@rescript/win32-x64commit: |
cristianoc commented Dec 15, 2025
OK it's 5.0.0. |
cristianocforce-pushed the reanalyze-reduce-global-state-ast branch from 1c05378 to 8632e45Comparecristianoc merged commit 699d02b into masterUh oh!
There was an error while loading. Please reload this page.
Summary
Add support for parallel processing of CMT files during dead code and exception analysis using OCaml 5 Domains.
Features
-parallel N: Process files using N parallel domains (0 = sequential, default)-parallel -1: Auto-detect number of CPU cores usingDomain.recommended_domain_count()-timing: Report internal timing breakdown by analysis phaseHow It Works
The parallelization targets the CMT file processing phase, which is the main bottleneck (typically 75-80% of total analysis time).
Parallel execution model:
Domain.join), the merged results are passed to the analysis phaseThe mutex is only held during the merge operation (list concatenation), not during file processing, which minimizes contention.
The analysis and reporting phases remain sequential as they require merged cross-file data.
Testing
Correctness Tests
Benchmark Tests
Observed Results (50 copies, ~4300 files, 12 cores)
Timing Breakdown
Key Observations
Usage Examples
Sample Timing Output