Skip to content

[flow] Fix file crawl hanging on symlink cycles - #9530

Closed
robhogan wants to merge 1 commit into
meta:mainfrom
robhogan:symlink-cycles
Closed

robhogan wants to merge 1 commit into
meta:mainfrom
robhogan:symlink-cycles

Conversation

@robhogan

Copy link
Copy Markdown
Contributor

Fixes #9529

Problem

When workspace packages are linked into each other's node_modules (pnpm with linkWorkspacePackages), the file crawl doesn't terminate in practice. make_next_files walks with jwalk's follow_links(true), and jwalk's loop check compares the raw read_link target with the paths of the directories above the entry, so it never matches a relative link:

https://github.com/Byron/jwalk/blob/a5b1ea634fcacee06f7ef6e9411e0cfecfccb128/src/core/dir_entry.rs#L219-L229

The walk only stops at ELOOP, after visiting every path through the link graph up to 32 links deep - exponential in the number of linked packages. Acyclic layouts are walked once per path too, so eg a pnpm store gets crawled once per dependent.

This change

For each directory symlink, the crawl now decides whether to descend:

  • If the target is somewhere the crawl reaches anyway without following links (inside the root, not under a pruned directory), the link isn't followed. Files reached through a link are already reported by their real path, so the crawl reports the same set of files.
  • Otherwise, the link is followed unless it leads back into a directory the walk is already inside, ie unless it closes a cycle.

Links that aren't followed are recorded in SymlinkMap as directory aliases, and try_resolve maps paths through them, so resolution through node_modules links works as before. The crawl also reports each canonicalised file once.

Outside the direct crawl (links out of the root, or flow ls <dir>), two links to the same target can be filtered differently by [ignore] - that's the ignored_alias case in node_modules_with_symlinks - so only cycles are cut there, not repeat visits.

Test plan

New node_modules_symlink_cycles test, covering workspace packages linked to each other, a .pnpm-style store with a dependency cycle and a cycle outside the root, through ls, check and status. New unit tests for the crawl and for resolving through chained aliases.

flow dev-tools runtests on macOS: 1084 passed, 0 failed, 9 skipped. I haven't run it on Windows - the new test creates its links with mklink there, the same way node_modules_with_symlinks does.

On react/metro under pnpm (react/metro#2110, macOS, cold flow check after flow stop):

main This PR
flow ls, with Metro's [ignore] workaround 125,443 lines (22,572 unique), 3.6s 22,573 lines, 0.6s
flow check, with workaround 8.2s, no errors 2.6s, no errors
flow ls, workaround removed hangs (no output after 120s) 22,576 lines, 0.8s
flow check, workaround removed hangs 4.1s, no errors

(The 3 extra files without the workaround are the npm metro-memory-fs package, which **/node_modules/metro*/** was hiding.)

Fixes meta#9529

## Problem

When workspace packages are linked into each other's `node_modules` (pnpm with `linkWorkspacePackages`), the file crawl doesn't terminate in practice. `make_next_files` walks with jwalk's `follow_links(true)`, and jwalk's loop check compares the raw `read_link` target with the paths of the directories above the entry, so it never matches a relative link:

https://github.com/Byron/jwalk/blob/a5b1ea634fcacee06f7ef6e9411e0cfecfccb128/src/core/dir_entry.rs#L219-L229

The walk only stops at `ELOOP`, after visiting every path through the link graph up to 32 links deep - exponential in the number of linked packages. Acyclic layouts are walked once per path too, so eg a pnpm store gets crawled once per dependent.

## This change

For each directory symlink, the crawl now decides whether to descend:
 - If the target is somewhere the crawl reaches anyway without following links (inside the root, not under a pruned directory), the link isn't followed. Files reached through a link are already reported by their real path, so the crawl reports the same set of files.
 - Otherwise, the link is followed unless it leads back into a directory the walk is already inside, ie unless it closes a cycle.

Links that aren't followed are recorded in `SymlinkMap` as directory aliases, and `try_resolve` maps paths through them, so resolution through `node_modules` links works as before. The crawl also reports each canonicalised file once.

Outside the direct crawl (links out of the root, or `flow ls <dir>`), two links to the same target can be filtered differently by `[ignore]` - that's the `ignored_alias` case in `node_modules_with_symlinks` - so only cycles are cut there, not repeat visits.

## Test plan

New `node_modules_symlink_cycles` test, covering workspace packages linked to each other, a `.pnpm`-style store with a dependency cycle and a cycle outside the root, through `ls`, `check` and `status`. New unit tests for the crawl and for resolving through chained aliases.

`flow dev-tools runtests` on macOS: 1084 passed, 0 failed, 9 skipped. I haven't run it on Windows - the new test creates its links with `mklink` there, the same way `node_modules_with_symlinks` does.

On react/metro under pnpm (react/metro#2110, macOS, cold `flow check` after `flow stop`):

| | `main` | This PR |
|---|---|---|
| `flow ls`, with Metro's `[ignore]` workaround | 125,443 lines (22,572 unique), 3.6s | 22,573 lines, 0.6s |
| `flow check`, with workaround | 8.2s, no errors | 2.6s, no errors |
| `flow ls`, workaround removed | hangs (no output after 120s) | 22,576 lines, 0.8s |
| `flow check`, workaround removed | hangs | 4.1s, no errors |

(The 3 extra files without the workaround are the npm `metro-memory-fs` package, which `**/node_modules/metro*/**` was hiding.)
@meta-cla meta-cla Bot added the CLA Signed label Nov 16, 2026
@meta-codesync

meta-codesync Bot commented Nov 16, 2026

Copy link
Copy Markdown

This pull request has been imported. If you are a Meta employee, you can view this in D132299746. (Because this pull request was imported automatically, there will not be any future comments.)

@robhogan
robhogan marked this pull request as ready for review November 16, 2026 16:53

@SamChou19815 SamChou19815 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Higher level review:

To make this easier to review, please break it down into several stacked PRs. Specifically for this part

If the target is somewhere the crawl reaches anyway without following links (inside the root, not under a pruned directory), the link isn't followed. Files reached through a link are already reported by their real path, so the crawl reports the same set of files.

Existing logic that decides whether a file belongs should be factored out and reused instead of rolling its own for this purpose

@@ -0,0 +1 @@
module.exports = 'ext2';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the symlinks in node_modules broke internal ci. The recommendation is to write test scripts to create them programmatically, so that the test is also runnable on windows

Comment on lines +1079 to +1186
/// Decides which directory symlinks the crawl descends into.
///
/// Following every directory symlink revisits the same real directory once per logical path,
/// and package managers such as pnpm create symlink cycles (workspace packages linked into each
/// other's `node_modules`). jwalk's own loop check misses these, because it compares the raw link
/// target with the logical paths of the directories above it, so the crawl recurses until the OS
/// refuses to resolve the path, through a number of logical paths that grows exponentially with
/// the number of linked packages. Instead:
///
/// - A symlink whose target the crawl reaches without following any symlink is not followed.
/// Files reached through a symlink are reported by their real path, so descending into it
/// reports nothing new.
/// - Any other symlink is followed unless its target is the real path of a directory the crawl is
/// already inside, i.e. unless it closes a cycle.
///
/// Each symlink that is not followed is recorded with a path its contents are crawled under, so
/// that module resolution can still map paths through it to real files.
struct DirSymlinkDedupe {
root: PathBuf,
/// The starting points as the walks see them, which bound the cycle check.
starting_points: Vec<PathBuf>,
canonical_starting_points: Vec<PathBuf>,
can_prune: bool,
all: bool,
include_libdef: bool,
/// Symlinks not followed, with the path their contents are crawled under.
skipped: Mutex<Vec<(PathBuf, PathBuf)>>,
}

impl DirSymlinkDedupe {
fn new(
root: &Path,
starting_points: &[PathBuf],
can_prune: bool,
all: bool,
include_libdef: bool,
) -> Self {
let canonical = |path: &Path| cached_canonicalize(path).unwrap_or_else(|_| path.into());
Self {
root: canonical(root),
starting_points: starting_points.to_vec(),
canonical_starting_points: starting_points.iter().map(|path| canonical(path)).collect(),
can_prune,
all,
include_libdef,
skipped: Mutex::new(Vec::new()),
}
}

/// Whether the crawl reads `real_path` and reports every file under it that a crawl through a
/// symlink to it can report, without following any symlink.
fn is_crawled_directly(&self, options: &FileOptions, real_path: &Path) -> bool {
if !(options.implicitly_include_root && real_path.starts_with(&self.root)) {
return false;
}
self.canonical_starting_points
.iter()
.filter(|starting_point| real_path.starts_with(starting_point))
.any(|starting_point| {
!self.can_prune
|| real_path
.ancestors()
.take_while(|dir| dir.starts_with(starting_point))
.all(|dir| wanted_filter(options, dir, self.all, self.include_libdef))
})
}

/// The directory containing `path` on the crawl's current walk whose real path is
/// `real_path`, if any.
fn enclosing_dir_with_real_path<'a>(
&self,
path: &'a Path,
real_path: &Path,
) -> Option<&'a Path> {
for dir in path.ancestors().skip(1) {
if cached_canonicalize(dir).is_ok_and(|dir_real_path| dir_real_path == real_path) {
return Some(dir);
}
if self
.starting_points
.iter()
.any(|starting_point| dir == starting_point)
{
break;
}
}
None
}

/// Whether the crawl descends into the directory symlink at `path`.
fn should_follow(&self, options: &FileOptions, path: &Path) -> bool {
let Ok(real_path) = cached_canonicalize(path) else {
return true;
};
let crawled_under = if self.is_crawled_directly(options, &real_path) {
real_path
} else if let Some(dir) = self.enclosing_dir_with_real_path(path, &real_path) {
dir.to_path_buf()
} else {
return true;
};
self.skipped
.lock()
.expect("skipped symlinks lock should not be poisoned")
.push((path.to_path_buf(), crawled_under));
false
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these changes look like ai slop duplicating existing logic

@robhogan

robhogan commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Hey @SamChou19815 - thanks for taking a look. Yeah that's fair, this can be split into 4, I'll open new PRs:

  1. Pure refactor to make crawl coverage helpers reusable: [flow] Share the crawl file filter between full and subdirectory crawls #9532

  2. Report each crawled file once (fixes flow ls duplicates): [flow] Report each crawled file once #9533

  3. Don't follow links into directories the crawl already covers (the bulk of this PR, fixes the hang from a root check/ls): [flow] Fix file crawl hanging on pnpm workspace links #9534

  4. Stop at symlink cycles outside the crawl (fix hang/over-reporting on flow ls <subdir>) [flow] Stop the file crawl at symlink cycles #9535

1 and 2 are independent. 3 then 4 will follow (I can't create a stack on GH from a fork)

@robhogan

Copy link
Copy Markdown
Contributor Author

Closing as superseded by the stack above.

@robhogan robhogan closed this Nov 19, 2026
meta-codesync Bot pushed a commit that referenced this pull request Nov 19, 2026
Summary:
Fixes #9529

Last part of the stack #9532, #9533, #9534 (split from #9530).

#9534 stops the file crawl following a symlink to a directory it reads anyway, but it still follows any other symlink, eg out of the root or out of the directory passed to `flow ls`, and goes round a cycle of them until the OS refuses to resolve the path. With a few packages linked to each other, eg `flow ls packages/a` in a pnpm workspace, that never ends in practice.

This stops following a symlink to a directory the crawl passed through to reach it. Every walk now ends, though a dense web of links out of the crawl is still walked once per path through it, which grows quickly with the number of links.

Pull Request resolved: #9535

Test Plan:
`symlinks_out_of_root` now also has a cycle out of the root, whose files `main` lists once per lap. The full `flow dev-tools runtests` suite passes (macOS).

On react/metro installed with pnpm (react/metro#2110), `flow ls packages/metro-config` hangs on `main`. With this, `flow ls` of each workspace package finishes in under a second, and `flow full-check` gives the same result in the same time.

Reviewed By: SamChou19815

Differential Revision: D133227488

fbshipit-source-id: 889fee75094aed805ff360dbd21939cbe8cf465b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flow hangs crawling pnpm workspaces (symlink cycles in node_modules)

2 participants