Repository navigation
Conversation
Fixes meta#9529 ## Problem When workspace packages are linked into each other's `node_modules` (pnpm with `linkWorkspacePackages`), the file crawl doesn't terminate in practice. `make_next_files` walks with jwalk's `follow_links(true)`, and jwalk's loop check compares the raw `read_link` target with the paths of the directories above the entry, so it never matches a relative link: https://github.com/Byron/jwalk/blob/a5b1ea634fcacee06f7ef6e9411e0cfecfccb128/src/core/dir_entry.rs#L219-L229 The walk only stops at `ELOOP`, after visiting every path through the link graph up to 32 links deep - exponential in the number of linked packages. Acyclic layouts are walked once per path too, so eg a pnpm store gets crawled once per dependent. ## This change For each directory symlink, the crawl now decides whether to descend: - If the target is somewhere the crawl reaches anyway without following links (inside the root, not under a pruned directory), the link isn't followed. Files reached through a link are already reported by their real path, so the crawl reports the same set of files. - Otherwise, the link is followed unless it leads back into a directory the walk is already inside, ie unless it closes a cycle. Links that aren't followed are recorded in `SymlinkMap` as directory aliases, and `try_resolve` maps paths through them, so resolution through `node_modules` links works as before. The crawl also reports each canonicalised file once. Outside the direct crawl (links out of the root, or `flow ls <dir>`), two links to the same target can be filtered differently by `[ignore]` - that's the `ignored_alias` case in `node_modules_with_symlinks` - so only cycles are cut there, not repeat visits. ## Test plan New `node_modules_symlink_cycles` test, covering workspace packages linked to each other, a `.pnpm`-style store with a dependency cycle and a cycle outside the root, through `ls`, `check` and `status`. New unit tests for the crawl and for resolving through chained aliases. `flow dev-tools runtests` on macOS: 1084 passed, 0 failed, 9 skipped. I haven't run it on Windows - the new test creates its links with `mklink` there, the same way `node_modules_with_symlinks` does. On react/metro under pnpm (react/metro#2110, macOS, cold `flow check` after `flow stop`): | | `main` | This PR | |---|---|---| | `flow ls`, with Metro's `[ignore]` workaround | 125,443 lines (22,572 unique), 3.6s | 22,573 lines, 0.6s | | `flow check`, with workaround | 8.2s, no errors | 2.6s, no errors | | `flow ls`, workaround removed | hangs (no output after 120s) | 22,576 lines, 0.8s | | `flow check`, workaround removed | hangs | 4.1s, no errors | (The 3 extra files without the workaround are the npm `metro-memory-fs` package, which `**/node_modules/metro*/**` was hiding.)
|
This pull request has been imported. If you are a Meta employee, you can view this in D132299746. (Because this pull request was imported automatically, there will not be any future comments.) |
SamChou19815
left a comment
There was a problem hiding this comment.
Higher level review:
To make this easier to review, please break it down into several stacked PRs. Specifically for this part
If the target is somewhere the crawl reaches anyway without following links (inside the root, not under a pruned directory), the link isn't followed. Files reached through a link are already reported by their real path, so the crawl reports the same set of files.
Existing logic that decides whether a file belongs should be factored out and reused instead of rolling its own for this purpose
| @@ -0,0 +1 @@ | |||
| module.exports = 'ext2'; | |||
There was a problem hiding this comment.
the symlinks in node_modules broke internal ci. The recommendation is to write test scripts to create them programmatically, so that the test is also runnable on windows
| /// Decides which directory symlinks the crawl descends into. | ||
| /// | ||
| /// Following every directory symlink revisits the same real directory once per logical path, | ||
| /// and package managers such as pnpm create symlink cycles (workspace packages linked into each | ||
| /// other's `node_modules`). jwalk's own loop check misses these, because it compares the raw link | ||
| /// target with the logical paths of the directories above it, so the crawl recurses until the OS | ||
| /// refuses to resolve the path, through a number of logical paths that grows exponentially with | ||
| /// the number of linked packages. Instead: | ||
| /// | ||
| /// - A symlink whose target the crawl reaches without following any symlink is not followed. | ||
| /// Files reached through a symlink are reported by their real path, so descending into it | ||
| /// reports nothing new. | ||
| /// - Any other symlink is followed unless its target is the real path of a directory the crawl is | ||
| /// already inside, i.e. unless it closes a cycle. | ||
| /// | ||
| /// Each symlink that is not followed is recorded with a path its contents are crawled under, so | ||
| /// that module resolution can still map paths through it to real files. | ||
| struct DirSymlinkDedupe { | ||
| root: PathBuf, | ||
| /// The starting points as the walks see them, which bound the cycle check. | ||
| starting_points: Vec<PathBuf>, | ||
| canonical_starting_points: Vec<PathBuf>, | ||
| can_prune: bool, | ||
| all: bool, | ||
| include_libdef: bool, | ||
| /// Symlinks not followed, with the path their contents are crawled under. | ||
| skipped: Mutex<Vec<(PathBuf, PathBuf)>>, | ||
| } | ||
|
|
||
| impl DirSymlinkDedupe { | ||
| fn new( | ||
| root: &Path, | ||
| starting_points: &[PathBuf], | ||
| can_prune: bool, | ||
| all: bool, | ||
| include_libdef: bool, | ||
| ) -> Self { | ||
| let canonical = |path: &Path| cached_canonicalize(path).unwrap_or_else(|_| path.into()); | ||
| Self { | ||
| root: canonical(root), | ||
| starting_points: starting_points.to_vec(), | ||
| canonical_starting_points: starting_points.iter().map(|path| canonical(path)).collect(), | ||
| can_prune, | ||
| all, | ||
| include_libdef, | ||
| skipped: Mutex::new(Vec::new()), | ||
| } | ||
| } | ||
|
|
||
| /// Whether the crawl reads `real_path` and reports every file under it that a crawl through a | ||
| /// symlink to it can report, without following any symlink. | ||
| fn is_crawled_directly(&self, options: &FileOptions, real_path: &Path) -> bool { | ||
| if !(options.implicitly_include_root && real_path.starts_with(&self.root)) { | ||
| return false; | ||
| } | ||
| self.canonical_starting_points | ||
| .iter() | ||
| .filter(|starting_point| real_path.starts_with(starting_point)) | ||
| .any(|starting_point| { | ||
| !self.can_prune | ||
| || real_path | ||
| .ancestors() | ||
| .take_while(|dir| dir.starts_with(starting_point)) | ||
| .all(|dir| wanted_filter(options, dir, self.all, self.include_libdef)) | ||
| }) | ||
| } | ||
|
|
||
| /// The directory containing `path` on the crawl's current walk whose real path is | ||
| /// `real_path`, if any. | ||
| fn enclosing_dir_with_real_path<'a>( | ||
| &self, | ||
| path: &'a Path, | ||
| real_path: &Path, | ||
| ) -> Option<&'a Path> { | ||
| for dir in path.ancestors().skip(1) { | ||
| if cached_canonicalize(dir).is_ok_and(|dir_real_path| dir_real_path == real_path) { | ||
| return Some(dir); | ||
| } | ||
| if self | ||
| .starting_points | ||
| .iter() | ||
| .any(|starting_point| dir == starting_point) | ||
| { | ||
| break; | ||
| } | ||
| } | ||
| None | ||
| } | ||
|
|
||
| /// Whether the crawl descends into the directory symlink at `path`. | ||
| fn should_follow(&self, options: &FileOptions, path: &Path) -> bool { | ||
| let Ok(real_path) = cached_canonicalize(path) else { | ||
| return true; | ||
| }; | ||
| let crawled_under = if self.is_crawled_directly(options, &real_path) { | ||
| real_path | ||
| } else if let Some(dir) = self.enclosing_dir_with_real_path(path, &real_path) { | ||
| dir.to_path_buf() | ||
| } else { | ||
| return true; | ||
| }; | ||
| self.skipped | ||
| .lock() | ||
| .expect("skipped symlinks lock should not be poisoned") | ||
| .push((path.to_path_buf(), crawled_under)); | ||
| false | ||
| } | ||
| } |
There was a problem hiding this comment.
these changes look like ai slop duplicating existing logic
|
Hey @SamChou19815 - thanks for taking a look. Yeah that's fair, this can be split into 4, I'll open new PRs:
1 and 2 are independent. 3 then 4 will follow (I can't create a stack on GH from a fork) |
|
Closing as superseded by the stack above. |
Summary: Fixes #9529 Last part of the stack #9532, #9533, #9534 (split from #9530). #9534 stops the file crawl following a symlink to a directory it reads anyway, but it still follows any other symlink, eg out of the root or out of the directory passed to `flow ls`, and goes round a cycle of them until the OS refuses to resolve the path. With a few packages linked to each other, eg `flow ls packages/a` in a pnpm workspace, that never ends in practice. This stops following a symlink to a directory the crawl passed through to reach it. Every walk now ends, though a dense web of links out of the crawl is still walked once per path through it, which grows quickly with the number of links. Pull Request resolved: #9535 Test Plan: `symlinks_out_of_root` now also has a cycle out of the root, whose files `main` lists once per lap. The full `flow dev-tools runtests` suite passes (macOS). On react/metro installed with pnpm (react/metro#2110), `flow ls packages/metro-config` hangs on `main`. With this, `flow ls` of each workspace package finishes in under a second, and `flow full-check` gives the same result in the same time. Reviewed By: SamChou19815 Differential Revision: D133227488 fbshipit-source-id: 889fee75094aed805ff360dbd21939cbe8cf465b
Fixes #9529
Problem
When workspace packages are linked into each other's
node_modules(pnpm withlinkWorkspacePackages), the file crawl doesn't terminate in practice.make_next_fileswalks with jwalk'sfollow_links(true), and jwalk's loop check compares the rawread_linktarget with the paths of the directories above the entry, so it never matches a relative link:https://github.com/Byron/jwalk/blob/a5b1ea634fcacee06f7ef6e9411e0cfecfccb128/src/core/dir_entry.rs#L219-L229
The walk only stops at
ELOOP, after visiting every path through the link graph up to 32 links deep - exponential in the number of linked packages. Acyclic layouts are walked once per path too, so eg a pnpm store gets crawled once per dependent.This change
For each directory symlink, the crawl now decides whether to descend:
Links that aren't followed are recorded in
SymlinkMapas directory aliases, andtry_resolvemaps paths through them, so resolution throughnode_moduleslinks works as before. The crawl also reports each canonicalised file once.Outside the direct crawl (links out of the root, or
flow ls <dir>), two links to the same target can be filtered differently by[ignore]- that's theignored_aliascase innode_modules_with_symlinks- so only cycles are cut there, not repeat visits.Test plan
New
node_modules_symlink_cyclestest, covering workspace packages linked to each other, a.pnpm-style store with a dependency cycle and a cycle outside the root, throughls,checkandstatus. New unit tests for the crawl and for resolving through chained aliases.flow dev-tools runtestson macOS: 1084 passed, 0 failed, 9 skipped. I haven't run it on Windows - the new test creates its links withmklinkthere, the same waynode_modules_with_symlinksdoes.On react/metro under pnpm (react/metro#2110, macOS, cold
flow checkafterflow stop):mainflow ls, with Metro's[ignore]workaroundflow check, with workaroundflow ls, workaround removedflow check, workaround removed(The 3 extra files without the workaround are the npm
metro-memory-fspackage, which**/node_modules/metro*/**was hiding.)