feat: allow configuring custom includes for prune subcommand

[krinoid]

Description

Extended turbo prune command and turbo.json config, so it's possible to specify additional fields that need to be copied during the prune operation. For example, it's possible to specify in the top-level turbo.json:

"prune": {
  "includes": ["**/README.md", "$TURBO_ROOT$/lib/shared-library/example.file"]
}

so that README.md of all pruned workspaces should be copied and if shared-library is part of the prunned tree, shared-library/example.file will also be copied.

It should also be possible to specify prune includes in the workspace-level turbo.json:

"prune": {
  "includes": ["example.file"]
}

Here the includes supports workspace-relative paths.

Motivation for this change
I have a use-case, where some of my workspaces have postinstall script performing some additional, required operations for a successful installation. By definition, postinstall is invoked directly after yarn/npm/pnpm install. I use turbo prune --docker, so before I can invoke install I need to manually copy relevant files first. While that works at a smaller scale, when this postinstall script is part of a shared library used by many apps/other libraries in the monorepo, every Dockerfile needs to manually copy these required files, which is annoying. Being able to configure that once in the turbo.json simplifies that use-case.

Related discussion: #7464

Testing Instructions

• I've added new unit tests validating the expected behavior of individual functions
• I've added a couple of integration tests for this new option

[vercel]

@krinoid is attempting to deploy a commit to the Vercel Team on Vercel.

A member of the Team first needs to authorize it.

[krinoid]

Note: I'm still learning Rust and I'm not familiar with the Turborepo codebase — I'm happy to completely refactor the feature, if needed, provided some guidance/feedback. The first question would be — is that a feature that you're willing to consider for adding to the Turborepo? I tried to explain my use-case and existing discussion indicates that others have similar problem. Let me know what you think.

[anthonyshew]

Hey, @krinoid, thanks for the PR. I'm curious, what is the use case where copy commands (e.g. cp) aren't enough to get the same outcome?

[krinoid]

Hey, @krinoid, thanks for the PR. I'm curious, what is the use case where copy commands (e.g. cp) aren't enough to get the same outcome?

@anthonyshew Fair question — cp or similar can work at smaller scale monorepos. The problem starts getting annoying with bigger scale. Let's imagine that we have shared-types package and for some reason it's required that packages/shared-types/some-script.sh is present during npm/yarn/pnpm install step, because it's invoked as part of the postinstall. Now, let's say that we have 100 apps in the monorepo, and 50 of those depend on shared-types. If we want to use turbo prune in our monorepo's Dockerfiles, then for each app's Dockerfile we'd need to have:

RUN turbo prune <some-app> --docker
RUN cp packages/shared-types/some-script.sh out/json/packages/shared-types/ # needed in every Dockerfile
RUN pnpm install

While you could argue that this is just copy & paste, it "leaks" some implementation details of shared-types. And that's just the simplest example. The biggest PITA is when the shared-library is not a direct dependency but a transitive one. Say our app-1 depends on package-1 and it's package-1 that depends on shared-types. Now we need to add

RUN cp packages/shared-types/some-script.sh out/json/packages/shared-types/ 

in app-1, even though it doesn't directly depend on shared-types. This is extra annoying. Even more so, when it's not just 1 package that requires some scripts/files to be present during the install step.

Let me know if is is an useful illustration of the problem. I'm happy to provide more details and answer any questions. To be clear, this is not a hypothetical scenario — I'm currently facing it in a monorepo with 50 apps and 100+ shared packages and it definitely increases the maintenance costs. I really like turbo and turbo prune proved to speed up our build times but I'd love to remove that small papercut 🙂

[krinoid]

Hi @anthonyshew, could you help me understand if the current approach/API makes sense? I'd love to have this feature in the turborepo, as it would really simplify the monorepo I'm working in. Thanks.

[krinoid]

@tknickman let me know if this feature is something you'd consider in general. I'm happy to rework the code, so that it fits into your vision around config etc.

[weyert]

Yes, this would be really helpful and avoid needing to mess around with post-scripts to copy over files to the pruned output directory. Such as copying over .dockerignore, some files in scripts etc

[krinoid]

@anthonyshew @tknickman Hi, do you think it's something that you might consider for turbo? I'm open to adapt the approach/conventions, if needed — but if possible, I'd like to get a decision whether we could go forward with this feature. Otherwise I'll have to look for other solutions for my use-case.