bundle templates of required input packages by teresaromero · Pull Request #3329 · elastic/elastic-package

teresaromero · 2026-03-03T08:47:45Z

Bundle input package templates for composable integrations

This PR adds support for composable integrations: packages that declare a
requires.input section in their manifest to pull agent input templates from a
separate input-type package at build time.

Dependencies

This PR requires the following change to be merged first:

What changed

Note: commit 4f25ab87 (chore: update package-spec) is temporary and must be reverted before merge once the package-spec release is available.

Commit	Summary
`4f25ab87`	Bump package-spec to pre-release version with `requires.input` support — revert before merge
`adf728b2`	Extend manifest types: `Requires`, `PackageDependency`, `RequiresOverride`, `TemplatePaths`; add `FindPackageInRepo`
`3ecf9d60`	Implement `bundleInputPackageTemplates` in the builder; add `DownloadPackage` to fetch packages from EPR; add unit tests and fixtures
`eba23df1`	Wire `RegistryURL` and `RequiresOverrides` through `FleetPackage` → `BuildOptions`; support per-test-type overrides in `_dev/test/config.yml`; inject registry client from app config
`b43bf5b1`	Add HOWTO guide for local/custom registry with composable integrations; update README and dependency management docs

How it works

When elastic-package build runs on an integration that declares requires.input,
it now:

Resolves each required input package — from the Elastic Package Registry or a
local source override configured in _dev/test/config.yml.
Copies the input package's agent input templates into
data_stream/<ds>/agent/stream/ with a <pkgname>- prefix to avoid
collisions.
Rewrites the data stream manifest to use template_paths, listing the input
package templates first and the integration's own template last.

Testing

Unit tests cover the YAML rewriting helpers (setStreamTemplatePaths) with
minimal fixtures under internal/builder/testdata/input_packages/.
Integration-style tests exercise the full bundleInputPackageTemplates path
using real on-disk fixture packages under test/packages/required_inputs/.
A source override in _dev/test/config.yml lets tests bypass the registry
and use a local fixture package instead.

Docs

A new HOWTO guide at docs/howto/local_package_registry.md explains how to
run a local registry when developing composable integrations.

Resolves Bundle templates of required input packages #3278

teresaromero · 2026-03-03T08:49:18Z

blocked for merge until 3.6 elastic/package-spec#1060

Replace the existing package-spec dependency with a specific pre-release version that includes support for the requires.input manifest section. NOTE: Remove the replace directive before merging and use the latest released version when available. Made-with: Cursor

Add Requires, PackageDependency, and RequiresOverride types. Extend Stream with PackageRef and TemplatePaths, PolicyTemplate with TemplatePaths, and PackageManifest with a Requires field. Add FindPackageInRepo to search the repository for a package by name, scanning up to 4 directory levels to accommodate monorepo layouts.

Add Client.DownloadPackage which fetches a package by name and version from the Elastic Package Registry, extracts the zip archive into a temporary directory, and returns the path to the extracted package root. Add bundleInputPackageTemplates, which runs after the package content is copied to the build directory. For each integration that declares requires.input, it resolves every required input package (from the registry or a local source override), copies agent input templates into data_stream/<ds>/agent/stream/ with a "<pkgname>-" prefix, and rewrites the data stream manifest to use template_paths. Add test fixtures (test_input_pkg, with_input_package_requires) and unit tests for the YAML rewriting helpers and the full bundling path.

… pipeline Thread RequiresOverrides and RegistryURL through FleetPackage, installer.Options, and BuildOptions so test runners can supply source or version overrides for required input packages. Support requires overrides at both top-level and per-type sections in the global test config (_dev/test/config.yml). Per-type entries override global entries by package name. Inject the registry client from app config values to enable local or custom package registries during the build. Add an integration test that exercises the full template bundling path using a local fixture package, bypassing the package registry.

Add a new HOWTO guide explaining how to configure elastic-package to use a local or custom package registry when developing composable integrations that declare required input packages. Update the build command description, README, and dependency management docs to document the relationship between the build process and the package registry for composable packages.

…ed before merge

…undle-templates-required-input-packages

mrodm · 2026-03-09T17:11:30Z

cmd/install.go

 		return fmt.Errorf("can't load configuration: %w", err)
 	}

+	eprClient := registry.NewClient(appConfig.PackageRegistryBaseURL())


As the installation would depend also on the profile being set, the EPR url should be also retrieved taken into account the one that could be set in the profile.

In the stack module, this is done via this method:

elastic-package/internal/stack/registry.go

Lines 34 to 44 in 997b47d

func packageRegistryBaseURL(profile *profile.Profile, appConfig *install.ApplicationConfiguration) string {

if registryURL := profile.Config(configElasticEPRURL, ""); registryURL != "" {

return registryURL

}

if appConfig != nil {

if registryURL := appConfig.PackageRegistryBaseURL(); registryURL != "" {

return registryURL

}

}

return registry.ProductionURL

}

i've made the function public and used it at install cmd. build command is still using appconfig as the command does not use profile

…undle-templates-required-input-packages

elasticmachine · 2026-03-10T14:07:10Z

💔 Build Failed

Buildkite Build
Commit: a690ef0

Failed CI Steps

:go: Run check-static

History

💔 Build #7423 failed 50c3b2b
💔 Build #7417 failed 7468026
💔 Build #7339 failed 09fdab8
💔 Build #7338 failed b43bf5b
💔 Build #7314 failed 9b55229

jsoriano

Nice, thanks!

jsoriano · 2026-03-10T18:31:07Z

docs/howto/dependency_management.md

+Composable (integration) packages can also depend on other packages — specifically **input
+packages** — by declaring them under `requires.input` in `manifest.yml`:


Nit. They can also depend on content packages, but at the moment they are resolved at runtime.

jsoriano · 2026-03-10T18:44:16Z

docs/howto/local_package_registry.md

+This guide explains how to point elastic-package at a local or custom registry, which is
+useful when the required input packages are still under development and not yet published to
+the production registry.


elastic-package stack up with the compose provider (the default), starts a local package registry, that serves the packages under build/packages. This can be used to serve local packages. You only need to remember building the ones you need.

This mechanism would even be improved after your change in elastic/package-registry#1482, what would allow reloading the packages without needing to restart the registry.

In the documentation here we are adding another method to serve local packages. We should probably converge both methods in a single one.

jsoriano · 2026-03-10T18:44:52Z

docs/howto/local_package_registry.md

+elastic-package build
+
+# Start a local registry serving the build/ directory
+docker run --rm -p 8080:8080 \


Starting a local registry on local port 8080 is going to conflict with the registry started with elastic-package stack up.

jsoriano · 2026-03-10T18:46:00Z

internal/builder/input_packages.go

Nit. Avoid using underscores in Go file or package names.

jsoriano · 2026-03-10T19:19:47Z

internal/builder/packages.go

 		return "", fmt.Errorf("resolving transform manifests failed: %w", err)
 	}

+	err = bundleInputPackageTemplates(options.PackageRoot, buildPackageRoot, options.RegistryClient, options.RequiresOverrides)


For a future refactor. We have several methods that receive options.PackageRoot to read the manifest. We could maybe make these methods members of an object that shares the manifest and maybe other parts of a package.

jsoriano · 2026-03-10T19:34:43Z

internal/builder/input_packages.go

+	dsRoot := filepath.Dir(dsManifestPath)
+	agentStreamDir := filepath.Join(dsRoot, "agent", "stream")
+
+	// Parse the YAML document preserving formatting for targeted modifications.


Preserving the format is nice, though we don't really need to keep the format here, as we are building the package, that is not intended for later human-driven modifications. I would be concerned of complicating things unnecessarily.

jsoriano · 2026-03-10T19:42:37Z

internal/builder/input_packages.go

+func removeKey(node *yaml.Node, key string) {
+	for i := 0; i+1 < len(node.Content); i += 2 {
+		if node.Content[i].Value == key {
+			node.Content = append(node.Content[:i], node.Content[i+2:]...)
+			return
+		}
+	}
+}


Nit. We can possibly reuse slices package here and in other helpers.

Suggested change

func removeKey(node *yaml.Node, key string) {

for i := 0; i+1 < len(node.Content); i += 2 {

if node.Content[i].Value == key {

node.Content = append(node.Content[:i], node.Content[i+2:]...)

return

}

}

}

func removeKey(node *yaml.Node, key string) {

node.Content = slices.DeleteFunc(node.Content, func(n *yaml.Node) bool {

return n.Value == key

})

}

jsoriano · 2026-03-10T19:46:28Z

internal/packages/installer/factory.go

+	RegistryClient    *registry.Client                     // Registry client for downloading input packages.
+	RequiresOverrides map[string]packages.RequiresOverride // Pre-merged requires overrides (test builds only).


Nit. Maybe for a future change. Instead of receiving the registry and the overrides, this could receive a higher level abstraction that gets dependencies. This abstraction would handle the different package sources, caches, overrides and so on.

jsoriano · 2026-03-10T19:47:38Z

internal/packages/packages.go

+// accommodate the common monorepo layout where packages live under a top-level
+// "packages/" directory. Returns the package root path and its manifest, or an
+// error when the package cannot be found.
+func FindPackageInRepo(repoRoot string, packageName string) (string, *PackageManifest, error) {


jsoriano · 2026-03-10T19:49:50Z

internal/registry/download.go

+	}
+	tmpFile.Close()
+
+	pkgRoot, err := extractZipPackage(tmpPath, destDir)


We should not need to extract the zips, it is possible to use a zip reader as a filesystem.

Even if we need to do it at the end, I think it should not be a responsibility of the DownloadPackage method to extract it.

teresaromero marked this pull request as ready for review March 3, 2026 08:49

teresaromero requested a review from a team as a code owner March 3, 2026 08:49

teresaromero marked this pull request as draft March 3, 2026 09:36

teresaromero added 5 commits March 3, 2026 14:50

teresaromero changed the title ~~builder: bundle templates of required input packages~~ bundle templates of required input packages Mar 4, 2026

teresaromero force-pushed the 3278-bundle-templates-required-input-packages branch from 9b55229 to b43bf5b Compare March 4, 2026 10:30

teresaromero added 2 commits March 4, 2026 11:39

chore: update with latests package-spec. this commit should be revert…

5da820b

…ed before merge

Merge branch 'main' of github.com:elastic/elastic-package into 3278-b…

09fdab8

…undle-templates-required-input-packages

teresaromero marked this pull request as ready for review March 4, 2026 10:42

mrodm reviewed Mar 9, 2026

View reviewed changes

teresaromero added 3 commits March 10, 2026 10:54

use profile epr url on install command. update readme

7468026

Revert "update readme"

50c3b2b

Merge branch 'main' of github.com:elastic/elastic-package into 3278-b…

a690ef0

…undle-templates-required-input-packages

mrodm requested a review from jsoriano March 10, 2026 17:10

jsoriano reviewed Mar 10, 2026

View reviewed changes

teresaromero marked this pull request as draft March 11, 2026 11:32

	func packageRegistryBaseURL(profile profile.Profile, appConfig install.ApplicationConfiguration) string {
	if registryURL := profile.Config(configElasticEPRURL, ""); registryURL != "" {
	return registryURL
	}
	if appConfig != nil {
	if registryURL := appConfig.PackageRegistryBaseURL(); registryURL != "" {
	return registryURL
	}
	}
	return registry.ProductionURL
	}

		Composable (integration) packages can also depend on other packages — specifically **input
		packages** — by declaring them under `requires.input` in `manifest.yml`:

		RegistryClient *registry.Client // Registry client for downloading input packages.
		RequiresOverrides map[string]packages.RequiresOverride // Pre-merged requires overrides (test builds only).

Conversation

teresaromero commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle input package templates for composable integrations

Dependencies

What changed

How it works

Testing

Docs

Uh oh!

teresaromero commented Mar 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

jsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

teresaromero commented Mar 3, 2026 •

edited

Loading

elasticmachine commented Mar 10, 2026 •

edited

Loading