Skip to content

hydra: build CUDA packages for the CUDA team #1335

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zimbatm merged 1 commit into from
Jul 4, 2024
Merged

hydra: build CUDA packages for the CUDA team #1335

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions terraform/hydra-projects.tf
View file
Open in desktop
Copy link
Contributor

@SomeoneSerge SomeoneSerge Jul 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zowoq so I'm considering just building the whole import <nixpkgs> { config.cudaSupport = true; }; this gives me something like

❯ nix-eval-jobs --expr 'import ./pkgs/top-level/release-cuda.nix { }' --force-recurse | wc -l
...
# eval errors, eval errors
...
138452

Does that sound unreasonable? I could in principle come up with a smaller, curated set of jobs.

Hexa also raises the concern that this would be effectively mirroring the NixOS Hydra:

hexa (UTC+1)
so all of them
if there was a cache behind nix-community hydra, than you'd be mirroring cache.nixos.org effectively
SomeoneSerge (UTC+3)
Yeah... Ideally we'd have a solution that evaluates the full DAGs for vanilla and cuda nixpkgs, starts building cuda from the leaves (ehhh, the roots), and always suspends the build if it hash matches the vanilla hash

Copy link
Contributor

@zowoq zowoq Jul 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

import <nixpkgs> { config.cudaSupport = true; }

TBH I doubt we have the capacity to handle that without it being detrimental to the other projects using this CI. Currently we only have two builders for linux that are shared by buildbot, hercules, hydra:

### `build03`

- Provider: Hetzner
- CPU: AMD Ryzen 9 3900 12-Core Processor
- RAM: 128GB DDR4 ECC
- Drives: 2 x 1.92 TB NVME in RAID 1

### `build04`

- Provider: Hetzner
- Instance type: [RX170](https://www.hetzner.com/dedicated-rootserver/rx170)
- CPU: Ampere Altra Q80-30 80-Core Processor
- RAM: 128GB DDR4 ECC
- Drives: 2 x 960 GB NVME in RAID 0

If you do want to build everything maybe we could have dedicated machines just for this package set? Not sure if raising the money for that is feasible?

if there was a cache behind nix-community hydra, than you'd be mirroring cache.nixos.org effectively

Not sure if I've misunderstood or not, we push everything to cachix but it skips existing nixos cache paths.

Copy link
Contributor

@SomeoneSerge SomeoneSerge Jul 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if I've misunderstood or not, we push everything to cachix but it skips existing nixos cache paths.

The concern is that if there is a phase shift between NixOS and the Community Hydras, and the latter starts building a certain derivation from a certain commit before the former, we'll have wasted some storage and compute

Copy link
Contributor

@SomeoneSerge SomeoneSerge Jul 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH I doubt we have the capacity to handle that without it being detrimental to the other projects using this CI. Currently we only have two builders for linux that are shared by buildbot, hercules, hydra:

Roger that. I'll push a smaller jobset tomorrow, based on what we've been building in https://github.com/SomeoneSerge/nixpkgs-cuda-ci.

How is the community builder funded? @ConnorBaker was asking on matrix if there's an opencollective

Copy link
Contributor

@zowoq zowoq Jul 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The concern is that if there is a phase shift between NixOS and the Community Hydras, and the latter starts building a certain derivation from a certain commit before the former, we'll have wasted some storage and compute

Yeah, we can't really avoid that with this approach. Could try adding the free deps as blockers for nixos-unstable or could try doing something similar in a repo here, flake update PRs with max-jobs = 0 so merging is blocked if they aren't cached?

How is the community builder funded? ConnorBaker was asking on matrix if there's an opencollective

We have an opencollective: https://opencollective.com/nix-community

Original file line number Diff line number Diff line change
Expand Up @@ -69,3 +69,55 @@ resource "hydra_project" "simple_nixos_mailserver" {
value = "https://gitlab.com/simple-nixos-mailserver/nixos-mailserver"
}
}

resource "hydra_project" "nixpkgs" {
name = "nixpkgs"
display_name = "nixpkgs"
description = "you know what this is"
homepage = "https://github.com/NixOS/nixpkgs"
owner = "admin"
enabled = true
visible = true
}

resource "hydra_jobset" "nixpkgs_cuda" {
project = hydra_project.nixpkgs.name
state = "enabled"
visible = true
name = "cuda"
type = "legacy"
description = "Testing CUDA support. Come help the CUDA team! https://nixos.org/community/teams/cuda/"

nix_expression {
file = "pkgs/top-level/release-cuda.nix"
input = "nixpkgs"
}

input {
name = "nixpkgs"
type = "git"
value = "https://github.com/NixOS/nixpkgs.git nixos-unstable-small"
notify_committers = false
}

input {
name = "officialRelease"
type = "boolean"
value = "false"
notify_committers = false
}

input {
name = "supportedSystems"
type = "nix"
value = "[ \"x86_64-linux\" ]"
notify_committers = false
}

check_interval = 1800
scheduling_shares = 3000
keep_evaluations = 1

email_notifications = false
email_override = ""
}