[NDT-561] Base orchestrator, PodRun, PodResult + LocalDockerOrchestrator implementation

[guzman-raphael]

Depends on #42, #43

Features

• Add orchestrator traits+ PodRun (see design below)
• Add mod docker that implements a local docker orchestrator (see design below)
• Add initial PodResult model (see design below). PodJob memory state is shared via container labels so it can reconstruct independently across client sessions e.g. calling orchestrator.list() or orchestrator.get_result(pod_run) on a different machine than you started on.
• Add optional environment variables to PodJob
• Add default compute_checksum definition for BlobInterface (passthrough)
• Allow DevContainer to set CPU/Memory limits for containers
• Use once_cell for setting RegEx's statically but with lazy evaluation (on first access)
• Add fixture and test support for a simple, real, python-based style transfer example
• Update dependencies
• Update tests

Project Management

My goal of this PR was to complete a MVP version of Orcapod that has enough features to be usable in alpha testing. That said, not everything we've discussed is implemented here but I propose we separate them out into individual issues that can be addressed independently without blocking a release for testing.

Specifically, I'm proposing the following changes:

Update the following issues such that this PR:

• Fixes Minimum Orchestrator trait #18: Update to reflect a minimal orchestrator trait and docker implementation.
• Fixes Minimum PodRun #12: Update to reflect a minimal PodRun
• Fixes Minimal PodResult Model #14: Update to reflect a minimal PodResult
• Fixes Revisit List Model output #41: Since list_model is stable

Open separate issues for:

• Orchestrator logging to trait + docker implementation + add toPodResult -> Orchestrator logging interface + Docker implementation #50
• BlobInterface: Evaluating checksums of Folder's + iterating over Collection's -> Evaluate checksums for Folder and Collection #54
• Evaluating and storing checksums for any outputs of PodResult -> Save compute output checksums in PodResult #55
• Orchestrator resource metrics to trait + docker implementation + add toPodResult -> Orchestrator resource metrics interface + Docker implementation #51
• Orchestrator load_image to trait + docker implementation -> Orchestrator load_image interface + Docker implementation #52
• Compute retry mechanism in PodJob -> Add compute retry mechanism to PodJob #53

Update the spec for the following issues:

• Capture image@digest on starting containers #9: Resolve start_with_altimage tag to an actual SHA256 digest + use it to launch container (image@digest)
• Implement thiserror #40: Redefine for thiserror since it is stable and provides the features we were looking for in anyhow

Design

mod model {
    pub struct PodResult {
        #[serde(skip)]
        pub annotation: Option<Annotation>,
        #[serde(skip)]
        pub hash: String,
        #[serde(
            serialize_with = "serialize_pod_job",
            deserialize_with = "deserialize_pod_job"
        )]
        pub pod_job: PodJob,
        /// Name given by orchestrator.
        pub assigned_name: String,
        /// Status of compute run when terminated.
        pub status: Status,
        /// Time in epoch when created in seconds.
        pub created: u64,
        /// Time in epoch when terminated in seconds.
        pub terminated: u64,
    }
}

mod orchestrator {
    pub enum ImageKind {
        Published(String),
        Tarball(PathBuf),
    }

    pub enum Status {
        Running,
        Completed,
        Failed(i16),
    }

    pub struct RunInfo {
        pub image: String,
        pub created: u64,
        pub terminated: Option<u64>,
        pub env_vars: HashMap<String, String>,
        pub command: String,
        pub status: Status,
        pub mounts: Vec<String>,
        pub labels: HashMap<String, String>,
        pub cpu_limit: f32,
        pub memory_limit: u64,
    }

    pub struct PodRun
    {
        pub pod_job: PodJob,
        pub orchestrator_source: String,
        pub assigned_name: String,
    }

    pub trait Orchestrator
    {
        fn start_with_altimage_blocking(&self, pod_job: &PodJob, image: &ImageKind) -> Result<PodRun>;
        fn start_blocking(&self, pod_job: &PodJob) -> Result<PodRun>;
        fn list_blocking(&self) -> Result<Vec<PodRun>>;
        fn delete_blocking(&self, pod_run: &PodRun) -> Result<()>;
        fn get_info_blocking(&self, pod_run: &PodRun) -> Result<RunInfo>;
        fn get_result_blocking(&self, pod_run: &PodRun) -> Result<PodResult>;
        fn start_with_altimage(
                &self,
                pod_job: &PodJob,
                image: &ImageKind,
            ) -> impl Future<Output = Result<PodRun>> + Send;
        fn start(&self, pod_job: &PodJob) -> impl Future<Output = Result<PodRun>> + Send;
        fn list(&self) -> impl Future<Output = Result<Vec<PodRun>>> + Send;
        fn delete(&self, pod_run: &PodRun) -> impl Future<Output = Result<()>> + Send;
        fn get_info(&self, pod_run: &PodRun) -> impl Future<Output = Result<RunInfo>> + Send;
        fn get_result(&self, pod_run: &PodRun) -> impl Future<Output = Result<PodResult>> + Send;
    }

    mod docker {
        pub struct LocalDockerOrchestrator {
            data_directory: PathBuf,
            api: Docker,
            async_driver: Runtime,
        }

        impl Orchestrator for LocalDockerOrchestrator {..}

        impl LocalDockerOrchestrator {
            pub fn new(data_directory: impl AsRef<Path>) -> Result<Self> {
                Ok(Self {
                    data_directory: fs::canonicalize(data_directory)?,
                    api: Docker::connect_with_local_defaults()?,
                    async_driver: Runtime::new()?,
                })
            }

            fn prepare_container_start_inputs(
                &self,
                pod_job: &PodJob,
                image: String,
            ) -> Result<(
                String,
                Option<CreateContainerOptions<String>>,
                Config<String>,
            )> {..}

            async fn list_containers(
                &self,
                filters: HashMap<String, Vec<String>>,
            ) -> Result<impl Iterator<Item = (String, RunInfo)>> {..}
        }
    }
}

[Synicix]

Should probably rename to just State due to the usage in pod result. Also perhaps move this to utils since it is used by mutiple modules?

[Synicix]

Also is there no option for queuing? Sometimes orcherstrator might have max jobs already running like in K8 and has to wait their turn

[Synicix]

Make an issue about either finding a way to deal with it via enum inheratince (don't think there either)
Other resolved

[Synicix]

Hmm, nice we don't have to redo thsi every time

[Synicix]

Need to look up how this works

[eywalker]

Marking it as a change request but it's really to just raise a few discussion points on which I commented.

[Synicix]

looks good with the updates