[PNE-7018] Add new evaluations API and deprecate PEWs.

docs/source/getting_started/ai_modules.rst

@@ -38,7 +38,7 @@ Design Workflow
 A Design Workflow combines a Design Space to define the materials of interest and a Predictor to predict material properties.
 They also include a :doc:`Score <../workflows/scores>` which codifies goals of the project.
 
-Predictor Evaluation Workflow
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Predictor Evaluation
+^^^^^^^^^^^^^^^^^^^^
 
-:doc:`Predictor Evaluation Workflows <../workflows/predictor_evaluation_workflows>` analyze the quality of a Predictor.
+:doc:`Predictor Evaluations <../workflows/predictor_evaluation_workflows>` analyze the quality of a Predictor.

docs/source/getting_started/basic_functionality.rst

@@ -49,14 +49,12 @@ It is often useful to know when a resource has completed validating, especially
     sintering_model = sintering_project.predictors.register(sintering_model)
     wait_while_validating(collection=sintering_project.predictors, module=sintering_model)
 
-Similarly, the ``wait_while_executing`` function will wait for a design or performance evaluation workflow to complete executing.
+Similarly, the ``wait_while_executing`` function will wait for a design or predictor evaluation to complete executing.
 
 .. code-block:: python
 
-    pew_workflow = sintering_project.predictor_evaluation_workflows.register(pew_workflow)
-    pew_workflow = wait_while_validating(collection=sintering_project.predictor_evaluation_workflows, module=pew_workflow)
-    pew_ex = pew_workflow.trigger(sintering_model)
-    wait_while_executing(collection=sintering_project.predictor_evaluation_executions, execution=pew_ex, print_status_info=True)
+    predictor_evaluation = project.predictor_evaluations.trigger_default(predictor_id=sintering_model.uid)
+    wait_while_executing(collection=sintering_project.predictor_evaluations, execution=predictor_evaluation, print_status_info=True)
 
 Checking Status
 ---------------

docs/source/workflows/getting_started.rst

@@ -11,9 +11,8 @@ These capabilities include generating candidates for Sequential Learning, identi
 Workflows Overview
 ------------------
 
-Currently, there are two workflows on the AI Engine: the :doc:`DesignWorkflow <design_workflows>` and the :doc:`PredictorEvaluationWorkflow <predictor_evaluation_workflows>`.
-Workflows employ reusable modules in order to execute.
-There are three different types of modules, and these are discussed in greater detail below.
+Currently, there are two workflows on the AI Engine: the :doc:`DesignWorkflow <design_workflows>` and the :doc:`PredictorEvaluation <predictor_evaluation_workflows>`.
+There are two different types of modules, and these are discussed in greater detail below.
 
 Design Workflow
 ***************
@@ -38,11 +37,11 @@ Branches
 A ``Branch`` is a named container which can contain any number of design workflows, and is purely a tool for organization.
 If you do not see branches in the Citrine Platform, you do not need to change how you work with design workflows. They will contain an additional field ``branch_id``, which you can ignore.
 
-Predictor Evaluation Workflow
-*****************************
+Predictor Evaluation
+********************
 
-The :doc:`PredictorEvaluationWorkflow <predictor_evaluation_workflows>` is used to analyze a :doc:`Predictor <predictors>`.
-This workflow helps users understand how well their predictor module works with their data: in essence, it describes the trustworthiness of their model.
+The :doc:`PredictorEvaluation <predictor_evaluation_workflows>` is used to analyze a :doc:`Predictor <predictors>`.
+They helps users understand how well their predictor module works with their data: in essence, it describes the trustworthiness of their model.
 These outcomes are captured in a series of response metrics.
 
 Modules Overview
@@ -80,17 +79,3 @@ Validation status can be one of the following states:
 -  **Error:** Validation did not complete. An error was raised during the validation process that prevented an invalid or ready status to be determined.
 
 Validation of a workflow and all constituent modules must complete with ready status before the workflow can be executed.
-
-Experimental functionality
-**************************
-
-Both modules and workflows can be used to access experimental functionality on the platform.
-In some cases, the module or workflow type itself may be experimental.
-In other cases, whether a module or workflow represents experimental functionality may depend on the specific configuration of the module or workflow.
-For example, a module might have an experimental option that is turned off by default.
-Another example could be a workflow that contains an experimental module.
-Because the experimental status of a module or workflow may not be known at registration time, it is computed as part
-of the validation process and then returned via two fields:
-
-- `experimental` is a Boolean field that is true when the module or workflow is experimental
-- `experimental_reasons` is a list of strings that describe what about the module or workflow makes it experimental

docs/source/workflows/predictor_evaluation_workflows.rst

@@ -1,15 +1,15 @@
-Predictor Evaluation Workflows
-==============================
+Predictor Evaluations
+=====================
 
-A :class:`~citrine.informatics.workflows.predictor_evaluation_workflow.PredictorEvaluationWorkflow` evaluates the performance of a :doc:`Predictor <predictors>`.
-Each workflow is composed of one or more :class:`PredictorEvaluators <citrine.informatics.predictor_evaluator.PredictorEvaluator>`.
+A :class:`~citrine.informatics.executions.predictor_evaluation.PredictorEvaluation` evaluates the performance of a :doc:`Predictor <predictors>`.
+Each evaluation utilizes one or more :class:`PredictorEvaluators <citrine.informatics.predictor_evaluator.PredictorEvaluator>`.
 
 Predictor evaluators
 --------------------
 
 A predictor evaluator defines a method to evaluate a predictor and any relevant configuration, e.g., k-fold cross-validation evaluation that specifies 3 folds.
 Minimally, each predictor evaluator specifies a name, a set of predictor responses to evaluate and a set of metrics to compute for each response.
-Evaluator names must be unique within a single workflow (more on that `below <#execution-and-results>`__).
+Evaluator names must be unique within a single evaluation (more on that `below <#execution-and-results>`__).
 Responses are specified as a set of strings, where each string corresponds to a descriptor key of a predictor output.
 Metrics are specified as a set of :class:`PredictorEvaluationMetrics <citrine.informatics.predictor_evaluation_metrics.PredictorEvaluationMetric>`.
 The evaluator will only compute the subset of metrics valid for each response, so the top-level metrics defined by an evaluator should contain the union of all metrics computed across all responses.
@@ -102,22 +102,21 @@ For categorical responses, performance metrics include the area under the receiv
 Execution and results
 ---------------------
 
-Triggering a Predictor Evaluation Workflow produces a :class:`~citrine.resources.predictor_evaluation_execution.PredictorEvaluationExecution`.
-This execution allows you to track the progress using its ``status`` and ``status_info`` properties.
-The ``status`` can be one of ``INPROGRESS``, ``READY``, or ``FAILED``.
-Information about the execution status, e.g., warnings or reasons for failure, can be accessed via ``status_info``.
+Once triggered, you can track the evaluation's progress using its ``status`` and ``status_detail`` properties.
+The ``status`` can be one of ``INPROGRESS``, ``SUCCEEDED``, or ``FAILED``.
+Information about the execution status, e.g., warnings or reasons for failure, can be accessed via ``status_detail``.
 
-When the ``status`` is ``READY``, results for each evaluation defined as part of the workflow can be accessed using the ``results`` method:
+When the ``status`` is ``SUCCEEDED``, results for each evaluator defined as part of the evaluation can be accessed using the ``results`` method:
 
 .. code:: python
 
-    results = execution.results('evaluator_name')
+    results = evaluation.results('evaluator_name')
 
-or by indexing into the execution object directly:
+or by indexing into the evaluation object directly:
 
 .. code:: python
 
-    results = execution['evaluator_name']
+    results = evaluation['evaluator_name']
 
 Both methods return a :class:`~citrine.informatics.predictor_evaluation_result.PredictorEvaluationResult`.
 
@@ -153,7 +152,7 @@ Each data point defines properties ``uuid``, ``identifiers``, ``trial``, ``fold`
 Example
 -------
 
-The following demonstrates how to create a :class:`~citrine.informatics.predictor_evaluator.CrossValidationEvaluator`, add it to a :class:`~citrine.informatics.workflows.predictor_evaluation_workflow.PredictorEvaluationWorkflow`, and use it to evaluate a :class:`~citrine.informatics.predictors.predictor.Predictor`.
+The following demonstrates how to create a :class:`~citrine.informatics.predictor_evaluator.CrossValidationEvaluator` and use it to evaluate a :class:`~citrine.informatics.predictors.predictor.Predictor`.
 
 The predictor we'll evaluate is defined below:
 
@@ -215,36 +214,19 @@ In this example we'll create a cross-validation evaluator for the response ``y``
         metrics={RMSE(), PVA()}
     )
 
-Then add the evaluator to a :class:`~citrine.informatics.workflows.predictor_evaluation_workflow.PredictorEvaluationWorkflow`, register it with your project, and wait for validation to finish:
+Then, trigger an evaluation and wait for the results to be ready:
 
 .. code:: python
 
-    from citrine.informatics.workflows import PredictorEvaluationWorkflow
-
-    workflow = PredictorEvaluationWorkflow(
-        name='workflow that evaluates y',
-        evaluators=[evaluator]
-    )
-
-    workflow = project.predictor_evaluation_workflows.register(workflow)
-    wait_while_validating(collection=project.predictor_evaluation_workflows, module=workflow)
-
-Trigger the workflow against a predictor to start an execution.
-Then wait for the results to be ready:
-
-.. code:: python
-
-    from citrine.jobs.waiting import wait_while_executing
-
-    execution = workflow.executions.trigger(predictor.uid, predictor_version=predictor.version)
-    wait_while_executing(collection=project.predictor_evaluation_executions, execution=execution, print_status_info=True)
+    evaluation = project.predictor_evaluations.trigger(evaluators=[evaluator], predictor_id=predictor.uid)
+    wait_while_executing(collection=project.predictor_evaluations, execution=evaluation, print_status_info=True)
 
 Finally, load the results and inspect the metrics and their computed values:
 
 .. code:: python
 
     # load the results computed by the CV evaluator defined above
-    cv_results = execution[evaluator.name]
+    cv_results = evaluation[evaluator.name]
 
     # load results for y
     y_results = cv_results['y']
@@ -280,18 +262,17 @@ Finally, load the results and inspect the metrics and their computed values:
 
 Archive and restore
 -------------------
-Both :class:`PredictorEvaluationWorkflows <citrine.informatics.workflows.predictor_evaluation_workflow.PredictorEvaluationWorkflow>` and :class:`PredictorEvaluationExecutions <citrine.resources.predictor_evaluation_execution.PredictorEvaluationExecution>` can be archived and restored.
-To archive a workflow:
+:class:`PredictorEvaluation <citrine.informatics.executions.predictor_evaluation.PredictorEvaluation>` can be archived and restored.
 
 .. code:: python
 
-    project.predictor_evaluation_workflows.archive(workflow.uid)
+    project.predictor_evaluation.archive(evaluation.uid)
 
-and to archive all executions associated with a predictor evaluation workflow:
+and to archive all evaluations associated with a predictor:
 
 .. code:: python
 
-    for execution in workflow.executions.list():
-        project.predictor_evaluation_executions.archive(execution.uid)
+    for evaluation in project.predictor_evaluations.list(predictor_id=predictor.uid):
+        project.predictor_evaluation.archive(evaluation.uid)
 
-To restore a workflow or execution, simply replace ``archive`` with ``restore`` in the code above.
+To restore an evaluation, simply replace ``archive`` with ``restore`` in the code above.

docs/source/workflows/predictors.rst

@@ -694,7 +694,7 @@ Predictor reports
 
 A :doc:`predictor report <predictor_reports>` describes a machine-learned model, for example its settings and what features are important to the model.
 It does not include predictor evaluation metrics.
-To learn more about predictor evaluation metrics, please see :doc:`PredictorEvaluationWorkflow <predictor_evaluation_workflows>`.
+To learn more about predictor evaluation metrics, please see :doc:`PredictorEvaluation <predictor_evaluation_workflows>`.
 
 Training data
 -------------

src/citrine/__version__.py

@@ -1 +1 @@
-__version__ = "3.23.1"
+__version__ = "3.24.0"

src/citrine/_rest/engine_resource.py

@@ -42,13 +42,14 @@ def is_archived(self):
         return self.archived_by is not None
 
     def _post_dump(self, data: dict) -> dict:
-        # Only the data portion of an entity is sent to the server.
-        data = data["data"]
-
-        if "instance" in data:
-            # Currently, name and description exists on both the data envelope and the config.
-            data["instance"]["name"] = data["name"]
-            data["instance"]["description"] = data["description"]
+        if data:
+            # Only the data portion of an entity is sent to the server.
+            data = data["data"]
+
+            if "instance" in data:
+                # Currently, name and description exists on both the data envelope and the config.
+                data["instance"]["name"] = data["name"]
+                data["instance"]["description"] = data["description"]
 
         return super()._post_dump(data)
 

src/citrine/informatics/executions/predictor_evaluation.py

@@ -0,0 +1,98 @@
+from functools import lru_cache
+from typing import List, Optional, Union
+from uuid import UUID
+
+from citrine.informatics.predictor_evaluation_result import PredictorEvaluationResult
+from citrine.informatics.predictor_evaluator import PredictorEvaluator
+from citrine.resources.status_detail import StatusDetail
+from citrine._rest.engine_resource import EngineResourceWithoutStatus
+from citrine._rest.resource import PredictorRef
+from citrine._serialization import properties
+from citrine._serialization.serializable import Serializable
+from citrine._utils.functions import format_escaped_url
+
+
+class PredictorEvaluatorsResponse(Serializable['EvaluatorsPayload']):
+    """Container object for a default predictor evaluator response."""
+
+    evaluators = properties.List(properties.Object(PredictorEvaluator), "evaluators")
+
+    def __init__(self, evaluators: List[PredictorEvaluator]):
+        self.evaluators = evaluators
+
+
+class PredictorEvaluationRequest(Serializable['EvaluatorsPayload']):
+    """Container object for a predictor evaluation request."""
+
+    predictor = properties.Object(PredictorRef, "predictor")
+    evaluators = properties.List(properties.Object(PredictorEvaluator), "evaluators")
+
+    def __init__(self,
+                 *,
+                 evaluators: List[PredictorEvaluator],
+                 predictor_id: Union[UUID, str],
+                 predictor_version: Optional[Union[int, str]] = None):
+        self.evaluators = evaluators
+        self.predictor = PredictorRef(predictor_id, predictor_version)
+
+
+class PredictorEvaluation(EngineResourceWithoutStatus['PredictorEvaluation']):
+    """The evaluation of a predictor's performance."""
+
+    uid: UUID = properties.UUID('id', serializable=False)
+    """:UUID: Unique identifier of the evaluation"""
+    evaluators = properties.List(properties.Object(PredictorEvaluator), "data.evaluators",
+                                 serializable=False)
+    """:List{PredictorEvaluator]:the predictor evaluators that were executed. These are used
+    when calling the ``results()`` method."""
+    predictor_id = properties.UUID('metadata.predictor_id', serializable=False)
+    """:UUID:"""
+    predictor_version = properties.Integer('metadata.predictor_version', serializable=False)
+    status = properties.String('metadata.status.major', serializable=False)
+    """:str: short description of the evaluation's status"""
+    status_description = properties.String('metadata.status.minor', serializable=False)
+    """:str: more detailed description of the evaluation's status"""
+    status_detail = properties.List(properties.Object(StatusDetail), 'metadata.status.detail',
+                                    default=[], serializable=False)
+    """:List[StatusDetail]: a list of structured status info, containing the message and level"""
+
+    def _path(self):
+        return format_escaped_url(
+            '/projects/{project_id}/predictor-evaluations/{evaluation_id}',
+            project_id=str(self.project_id),
+            evaluation_id=str(self.uid)
+        )
+
+    @lru_cache()
+    def results(self, evaluator_name: str) -> PredictorEvaluationResult:
+        """
+        Get a specific evaluation result by the name of the evaluator that produced it.
+
+        Parameters
+        ----------
+        evaluator_name: str
+            Name of the evaluator for which to get the results
+
+        Returns
+        -------
+        PredictorEvaluationResult
+            The evaluation result from the evaluator with the given name
+
+        """
+        params = {"evaluator_name": evaluator_name}
+        resource = self._session.get_resource(self._path() + "/results", params=params)
+        return PredictorEvaluationResult.build(resource)
+
+    @property
+    def evaluator_names(self):
+        """Names of the predictor evaluators. Used when calling the ``results()`` method."""
+        return list(iter(self))
+
+    def __getitem__(self, item):
+        if isinstance(item, str):
+            return self.results(item)
+        else:
+            raise TypeError("Results are accessed by string names")
+
+    def __iter__(self):
+        return iter(e.name for e in self.evaluators)

src/citrine/informatics/executions/predictor_evaluation_execution.py

@@ -8,7 +8,7 @@
 
 
 class PredictorEvaluationExecution(Resource['PredictorEvaluationExecution'], Execution):
-    """The execution of a PredictorEvaluationWorkflow.
+    """[DEPRECATED] The execution of a PredictorEvaluationWorkflow.
 
     Possible statuses are INPROGRESS, SUCCEEDED, and FAILED.
     Predictor evaluation executions also have a ``status_description`` field with more information.

src/citrine/informatics/predictor_evaluator.py

@@ -112,7 +112,8 @@ class CrossValidationEvaluator(Serializable["CrossValidationEvaluator"], Predict
     typ = properties.String("type", default="CrossValidationEvaluator", deserializable=False)
 
     def __init__(self,
-                 name: str, *,
+                 name: str,
+                 *,
                  description: str = "",
                  responses: Set[str],
                  n_folds: int = 5,

src/citrine/informatics/workflows/predictor_evaluation_workflow.py

@@ -12,7 +12,7 @@
 
 class PredictorEvaluationWorkflow(Resource['PredictorEvaluationWorkflow'],
                                   Workflow, AIResourceMetadata):
-    """A workflow that evaluations a predictor.
+    """[DEPRECATED] A workflow that evaluates a predictor.
 
     Parameters
     ----------

src/citrine/resources/design_space.py

@@ -129,15 +129,15 @@ def _list_base(self, *, per_page: int = 100, archived: Optional[bool] = None):
                                         per_page=per_page)
 
     def list_all(self, *, per_page: int = 20) -> Iterable[DesignSpace]:
-        """List the most recent version of all design spaces."""
+        """List all design spaces."""
         return self._list_base(per_page=per_page)
 
     def list(self, *, per_page: int = 20) -> Iterable[DesignSpace]:
-        """List the most recent version of all non-archived design spaces."""
+        """List non-archived design spaces."""
         return self._list_base(per_page=per_page, archived=False)
 
     def list_archived(self, *, per_page: int = 20) -> Iterable[DesignSpace]:
-        """List the most recent version of all archived predictors."""
+        """List archived design spaces."""
         return self._list_base(per_page=per_page, archived=True)
 
     def create_default(self,