autointent.context.optimization_info.OptimizationInfo#

class autointent.context.optimization_info.OptimizationInfo#

Tracks optimization results, including trials, artifacts, and modules.

This class provides methods for logging optimization results, retrieving the best-performing modules and artifacts, and generating configuration for inference nodes.

artifacts#
trials#
modules#
pipeline_metrics: dict[str, float]#
log_module_optimization(node_type, module_name, module_params, metric_value, metric_name, artifact, module_dump_dir, module=None)#

Log optimization results for a module.

Parameters:

  • node_type (str) – Type of the node being optimized.

  • module_name (str) – Type of the module.

  • module_params (dict[str, Any]) – Parameters of the module for the trial.

  • metric_value (float) – Metric value achieved by the module.

  • metric_name (str) – Name of the evaluation metric.

  • artifact (autointent.context.optimization_info._data_models.Artifact) – Artifact generated by the module.

  • module_dump_dir (str | None) – Directory where the module is dumped.

  • module (Module | None) – The module instance, if available.

Return type:

None

get_best_embedder()#

Retrieve the name of the best embedder from the retriever node.

Returns:

Name of the best embedder.

Return type:

str

get_best_train_scores()#

Retrieve the train scores from the best scorer node.

Returns:

Train scores as a numpy array.

Return type:

numpy.typing.NDArray[numpy.float64] | None

get_best_validation_scores()#

Retrieve the validation scores from the best scorer node.

Returns:

Validation scores as a numpy array.

Return type:

numpy.typing.NDArray[numpy.float64] | None

get_best_test_scores()#

Retrieve the test scores from the best scorer node.

Returns:

Test scores as a numpy array.

Return type:

numpy.typing.NDArray[numpy.float64] | None

get_best_oos_scores(split)#

Retrieve the out-of-scope scores from the best scorer node.

Parameters:

split (Literal['train', 'validation', 'test']) – The data split for which to retrieve the OOS scores. Must be one of “train”, “validation”, or “test”.

Returns:

A numpy array containing OOS scores for the specified split, or None if no OOS scores are available.

Return type:

numpy.typing.NDArray[numpy.float64] | None

dump_evaluation_results()#

Dump evaluation results for all nodes.

Returns:

Dictionary containing metrics and configurations for all nodes.

Return type:

dict[str, Any]

get_inference_nodes_config(asdict=False)#

Generate configuration for inference nodes based on the best trials.

Returns:

List of InferenceNodeConfig objects for inference nodes.

Parameters:

asdict (bool)

Return type:

list[autointent.configs.InferenceNodeConfig]

get_best_modules()#

Retrieve the best modules for all node types.

Returns:

Dictionary of the best modules for each node type.

Return type:

dict[autointent.custom_types.NodeType, autointent.modules.abc.Module]