autointent.modules.base.BaseScorer#

class autointent.modules.base.BaseScorer#

Bases: autointent.modules.base.BaseModule, abc.ABC

Abstract base class for scoring modules.

Scoring modules predict scores for utterances and evaluate their performance using a scoring metric.

supports_oos = False#

Whether the module supports oos data

abstract fit(utterances, labels)#

Fit the scoring module to the training data.

Parameters:

  • utterances (list[str]) – List of training utterances.

  • labels (autointent.custom_types.ListOfLabels) – List of training labels.

Return type:

None

score_ho(context, metrics)#

Evaluate the scorer on a test set and compute the specified metric.

Parameters:

  • context (autointent.Context) – Context containing test set and other data.

  • metrics (list[str]) – List of metrics to compute.

Returns:

Computed metrics value for the test set or error code of metrics.

Return type:

dict[str, float]

score_cv(context, metrics)#

Evaluate the scorer on a test set and compute the specified metric.

Parameters:

  • context (autointent.Context) – Context containing test set and other data.

  • metrics (list[str]) – List of metrics to compute.

Returns:

Computed metrics value for the test set or error code of metrics.

Return type:

dict[str, float]

get_assets()#

Retrieve assets generated during scoring.

Returns:

ScorerArtifact containing test, validation and test scores.

Return type:

autointent.context.optimization_info.ScorerArtifact

get_train_data(context)#

Get train data.

Parameters:

context (autointent.Context) – Context to get train data from

Returns:

Tuple of train utterances and train labels

Return type:

tuple[list[str], autointent.custom_types.ListOfLabels]

abstract predict(utterances)#

Predict scores for a list of utterances.

Parameters:

utterances (list[str]) – List of utterances to score.

Returns:

Array of predicted scores.

Return type:

numpy.typing.NDArray[Any]