autointent.advisor.dataset_stats

autointent.advisor.dataset_stats(dataset)

Summarize an in-memory Dataset for the advisor.

Reads the train split (train, or train_0 once the dataset has been split) to count samples and measure utterance length — average and 95th percentile word counts, over at most the first 1000 rows — and takes n_classes, multilabel and has_descriptions from the dataset itself. Returns a placeholder when no train split is present.

This is how a caller gets from a Dataset to the DatasetStats that run_preflight() and reduce_to_fit() require. Use DatasetStats.placeholder() instead when no dataset exists yet and you only want to size a search space against hypothetical numbers.

Parameters:

dataset (autointent.Dataset) – the dataset the pipeline would be fitted on.

Returns:

Stats describing that dataset, with source="dataset:in-memory".

Return type:

autointent.advisor._report.DatasetStats