Basic Config

This module contains basic benchmark configurations.

It defines a simple configurations class (BasicBenchmarkConfig) as well as a set of configurations that covers different dialogs a user might have and some edge-cases (basic_configurations).

get_dict(dimensions)

Return misc dictionary build in dimensions dimensions.

Parameters:

dimensions (Tuple[int, ...]) –

Dimensions of the dictionary. Each element of the dimensions tuple is the number of keys on the corresponding level of the dictionary. The last element of the dimensions tuple is the length of the string values of the dict.

e.g. dimensions=(1, 2) returns a dictionary with 1 key that points to a string of len 2. whereas dimensions=(1, 2, 3) returns a dictionary with 1 key that points to a dictionary with 2 keys each of which points to a string of len 3.

So, the len of dimensions is the depth of the dictionary, while its values are the width of the dictionary at each level.

get_message(message_dimensions)

Return message with a non-empty misc field.

Parameters:

message_dimensions (Tuple[int, ...]) – Dimensions of the misc field of the message. See get_dict().

get_context(dialog_len, message_dimensions, misc_dimensions)

Return context with a non-empty misc, labels, requests, responses fields.

Parameters:

• dialog_len (int) – Number of labels, requests and responses.

• message_dimensions (Tuple[int, ...]) – A parameter used to generate messages for requests and responses. See get_message().

• misc_dimensions (Tuple[int, ...]) – A parameter used to generate misc field. See get_dict().

Return type:

Context

class BasicBenchmarkConfig(**data)

Bases: BenchmarkConfig

A simple benchmark configuration that generates contexts using two parameters:

• message_dimensions – to configure the way messages are generated.

• misc_dimensions – to configure size of context’s misc field.

Dialog length is configured using from_dialog_len, to_dialog_len, step_dialog_len.

context_num: int

Number of times the contexts will be benchmarked. Increasing this number decreases standard error of the mean for benchmarked data.

from_dialog_len: int

Starting dialog len of a context.

to_dialog_len: int

Final dialog len of a context. context_updater() will return contexts until their dialog len is less then to_dialog_len.

step_dialog_len: int

Increment step for dialog len. context_updater() will return contexts increasing dialog len by step_dialog_len.

message_dimensions: Tuple[int, ...]

Dimensions of misc dictionaries inside messages. See get_message().

model_computed_fields: ClassVar[dict[str, ComputedFieldInfo]] = {}

A dictionary of computed field names and their corresponding ComputedFieldInfo objects.

model_config: ClassVar[ConfigDict] = {'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

misc_dimensions: Tuple[int, ...]

Dimensions of misc dictionary. See get_dict().

get_context()

Return context with from_dialog_len, message_dimensions, misc_dimensions.

Wraps get_context().

Return type:

Context

info()

Return fields of this instance and sizes of objects defined by this config.

Returns:

A dictionary with two keys. Key “params” stores fields of this configuration. Key “sizes” stores string representation of following values:

• ”starting_context_size” – size of a context with from_dialog_len.

• ”final_context_size” – size of a context with to_dialog_len. A context of this size will never actually be benchmarked.

• ”misc_size” – size of a misc field of a context.

• ”message_size” – size of a misc field of a message.

context_updater(context)

Update context to have step_dialog_len more labels, requests and responses, unless such dialog len would be equal to to_dialog_len or exceed than it, in which case None is returned.

Return type:

Optional[Context]

basic_configurations

Configuration that covers many dialog cases (as well as some edge-cases).