autointent.modules.LLMDescriptionScorer#

class autointent.modules.LLMDescriptionScorer(generator_config=None, temperature=1.0, max_concurrent=15, max_per_second=10, max_retries=3, multilabel=False)#

Bases: autointent.modules.scoring._description.base.BaseDescriptionScorer

LLM-based description scorer for zero-shot intent classification using structured output.

This scorer uses a Large Language Model (LLM) with structured output to perform zero-shot intent classification. The LLM is prompted to categorize intent descriptions into three probability levels for each utterance: - Most probable (probability 1.0): Intents that are most likely to match the utterance - Promising (probability 0.5): Intents that are plausible but less confident - Unlikely (probability 0.0): All other intents (implicit)

This approach leverages the reasoning capabilities of LLMs to understand complex relationships between utterances and intent descriptions, potentially achieving high accuracy for nuanced classification tasks. However, it requires API access to LLM services and can be slower and more expensive than encoder-based methods.

Parameters:

  • generator_config (dict[str, Any] | None) – Configuration for the Generator instance (LLM model settings)

  • temperature (pydantic.PositiveFloat) – Temperature parameter for scaling classifier logits (default: 1.0)

  • max_concurrent (pydantic.PositiveInt | None) – Maximum number of concurrent async calls to LLM (default: 15)

  • max_per_second (pydantic.PositiveInt) – Maximum number of API calls per second for rate limiting (default: 10)

  • max_retries (pydantic.PositiveInt) – Maximum number of retry attempts for failed API calls (default: 3)

  • multilabel (bool) – Flag indicating classification task type

Example:#

from autointent.modules.scoring import LLMDescriptionScorer

# Initialize LLM scorer with OpenAI GPT
scorer = LLMDescriptionScorer(
    temperature=1.0,
    max_concurrent=10,
    max_per_second=5,
    max_retries=2
)

# Zero-shot classification with intent descriptions
descriptions = [
    "User wants to book or reserve transportation like flights, trains, or hotels",
    "User wants to cancel an existing booking or reservation",
    "User asks about weather conditions or forecasts"
]

# Fit using descriptions only (zero-shot approach)
scorer.fit([], [], descriptions)

# Make predictions on new utterances
test_utterances = ["Reserve a hotel room", "Delete my booking"]
probabilities = scorer.predict(test_utterances)
name = 'description_llm'#

Name of the module to reference in search space configuration.

generator_config#
max_concurrent = 15#
max_per_second = 10#
max_retries = 3#
classmethod from_context(context, temperature=1.0, generator_config=None, max_concurrent=15, max_per_second=10, max_retries=3)#

Initialize self from context.

Parameters:

  • context (autointent.Context) – Context to init from

  • **kwargs – Additional kwargs

  • temperature (pydantic.PositiveFloat)

  • generator_config (dict[str, Any] | None)

  • max_concurrent (pydantic.PositiveInt | None)

  • max_per_second (pydantic.PositiveInt)

  • max_retries (pydantic.PositiveInt)

Returns:

Initialized module

Return type:

LLMDescriptionScorer

get_implicit_initialization_params()#

Return default params used in __init__ method.

Some parameters of the module may be inferred using context rather from __init__ method. But they need to be logged for reproducibility during loading from disk.

Returns:

Dictionary of default params

Return type:

dict[str, Any]

clear_cache()#

Clear cached data in memory used by the generator.

Return type:

None

dump(path)#

Dump all data needed for inference.

Parameters:

path (str) – Path to dump

Return type:

None

classmethod load(path, embedder_config=None, cross_encoder_config=None)#

Load data from file system.

Parameters:

  • path (str) – Path to load

  • embedder_config (autointent.configs._transformers.EmbedderConfig | None) – one can override presaved settings

  • cross_encoder_config (autointent.configs._transformers.CrossEncoderConfig | None) – one can override presaved settings

Return type:

LLMDescriptionScorer