Test Case Generator

class mixedvoices.evaluation.test_case_generator.TestCaseGenerator(prompt: str, user_demographic_info: str | None = None)[source]

Bases: object

Generate test cases for evaluation based on the prompt and user demographic info

Parameters:

  • prompt (str) – The prompt of the agent to generate test for

  • user_demographic_info (Optional[str]) – The user demographic info. Include things like age group, country, accents etc

add_edge_cases(count: int) → TestCaseGenerator[source]

Create test cases for edge cases where bot might fail or behave unexpectedly

Parameters:

count (int) – Number of test cases to add

add_from_descriptions(descriptions: List[str]) → TestCaseGenerator[source]

Add test cases from rough descriptions. 1 test case will be generated for each description

Parameters:

descriptions (List[str]) – List of descriptions

add_from_project(project: Project, cases_per_path: int = 1) → TestCaseGenerator[source]

Add test cases from a project. 1 test case will be generated for each path in the project

Parameters:

  • project (Project) – Project object

  • cases_per_path (int, optional) – Number of test cases to generate for each path. Defaults to 1.

add_from_recordings(recording_paths: List[str], user_channel: Literal['left', 'right'] = 'left') → TestCaseGenerator[source]

Add test cases from recordings. 1 test case will be generated for each recording

Parameters:

  • recording_paths (List[str]) – List of recording paths. Use stereo recordings with user and bot on different channels.

  • user_channel (str, optional) – Channel of the user in the recording. Can be “left” or “right”. Defaults to “left”.

add_from_transcripts(transcripts: List[str]) → TestCaseGenerator[source]

Add test cases from transcripts. 1 test case will be generated for each transcript

Parameters:

transcripts (List[str]) – List of transcripts. Transcript should have labels for each utterance . Use ‘user:’, ‘bot:’ labels”

add_from_version(version: Version, cases_per_path: int = 1) → TestCaseGenerator[source]

Add test cases from a version. 1 test case will be generated for each path in the version

Parameters:

  • version (Version) – Version object

  • cases_per_path (int, optional) – Number of test cases to generate for each path. Defaults to 1.

generate(show_progress=True)[source]

Generate test cases from all the given inputs

property num_cases: int