notice

This is unreleased documentation for Rasa Documentation Main/Unreleased version.
For the latest released documentation, see the latest version (3.x).

Version: Main/Unreleased

rasa.model_testing

ClassificationReportException Objects

class ClassificationReportException(Exception)

Raised when clf_report doesn't correctly set accuracy and/or micro avg.

sklearn.metrics.classification_report should provide either accuracy or micro avg.

test_core_models_in_directory

async def test_core_models_in_directory(
model_directory: Text,
stories: Text,
output: Text,
use_conversation_test_files: bool = False) -> None

Evaluates a directory with multiple Core models using test data.

Arguments:

  • model_directory - Directory containing multiple model files.
  • stories - Path to a conversation test file.
  • output - Output directory to store results to.
  • use_conversation_test_files - True if conversation test files should be used for testing instead of regular Core story files.

plot_core_results

def plot_core_results(output_directory: Text,
number_of_examples: List[int]) -> None

Plot core model comparison graph.

Arguments:

  • output_directory - path to the output directory
  • number_of_examples - number of examples per run

test_core_models

async def test_core_models(models: List[Text],
stories: Text,
output: Text,
use_conversation_test_files: bool = False) -> None

Compares multiple Core models based on test data.

Arguments:

  • models - A list of models files.
  • stories - Path to test data.
  • output - Path to output directory for test results.
  • use_conversation_test_files - True if conversation test files should be used for testing instead of regular Core story files.

test_core

async def test_core(model: Optional[Text] = None,
stories: Optional[Text] = None,
output: Text = DEFAULT_RESULTS_PATH,
additional_arguments: Optional[Dict] = None,
use_conversation_test_files: bool = False) -> None

Tests a trained Core model against a set of test stories.

test_nlu

async def test_nlu(model: Optional[Text],
nlu_data: Optional[Text],
output_directory: Text = DEFAULT_RESULTS_PATH,
additional_arguments: Optional[Dict] = None,
domain_path: Optional[Text] = None) -> None

Tests the NLU Model.

compare_nlu_models

async def compare_nlu_models(configs: List[Text], test_data: TrainingData,
output: Text, runs: int,
exclusion_percentages: List[int]) -> None

Trains multiple models, compares them and saves the results.

plot_nlu_results

def plot_nlu_results(output_directory: Text,
number_of_examples: List[int]) -> None

Plot NLU model comparison graph.

Arguments:

  • output_directory - path to the output directory
  • number_of_examples - number of examples per run

perform_nlu_cross_validation

async def perform_nlu_cross_validation(
config: Dict[Text, Any], data: TrainingData, output: Text,
additional_arguments: Optional[Dict[Text, Any]]) -> None

Runs cross-validation on test data.

Arguments:

  • config - The model configuration.
  • data - The data which is used for the cross-validation.
  • output - Output directory for the cross-validation results.
  • additional_arguments - Additional arguments which are passed to the cross-validation, like number of disable_plotting.

get_evaluation_metrics

def get_evaluation_metrics(
targets: Iterable[Any],
predictions: Iterable[Any],
output_dict: bool = False,
exclude_label: Optional[Text] = None
) -> Tuple[Union[Text, Dict[Text, Dict[Text, float]]], float, float, float]

Compute the f1, precision, accuracy and summary report from sklearn.

Arguments:

  • targets - target labels
  • predictions - predicted labels
  • output_dict - if True sklearn returns a summary report as dict, if False the report is in string format
  • exclude_label - labels to exclude from evaluation

Returns:

Report from sklearn, precision, f1, and accuracy values.

make_classification_report_complete

def make_classification_report_complete(report: dict, accuracy: float) -> dict

Completes the sklearn classification report with accuracy xor micro avg.

Arguments:

  • report - Report generated by metrics.classification_report with output_dict=True
  • accuracy - Model accuracy

Raises:

  • Exception - When sklearn.metrics.classification_report behaves different to our expectation.

Returns:

  • report - Report generated by metrics.classification_report enhanced with accuracy xor micro avg.

clean_labels

def clean_labels(labels: Iterable[Text]) -> List[Text]

Remove None labels. sklearn metrics do not support them.

Arguments:

  • labels - list of labels

Returns:

Cleaned labels.

get_unique_labels

def get_unique_labels(targets: Iterable[Text],
exclude_label: Optional[Text]) -> List[Text]

Get unique labels. Exclude 'exclude_label' if specified.

Arguments:

  • targets - labels
  • exclude_label - label to exclude

Returns:

Unique labels.