skit_pipelines.pipelines.evaluate_slu package

Module contents

evaluate_slu(*, repo_name: str, repo_branch: str = 'master', compare_branch: str = '', job_ids: str = '', test_dataset_path: str = '', labelstudio_project_ids: str = '', job_start_date: str = '', job_end_date: str = '', remove_intents: str = '', alias_yaml_path: str = '', notify: str = '', channel: str = '', slack_thread: str = '', core_slu_repo_name: str = 'core-slu-service', core_slu_repo_branch: str = 'master', customization_repo_name: str = 'customization', customization_repo_branch: str = 'master')[source]

A pipeline to evaluate an existing SLU model.

Example payload to invoke via slack integrations:

A minimal example:

@charon run evaluate_slu

{
    "repo_name": "slu_repo_name",
    "labelstudio_project_ids": "10,13",
    "test_dataset_path":"s3://bucket/data.csv"
}

A full available parameters example:

@charon run evaluate_slu

{
    "repo_name": "slu_repo_name",
    "repo_branch": "master",
    "test_dataset_path": "s3://bucket-name/path1/to1/data1.csv,s3://bucket-name/path2/to2/data2.csv",
    "job_ids": "4011,4012",
    "labelstudio_project_ids": "10,13",
    "job_start_date": "2022-08-01",
    "job_end_date": "2022-09-19",
    "remove_intents": "_confirm_,_oos_,audio_speech_unclear,ood",
    "alias_yaml_path": "intents/oppo/alias.yaml"
}
Parameters

  • repo_name (str, optional) – SLU repository name under /vernacularai/ai/clients org in gitlab.

  • repo_branch – The branch name in the SLU repository one wants to use, defaults to master.

  • test_dataset_path – The S3 URI or the S3 key for the tagged dataset (can be multiple - comma separated).

  • job_ids (str) – The job ids as per tog. Optional if labestudio_project_ids is provided.

  • labelstudio_project_ids (str) – The labelstudio project id (this is a number) since this is optional, defaults to “”.

  • job_start_date (str, optional) – The start date range (YYYY-MM-DD) to filter tagged data.

  • job_end_date (str, optional) – The end date range (YYYY-MM-DD) to filter tagged data

  • remove_intents (str, optional) – Comma separated list of intents to remove from dataset while training.

  • alias_yaml_path (str, optional) –

    eevee’s intent_report alias.yaml, refer docs here . Upload your yaml to eevee-yamls repository here & pass the relative path of the yaml from base of the repository.

  • core_slu_repo_name (str, optional) – Name of repository for core slu service. Defaults to core-slu-service

  • core_slu_repo_branch (str, optional) – Branch to check out for core slu repository. Defaults to master

  • customization_repo_name (str, optional) – Name of repository for customization service. Defaults to customization

  • customization_repo_branch (str, optional) – Branch to check out for customization service repository. Defaults to master

  • notify (str, optional) – Whether to send a slack notification, defaults to “”

  • channel (str, optional) – The slack channel to send the notification, defaults to “”

  • slack_thread (str, optional) – The slack thread to send the notification, defaults to “”