ts_benchmark.report.utils package

ts_benchmark.report.utils.leaderboard module

Functions

get_leaderboard(log_data, report_metrics, ...)

Generate a leaderboard from benchmarking records.
get_leaderboard(log_data: pandas.DataFrame, report_metrics: str | List[str], aggregate_type: str, fill_type: str, nan_threshold: float) → pandas.DataFrame[source]

Generate a leaderboard from benchmarking records.

Parameters:

  • log_data – Benchmarking records.

  • report_metrics –

    The (list of) metrics that should be included in the leaderboard, each item can be in either format:

    • exact names: When there exists “;” symbols in name (i.e. parametrized metrics), the name is compared with record_metrics using exact match;

    • stems: When there’s no “;” symbol in the name, the name is considered as a stem name (i.e. metric name without parameters), and is compared with stem names in the record_metrics;

  • aggregate_type – Aggregation method, optional values include “mean”, “median”, “max”.

  • fill_type – Fill method, optional values include “mean_value”.

  • nan_threshold – The metric for any algorithm will be set to NaN if the ratio of NaN values from that algorithm exceeds this threshold.

Returns:

The leaderboard in DataFrame format.