ts_benchmark.evaluation.strategy package

ts_benchmark.evaluation.strategy.constants module

Classes

FieldNames()

class FieldNames

Bases: object

ACTUAL_DATA = 'actual_data'

FILE_NAME = 'file_name'

FIT_TIME = 'fit_time'

INFERENCE_DATA = 'inference_data'

INFERENCE_TIME = 'inference_time'

LOG_INFO = 'log_info'

MODEL_NAME = 'model_name'

MODEL_PARAMS = 'model_params'

STRATEGY_ARGS = 'strategy_args'

classmethod all_fields() → List[str]

ts_benchmark.evaluation.strategy.fixed_forecast module

Classes

FixedForecast(strategy_config, evaluator)

Fixed forecast strategy class

class FixedForecast(strategy_config: Dict, evaluator: Evaluator)

Bases: ForecastingStrategy

Fixed forecast strategy class

This strategy defines a forecasting task with fixed prediction length.

The required strategy configs include:

• horizon (int): The length to predict, i.e. the length of the test series;

• train_ratio_in_tv (float): The ratio of the training series when performing train-validation split.

The accepted metrics include all regression metrics.

The return fields other than the specified metrics are (in order):

• FieldNames.FILE_NAME: The name of the series;

• FieldNames.FIT_TIME: The training time;

• FieldNames.INFERENCE_TIME: The inference time;

• FieldNames.ACTUAL_DATA: The true test data, encoded as a string.

• FieldNames.INFERENCE_DATA: The predicted data, encoded as a string.

• FieldNames.LOG_INFO: Any log returned by the evaluator.

REQUIRED_CONFIGS = ['horizon', 'train_ratio_in_tv']

static accepted_metrics()

Gets the accepted metrics by this strategy

property field_names: List[str]

Gets the field names of the result records

ts_benchmark.evaluation.strategy.forecasting module

Classes

ForecastingStrategy(strategy_config, evaluator)

The base class for forecasting strategies

class ForecastingStrategy(strategy_config: Dict, evaluator: Evaluator)

Bases: Strategy

The base class for forecasting strategies

REQUIRED_CONFIGS = ['seed']

execute(series_name: str, model_factory: ModelFactory) → Any

The primary interface to execute a forecasting strategy

In this method:

• Random seeds are set;

• Target series and corresponding meta-info are prepared;

• Exceptions are handled;

Parameters:

• series_name – The name of a series data to evaluate.

• model_factory – A model factory that creates a new model with each invocation.

Returns:

The results generated by evaluating a model on a series.

ts_benchmark.evaluation.strategy.rolling_forecast module

Classes

RollingForecast(strategy_config, evaluator)	Rolling forecast strategy class
RollingForecastEvalBatchMaker(series, index_list)
RollingForecastPredictBatchMaker(batch_maker)

class RollingForecast(strategy_config: Dict, evaluator: Evaluator)

Bases: ForecastingStrategy

Rolling forecast strategy class

This strategy defines a forecasting task that fits once on the training set and forecasts on the testing set in a rolling window style.

The required strategy configs include:

• horizon (int): The length of each prediction;

• tv_ratio (float): The ratio of the train-validation series when performing train-test split;

• train_ratio_in_tv (float): The ratio of the training series when performing train-validation split;

• stride (int): Rolling stride, i.e. the interval between two windows;

• num_rollings (int): The maximum number of steps to forecast;

The accepted metrics include all regression metrics.

The return fields other than the specified metrics are (in order):

• FieldNames.FILE_NAME: The name of the series;

• FieldNames.FIT_TIME: The training time;

• FieldNames.INFERENCE_TIME: The inference time;

• FieldNames.ACTUAL_DATA: The true test data, encoded as a string.

• FieldNames.INFERENCE_DATA: The predicted data, encoded as a string.

• FieldNames.LOG_INFO: Any log returned by the evaluator.

REQUIRED_CONFIGS = ['horizon', 'tv_ratio', 'train_ratio_in_tv', 'stride', 'num_rollings']

static accepted_metrics() → List[str]

Gets the accepted metrics by this strategy

property field_names: List[str]

Gets the field names of the result records

class RollingForecastEvalBatchMaker(series: pandas.DataFrame, index_list: List[int])

Bases: object

has_more_batches() → bool

Check if there are more batches to process.

Returns:

True if there are more batches, False otherwise.

make_batch_eval(horizon: int) → dict

Return all data to be used for batch evaluation.

Parameters:

horizon – The size of horizon.

Returns:

All data to be used for batch evaluation.

make_batch_predict(batch_size: int, win_size: int) → dict

Return a batch of data with index and column to be used for batch prediction.

Parameters:

• batch_size – The size of batch.

• win_size – The length of data used for prediction.

Returns:

a batch of data and its time stamps.

class RollingForecastPredictBatchMaker(batch_maker: RollingForecastEvalBatchMaker)

Bases: BatchMaker

has_more_batches() → bool

Check if there are more batches to process.

Returns:

True if there are more batches, False otherwise.

make_batch(batch_size: int, win_size: int) → dict

Return a batch of data to be used for batch prediction.

Parameters:

• batch_size – The size of batch.

• win_size – The length of data used for prediction.

Returns:

A batch of data.

ts_benchmark.evaluation.strategy.strategy module

Classes

ResultCollector()	Result collector
Strategy(strategy_config, evaluator)	The base class of strategies

class ResultCollector

Bases: object

Result collector

Result collectors helps to gather outputs returned by strategy classes, it is helpful define a custom result collector when the strategy class has irregular returns (e.g. returning multiple records in one evaluation).

add(result: Any) → NoReturn

Adds the output of a strategy to the collection

Parameters:

result – The return value of a strategies’ execute() method.

collect() → List

Returns the current result collection

get_size() → int

Gets the number of collected results

reset() → NoReturn

Resets the current result collection

class Strategy(strategy_config: Dict, evaluator: Evaluator)

Bases: object

The base class of strategies

A strategy defines the evaluation pipeline of the specific time-series analysis task.

Warning

Strategies are currently using pickle to store Python objects in the evaluation results, which is known to be unsafe during decoding. Although reading the evaluation log itself is safe, please DO NOT decode any pickled columns in the log file if the data source is untrusted.

DEFAULT_CONFIG_KEY = '__default__'

REQUIRED_CONFIGS = ['strategy_name']

abstract static accepted_metrics() → List[str]

Gets the accepted metrics by this strategy

abstract execute(series_name: str, model_factory: ModelFactory) → Any

The primary interface to execute a strategy

Parameters:

• series_name – The name of a series data to evaluate.

• model_factory – A model factory that creates a new model with each invocation.

Returns:

The results generated by evaluating a model on a series.

abstract property field_names: List[str]

Gets the field names of the result records

get_collector() → ResultCollector

Creates a new compatible result collector

get_config_str(required_configs_only: bool = False) → str

Gets the string representation of the strategy config

Parameters:

required_configs_only – If True, includes only the keys specified by REQUIRED_CONFIGS in the string, otherwise, encode the strategy config as is.

Returns:

A string representation of the strategy config.

get_default_result(**kwargs) → List

Gets the default result when the strategy fails to execute

Parameters:

kwargs – Each key-value pair updates the key field with value value of the return value.

classmethod get_required_configs() → List[str]

Gets the required configs from the current class and all super classes