Audit and Extend a Time-Series ML Pipeline

Time-Series ML Pipeline Audit, Tuning, and Inference Hard · Machine Learning, Time Series, Feature Leakage, Model Validation, MLOps · Voleon · Reconstruct one prediction timestamp and trace every input used by a suspect feature.

Imagine you inherit an ML pipeline that ingests a timestamped CSV and trains a model. Your job is to inspect the implementation for correctness errors and temporal leakage, adjust model settings through an evaluation workflow that avoids contaminating the evaluation window, and add an inference path for unseen data. Before judging any feature, define exactly when each prediction is made, how far ahead the label sits, and which columns are knowable at that moment.

Example 1:

Input: prediction_time = 2024-03-01 00:00, feature = avg_sales_7d, rows_used = [2024-02-23, ..., 2024-03-01 12:00]
Output: Potential leakage in avg_sales_7d

Explanation: The feature uses a timestamp later than the prediction instant, so it consumes information that would not yet be available.

Example 2:

Input: chronological split ending training at 2024-05-01; target_horizon = 3 days; test row timestamp = 2024-05-02; feature = rolling_mean_14d
Output: Potential leakage in rolling_mean_14d for the first test rows

Explanation: The rolling window reaches beyond the test row timestamp and can include early evaluation-period observations, violating training/inference parity.

Constraints & Assumptions:

• The dataset, source code, target column, and a confirmed defect are not supplied; leakage should be treated as a hypothesis to investigate.
• Before validating features, specify the prediction timestamp, target horizon, and the information that is actually available.
• This is a pipeline engineering and modeling discussion, covering file ingestion, transformations, model fitting, and inference, not a single read-only query or deterministic console algorithm.
• If the file contains multiple entities, preserve entity boundaries alongside chronological ordering.

Part 1 — Audit data and features

Explain how you would search for implementation bugs and decide whether any feature uses future information. Your plan should address:

• Schema inspection, timestamp validation, ordering, duplicates, missing values, and entity grouping.
• Leakage risks from target construction, lags, rolling aggregates, as-of joins, and fit/transform boundaries.
• Reproducing a suspected bug with a compact, traceable example that demonstrates whether a feature value was knowable at the prediction timestamp.

Part 2 — Tune and evaluate

Explain how to compare parameter or configuration changes without letting evaluation-period information leak into the modeling process. Address:

• Chronological or walk-forward splits that respect the forecasting horizon.
• Fitting learned preprocessing, including scaling, imputation, and target encoding, only on training time ranges, and selecting hyperparameters only with validation periods.
• Building a naive baseline, choosing metrics aligned with the business objective, and reserving a final evaluation window that remains untouched until the end.

Part 3 — Add inference

Define the saved artifacts and data contract required to replicate training-time feature construction on new inputs. Address:

• Persisting the fitted model, preprocessing state, feature order, schema version, and pipeline version metadata.
• Obtaining sufficient historical context before each prediction timestamp and constructing features exactly as of that timestamp.
• Handling missing columns or missing history, formatting prediction outputs, and performing parity checks between training-time features and inference-time features.

Hint: Reconstruct one prediction timestamp

For a suspect feature, enumerate every row and column used in its calculation. Then decide whether each input existed at the time that prediction would have been made.

What a Strong Answer Covers

• A complete audit, tuning, and inference plan that consistently respects temporal and entity information availability.
• Concrete evidence for any claimed leakage defect, with fixes separated from hyperparameter improvements.
• Reproducible evaluation and a deployable feature/model contract.

Follow-up Questions

• Why can a rolling mean still leak after a chronological train/test split?
• If offline metrics look strong but inference performance deteriorates immediately, what would you inspect?

Overview:

Audit a time-series ML pipeline for leakage, tune it using chronological validation, and add inference with saved preprocessing and timestamp-correct feature computation.