From Past to Present: Building Accurate Time Series Forecasts#

What Is a Time Series?#

A time series is a sequence of data points collected at regular intervals over time. For instance, the daily closing prices of a stock, hourly temperature readings, or monthly sales data for a retail storefront are all examples of time series. The objective of time series forecasting is to capture the trends, cyclicity, and occasional randomness in the data to predict future values.

Why Is Time Series Forecasting Important?#

1. Predictive Planning: Businesses rely on sales forecasts to plan inventory, staffing, and marketing campaigns.
2. Financial Modeling: Investors and analysts forecast currency exchange rates, stock prices, and economic indicators.
3. Resource Allocation: Utility companies forecast energy consumption, while governments model transport infrastructure needs.
4. Risk Management: In finance, better predictive models lead to improved risk-adjusted returns.

Types of Time Series Data#

• Univariate Time Series: A series with a single variable or measurement (e.g., daily closing price).
• Multivariate Time Series: A series where multiple variables or features are collected simultaneously (e.g., daily stock closing price plus associated trading volume, and/or economic indicators).

Working with time series data involves not just reading in data but also understanding trends, analyzing structure, and preparing it for modeling through adequate transformations and splitting into training and testing sets chronologically.

Trend#

Trend refers to the long-term increase or decrease in the data. For instance, a companys sales typically show upward trends over years if the business is growing. Sometimes a trend can be negative if the market is declining.

Seasonality#

Seasonality is a pattern that repeats at regular intervals. Common seasonal periodicities include:

• Daily (for data collected by the hour)
• Weekly (for data with notable weekday/weekend patterns)
• Monthly (e.g., retail or e-commerce often sees spikes in December)
• Quarterly or Annual

Cyclical Patterns#

Cyclical behavior refers to fluctuations that dont have a fixed frequency but often occur due to macroeconomic conditions or longer business cycles.

Irregular/Noise#

The random, unpredictable part of a time series is the irregular or noise?component. Its the random fluctuation that modeling aims to account for or smooth out.

A conceptual table:

Understanding and appropriately modeling these components is crucial for distinct forecasting methods.

Naive Forecast#

A naive forecast simply takes the last observed value and uses it as an estimate for future values. This is often used as a baseline measure.

• Naive: y_{t+1} = y_t

Moving Average#

The moving average forecast calculates the average of the most recent observations. For instance, a 3-period moving average forecast would be:

• Moving Average:

  1
  y_{t+1} = (y_t + y_{t-1} + y_{t-2}) / 3
  

Simple Exponential Smoothing#

Simple Exponential Smoothing (SES) applies a weighted average of past observations, giving more weight to the most recent observations:

• SES:

  1
  S_t =  y_t + (1 - ) S_{t-1}
  

  where is the smoothing parameter (0 < < 1).

Strengths and Weaknesses#

• These methods are easy to implement and interpret.
• They often work well for short-term forecasts if the time series is relatively stable.
• They may not capture complex trends or seasonality effectively, which limits their accuracy in more nuanced series.

Train-Test Split in Time Series#

Unlike classic machine learning, time series forecasting requires that training data precede test data in time. Random splitting can violate the chronological order and lead to unrealistic performance estimates.

Common Error Metrics#

1. Mean Absolute Error (MAE):

   1
   MAE = (1/n) *  |y_t - _t|
   

   Emphasizes absolute differences between predicted and actual values.

2. Mean Squared Error (MSE):

   1
   MSE = (1/n) *  (y_t - _t)^2
   

   Penalizes larger errors more because of the squaring.

3. Mean Absolute Percentage Error (MAPE):

   1
   MAPE = (100% / n) *  |(y_t - _t) / y_t|
   

   Measures the error as a percentage, which can be useful for comparing performance across datasets at different scales.

4. Root Mean Squared Error (RMSE):

   1
   RMSE = ?MSE)
   

   Similar to MSE but returns the error in the same units as the forecasted variable.

Its best practice to use multiple metrics. Depending on the application, penalizing larger errors might matter more (leading you to prefer MSE/RMSE), or the magnitude of errors in percentage terms might be more important (leading you toward MAPE).

Autoregressive (AR) Model#

An Autoregressive (AR) model uses a linear combination of past values to forecast the future. An AR(p) model is:

1
y_t = c + _1 y_{t-1} + _2 y_{t-2} + ... + _p y_{t-p} + _t

where:

• p denotes the number of autoregressive lags,
• values are model coefficients,
• _t is white noise.

Moving Average (MA) Model#

A Moving Average (MA) model uses past forecast errors to predict future values. An MA(q) model is:

1
y_t = c + _1 _{t-1} + _2 _{t-2} + ... + _q _{t-q} + _t

ARMA and ARIMA#

• ARMA(p, q) models combine AR(p) and MA(q) components.
• ARIMA(p, d, q) extends this by introducing d?differences to make the time series stationary. The differencing step is crucial for series with a trend or non-stationary behavior.

SARIMA and Seasonal Patterns#

For time series with strong seasonality, Seasonal ARIMA (SARIMA) extends ARIMA by adding seasonal terms: (P, D, Q)s. This helps capture repeating seasonal structures.

In practice, classical models like ARIMA and SARIMA are robust go-to techniques, often forming a strong benchmark or even final solution in many business forecasting contexts.

Why Machine Learning?#

Statistical models like ARIMA can be powerful but may struggle when we have:

• Additional exogenous variables or multiple external predictors.
• Non-linear relationships not well-captured by linear modeling.
• Complex patterns that vary over time or do not follow the typical ARIMA structure.

Popular Machine Learning Algorithms#

1. Random Forest: Leverages ensembles of decision trees to capture non-linearities and interactions.
2. Gradient Boosted Trees: Models complex patterns by iteratively improving upon weak learners.
3. Support Vector Regression (SVR): Effective for smaller datasets, though not always easy to tune.

Feature Engineering for ML#

When using ML, you must transform the time series into a tabular structure with features, such as:

• Lags of the target variable (e.g., y_{t-1}, y_{t-2}, ?.
• Rolling averages or rolling standard deviations to capture trends/volatility.
• Time-based features like day of the week, month, or holiday indicators.
• External features (e.g., marketing spend, temperature, economic indicators) if relevant.

Sliding Window Approach#

Instead of directly predicting y_{t+1} from y_t, you create input-output pairs:

• Input: [y_{t-1}, y_{t-2}, ? y_{t-n}]
• Output: y_t

Then you use a regression model to learn the mapping from these inputs to the output. This approach is flexible but can be data-hungry.

Why Use Neural Networks for Time Series?#

Neural networks automatically learn non-linear representations from the data. They can:

• Capture complex, non-linear relationships.
• Integrate multiple features and exogenous variables easily.
• Handle large-scale data sets and are suitable for sequence modeling.

Feedforward Networks#

A simple approach is to use a feedforward neural network with lag-based features. For example, you might feed the values [y_{t-1}, y_{t-2}, ? y_{t-n}] into a multi-layer perceptron (MLP) to predict y_t. However, MLPs do not inherently model sequential dependencies without additional structural components.

Recurrent Neural Networks (RNN)#

Recurrent Neural Networks (RNNs):

• Have hidden states that update over time steps.
• Are specifically designed to capture sequence information.
• Vanilla RNNs can suffer from vanishing or exploding gradients when dealing with longer sequences.

LSTM and GRU#

These are specialized RNN architectures designed to handle long-term dependencies:

• LSTM (Long Short-Term Memory): Uses gates (input, forget, output) to add or remove information over time steps.
• GRU (Gated Recurrent Unit): A simpler variant of LSTM with fewer parameters.

Common steps for an LSTM-based time series model:

1. Prepare the dataset with a sliding window.
2. Scale/normalize the data.
3. Build a model using one or more LSTM layers.
4. Train for multiple epochs, validating on a separate portion of your data.
5. Evaluate on the test set.

Convolutional Neural Networks (CNN) for Time Series#

Though typically used for image and audio processing, 1D CNNs can also model sequence-based data. CNNs can identify local patterns within the series (e.g., repeated shapes over time), and they can be combined with RNN or LSTM layers for hybrid architectures.

Incorporating Exogenous Variables#

Many real-world forecasting problems benefit from additional inputs, known as exogenous variables or covariates. For instance, a model predicting electricity load might include weather patterns, or a sales forecast might include promotions or marketing spend. Tools like ARIMAX (ARIMA with exogenous variables) can be used in classical modeling, while machine learning and neural network approaches incorporate them as additional features.

Hybrid Models#

Hybrid models combine classical and ML or deep learning approaches to get the best of both worlds. One strategy could be:

1. Decompose the series into trend/seasonality using classical methods like STL decomposition.
2. Model the residual with a machine learning model to capture nonlinear effects.
3. Combine the classical estimate + ML residual prediction for a final forecast.

By blending models, you can capture fundamental seasonality with well-proven statistical techniques while letting modern, more flexible models handle complex relationships and interactions.

Pitfall 1: Ignoring Seasonality or Trend#

Failing to account for seasonality will often lead to poor forecasts if the series clearly exhibits such patterns.

Pitfall 2: Overfitting#

Complex machine learning models can overfit the training set, especially with limited data. A robust pipeline includes:

• Proper train-test splits by time.
• Potential cross-validation strategies like TimeSeriesSplit.
• Early stopping or regularization in machine learning and neural networks.

Pitfall 3: Improper Scaling#

When using methods like neural networks or gradient-boosting, data often needs to be scaled (e.g., min-max or standardization). Misaligned scaling between training and test sets can sabotage forecasts.

Pitfall 4: Data Leakage#

Inadvertently using future information in the training process. Always ensure that features come only from historical data at the time of prediction.

Pitfall 5: Over-simplification of Errors#

Often, the forecasting environment includes unpredictable external events (e.g., pandemics, system outages, market shifts). Over-simplifying error metrics or ignoring out-of-distribution occurrences can minimize the realism of your predictions. Always re-evaluate forecasts under changing conditions.

Example Dataset#

Lets assume a CSV file named monthly_sales.csv with two columns: date (YYYY-MM) and sales.

1
import pandas as pd
2
import matplotlib.pyplot as plt
3

4
# Load dataset
5
df = pd.read_csv("monthly_sales.csv", parse_dates=["date"])
6
df.set_index("date", inplace=True)
7

8
# Check the first few rows
9
print(df.head())
10

11
# Plot the data
12
plt.plot(df.index, df["sales"], label="Monthly Sales")
13
plt.title("Monthly Sales over Time")
14
plt.xlabel("Date")
15
plt.ylabel("Sales")
16
plt.legend()
17
plt.show()

Classical ARIMA#

1
from pmdarima import auto_arima
2
import numpy as np
3

4
# Split data into training (80%) and test (20%)
5
train_size = int(len(df) * 0.8)
6
train_data = df.iloc[:train_size]
7
test_data = df.iloc[train_size:]
8

9
# Use auto_arima to find optimal (p, d, q)
10
arima_model = auto_arima(train_data["sales"], seasonal=True, m=12,  # monthly seasonality
11
                         trace=True, error_action='ignore', suppress_warnings=True)
12
arima_model.fit(train_data["sales"])
13

14
# Forecast the length of the test data
15
n_periods = len(test_data)
16
forecast_arima = arima_model.predict(n_periods=n_periods)
17

18
# Compare with actual
19
test_index = test_data.index
20
forecast_df_arima = pd.DataFrame(forecast_arima, index=test_index, columns=["forecast"])
21

22
# Plot
23
plt.figure(figsize=(10,6))
24
plt.plot(train_data.index, train_data["sales"], label="Train")
25
plt.plot(test_data.index, test_data["sales"], label="Test")
26
plt.plot(forecast_df_arima.index, forecast_df_arima["forecast"], label="ARIMA Forecast")
27
plt.legend()
28
plt.show()
29

30
# Evaluate
31
from sklearn.metrics import mean_squared_error
32
mse_arima = mean_squared_error(test_data["sales"], forecast_df_arima["forecast"])
33
rmse_arima = np.sqrt(mse_arima)
34
print("ARIMA RMSE:", rmse_arima)

Simple LSTM#

1
import tensorflow as tf
2
from sklearn.preprocessing import MinMaxScaler
3
import numpy as np
4

5
# Prepare data for LSTM
6
# Using one-step prediction, we'll convert the time series to (X, y) pairs
7
data = df["sales"].values.reshape(-1, 1)
8
scaler = MinMaxScaler(feature_range=(0, 1))
9
scaled_data = scaler.fit_transform(data)
10

11
window_size = 12  # using the last 12 months to predict the next
12
X = []
13
y = []
14
for i in range(window_size, len(scaled_data)):
15
    X.append(scaled_data[i-window_size:i, 0])
16
    y.append(scaled_data[i, 0])
17

18
X = np.array(X)
19
y = np.array(y)
20

21
train_size = int(len(X) * 0.8)
22
X_train, X_test = X[:train_size], X[train_size:]
23
y_train, y_test = y[:train_size], y[train_size:]
24

25
# Reshape X for LSTM: [samples, timesteps, features]
26
X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], 1)
27
X_test = X_test.reshape(X_test.shape[0], X_test.shape[1], 1)
28

29
model = tf.keras.Sequential([
30
    tf.keras.layers.LSTM(64, return_sequences=False, input_shape=(X_train.shape[1], 1)),
31
    tf.keras.layers.Dense(1)
32
])
33

34
model.compile(optimizer='adam', loss='mse')
35
hist = model.fit(X_train, y_train, epochs=50, batch_size=16,
36
                 validation_split=0.1, verbose=0)
37

38
# Forecast on the test set
39
predictions = model.predict(X_test)
40
predictions_rescaled = scaler.inverse_transform(predictions)
41

42
y_test_rescaled = scaler.inverse_transform(y_test.reshape(-1, 1))
43

44
plt.figure(figsize=(10,6))
45
plt.plot(range(len(y_test)), y_test_rescaled, label="Actual")
46
plt.plot(range(len(y_test)), predictions_rescaled, label="Predicted")
47
plt.legend()
48
plt.show()
49

50
mse_lstm = mean_squared_error(y_test_rescaled, predictions_rescaled)
51
rmse_lstm = np.sqrt(mse_lstm)
52
print("LSTM RMSE:", rmse_lstm)

In professional practice, youd want to further refine hyperparameters, possibly add multiple layers, or incorporate exogenous variables. But the fundamental approach remains consistent.

Model Stability and Monitoring#

Forecasting models can drift in performance over time if new behaviors or structural changes arise. Introduce a model monitoring system:

1. Periodically retrain or update the model with the latest data.
2. Track both training and test performance metrics over time.
3. Implement alerts if the forecast error surpasses acceptable thresholds.

Cross-Validation Approaches for Time Series#

• Rolling Origin or Expanding Window approach: Re-train as you move your time window forward, ensuring that training always precedes test data.
• Blocked Time Series Split: Typically used for multiple validation folds, blocking out historical segments.

Hyperparameter Tuning#

• For ARIMA-like models, rely on techniques like grid search or automated selection (e.g., auto_arima).
• For ML or neural networks, use Bayesian optimization or grid search with specialized time series cross-validation to avoid data leakage.

Transfer Learning in Time Series#

When you have multiple related series, you can train a single deep learning model that learns generic patterns of seasonality or trends across them and then fine-tune to a specific series. This can be especially advantageous if each individual series has limited historical data.

Hierarchical and Grouped Time Series#

In large organizations, you might forecast at multiple hierarchical levels (e.g., total sales across a country, region-level sales, store-level sales). Reconciling these forecasts so that the sum of store-level forecasts matches the region-level forecast is part of hierarchical forecasting.

Probabilistic Forecasting#

Instead of a point prediction, consider forecasting a distribution or confidence interval around your predictions. Methods for probabilistic forecasting include:

• Bayesian approaches in ARIMA or structural time series models.
• Quantile regression in machine learning.
• Deep learning approaches that output probability distributions (e.g., using likelihood-based loss functions).

Automated Forecasting Tools#

Platforms such as Facebook (Meta) Prophet or Amazon Forecast aim to reduce the complexity of model tuning. These can be a good starting point for time series novices. However, for large-scale or highly sophisticated needs, custom solutions and deeper knowledge are invaluable.