Qlibs Extension Points: Customizing for Your Workflow#

Project Structure and Configuration#

When you first install Qlib, you typically initialize a directory that holds your data, configuration files, and scripts. A common directory structure might look like this:

1
my_qlib_project/
2
? data/
3
?   csv_data/
4
?   qlib_data/
5
? scripts/
6
?   run_train.py
7
?   run_backtest.py
8
?   run_inference.py
9
? config/
10
?   data_handler.yaml
11
?   model_config.yaml
12
? custom_modules/
13
?   data_handler_custom.py
14
?   model_custom.py
15
?   ...
16
? logs/

Inside your scripts, you will initialize Qlib:

1
import qlib
2
from qlib.config import C
3

4
def init_qlib():
5
    qlib.init(
6
        provider_uri="./data/qlib_data",  # points to your local Qlib data
7
        region="cn",  # region of the data
8
        expression_cache=None,
9
        dataset_cache=None,
10
        ... # additional parameters
11
    )

Then in a script like run_train.py, you might do:

1
if __name__ == "__main__":
2
    init_qlib()
3
    # Proceed with data loading, model training, etc.

Selecting and Switching Data Sources#

When you run qlib.init(), you can specify multiple data providers via arguments. For instance, you can configure to use a local CSV data provider or a third-party feed. Switching between providers can be as simple as modifying the provider_uri in your initialization or updating a config file.

For example, if you have a Postgres-based data store, you might do something like:

1
qlib.init(
2
    provider_uri="postgresql://myuser:mypassword@localhost/mydatabase",
3
    region="cn"
4
)

Qlibs design abstracts the data access layer from the rest of the pipeline, so your backtesting scripts wont need rewriting if you swap data providersonly the initialization needs changing.

Creating a Simple Data Handler#

One of the first extension points youll likely use is the DataHandler. A DataHandler fetches raw data from Qlibs storage and transforms it into features. For example, you can create a simple DataHandler that calculates a 7-day moving average of closing prices.

1
import numpy as np
2
from qlib.data.dataset.handler import DataHandlerLP
3

4
class MySimpleDataHandler(DataHandlerLP):
5
    def __init__(self, **kwargs):
6
        super().__init__(**kwargs)
7

8
    def features(self, df):
9
        # df is your DataFrame for a single instrument
10
        df["ma_7"] = df["close"].rolling(7).mean()
11
        return df[["ma_7"]]

Youd register this class in your config or reference it directly in your pipeline:

1
from qlib.data.dataset import DatasetD
2

3
my_handler = MySimpleDataHandler()
4
my_dataset = DatasetD(handler=my_handler)
5
# my_dataset can now be used by the model or other modules

With this, youve already extended Qlib by introducing a new data transformation process. Its minimal, but it shows how easy it is to plug into the data pipeline.

Customizing Data Handlers#

Although the simple example above demonstrates how to transform a single DataFrame, real-world data handlers often need a comprehensive pipeline: retrieving multiple features, merging data from multiple sources, handling outliers, or applying advanced math transformations.

Inside your custom DataHandler, you can override methods like:

• init_load(): For initial data loading configurations
• prepare_data(): To process raw data before feature calculations
• features(): The main transformation logic

A more complex DataHandler might look like this:

1
from qlib.data.dataset import DatasetD
2
from qlib.data.dataset.handler import DataHandlerLP
3

4
class CustomFeatureDataHandler(DataHandlerLP):
5
    def __init__(self, moving_avg_window=20, **kwargs):
6
        super().__init__(**kwargs)
7
        self.moving_avg_window = moving_avg_window
8

9
    def prepare_data(self, df):
10
        # Example: fill missing values, drop duplicates
11
        df = df.fillna(method="ffill").drop_duplicates()
12
        return df
13

14
    def features(self, df):
15
        # Create multiple features
16
        df[f"ma_{self.moving_avg_window}"] = df["close"].rolling(self.moving_avg_window).mean()
17
        df["volume_log"] = np.log1p(df["volume"])
18
        df["price_diff"] = df["close"].diff()
19
        return df

Now you can pass parameters to your data handler (like moving_avg_window=20), giving you a flexible starting point for more nuanced feature sets.

Implementing Your Own Model#

Qlib comes with many built-in model classes (e.g., GBDT, LSTM, AutoML). However, custom modeling is often a high priority. Suppose you want to plug in a new variant of a time-series neural network. By subclassing Qlibs Model interface, you can define how training, prediction, and parameter saving/loading work.

A typical template:

1
import torch
2
import torch.nn as nn
3
from qlib.model.base import Model
4

5
class MyCustomNN(Model):
6
    def __init__(self, input_dim=10, hidden_dim=20, output_dim=1, **kwargs):
7
        super().__init__(**kwargs)
8
        self.net = nn.Sequential(
9
            nn.Linear(input_dim, hidden_dim),
10
            nn.ReLU(),
11
            nn.Linear(hidden_dim, output_dim)
12
        )
13
        self.loss_fn = nn.MSELoss()
14
        self.optimizer = torch.optim.Adam(self.net.parameters())
15

16
    def fit(self, dataset, **kwargs):
17
        # Extract features/labels from dataset
18
        data_x, data_y = dataset.get_data()
19
        self.net.train()
20
        for epoch in range(10):  # simplistic training loop
21
            self.optimizer.zero_grad()
22
            pred = self.net(data_x)
23
            loss = self.loss_fn(pred, data_y)
24
            loss.backward()
25
            self.optimizer.step()
26

27
    def predict(self, dataset, **kwargs):
28
        self.net.eval()
29
        data_x, _ = dataset.get_data(inference=True)
30
        with torch.no_grad():
31
            predictions = self.net(data_x)
32
        return predictions.numpy()

In your training script:

1
from qlib.data.dataset import DatasetD
2
from custom_modules.data_handler_custom import CustomFeatureDataHandler
3
from custom_modules.model_custom import MyCustomNN
4

5
if __name__ == "__main__":
6
    qlib.init(provider_uri="./data/qlib_data")
7

8
    # Prepare dataset
9
    handler = CustomFeatureDataHandler(moving_avg_window=30)
10
    dataset = DatasetD(handler=handler)
11

12
    # Instantiate model
13
    model = MyCustomNN(input_dim=3, hidden_dim=20, output_dim=1)
14

15
    # Fit model
16
    model.fit(dataset)
17

18
    # Prediction
19
    preds = model.predict(dataset)
20
    print(preds)

By following Qlibs structure, you benefit from:

• Unified dataset handling
• A consistent interface for training and prediction
• Easy integration with Qlibs evaluation tools

Extending the Workflow#

Workflow extends beyond data and modeling. You can integrate custom analyses or schedule tasks in production. For instance, you might create a daily pipeline:

1. Fetch new data from a custom feed.
2. Generate fresh features using CustomFeatureDataHandler.
3. Retrain or fine-tune your MyCustomNN.
4. Evaluate the performance with built-in metrics.
5. Store predictions or signals in a database.

This can be done through Qlibs workflow by script?approach, as in the snippet below, or by using a well-known external scheduler (e.g., Airflow) that calls Qlib tasks.

1
if __name__ == "__main__":
2
    # Step 1: Data update
3
    update_data_feed()
4

5
    # Step 2: Feature generation
6
    custom_handler = CustomFeatureDataHandler(moving_avg_window=15)
7
    dataset = DatasetD(handler=custom_handler)
8

9
    # Step 3: Model training
10
    model = MyCustomNN(input_dim=4, hidden_dim=32, output_dim=1)
11
    model.fit(dataset)
12

13
    # Step 4: Evaluation
14
    # Qlib's built-in backtesting or custom evaluation can be invoked here
15

16
    # Step 5: Save or deploy predictions
17
    predictions = model.predict(dataset)
18
    store_predictions(predictions)

In practice, you might refine each step to ensure continuity (e.g., reusing the same model weights, safe checkpointing, or versioned data updates), but the overall concept remains the same.

Creating Custom Features and Operators#

In addition to DataHandler-level transformations, Qlib offers an operator-based approach to feature engineering. You can define various operators (e.g., rolling window calculations, special mathematical transformations) and chain them in an expression.

For instance, Qlibs expression syntax might look like:
"Ref($close, 1) / $close - 1"

To create a custom operator, register it as:

1
import numpy as np
2
from qlib.data.dataset.handler import Operator
3

4
class MyCustomOperator(Operator):
5
    def __init__(self, factor_offset=1):
6
        super().__init__()
7
        self.factor_offset = factor_offset
8

9
    def __call__(self, series):
10
        # Example transformation
11
        return series / (series.shift(self.factor_offset)) - 1

You could integrate it into your DataHandler expressions or use it directly:

1
from qlib.data.dataset import DatasetD
2
from qlib.data.dataset.handler import DataHandlerLP
3

4
class ExtendedHandler(DataHandlerLP):
5
    def __init__(self):
6
        super().__init__()
7

8
    def features(self, df):
9
        custom_operator = MyCustomOperator(factor_offset=2)
10
        df["my_factor"] = custom_operator(df["close"])
11
        return df

By creating and registering custom operators in Qlib, your pipeline can reuse them across multiple DataHandlers, ensuring modularity and consistency.

Plugging Into Qlibs Forecasting Flow#

Forecasting exotic instruments or creating multi-horizon forecasts may require hooking into Qlibs forecasting utilities. For example, you can merge your external risk model with your main model outputs:

1. Obtain model outputs for raw predictions.
2. Incorporate external risk signals from a separate pipeline.
3. *Combine or overlay? final signals.

In pseudo-code:

1
raw_preds = main_model.predict(dataset)
2
risk_signals = risk_model.predict(external_dataset)
3

4
enhanced_signals = (raw_preds * (1 - risk_signals))  # simplistic merging

This combined signal can then be fed to Qlibs strategy or backtesting module using standard calls. You might also integrate these steps into a single predict method within a specialized model class, ensuring a neat, end-to-end pipeline.

Designing Interactive Tools with Qlib#

If you need an interactive environment (e.g., a Jupyter notebook or a web dashboard) to let traders or analysts adjust parameters, you can integrate Qlibs Python API accordingly:

1
# Sample Jupyter notebook snippet
2
from ipywidgets import interact
3

4
def interactive_analysis(mov_avg=7, hidden_dim=16):
5
    qlib.init(provider_uri="./data/qlib_data")
6
    dataset = DatasetD(handler=CustomFeatureDataHandler(moving_avg_window=mov_avg))
7
    model = MyCustomNN(input_dim=2, hidden_dim=hidden_dim, output_dim=1)
8
    model.fit(dataset)
9
    preds = model.predict(dataset)
10
    # plot or display results dynamically
11
    return preds
12

13
interact(interactive_analysis, mov_avg=(5,30), hidden_dim=(8,64))

This approach fosters user experimentation, letting them tune parameters in real-time while leveraging Qlibs back-end capabilities.

High-Performance Data Caching Strategies#

For large-scale quantitative workflows, performance is critical. Qlib includes caching mechanisms at multiple layers (expression cache, dataset cache, etc.). However, you might want to create a custom caching strategy:

• Distributed Cache on a high-speed cluster file system or Redis
• On-Disk Parquet for chunked I/O
• In-Memory for the most frequently accessed data segments

Qlibs dataset caching can be extended by supplying custom caching logic in your DataHandler or by configuring the cache manager. For example:

1
dataset_cache:
2
  class: "my_qlib_project.custom_cache.CustomDatasetCache"
3
  force_update: false

Where CustomDatasetCache might implement a specialized in-memory or distributed cache.

Scalable Parallelism and Distributed Training#

When your dataset grows or your models become more computationally intensive, single-machine training can become a bottleneck. You can integrate Qlib with distributed frameworks such as:

• Horovod for distributed Deep Learning
• Ray or Dask for scaling parallel computations

Within your model class, you could incorporate distributed training logic:

1
from qlib.model.base import Model
2
import horovod.torch as hvd
3

4
class DistributedNN(Model):
5
    def __init__(self, ...):
6
        super().__init__(...)
7
        hvd.init()  # initialize Horovod
8
        # build model, define optimizer, etc.
9

10
    def fit(self, dataset, **kwargs):
11
        # Adjust your training loop to account for distributed setup
12
        hvd.broadcast_parameters(self.net.state_dict(), root_rank=0)
13
        ...

Additionally, you can run multiple Qlib processes for data preparation or backtesting concurrently, offloading CPU/GPU workload across a cluster. As you scale, ensure all nodes share access to the data store (or replicate data appropriately).

Advanced Model Serving and Automation#

Once your model is trained, you might want to serve predictions in real-time or near-real-time to a trading desk or an application. Qlib doesnt force a particular approach, but you can:

1. Save your trained model (e.g., TorchScript for PyTorch) in a standard format.
2. Deploy the service using a web framework like Flask, FastAPI, or specialized serving solutions like TorchServe or TensorFlow Serving.
3. Automate the pipeline to regularly update models with fresh data.

A possible step in your run_inference.py might look like:

1
model = MyCustomNN.load_checkpoint("./checkpoints/latest_model.ckpt")
2
new_data = fetch_latest_data()
3
dataset = DatasetD(handler=CustomFeatureDataHandler())
4
predictions = model.predict(dataset)
5

6
# Serve predictions
7
serve_predictions_via_rest_api(predictions)

You can combine this with containerization (e.g., Docker) and orchestration (Kubernetes) for production-grade setups. Qlib just needs to be installed within that environment, and data references need to be properly configured.