MetaSpore Introduction

MetaSpore is a one-stop end-to-end machine learning development platform that provides a full-cycle framework and development interface for from data preprocessing, model training, offline experiments, online predictions to online experiment traffic bucketization and ab-testing.

MetaSpore is developed and opensourced by DMetaSoul team. You could also join our slack user discussion space.

Core Features

MetaSpore has the following features:

1. One-stop end-to-end development, from offline model training to online prediction and bucketing experiments, with a unified development experience across the entire process;
2. Deep learning training framework, compatible with PyTorch ecology, supports distributed large-scale sparse feature learning;
3. The training framework is connected with PySpark to seamlessly read the training data from the data lake and data warehouse;
4. High-performance online prediction service, supports fast inference for neural network, decision tree, Spark ML, SKLearn and other models; supports heterogeneous hardware inference acceleration;
5. In the offline unified feature extraction framework, the online feature reading logic is automatically generated, and the feature extraction logic is unified cross offline and online;
6. Online algorithm application framework, providing model prediction, experiment bucketing and traffic splitting, dynamic hot loading of parameters and rich debug functions;
7. Rich industry algorithm examples and end-to-end solutions.

Examples

• Offline Training Getting Started Tutorial

• Online Algorithm Application (Java implementation)

  • Online Model Serving
  • Online Feature Extraction
  • Online Experiment Pipeline

• A MovieLens end-to-end recommender system demo, including

  • Offline models
  • Online algorithm application (Java implementation)

Installation package download

Training package

We provide precompiled offline training wheel package on pypi, install it via pip:

pip install metaspore

The minimum Python version required is 3.8.

After installation, also install pytorch and pyspark (they are not included as depenencies of metaspore wheel so you could choose pyspark and pytorch versions as needed):

pip install pyspark
pip install torch==1.11.0+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html

Serving package

We provide prebuilt docker images for MetaSpore Serving Service:

CPU only image

docker pull dmetasoul/metaspore-serving-release:cpu-v1.0.1

GPU image

docker pull dmetasoul/metaspore-serving-release:gpu-v1.0.1

See Run MetaSpore Serving for details.

Development

• Offline training framework compilation