MetaSpore Introduction
MetaSpore is a one-stop end-to-end machine learning development platform that provides a full-cycle framework and development interface for from data preprocessing, model training, offline experiments, online predictions to online experiment traffic bucketization and ab-testing.
MetaSpore is developed and opensourced by DMetaSoul team. You could also join our slack user discussion space.
Core Features
MetaSpore has the following features:
- One-stop end-to-end development, from offline model training to online prediction and bucketing experiments, with a unified development experience across the entire process;
- Deep learning training framework, compatible with PyTorch ecology, supports distributed large-scale sparse feature learning;
- The training framework is connected with PySpark to seamlessly read the training data from the data lake and data warehouse;
- High-performance online prediction service, supports fast inference for neural network, decision tree, Spark ML, SKLearn and other models; supports heterogeneous hardware inference acceleration;
- In the offline unified feature extraction framework, the online feature reading logic is automatically generated, and the feature extraction logic is unified cross offline and online;
- Online algorithm application framework, providing model prediction, experiment bucketing and traffic splitting, dynamic hot loading of parameters and rich debug functions;
- Rich industry algorithm examples and end-to-end solutions.
Examples
Installation package download
Training package
We provide precompiled offline training wheel package on pypi, install it via pip:
pip install metaspore
The minimum Python version required is 3.8.
After installation, also install pytorch and pyspark (they are not included as depenencies of metaspore wheel so you could choose pyspark and pytorch versions as needed):
pip install pyspark
pip install torch==1.11.0+cpu -f https://download.pytorch.org/whl/cpu/torch_stable.html
Serving package
We provide prebuilt docker images for MetaSpore Serving Service:
CPU only image
docker pull dmetasoul/metaspore-serving-release:cpu-v1.0.1
GPU image
docker pull dmetasoul/metaspore-serving-release:gpu-v1.0.1
See Run MetaSpore Serving for details.