Overview

Model serving overview

Kubeflow supports two model serving systems that allow multi-framework model serving: KFServing and Seldon Core. Alternatively, you can use a standalone model serving system. This page gives an overview of the options, so that you can choose the framework that best supports your model serving requirements.

Multi-framework serving with KFServing or Seldon Core

KFServing and Seldon Core are both open source systems that allow multi-framework model serving. The following table compares KFServing and Seldon Core. A check mark () indicates that the system (KFServing or Seldon Core) supports the feature specified in that row.

Feature Sub-feature KFServing Seldon Core
Framework TensorFlow sample docs
XGBoost sample docs
scikit-learn sample docs
NVIDIA Triton Inference Server sample docs
ONNX sample
PyTorch sample
Graph Transformers sample docs
Combiners Roadmap sample
Routers including MAB Roadmap docs
Analytics Explanations sample docs
Scaling Knative sample
GPU AutoScaling sample
HPA HPA vs KPA docs
Custom Container sample docs
Language Wrappers Python, Java, R
Multi-Container docs
Rollout Canary sample docs
Shadow
Istio

Notes:

  • KFServing and Seldon Core share some technical features, including explainability (using Seldon Alibi Explain) and payload logging, as well as other areas.
  • A commercial product, Seldon Deploy, supports both KFServing and Seldon in production.
  • KFServing is part of the Kubeflow project ecosystem. Seldon Core is an external project supported within Kubeflow.

Further information:

TensorFlow Serving

For TensorFlow models you can use TensorFlow Serving for real-time prediction. However, if you plan to use multiple frameworks, you should consider KFServing or Seldon Core as described above.

NVIDIA Triton Inference Server

NVIDIA Triton Inference Server is a REST and GRPC service for deep-learning inferencing of TensorRT, TensorFlow, Pytorch, ONNX and Caffe2 models. The server is optimized to deploy machine learning algorithms on both GPUs and CPUs at scale. Triton Inference Server was previously known as TensorRT Inference Server.

You can use NVIDIA Triton Inference Server as a standalone system, but you should consider KFServing as described above. KFServing includes support for NVIDIA Triton Inference Server.

BentoML

BentoML is an open-source platform for high-performance ML model serving. It makes building production API endpoint for your ML model easy and supports all major machine learning training frameworks, including Tensorflow, Keras, PyTorch, XGBoost, scikit-learn and etc.

BentoML comes with a high-performance API model server with adaptive micro-batching support, which achieves the advantage of batch processing in online serving. It also provides model management and model deployment functionality, giving ML teams an end-to-end model serving workflow, with DevOps best practices baked in.