Inference Model Deployment

This tutorial shows you how to deploy a simple linear regression model on Seaplane and make it accessible through an API endpoint. This serves as an example of how Seaplane can turn any trained model into an inference API endpoint. Additionally, the tutorial uses a feature flag to allow you to hot-swap models without redeploying the application. We included a video walkthrough at the bottom for the visual learners.

This tutorial assumes you have:

• A seaplane account, you can sign up here.
• The Seaplane CLI installed on your machine. You can learn more in our documentation.
• A trained model stored as a pickle.
• A basic understanding of Seaplane and Python.

Let's get started.

Create Seaplane Project

Create a new Seaplane application by executing the following command in your terminal.

seaplane init regression_app

Navigate into your project and open the main.py file. Remove the boilerplate code. Import all the required packages by adding the following lines to the top of the file.

from seaplane.apps import App
from seaplane.object import object_store
from seaplane.kv import kv_store
import json
import pickle

We imported the main Seaplane SDK, the Seaplane object store (this will hold your trained model), the Seaplane KV store (this will hold a feature flag to hot-swap models) and some auxiliary packages.

This tutorial creates a simple app with only one task, however you can easily extend it with additional tasks for pre- or post-processing.

Prediction Task

Create the single required task definition by creating a new function called make_prediction as follows in main.py. The function takes one parameter as input called msg this is the input data the task receives through the API endpoint.

def make_prediction(msg):
    # load the input data
    data = json.loads(msg.body)

Extend the code to load the trained model Pickle file from local storage.

def make_prediction(msg):
    # load the input data
    data = json.loads(msg.body)

    # open the local model if its still the latest version else download
    try:
        with open(f"regression-models/{model_name}", "rb") as f:
            model = pickle.load(f)
    except FileNotFoundError:
        # code here to deal with none existent model

This app could function as is, however, it would hard code the model into the application. Model updates would require a complete redeployment of the app. Not the most convenient setup. To overcome this implement a feature flag using the Seaplane low latency KV store to check which model to use and update the model if it is outdated.

def make_prediction(msg):
    # load the input data
    data = json.loads(msg)

    # check KV store for which model to use
    model_name = kv_store.get_key("regression-models", "model-name").decode()

    # open the local model if its still the latest version else download
    try:
        with open(f"regression_model/{model_name}", "rb") as f:
            model = pickle.load(f)
    except FileNotFoundError:
        data = object_store.download("my-models", model_name)
        with open(f"regression_model/{model_name}", "wb") as f:
            f.write(data)

        # load the model
        model = pickle.load(data)

Let's unpack the code above. Line 6 loads the model name from the low-latency KV store. Lines 8-18 load the model from local storage based on the model name. If that model does not exist it is downloaded from the object store and stored locally. This allows you to hot-swap models by uploading a new model in the object store and then updating the model name in the KV store.

Finally, add the following lines to make the prediction and yield the results out of the task and back to the user.

def make_prediction(msg):
    # load the input data
    data = json.loads(msg)

    # check KV store for which model to use
    model_name = kv_store.get_key("regression-models", "model-name").decode()

    # open the local model if its still the latest version else download
    try:
        with open(f"regression_model/{model_name}", "rb") as f:
            model = pickle.load(f)
    except FileNotFoundError:
        data = object_store.download("my-models", model_name)
        with open(f"regression_model/{model_name}", "wb") as f:
            f.write(data)

        # load the model
        model = pickle.load(data)

    # make the prediction
    prediction = model.predict(data["input"])

    # send the prediciton back through the endpoint
    yield json.dumps({"prediction": prediction[0]})

Creating And Deploying The Application

Now that you have a task definition it is time to wire it into an application and deploy it on Seaplane. Below the make_prediction function add the following lines

# create the app and DAG
app = App("linear-regression-app")
dag = app.dag("regression-dag")
prediction = dag.task(make_prediction, [app.input()], instance_name="make-prediction")

# send the reponse back to the user
app.respond(prediction)

# run the app
app.run()

Let's unpack what we did here. You created a new application called linear-regression-app on line 2. On line 3 you created a new directed acyclic graph (DAG) that defines the flow of the data. If you want to add pre-processing or post-processing tasks this would be the place to do so. Line 4 creates a new task instance i.e a deployed compute container based on the make-prediction task definition. Line 7 sets the output of the application to the output of the task. In practice, this means that you are sending the JSON {"prediction": prediction} back to the user. Finally, on line 10 you start the application.

Completed Code Snippet

from seaplane.apps import App
from seaplane.object import object_store
from seaplane.kv import kv_store
import json
import pickle


def make_prediction(msg):
# load the input data
data = json.loads(msg)

# check KV store for which model to use
model_name = kv_store.get_key("regression-models", "model-name").decode()

# open the local model if it is still the latest version else download
try:
    with open(f"regression_model/{model_name}", "rb") as f:
        model = pickle.load(f)
except FileNotFoundError:
    data = object_store.download("my-models", model_name)
    with open(f"regression_model/{model_name}", "wb") as f:
        f.write(data)

    # load the model
    model = pickle.load(data)

# make the prediction
prediction = model.predict(data["input"])

# send the prediciton back through the endpoint
yield json.dumps({"prediction": prediction[0]})


# create the app and DAG
app = App("linear-regression-app")
dag = app.dag("regression-dag")
prediction = dag.task(make_prediction, [app.input()], instance_name="make-prediction")

# send the reponse back to the user
app.respond(prediction)

# run the app
app.run()

Now get ready for deployment. Open the .env file and replace sp-your-api-key with your Seaplane API key which you can get from the flightdeck.

Since this is the first time deploying the application you need to install the required packages using poetry. Run the following two commands in your project root directory.

poetry lock
poetry install

With all packages installed, you are ready to deploy the application. Run the following command in your terminal in your project's root directory.

seaplane deploy

A few seconds later your application is deployed and ready to serve production traffic.

Initial Model Setup

But before we serve traffic there is one more thing to do. Add your preferred model to the object store and set it as the model to use in the KV store. Navigate to the directory where your trained model is stored. This application assumes models are stored as Pickel files.

Create a new bucket named regression-models by executing the following plane command. If this is the first time using plane you need to create a new context first. You can learn more about creating contexts in our documentation.

plane object create-bucket regression-models

Upload your model to the newly created bucket by executing the following command in your terminal

plane object upload regression-models model.pkl remote-model.pkl

Where model.pkl is the local file name and remote-model.pkl is the remote object name.

Now that the model is uploaded set it as the model to use in the KV store. First, create a new KV store using the Seaplane CLI by executing the following command.

plane kv create-store regression-models

Next set the model to use to the model you just uploaded by executing the following command. Make sure you match the model name to the model name in the object store.

plane kv set regression-models mode-name remote-model.pkl

Hot-Swapping New Models

Hot swapping a new model is as simple as uploading a new model to the object store and setting it as the model to use in the KV store. Make sure to use unique names for your models as the application looks for the specific model name in the local data directory.

plane object upload regression-models new-model.pkl remote-new-model.pkl
plane kv set regression-models mode-name new-remote-model.pkl

Calling your model

With the deployed model we can call it directly using the provided URL.

curl -X POST -H 'Content-Type: application/octet-stream' \
--header "Authorization: Bearer $(curl https://flightdeck.cplane.cloud/identity/token --request POST  --header "Authorization: Bearer <YOUR-API-KEY>")" \
-d '{"input": [[3.25000000,3.90000000,4.50320513,1.07371795,1.10900000,1.77724359,3.40600000,-1.18360000]]}' https://carrier.cplane.cloud/v1/endpoints/linear-regression-app/request

Video Tutorial