Seaplane Model Hub

Seaplane currently offers access to over 40 large generative AI models. These models are accessed through the Seaplane model hub. This document describes how to use these models inside of your Seaplane application.

With the Seaplane CLI installed, run seaplane models in your terminal to get a full list of all available models.

tip

The generative AI field is moving fast and new models are available almost every week. If you require a specific model that is not currently listed let us know by contacting support@seaplane.io.

We add models quickly and can likely make them available within a few days.

Accessing models

All models on Seaplane are accessible through the model hub hub. A seaplane-created reusable DAG (Directed Acyclic Graph) that runs model requests asynchronously. The code snippet below provides a basic instantiation of the model hub inside an application.

The model hub takes two required input parameters and 1 optional parameter.

• name (str) - A unique name for your model hub instance set to model-hub in the example below (required).
• input (list) - A list of inputs consisting of one or more outputs from upstream tasks or DAGs (required).
• simplify (bool) - By default, this parameter is set to True. By enabling this feature Seaplane simplifies the input and output messages of the models. This allows you to send a standard JSON input and always receive the same output. It is not recommended to turn this off unless you know what you are doing or by recommendation of our support team.

Seaplane app using the Seaplane model hub

from seaplane.apps import App

# create an app instance
app = App("model-app")

# initialize a substation instance
model_output = app.model_hub("model-hub", [app.input()])
app.respond(substation_output)

# run the app
app.run()

Input

The model hub takes a JSON object as input. The JSON object should contain the model you want to use and a set of model parameters depending on the model you use. For example, a request to Llama-70b might look something like this.

{
  "model": "llama-2-70b-chat",
  "prompt": "what is a seaplane?"
}

The following input parameters are available. However, not all models support all parameters. You can read more about the available parameters per model in our parameter documentation.

• model (required)
• prompt (required)
• system_prompt (optional)
• temperature (optional)
• max_new_tokens (optional)
• min_new_tokens (optional)
• top_p (optional)
• top_k (optional)
• stop_sequences (optional)
• length_penalty (optional)
• presence_penalty (optional)
• frequency_penalty (optional)
• repeat_penalty (optional)
• seed (optional)
• use_prompt_template (optional)

By default, Seaplane wraps the prompt in the relevant prompt template for each model. As a result, your output data prompt can look different than the provided input prompt. For example, the input above uses llama-2-70b-chat. Seaplane automatically changes your prompt from What is a seaplane? to <s>[INST]<<SYS>>\\n\\n<</SYS>>\\n\\nWhat is a Seaplane? [/INST].

You can turn this off by setting use_prompt_template to False.

Example input without prompt template

{
    "model" : "llama-2-70b-chat",
    "prompt" : "what is a seaplane?",
    "use_prompt_template" : false
}

Output

Substation returns a JSON object that includes the following elements. This object is sent to the next Task as defined in your DAG.

• input_data - A JSON object containing the input message into Substation
• output - The generated response by the model.

example output

{
    "input_data": {
        "model": "llama-2-70b-chat",
        "prompt": "<s>[INST]<<SYS>>\\n\\n<</SYS>>\\n\\nWhat is a Seaplane? [/INST]"
    },
    "output": " A seaplane is a type of aircraft that is designed to take off and land on water. It has a specialized fuselage and wings that allow it to float on the surface of the water, and it is equipped with a propeller or jet engines that provide thrust for takeoff and landing. Seaplanes are also known as \\"flying boats\\" because they have a hull that resembles a boat, allowing them to float on the water. They are commonly used for transportation over long distances, particularly in areas where there are no suitable airports or runways. Seaplanes can"
}