Project Creation And Default Structure
This document describes the default project structure of Seaplane applications. It assumes you have the Seaplane SDK and CLI tool installed. You can learn more about installation and setup here.
Creating A New Project​
To create a new Seaplane application run the following command inside your terminal. We recommend you set up a virtual environment to avoid dependency conflicts.
seaplane init my-project-name
Seaplane automatically generates the default project structure and populates the
main.py
file with a "hello world" application.
Creating Virtual Environments​
Run the following commands to create a virtual environment. You can learn more about virtual environments here.
- Mac and Linux
- Windows
python3 -m venv my-seaplane-project
source my-seaplane-project/bin/activate
python3 -m venv myenv
Using Command Prompt:
myenv\Scripts\activate
Using Powershell:
myenv\Scripts\Activate.ps1
Default Project Structure and Files​
A default seaplane project consists of the following elements.
my-project-name/
├── my-project-name/
│ └── main.py
├── .env
└── pyproject.toml
The main my-project-name
directory contains all the files associated with this
project. Including a directory for the project's Python scripts, a
pyproject.toml
file for build configuration and a .env
file for declaring
project secrets.
Python File Directory​
Each Seaplane project requires another directory by the same name containing at
least one Python script indicated as the main Python script by the
pyproject.toml
file. You can learn more about the Pyproject file in this section.
The only requirement for the main Python script is that it includes a new app
instance by using App()
, a response to return via app.respond()
, an app.run()
command to tell Seaplane to run the app, a DAG
(Directed Acyclic Graph), and a task
. The init function generates a sample project with one file
main.py
containing all the components of an application, a DAG
, and a task
.
For larger projects, we recommend you define your App()
and your main DAG
inside your main.py
.
Create a new file for all other DAGs
and tasks
to enable easier collaboration and to keep
your project structured and easy to understand.
The application, DAG, and task names are currently limited to 28 characters and can only include the following characters:
a-z
0-9
-
A sample project structure might look something like this.
my-project-name/
├── my-project-name/
│ ├── task1.py
│ ├── task2.py
│ ├── task3.py
│ ├── dag1.py
│ └── main.py
├── .env
└── pyproject.toml
Pyproject​
The pyproject.toml
file is the main configuration file. It contains basic
information about your app and its dependencies, and it indicates which Python
script in your project contains the app.run()
command. More information on each
section below.
[tool.poetry]
- Includes the name, version and optionally a description and authors of the project.[tool.seaplane]
- Indicates which Python script includes theapp.run()
command. By default, it is set tomain.py
but you can use any Python script you like as long as it contains the start command.[tool.poetry.dependencies]
- This section should include all dependencies of your project. Add each dependency as a new line. To avoid dependency conflicts we recommend using loosely defined dependencies i.e'package>=1.0.0'
instead of'package==1.0.0'
[build-system]
- This section defines the build backend and relevant configuration options for building your project.
Environment Variables​
Use the .env
file to define application secrets. During deployment, Seaplane
automatically creates environment variables for all secrets defined in .env
.
Make sure you include the following line inside your .gitignore
file to avoid
pushing application secrets to your GitHub repository.
.env
You can load environment secrets inside your application using
os.getenv('SECRET_NAME')
. For example, assume you have the following .env
file.
SEAPLANE_API_KEY=<YOUR-SEAPLANE-KEY>
OPEN_AI_API_KEY=<API-KEY-1>
MY_EXTERNAL_SERVICE=<API-KEY-2>
During deployment, Seaplane automatically creates the required secrets as
environment variables. You can load them inside a sql_access
object as
follows.
import os
def my_sql_task(msg):
# load API keys from environment variables
open_ai_key = os.getenv('OPEN_AI_API_KEY')
external_key = os.getenv('MY_EXTERNAL_SERVICE')