Python SDK

A simple, intuitive SDK for AI engineers to work with the Datalayer platform.

This SDK provides a unified interface for authentication, runtime creation, and code execution in Datalayer Runtimes.

Features

• Simple Authentication: Easy token-based authentication with environment variable support
• Runtime Management: Create and manage compute runtimes (CPU/GPU) for code execution
• Runtime Snapshot Management: Create and manage compute snapshots of your runtimes
• Context Managers: Clean resource management with Python context managers

Installation

pip install datalayer_core

Quick Start

1. Authentication

Set your Datalayer token as an environment variable:

export DATALAYER_TOKEN="your-token-here"

Or pass it directly to the SDK:

from datalayer_core import DatalayerClient

client = DatalayerClient(token="your-token-here")
if client.authenticate():
    print("Successfully authenticated!")

2. Execute Code in a Runtime

You can use a context manager to create a Runtime and ensure it is correctly terminated after code execution.

from datalayer_core import DatalayerClient

client = DatalayerClient()
with client.create_runtime() as runtime:
    response = runtime.execute("Hello world!")
    print(response.stdout)

3. Save a Runtime Snapshot

Runtime snapshots allow you to save the state of a runtime (including installed packages, files, and variables) for later reuse.

from datalayer_core import DatalayerClient

client = DatalayerClient()

# Create a runtime and install some packages
with client.create_runtime() as runtime:
    runtime.execute("!pip install pandas numpy")
    runtime.execute("import pandas as pd; df = pd.DataFrame({'a': [1, 2, 3]})")
    
    # Create a snapshot of the current runtime state
    snapshot = runtime.create_snapshot(
        name="my-data-analysis-env",
        description="Runtime with pandas and sample data",
        stop=False  # Keep runtime running after snapshot
    )
    print(f"Created snapshot: {snapshot.name}")

# Later, create a new runtime from the snapshot
with client.create_runtime(snapshot_name="my-data-analysis-env") as runtime:
    # The runtime starts with pandas already installed and df variable available
    result = runtime.execute("print(df.head())")
    print(result.stdout)

You can also list and manage existing snapshots:

# List all available snapshots
snapshots = client.list_snapshots()
for snapshot in snapshots:
    print(f"Name: {snapshot.name}, Environment: {snapshot.enviroment}")

# Delete a snapshot when no longer needed
client.delete_snapshot(snapshot)

4. Set Secrets

Secrets allow you to securely store and use sensitive information like API keys, passwords, and tokens in your runtimes.

from datalayer_core import DatalayerClient

client = DatalayerClient()

# Create a new secret
created_secret = client.create_secret(
    name="OPENAI_API_KEY",
    description="OpenAI API key for GPT models",
    value="sk-your-secret-api-key-here"
)
print(f"Created secret: {created_secret.name}")

# Secrets are automatically available as environment variables in runtimes
with client.create_runtime() as runtime:
    result = runtime.execute("""
import os
api_key = os.environ.get('OPENAI_API_KEY')
print(f"API key length: {len(api_key) if api_key else 'Not found'}")
""")
    print(result.stdout)

You can also manage existing secrets:

# List all secrets
secrets = client.list_secrets()
for secret in secrets:
    print(f"Name: {secret.name}, Description: {secret.description}")

# Delete a secret when no longer needed
client.delete_secret(created_secret)
print("Secret deleted successfully")

Decorator Usage

The @datalayer decorator allows you to execute Python functions directly in Datalayer runtimes with minimal code changes:

from datalayer_core.sdk.decorators import datalayer

@datalayer
def process_data(x: float, y: float) -> float:
    """Process two numbers and return their sum."""
    return x + y

# This function will execute in a Datalayer runtime
result = process_data(10.5, 20.3)
print(result)  # 30.8

You can also specify advanced options:

@datalayer(
    snapshot_name="ml-environment",
    inputs=["data", "model_params"],
    output="predictions"
)
def train_model(data: list, model_params: dict) -> dict:
    """Train a machine learning model."""
    import numpy as np
    from sklearn.linear_model import LinearRegression
    
    X = np.array(data).reshape(-1, 1)
    y = np.array([x * 2 + 1 for x in data])
    
    model = LinearRegression()
    model.fit(X, y)
    
    return {
        "coefficients": model.coef_.tolist(),
        "intercept": model.intercept_,
        "score": model.score(X, y)
    }

# Execute with custom parameters
result = train_model([1, 2, 3, 4, 5], {"regularization": 0.1})
print(result)

Advanced Usage

Working with Files

You can execute Python files directly in runtimes:

from datalayer_core import DatalayerClient

client = DatalayerClient()
with client.create_runtime() as runtime:
    # Execute a Python file
    response = runtime.execute("/path/to/your/script.py")
    print(response.stdout)
    
    # Pass variables to the script
    variables = {"input_data": [1, 2, 3, 4, 5]}
    response = runtime.execute("/path/to/your/script.py", variables=variables)
    
    # Get a specific variable back
    result = runtime.execute("result = sum([1, 2, 3])", output="result")
    print(result)  # 6

Error Handling

from datalayer_core import DatalayerClient

client = DatalayerClient()

try:
    with client.create_runtime() as runtime:
        response = runtime.execute("1 / 0")  # Division by zero
        if response.stderr:
            print(f"Error occurred: {response.stderr}")
        else:
            print(response.stdout)
except Exception as e:
    print(f"Runtime error: {e}")

Runtime Management

# List all running runtimes
runtimes = client.list_runtimes()
for runtime in runtimes:
    print(f"Runtime: {runtime.name}, Environment: {runtime._environment_name}")

# Terminate a specific runtime
client.terminate_runtime(runtime)

Contributing

This SDK is designed to be simple and extensible. Feel free to submit issues and enhancement requests!

License

This SDK is open source software licensed under the BSD 3-Clause License.