DuckDB (dagster-duckdb)¶

This library provides an integration with the DuckDB database.

dagster_duckdb.build_duckdb_io_manager(type_handlers)[source]¶

Builds an IO manager definition that reads inputs from and writes outputs to DuckDB.

Note that the DuckDBIOManager cannot load partitioned assets.

Parameters:

type_handlers (Sequence[DbTypeHandler]) – Each handler defines how to translate between DuckDB tables and an in-memory type - e.g. a Pandas DataFrame.

Returns:

IOManagerDefinition

Examples

from dagster_duckdb import build_duckdb_io_manager
from dagster_duckdb_pandas import DuckDBPandasTypeHandler

duckdb_io_manager = build_duckdb_io_manager([DuckDBPandasTypeHandler()])

@job(resource_defs={'io_manager': duckdb_io_manager})
def my_job():
    ...

You may configure the returned IO Manager as follows:

resources:
    io_manager:
        config:
            base_path: path/to/store/files  # all data will be stored at this path
            duckdb_path: path/to/database.duckdb  # path to the duckdb database

class dagster_duckdb.DbTypeHandler(*args, **kwds)[source]¶