sscdev.databricks

Classes

DatabricksClient

Unified client for interacting with Databricks SQL and Workspace Files API.

Module Contents

class sscdev.databricks.DatabricksClient(config: str | None = None, base_path: str | None = None)

Unified client for interacting with Databricks SQL and Workspace Files API. Uses secure credentials from Azure KeyVault under the ‘Databricks’ secret name by default. Automatically prepends a configurable base path to all file operations.

_full_path(relative_path: str) → str

Constructs the full path by prepending the base path.

query(query: str) → pandas.DataFrame

Executes a SQL query and returns results as a DataFrame.

mkdir(path: str)

Creates a new directory at the specified relative path.

rm(path: str, recurse: bool = False)

Deletes a file or directory at the specified relative path. If recurse=True and the path is a directory, deletes all contents recursively before removing the directory.

Parameters:

path (str): Relative path to the file or directory. recurse (bool): Whether to recursively delete directory contents. Defaults to False.

read(path: str) → bytes | None

Reads a file from the specified relative path and returns its raw bytes.

upload(path: str, contents: bytes, overwrite: bool = True)

Uploads a file to the specified relative path.

ls(path: str | None = '/', recurse: bool = False) → pandas.DataFrame

Lists contents of a directory at the specified relative path and returns as a Pandas DataFrame. If recurse=True, traverses subdirectories recursively.

Parameters:

path (str): Relative path within the base directory. recurse (bool): Whether to list contents recursively. Defaults to False.

Returns:

pd.DataFrame: Directory contents with metadata and relative paths.