Using Data Adapters¶

This is an index of all the available data adapters, both savers and loaders. Note that some savers and loaders are the same (certain classes can handle both), but some are different. You will want to reference this when calling out to any of the following:

Using save_to.
Using load_from.
Using materializers

To read these tables, you want to first look at the key to determine which format you want – these should be human-readable and familiar to you. Then you’ll want to look at the types field to figure out which is the best for your case (the object you want to load from or save to).

Finally, look up the adapter params to see what parameters you can pass to the data adapters. The optional params come with their default value specified.

If you want more information, click on the module, it will send you to the code that implements it to see how the parameters are used.

As an example, say we wanted to save a pandas dataframe to a CSV file. We would first find the key csv, which would inform us that we want to call save_to.csv (or to.csv in the case of materialize). Then, we would look at the types field, finding that there is a pandas dataframe adapter. Finally, we would look at the params field, finding that we can pass path, and (optionally) sep (which we’d realize defaults to , when looking at the code).

All together, we’d end up with:

import pandas as pd
from hamilton.function_modifiers import value, save_to

@save_to.csv(path=value("my_file.csv"))
def my_data(...) -> pd.DataFrame:
    ...

And we’re good to go!

If you want to extend these, see Using Data Adapters for documentation, and the example in the repository for an example of how to do so.

Note that you will need to call registry.register_adapters (or import a module that does that) prior to dynamically referring to these in the code – otherwise we won’t know about them, and won’t be able to access that key!

Data Loaders¶

key	loader params	types	module
json	path`str`	`dict` `list`	hamilton.io.default_data_loaders
json	filepath_or_buffer`Union` chunksize`Optional=None` compression`Union=infer` convert_axes`Optional=None` convert_dates`Union=True` date_unit`Optional=None` dtype`Union=None` dtype_backend`Optional=None` encoding`Optional=None` encoding_errors`Optional=strict` engine`str=ujson` keep_default_dates`bool=True` lines`bool=False` nrows`Optional=None` orient`Optional=None` precise_float`bool=False` storage_options`Optional=None` typ`str=frame`	`DataFrame`	hamilton.plugins.pandas_extensions
json	source`Union` schema`Union=None` schema_overrides`Union=None`	`DataFrame`	hamilton.plugins.polars_extensions
json	path`Union`	`XGBModel` `Booster`	hamilton.plugins.xgboost_extensions
literal	value`Any`	`Any`	hamilton.io.default_data_loaders
file	path`str` encoding`str=utf-8`	`str`	hamilton.io.default_data_loaders
file	path`Union`	`LGBMModel` `Booster` `CVBooster`	hamilton.plugins.lightgbm_extensions
pickle	path`str`	`object` `Any`	hamilton.io.default_data_loaders
pickle	filepath_or_buffer`Union=None` path`Union=None` compression`Union=infer` storage_options`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
environment	names`Tuple`	`dict`	hamilton.io.default_data_loaders
yaml	path`Union`	`str` `int` `float` `bool` `dict` `list`	hamilton.plugins.yaml_extensions
npy	path`Union` mmap_mode`Optional=None` allow_pickle`Optional=None` fix_imports`Optional=None` encoding`Literal=ASCII`	`ndarray`	hamilton.plugins.numpy_extensions
csv	path`Union` sep`Optional=,` delimiter`Optional=None` header`Union=infer` names`Optional=None` index_col`Union=None` usecols`Union=None` dtype`Union=None` engine`Optional=None` converters`Optional=None` true_values`Optional=None` false_values`Optional=None` skipinitialspace`Optional=False` skiprows`Union=None` skipfooter`int=0` nrows`Optional=None` na_values`Union=None` keep_default_na`bool=True` na_filter`bool=True` verbose`bool=False` skip_blank_lines`bool=True` parse_dates`Union=False` keep_date_col`bool=False` date_format`Optional=None` dayfirst`bool=False` cache_dates`bool=True` iterator`bool=False` chunksize`Optional=None` compression`Union=infer` thousands`Optional=None` decimal`str=.` lineterminator`Optional=None` quotechar`Optional=None` quoting`int=0` doublequote`bool=True` escapechar`Optional=None` comment`Optional=None` encoding`str=utf-8` encoding_errors`Union=strict` dialect`Union=None` on_bad_lines`Union=error` delim_whitespace`bool=False` low_memory`bool=True` memory_map`bool=False` float_precision`Optional=None` storage_options`Optional=None` dtype_backend`Literal=numpy_nullable`	`DataFrame`	hamilton.plugins.pandas_extensions
csv	file`Union` has_header`bool=True` columns`Union=None` new_columns`Sequence=None` separator`str=,` comment_char`str=None` quote_char`str="` skip_rows`int=0` dtypes`Union=None` null_values`Union=None` missing_utf8_is_empty_string`bool=False` ignore_errors`bool=False` try_parse_dates`bool=False` n_threads`int=None` infer_schema_length`int=100` batch_size`int=8192` n_rows`int=None` encoding`Union=utf8` low_memory`bool=False` rechunk`bool=True` use_pyarrow`bool=False` storage_options`Dict=None` skip_rows_after_header`int=0` row_count_name`str=None` row_count_offset`int=0` sample_size`int=1024` eol_char`str=` raise_if_empty`bool=True`	`DataFrame`	hamilton.plugins.polars_extensions
csv	file`Union` has_header`bool=True` columns`Union=None` new_columns`Sequence=None` separator`str=,` comment_char`str=None` quote_char`str="` skip_rows`int=0` dtypes`Union=None` null_values`Union=None` missing_utf8_is_empty_string`bool=False` ignore_errors`bool=False` try_parse_dates`bool=False` n_threads`int=None` infer_schema_length`int=100` batch_size`int=8192` n_rows`int=None` encoding`Union=utf8` low_memory`bool=False` rechunk`bool=True` use_pyarrow`bool=False` storage_options`Dict=None` skip_rows_after_header`int=0` row_count_name`str=None` row_count_offset`int=0` eol_char`str=` raise_if_empty`bool=True`	`LazyFrame`	hamilton.plugins.polars_lazyframe_extensions
csv	spark`SparkSession` path`str` header`bool=True` sep`str=,`	`DataFrame`	hamilton.plugins.spark_extensions
parquet	path`Union` engine`Literal=auto` columns`Optional=None` storage_options`Optional=None` use_nullable_dtypes`bool=False` dtype_backend`Literal=numpy_nullable` filesystem`Optional=None` filters`Union=None`	`DataFrame`	hamilton.plugins.pandas_extensions
parquet	file`Union` columns`Union=None` n_rows`int=None` use_pyarrow`bool=False` memory_map`bool=True` storage_options`Dict=None` parallel`Any=auto` row_count_name`str=None` row_count_offset`int=0` low_memory`bool=False` pyarrow_options`Dict=None` use_statistics`bool=True` rechunk`bool=True`	`DataFrame`	hamilton.plugins.polars_extensions
parquet	file`Union` columns`Union=None` n_rows`int=None` use_pyarrow`bool=False` memory_map`bool=True` storage_options`Dict=None` parallel`Any=auto` row_count_name`str=None` row_count_offset`int=0` low_memory`bool=False` use_statistics`bool=True` rechunk`bool=True`	`LazyFrame`	hamilton.plugins.polars_lazyframe_extensions
parquet	spark`SparkSession` path`str`	`DataFrame`	hamilton.plugins.spark_extensions
sql	query_or_table`str` db_connection`Union` chunksize`Optional=None` coerce_float`bool=True` columns`Optional=None` dtype`Union=None` dtype_backend`Optional=None` index_col`Union=None` params`Union=None` parse_dates`Union=None`	`DataFrame`	hamilton.plugins.pandas_extensions
xml	path_or_buffer`Union` xpath`Optional=./*` namespace`Optional=None` elems_only`Optional=False` attrs_only`Optional=False` names`Optional=None` dtype`Optional=None` converters`Optional=None` parse_dates`Union=False` encoding`Optional=utf-8` parser`str=lxml` stylesheet`Union=None` iterparse`Optional=None` compression`Union=infer` storage_options`Optional=None` dtype_backend`str=numpy_nullable`	`DataFrame`	hamilton.plugins.pandas_extensions
html	io`Union` match`Optional=.+` flavor`Union=None` header`Union=None` index_col`Union=None` skiprows`Union=None` attrs`Optional=None` parse_dates`Optional=None` thousands`Optional=,` encoding`Optional=None` decimal`str=.` converters`Optional=None` na_values`Iterable=None` keep_default_na`bool=True` displayed_only`bool=True` extract_links`Optional=None` dtype_backend`Literal=numpy_nullable` storage_options`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
stata	filepath_or_buffer`Union` convert_dates`bool=True` convert_categoricals`bool=True` index_col`Optional=None` convert_missing`bool=False` preserve_dtypes`bool=True` columns`Optional=None` order_categoricals`bool=True` chunksize`Optional=None` iterator`bool=False` compression`Union=infer` storage_options`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
feather	path`Union` columns`Optional=None` use_threads`bool=True` storage_options`Optional=None` dtype_backend`Literal=numpy_nullable`	`DataFrame`	hamilton.plugins.pandas_extensions
feather	source`Union` columns`Union=None` n_rows`Optional=None` use_pyarrow`bool=False` memory_map`bool=True` storage_options`Optional=None` row_count_name`Optional=None` row_count_offset`int=0` rechunk`bool=True`	`DataFrame`	hamilton.plugins.polars_extensions
feather	source`Union` columns`Union=None` n_rows`Optional=None` use_pyarrow`bool=False` memory_map`bool=True` storage_options`Optional=None` row_count_name`Optional=None` row_count_offset`int=0` rechunk`bool=True`	`LazyFrame`	hamilton.plugins.polars_lazyframe_extensions
orc	path`Union` columns`Optional=None` dtype_backend`Literal=numpy_nullable` filesystem`Union=None`	`DataFrame`	hamilton.plugins.pandas_extensions
excel	path`Union=None` sheet_name`Union=0` header`Union=0` names`Optional=None` index_col`Union=None` usecols`Union=None` dtype`Union=None` engine`Optional=None` converters`Union=None` true_values`Optional=None` false_values`Optional=None` skiprows`Union=None` nrows`Optional=None` keep_default_na`bool=True` na_filter`bool=True` verbose`bool=False` parse_dates`Union=False` date_format`Union=None` thousands`Optional=None` decimal`str=.` comment`Optional=None` skipfooter`int=0` storage_options`Optional=None` dtype_backend`Literal=numpy_nullable` engine_kwargs`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
table	filepath_or_buffer`Union` sep`Optional=None` delimiter`Optional=None` header`Union=infer` names`Optional=None` index_col`Union=None` usecols`Optional=None` dtype`Union=None` engine`Optional=None` converters`Optional=None` true_values`Optional=None` false_values`Optional=None` skipinitialspace`bool=False` skiprows`Union=None` skipfooter`int=0` nrows`Optional=None` na_values`Union=None` keep_default_na`bool=True` na_filter`bool=True` verbose`bool=False` skip_blank_lines`bool=True` parse_dates`Union=False` infer_datetime_format`bool=False` keep_date_col`bool=False` date_parser`Optional=None` date_format`Optional=None` dayfirst`bool=False` cache_dates`bool=True` iterator`bool=False` chunksize`Optional=None` compression`Union=infer` thousands`Optional=None` decimal`str=.` lineterminator`Optional=None` quotechar`Optional="` quoting`int=0` doublequote`bool=True` escapechar`Optional=None` comment`Optional=None` encoding`Optional=None` encoding_errors`Optional=strict` dialect`Optional=None` on_bad_lines`Union=error` delim_whitespace`bool=False` low_memory`bool=True` memory_map`bool=False` float_precision`Optional=None` storage_options`Optional=None` dtype_backend`Literal=numpy_nullable`	`DataFrame`	hamilton.plugins.pandas_extensions
spss	path`Union` usecols`Union=None` convert_categoricals`bool=True` dtype_backend`Literal=numpy_nullable`	`DataFrame`	hamilton.plugins.pandas_extensions
avro	file`Union` columns`Union=None` n_rows`Optional=None`	`DataFrame`	hamilton.plugins.polars_extensions
database	query`str` connection`str` iter_batches`bool=False` batch_size`Optional=None` schema_overrides`Optional=None` infer_schema_length`Optional=None` execute_options`Optional=None`	`DataFrame`	hamilton.plugins.polars_extensions
spreadsheet	source`Union` sheet_id`Union=None` sheet_name`Union=None` engine`Literal=xlsx2csv` engine_options`Optional=None` read_options`Optional=None` schema_overrides`Optional=None` raise_if_empty`bool=True`	`DataFrame`	hamilton.plugins.polars_extensions
dlt	resource`DltResource`	`DataFrame`	hamilton.plugins.dlt_extensions

Data Savers¶

key	saver params	types	module
json	path`str`	`dict` `list`	hamilton.io.default_data_loaders
json	filepath_or_buffer`Union` compression`str=infer` date_format`str=epoch` date_unit`str=ms` default_handler`Optional=None` double_precision`int=10` force_ascii`bool=True` index`Optional=None` indent`int=0` lines`bool=False` mode`str=w` orient`Optional=None` storage_options`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
json	file`Union` pretty`bool=False` row_oriented`bool=False`	`DataFrame` `LazyFrame`	hamilton.plugins.polars_extensions
json	path`Union`	`XGBModel` `Booster`	hamilton.plugins.xgboost_extensions
file	path`str` encoding`str=utf-8`	`str`	hamilton.io.default_data_loaders
file	path`Union`	`bytes` `BytesIO`	hamilton.io.default_data_loaders
file	path`Union` num_iteration`Optional=None` start_iteration`int=0` importance_type`Literal=split`	`LGBMModel` `Booster` `CVBooster`	hamilton.plugins.lightgbm_extensions
pickle	path`str`	`object`	hamilton.io.default_data_loaders
pickle	path`Union` compression`Union=infer` protocol`int=5` storage_options`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
memory		`Any`	hamilton.io.default_data_loaders
yaml	path`Union`	`str` `int` `float` `bool` `dict` `list`	hamilton.plugins.yaml_extensions
npy	path`Union` allow_pickle`Optional=None` fix_imports`Optional=None`	`ndarray`	hamilton.plugins.numpy_extensions
csv	path`Union` sep`Optional=,` na_rep`str=` float_format`Union=None` columns`Optional=None` header`Union=True` index`Optional=False` index_label`Union=None` mode`str=w` encoding`Optional=None` compression`Union=infer` quoting`Optional=None` quotechar`Optional="` lineterminator`Optional=None` chunksize`Optional=None` date_format`Optional=None` doublequote`bool=True` escapechar`Optional=None` decimal`str=.` errors`str=strict` storage_options`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
csv	file`Union` has_header`bool=True` separator`str=,` line_terminator`str=` quote`str="` batch_size`int=1024` datetime_format`str=None` date_format`str=None` time_format`str=None` float_precision`int=None` null_value`str=None` quote_style`Literal=None`	`DataFrame` `LazyFrame`	hamilton.plugins.polars_extensions
parquet	path`Union` engine`Literal=auto` compression`Optional=snappy` index`Optional=None` partition_cols`Optional=None` storage_options`Optional=None` extra_kwargs`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
parquet	file`Union` compression`Any=zstd` compression_level`int=None` statistics`bool=False` row_group_size`int=None` use_pyarrow`bool=False` pyarrow_options`Dict=None`	`DataFrame` `LazyFrame`	hamilton.plugins.polars_extensions
sql	table_name`str` db_connection`Any` chunksize`Optional=None` dtype`Union=None` if_exists`str=fail` index`bool=True` index_label`Union=None` method`Union=None` schema`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
xml	path_or_buffer`Union` index`bool=True` root_name`str=data` row_name`str=row` na_rep`Optional=None` attr_cols`Optional=None` elems_cols`Optional=None` namespaces`Optional=None` prefix`Optional=None` encoding`str=utf-8` xml_declaration`bool=True` pretty_print`bool=True` parser`str=lxml` stylesheet`Union=None` compression`Union=infer` storage_options`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
html	buf`Union=None` columns`Optional=None` col_space`Union=None` header`Optional=True` index`Optional=True` na_rep`Optional=NaN` formatters`Union=None` float_format`Optional=None` sparsify`Optional=True` index_names`Optional=True` justify`str=None` max_rows`Optional=None` max_cols`Optional=None` show_dimensions`bool=False` decimal`str=.` bold_rows`bool=True` classes`Union=None` escape`Optional=True` notebook`Literal=False` border`int=None` table_id`Optional=None` render_links`bool=False` encoding`Optional=utf-8`	`DataFrame`	hamilton.plugins.pandas_extensions
stata	path`Union=None` convert_dates`Optional=None` write_index`bool=True` byteorder`Optional=None` time_stamp`Optional=None` data_label`Optional=None` variable_labels`Optional=None` version`Literal=114` convert_strl`Optional=None` compression`Union=infer` storage_options`Optional=None` value_labels`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
feather	path`Union` dest`Optional=None` compression`Literal=None` compression_level`Optional=None` chunksize`Optional=None` version`Optional=2`	`DataFrame`	hamilton.plugins.pandas_extensions
feather	file`Union=None` compression`Literal=uncompressed`	`DataFrame` `LazyFrame`	hamilton.plugins.polars_extensions
orc	path`Union` engine`Literal=pyarrow` index`Optional=None` engine_kwargs`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
excel	path`Union` sheet_name`str=Sheet1` na_rep`str=` float_format`Optional=None` columns`Optional=None` header`Union=True` index`bool=True` index_label`Union=None` startrow`int=0` startcol`int=0` engine`Optional=None` merge_cells`bool=True` inf_rep`str=inf` freeze_panes`Optional=None` storage_options`Optional=None` engine_kwargs`Optional=None`	`DataFrame`	hamilton.plugins.pandas_extensions
avro	file`Union` compression`Any=uncompressed`	`DataFrame` `LazyFrame`	hamilton.plugins.polars_extensions
database	table_name`str` connection`str` if_table_exists`Literal=fail` engine`Literal=sqlalchemy`	`DataFrame` `LazyFrame`	hamilton.plugins.polars_extensions
spreadsheet	workbook`Union` worksheet`Optional=None` position`Union=A1` table_style`Union=None` table_name`Optional=None` column_formats`Optional=None` dtype_formats`Optional=None` conditional_formats`Optional=None` header_format`Optional=None` column_totals`Union=None` column_widths`Union=None` row_totals`Union=None` row_heights`Union=None` sparklines`Optional=None` formulas`Optional=None` float_precision`int=3` include_header`bool=True` autofilter`bool=True` autofit`bool=False` hidden_columns`Union=None` hide_gridlines`bool=None` sheet_zoom`Optional=None` freeze_panes`Union=None`	`DataFrame` `LazyFrame`	hamilton.plugins.polars_extensions
dlt	pipeline`Pipeline` table_name`str` primary_key`Optional=None` write_disposition`Optional=None` columns`Optional=None` schema`Optional=None` loader_file_format`Optional=None`	`Iterable` `DataFrame` `Table` `RecordBatch`	hamilton.plugins.dlt_extensions