Data Adapters¶

Reference for data adapter base classes:

class hamilton.io.data_adapters.DataLoader¶

Base class for data loaders. Data loaders are used to load data from a data source. Note that they are inherently polymorphic – they declare what type(s) they can load to, and may choose to load differently depending on the type they are loading to.

abstract classmethod applicable_types() → Collection[Type]¶

Returns the types that this data loader can load to. These will be checked against the desired type to determine whether this is a suitable loader for that type.

Note that a loader can load to multiple types. This is the function to override if you want to add a new type to a data loader.

Note if you have any specific requirements for loading types (generic/whatnot), you can override applies_to as well, but it will make it much harder to document/determine what is happening.

Returns:

classmethod applies_to(type_: Type[Type]) → bool¶

Tells whether or not this data loader can load to a specific type. For instance, a CSV data loader might be able to load to a dataframe, a json, but not an integer.

I.e. is the adapter type a subclass of the passed in type?

This is a classmethod as it will be easier to validate, and we have to construct this, delayed, with a factory.

Parameters:: type – Candidate type
Returns:: True if this data loader can load to the type, False otherwise.

classmethod can_load() → bool¶

Returns whether this adapter can “load” data. Subclasses are meant to implement this function to tell the framework what to do with them.

Returns:

classmethod can_save() → bool¶

Returns whether this adapter can “save” data. Subclasses are meant to implement this function to tell the framework what to do with them.

Returns:

classmethod get_optional_arguments() → Dict[str, Type[Type]]¶

Gives the optional arguments for the class. Note that this just uses the type hints from the dataclass.

Returns:: The optional arguments for the class.

classmethod get_required_arguments() → Dict[str, Type[Type]]¶

Gives the required arguments for the class. Note that this just uses the type hints from the dataclass.

Returns:: The required arguments for the class.

abstract load_data(type_: Type[Type]) → Tuple[Type, Dict[str, Any]]¶

Loads the data from the data source. Note this uses the constructor parameters to determine how to load the data.

Returns:: The type specified

abstract classmethod name() → str¶

Returns the name of the data loader. This is used to register the data loader with the load_from decorator.

Returns:: The name of the data loader.

class hamilton.io.data_adapters.DataSaver¶

Base class for data savers. Data savers are used to save data to a data source. Note that they are inherently polymorphic – they declare what type(s) they can save from, and may choose to save differently depending on the type they are saving from.

abstract classmethod applicable_types() → Collection[Type]¶

Returns the types that this data loader can load to. These will be checked against the desired type to determine whether this is a suitable loader for that type.

Note that a loader can load to multiple types. This is the function to override if you want to add a new type to a data loader.

Note if you have any specific requirements for loading types (generic/whatnot), you can override applies_to as well, but it will make it much harder to document/determine what is happening.

Returns:

classmethod applies_to(type_: Type[Type]) → bool¶

Tells whether or not this data saver can ingest a specific type to save it.

I.e. is the adapter type a superclass of the passed in type?

This is a classmethod as it will be easier to validate, and we have to construct this, delayed, with a factory.

Parameters:: type – Candidate type
Returns:: True if this data saver can handle to the type, False otherwise.

classmethod can_load() → bool¶

Returns whether this adapter can “load” data. Subclasses are meant to implement this function to tell the framework what to do with them.

Returns:

classmethod can_save() → bool¶

Returns whether this adapter can “save” data. Subclasses are meant to implement this function to tell the framework what to do with them.

Returns:

classmethod get_optional_arguments() → Dict[str, Type[Type]]¶

Gives the optional arguments for the class. Note that this just uses the type hints from the dataclass.

Returns:: The optional arguments for the class.

classmethod get_required_arguments() → Dict[str, Type[Type]]¶

Gives the required arguments for the class. Note that this just uses the type hints from the dataclass.

Returns:: The required arguments for the class.

abstract classmethod name() → str¶

Returns the name of the data loader. This is used to register the data loader with the load_from decorator.

Returns:: The name of the data loader.

abstract save_data(data: Any) → Dict[str, Any]¶

Saves the data to the data source.: Note this uses the constructor parameters to determine how to save the data.

Returns:: Any relevant metadata. This is up the the data saver, but will likely include the URI, etc… This is going to be similar to the metadata returned by the data loader in the loading tuple.

class hamilton.io.data_adapters.AdapterCommon¶

abstract classmethod applicable_types() → Collection[Type]¶

Returns the types that this data loader can load to. These will be checked against the desired type to determine whether this is a suitable loader for that type.

Note that a loader can load to multiple types. This is the function to override if you want to add a new type to a data loader.

Note if you have any specific requirements for loading types (generic/whatnot), you can override applies_to as well, but it will make it much harder to document/determine what is happening.

Returns:

abstract classmethod applies_to(type_: Type[Type]) → bool¶

Tells whether or not this adapter applies to the given type.

Note: you need to understand the edge direction to properly determine applicability. For loading data, the loader type needs to be a subclass of the type being loaded into. For saving data, the saver type needs to be a superclass of the type being passed in.

This is a classmethod as it will be easier to validate, and we have to construct this, delayed, with a factory.

Parameters:: type – Candidate type
Returns:: True if this adapter can be used with that type, False otherwise.

classmethod can_load() → bool¶

Returns whether this adapter can “load” data. Subclasses are meant to implement this function to tell the framework what to do with them.

Returns:

classmethod can_save() → bool¶

Returns whether this adapter can “save” data. Subclasses are meant to implement this function to tell the framework what to do with them.

Returns:

classmethod get_optional_arguments() → Dict[str, Type[Type]]¶

Gives the optional arguments for the class. Note that this just uses the type hints from the dataclass.

Returns:: The optional arguments for the class.

classmethod get_required_arguments() → Dict[str, Type[Type]]¶

Gives the required arguments for the class. Note that this just uses the type hints from the dataclass.

Returns:: The required arguments for the class.

abstract classmethod name() → str¶

Returns the name of the data loader. This is used to register the data loader with the load_from decorator.

Returns:: The name of the data loader.