iterableinterface for working with the parsed data (with support from a mapper mixin)
To accomplish these functions, BaseIO contains a number of methods and properties:
||Triggers the load and parse mixins to ensure the dataset is ready for iteration. Called automatically when the class is initialized.|
||Copy the entire dataset to another IO instance, which presumably uses a different loader or parser. This method provides a means of converting data between formats. Any existing data on the other IO instance will be erased. If
||Generates a Pandas DataFrame containing the data in the IO instance. Useful for more complex data analysis tasks. Requires Pandas which is not installed by default.|
||The field or column names in the dataset. This can usually be determined automatically.|
||A "primary key" on the dataset. If
||Boolean indicating whether the IO has a two-tiered API (see below).|
||Boolean indicating whether the dataset comes from an inherently tabular file format (e.g. a spreadsheet). See Parsers for more details.|
Most properties (including mixin properties) can be set by passing them as arguments when initializing the class. However, in general it is better to create a subclass with the properties pre-set.
# Works, but less re-usable instance = CustomIO(field_names=['id','name']) # Usually better class MyCustomIO(CustomIO) field_names = ['id', 'name'] instance = MyCustomIO()
The main exception to this rule is for properties that are almost guaranteed to be different every time the IO is instantiated, e.g. FileLoader's
wq.io supports the notion of "nested" IOs containing two levels of iteration. This is best illustrated by example:
instance = MyNestedIO(option1=value) for group in instance: print group.group_name for row in group.data: print row.date, row.value
For compatibility with tools that expect only a single level IO (e.g. the dbio module), nested IOs can be "flattened" using a function from
from wq.io.util import flattened instance = flattened(MyNestedIO, option1=value) for row in instance: print row.group_name, row.date, row.value
To be compatible with
flattened(), nested IOs need to have the following characteristics:
nested = True
3. Each mapped row should have a
data property pointing to a nested IO class instance.
Note that none of the pre-mixed IO classes in wq.io are nested. The climata library provides a number of examples of nested IO classes.