Loader mixin classes facilitate loading an external resource from the local filesystem or from the web into a file-like object. A loader is essentially just a class with
save() methods defined. The canonical example is
FileLoader, represented in its entirely below:
class FileLoader(BaseLoader): filename = None read_mode = 'r' write_mode = 'w+' def load(self): try: self.file = open(self.filename, self.read_mode) self.empty_file = False except IOError: self.file = StringIO() self.empty_file = True def save(self): file = open(self.filename, self.write_mode) self.dump(file) file.close()
As can be seen above, every
load() method should take no arguments, instead determining what to load based on properties on the class instance. (Remember that the BaseIO class provides a convenient method for setting class properties on initialization).
load() should set two properties on the class:
file, a file-like object that will be accessed by the parser
empty_file, a boolean indicating that the file was empty or nonexistent (used to short-circuit avoid parser errors)
To support file output, loaders should define a
save() method, which should prepare a file-like object for writing, call
self.dump() with the output file, and perform any needed wrap-up operations.
There are six built-in loader classes defined in wq.io.loaders.
||Loads text data from a local file (e.g a CSV or XML file). Expects a
||Loads binary data from the local filesystem (e.g. an Excel spreadsheet). Expects a
||Opens a local zip file and extracts a single inner file. If there is more than one inner file, the
||Loads data to and from a
||Loads data over HTTP(S) and expects a
||Loads a zip file over HTTP(S) and extracts a single inner file. If there is more than one inner file, the
ZipNetLoader support optional HTTP
password, and URL
params properties. Note that
save() is not implemented for these loaders.
The built in loaders should be enough for many use cases. The most common use for a custom loader is to encapsulate a number of
NetLoader options into a reusable mixin class. For example, the climata library defines a
WebserviceLoader for this purpose.