7.3.4. IO Module

io module

Helper functions to load or write files

class parflow.tools.io.DataAccessor(run, selector=None)

Bases: object

Helper for extracting numpy array from a given run

__init__(run, selector=None): Create DataAccessor from a Run instance

clm_forcing(name)

property clm_forcing_apcp: Precipitation rate [mm/s]

property clm_forcing_dlwr: Downward Infa-Red or Long-Wave radiation [W/m2]

property clm_forcing_dswr: Downward Visible or Short-Wave radiation [W/m2]

property clm_forcing_press: Atmospheric Pressure [pa]

property clm_forcing_spfh: Water-vapor specific humidity [kg/kg]

property clm_forcing_temp: Air temperature [K]

property clm_forcing_ugrd: West-to-East or U-component of wind [m/s]

property clm_forcing_vgrd: South-to-North or V-component of wind [m/s]

property clm_map_clay

property clm_map_color

clm_map_land_fraction(name)

property clm_map_latitude

property clm_map_longitude

property clm_map_sand

clm_output(field, layer=-1)

property clm_output_diagnostics

property clm_output_eflx_lh_tot

property clm_output_eflx_lwrad_out

property clm_output_eflx_sh_tot

property clm_output_eflx_soil_grnd

property clm_output_qflx_evap_grnd

property clm_output_qflx_evap_soi

property clm_output_qflx_evap_tot

property clm_output_qflx_evap_veg

property clm_output_qflx_infl

property clm_output_qflx_top_soil

property clm_output_qflx_tran_veg

property clm_output_swe_out

property clm_output_t_grnd

property clm_output_variables

property computed_permeability_x

property computed_permeability_y

property computed_permeability_z

property computed_porosity

property dx

property dy

property dz

property elevation

property et

property forcing_time

property mannings

property mask

overland_flow(flow_method='OverlandKinematic', epsilon=1e-05)

overland_flow_grid(flow_method='OverlandKinematic', epsilon=1e-05)

property pressure

property pressure_boundary_conditions

property pressure_initial_condition

property process_id

property saturation

property selector

property shape

property slope_x

property slope_y

property specific_storage

property subsurface_storage

property surface_storage

property time

property times

property wtd

class parflow.tools.io.ParflowBinaryReader(file: str, precompute_subgrid_info: bool = True, p: int = None, q: int = None, r: int = None, header: Mapping[str, Number] = None, read_sg_info: bool = False)

Bases: object

The ParflowBinaryReader, unsurprisingly, provides functionality for reading parflow binary files. It is designed to separate the header reading and metadata from subgrids from the reading of the underlying subgrid data in an efficient and flexible way. The ParflowBinaryReader only ever stores state about the header and/or subgrid headers. When reading data it is immediately returned to the user in the form of a numpy array. The ParflowBinaryReader implements a simple Context Manager so it is recommended to use with the standard idiom:

:: with ParflowBinaryReader(file) as pfb:

data = pfb.read_all_subgrids()

Parameters:

file – The pfb file to read
precompute_subgrid_info – Whether or not to precompute subgrid information. This defaults to True but can be turned off for reading multiple pfb files to reduce the amount of IO overhead when reading a sequence of pfb files. This computes the subgrid offset bytes, subgrid locations, subgrid indices, subgrid shapes, as well as subgrid coordinates and chunk sizes
p – The number of subgrids along the x dimension. This is an optional input, if it is not given we will try to precompute it.
q – The number of subgrids along the y dimension. This is an optional input, if it is not given we will try to precompute it.
r – The number of subgrids along the z dimension. This is an optional input, if it is not given we will try to precompute it.
header – A dictionary representing the header of the pfb file. This is an optional input, if it is not given we will read it from the pfb file directly.
read_sg_info – Whether or not to read subgrid information directly from the pfb file. Precalculating subgrid information does not always work correctly for some files, especially velocity files. This reads the subgrid offset bytes, subgrid indices and subgrid shapes, then computes the subgrid coordinates subgrid locations, and chunk sizes as normal

__init__(file: str, precompute_subgrid_info: bool = True, p: int = None, q: int = None, r: int = None, header: Mapping[str, Number] = None, read_sg_info: bool = False)

close()

compute_subgrid_info(): Computes the subgrid information

iloc_subgrid(idx: int) → ndarray

Read a subgrid at some scalar index.

Parameters:: idx – The index of the subgrid to read
Returns:: The data from the idx’th subgrid.

loc_subgrid(sg_p: int, sg_q: int, sg_r: int) → ndarray

Read a subgrid given it’s (sg_p, sg_q, sg_r) coordinate in the subgrid-grid.

Parameters:

sg_p – Index in the p subgrid to read.
sg_q – Index in the q subgrid to read.
sg_r – Index in the r subgrid to read.

Returns:

The data from the (sg_p, sg_q, sg_r)’th subgrid.

read_all_subgrids(mode: str = 'full', z_first: bool = True) → Iterable[ndarray] | ndarray

Read all of the subgrids in the file.

Parameters:

mode – Specifies how to arange the data from the subgrids before returning.
z_first – Whether the z dimension should be first. If true returned arrays have dimensions (‘z’, ‘y’, ‘x’) else (‘x’, ‘y’, ‘z’)

Returns:

A numpy array or iterable of numpy arrays, depending on how mode is set. If full the returned array will be of dimensions (nx, ny, nz). If flat the returned data will be a list of each of the subgrid arrays. If tiled the returned data will be a numpy array with dimensions (p, q, r) where each index of the array contains the subgrid data which also will be numpy array of floats with dimensions (sg_nx, sg_ny, sg_nz) where each of sg_nx, sg_ny, and sg_nz are the size of the subgrid array.

read_header(): Reads the header

read_subarray(start_x: int, start_y: int, start_z: int = 0, nx: int = 1, ny: int = 1, nz: int = None, z_first: bool = True) → ndarray

Read a subsection of the full pfb file. For an example of what happens here consider the following image:

:: +——-+——-+ | | | | x|xx | +——-+——-+ | x|xx | | x|xx | +——-+——-+

Where each of the borders of the big grid are the four subgrids (2,2) that we are trying to index data from. The data to be selected falls in each of these subgrids, as denoted by the ‘x’ marks.

Parameters:

start_x – The index to start at in the x dimension.
start_y – The index to start at in the y dimension.
start_z – The index to start at in the z dimension. This is optional, and if not provided is 0.
nx – The number of values to read in the x dimension. This is optional, and if not provided is 1.
ny – The number of values to read in the y dimension. This is optional, and if not provided is 1.
nz – The number of values to read in the z dimension. This is optional, and if not provided is None, which indicates to read all of the values.
z_first – Whether the z dimension should be first. If true returned arrays have dimensions (‘z’, ‘y’, ‘x’) else (‘x’, ‘y’, ‘z’)

Returns:

A nd array with shape (nx, ny, nz).

read_subgrid_header(skip_bytes: int = 64): Reads a subgrid header at the position skip_bytes

read_subgrid_info(): Read the header for each subgrid directly from the pfb file, rather than calculating it via precalculate_subgrid_info

parflow.tools.io.get_maingrid_and_remainder(nx, ny, nz, p, q, r)

Determines the sizes of the subgrids. Maingrid sizes are simprm_ny the integer value of the number of cells divided by the number of subgrids along each axis. The remainder is the modulus.

Parameters:

nx – The length of the array along the x axis.
ny – The length of the array along the y axis.
nz – The length of the array along the z axis.
p – The number of subgrids along the x axis.
q – The number of subgrids along the y axis.
r – The number of subgrids along the z axis.

parflow.tools.io.get_subgrid_loc(sel_subgrid, p, q, r)

Translate an integer subgrid to the location in 3d subgrid space.

Parameters:

sel_subgrid – The scalar index of the subgrid of interest.
p – The number of subgrids along the x axis.
q – The number of subgrids along the y axis.
r – The number of subgrids along the z axis.

parflow.tools.io.load_patch_matrix_from_asc_file(file_name)

parflow.tools.io.load_patch_matrix_from_image_file(file_name, color_to_patch=None, fall_back_id=0)

parflow.tools.io.load_patch_matrix_from_sa_file(file_name)

parflow.tools.io.precalculate_subgrid_info(nx, ny, nz, p, q, r)

Computes all necessary subgrid information to index and read or write a pfb file.

Parameters:

nx – Number of cells along the x-axis.
ny – Number of cells along the y-axis.
nz – Number of cells along the z-axis.
p – Number of subgrids along the x-axis.
q – Number of subgrids along the y-axis.
r – Number of subgrids along the z-axis.

Returns:

A tuple of arrays, (: subgrid_offsets, subgid_locations, subgrid_lower_left_indices, subgrid_shapes

)

parflow.tools.io.read_clm(file_name, type='clmin')

parflow.tools.io.read_pfb(file: str, keys: dict = None, mode: str = 'full', z_first: bool = True, read_sg_info: bool = True)

Read a single pfb file, and return the data therein

Parameters:

file – The file to read.
keys –
A set of keys for indexing subarrays of the full pfb. Optional. This is mainly a trick for interfacing with xarray, but the format of the keys is:

:: {‘x’: {‘start’: start_x, ‘stop’: end_x},

’y’: {‘start’: start_y, ‘stop’: end_y}, ‘z’: {‘start’: start_z, ‘stop’: end_z}}
mode – The mode for the reader. See ParflowBinaryReader::read_all_subgrids for more information about what modes are available.
read_sg_info – Precalculating subgrid information does not always work correctly for some files, especially velocity files. If True, read subgrid info directly from the pfb file (slower, but more robust)

Returns:

An nd array containing the data from the pfb file.

parflow.tools.io.read_pfb_sequence(file_seq: Iterable[str], keys=None, z_first: bool = True, z_is: str = 'z', read_sg_info: bool = False)

An efficient wrapper to read a sequence of pfb files. This approach is faster than looping over the read_pfb function because it caches the subgrid information from the first pfb file and then uses that to initialize all other readers.

Parameters:

file_seq – An iterable sequence of file names to be read.
keys –
A set of keys for indexing subarrays of the full pfb. Optional. This is mainly a trick for interfacing with xarray, but the format of the keys is:

:: {‘x’: {‘start’: start_x, ‘stop’: end_x},

’y’: {‘start’: start_y, ‘stop’: end_y}, ‘z’: {‘start’: start_z, ‘stop’: end_z}}
z_first – Whether the z dimension should be first. If true returned arrays have dimensions (‘z’, ‘y’, ‘x’) else (‘x’, ‘y’, ‘z’)
z_is – A descriptor of what the z axis represents. Can be one of ‘z’, ‘time’, ‘variable’. Default is ‘z’.
read_sg_info – Precalculating subgrid information does not always work correctly for some files, especially velocity files. If True, read subgrid info directly from the pfb file (slower, but more robust)

Returns:

An nd array containing the data from the files.

parflow.tools.io.read_pfidb(file_path): Load pfidb file into a Python dict

parflow.tools.io.read_pfsb(file_path)

Read a ParFlow scattered binary file (pfsb) and return the data.

The data will be returned as a numpy array of shape (nz, ny, nx). To avoid confusion, there is no option to toggle ‘z_first’ on and off.

parflow.tools.io.read_yaml(file_path): Load yaml file into a Python dict

parflow.tools.io.subgrid_lower_left(mg_nx, mg_ny, mg_nz, sg_p, sg_q, sg_r, rm_nx, rm_ny, rm_nz)

Get the index of the lower left corner of a subgrid.

Parameters:

mg_nx – The standard length of a subgrid on the x-axis
mg_ny – The standard length of a subgrid on the y-axis
mg_nz – The standard length of a subgrid on the z-axis
sg_p – The index of the subgrid on the x-axis.
sg_q – The index of the subgrid on the y-axis.
sg_r – The index of the subgrid on the z-axis.
rm_nx – Remainder from the maingrid calculation on the x-axis
rm_ny – Remainder from the maingrid calculation on the y-axis
rm_nz – Remainder from the maingrid calculation on the z-axis

parflow.tools.io.subgrid_size(mg_nx, mg_ny, mg_nz, sg_p, sg_q, sg_r, rm_nx, rm_ny, rm_nz)

Get the size of a subgrid

Parameters:

mg_nx – The standard length of a subgrid on the x-axis
mg_ny – The standard length of a subgrid on the y-axis
mg_nz – The standard length of a subgrid on the z-axis
sg_p – The index of the subgrid on the x-axis.
sg_q – The index of the subgrid on the y-axis.
sg_r – The index of the subgrid on the z-axis.
rm_nx – Remainder from the maingrid calculation on the x-axis
rm_ny – Remainder from the maingrid calculation on the y-axis
rm_nz – Remainder from the maingrid calculation on the z-axis

parflow.tools.io.to_native_type(string): Converting a string to a value in native format. Used for converting .pfidb files

parflow.tools.io.undist(undis_name: str)

Undistribute PFB files. Distributed PFB files can be in two formats. The split file model creates a file per rank; each rank file has the rank index appended to the fileneme. The sequential model writes all data to a single file and creates and index file (.dist extension) to aid in locating offsets.

The Python interface currently can’t create the split file model but this should undistribute the split files, which is just a cat operation.

Parameters:: undis_name – This can be the full undis_name used, in this case all files

will be undistributed. Also a specific PFB file can be specified.

parflow.tools.io.write_dict(dict_obj, file_name): Write a Python dict into a file_name using the extension to determine its format.

parflow.tools.io.write_dict_as_json(dict_obj, file_name): Write a Python dict in a json format inside the provided file_name

parflow.tools.io.write_dict_as_pfidb(dict_obj, file_name): Write a Python dict in a pfidb format inside the provided file_name

parflow.tools.io.write_dict_as_yaml(dict_obj, file_name): Write a Python dict in a pfidb format inside the provided file_name

parflow.tools.io.write_dist(file, sg_offs)

Write a distfile.

Parameters:

file – The path of the file to be written.
sg_offs – The subgrid offsets.

parflow.tools.io.write_patch_matrix_as_asc(matrix, file_name, xllcorner=0.0, yllcorner=0.0, cellsize=1.0, NODATA_value=0, **kwargs): Write asc for pfsol

parflow.tools.io.write_patch_matrix_as_sa(matrix, file_name, **kwargs): Write asc for pfsol

parflow.tools.io.write_pfb(file, array, p=1, q=1, r=1, x=0.0, y=0.0, z=0.0, dx=1.0, dy=1.0, dz=1.0, z_first=True, dist=True, **kwargs)

Write a single pfb file. The data must be a 3D numpy array with float64 values. The number of subgrids in the saved file will be p * q * r. This is regardless of the number of subgrids in the PFB file loaded by the ParflowBinaryReader into the numpy array. Therefore, loading a file with ParflowBinaryReader and saving it with this method may restructure the file into a different number of subgrids if you change these values.

If dist is True then also write a file with the .dist extension added to the file_name. The .dist file will contain one line per subgrid with the offset of the subgrid in the .pfb file.

Parameters:

file – The name of the file to write the array to.
array – The array to write.
p – Number of subgrids in the x direction.
q – Number of subgrids in the y direction.
r – Number of subgrids in the z direction.
x – The length of the x-axis
y – The length of the y-axis
z – The length of the z-axis
dx – The spacing between cells in the x direction
dy – The spacing between cells in the y direction
dz – The spacing between cells in the z direction
z_first – Whether the z-axis should be first or last.
dist – Whether to write the distfile in addition to the pfb.
kwargs – Extra keyword arguments, primarily to eat unnecessary args by passing in a dictionary with **dict.