mapchete_eo.io package

Submodules

mapchete_eo.io.assets module

class mapchete_eo.io.assets.STACRasterBandProperties(*, nodata: float | None = None, data_type: str | None = None, scale: float = 1.0, offset: float = 0.0)

Bases: BaseModel

data_type: str | None

static from_asset(asset: Asset, nodataval: float | None = None) → STACRasterBandProperties

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

nodata: NodataVal | None

offset: float

scale: float

mapchete_eo.io.assets.asset_to_np_array(item: Item, asset: str, indexes: List[int] | int = 1, grid: GridProtocol | None = None, resampling: Resampling = Resampling.nearest, nodataval: float | None = None, apply_offset: bool = True) → MaskedArray

Read grid window of STAC Items and merge into a 2D ma.MaskedArray.

This is the main read method which is one way or the other being called from everywhere whenever a band is being read!

mapchete_eo.io.assets.convert_asset(item: Item, asset: str, dst_dir: MPath, src_fs: AbstractFileSystem = None, overwrite: bool = False, resolution: None | float | int = None, profile: Profile | None = None, item_href_in_dst_dir: bool = True, ignore_if_exists: bool = False) → Item

Convert asset to a different format.

mapchete_eo.io.assets.convert_raster(src_path: MPath, dst_path: MPath, resolution: None | float | int = None, profile: Profile | None = None) → None

mapchete_eo.io.assets.copy_asset(item: Item, asset: str, dst_dir: MPath, src_fs: AbstractFileSystem = None, overwrite: bool = False, item_href_in_dst_dir: bool = True, ignore_if_exists: bool = False) → Item

Copy asset from one place to another.

mapchete_eo.io.assets.get_assets(item: Item, assets: List[str], dst_dir: MPath, src_fs: AbstractFileSystem = None, overwrite: bool = False, resolution: None | float | int = None, convert_profile: Profile | None = None, item_href_in_dst_dir: bool = True, ignore_if_exists: bool = False) → Item

Copy or convert assets depending on settings.

Conversion is triggered if either resolution or convert_profile is provided.

mapchete_eo.io.assets.get_metadata_assets(item: Item, dst_dir: MPath, overwrite: bool = False, metadata_parser_classes: tuple | None = None, resolution: None | float | int = None, convert_profile: Profile | None = None, metadata_asset_names: List[str] = ['metadata', 'granule_metadata'])

Copy STAC item metadata and its metadata assets.

mapchete_eo.io.assets.read_mask_as_raster(path: ~mapchete.path.MPath, indexes: ~typing.List[int] | None = None, dst_grid: ~mapchete.protocols.GridProtocol | None = None, resampling: ~rasterio.enums.Resampling = Resampling.nearest, rasterize_value_func: ~typing.Callable = <function <lambda>>, rasterize_feature_filter: ~typing.Callable = <function <lambda>>, dtype: ~numpy.dtype[~typing.Any] | None | type[~typing.Any] | ~numpy._typing._dtype_like._SupportsDType[~numpy.dtype[~typing.Any]] | str | tuple[~typing.Any, int] | tuple[~typing.Any, ~typing.SupportsIndex | ~collections.abc.Sequence[~typing.SupportsIndex]] | list[~typing.Any] | ~numpy._typing._dtype_like._DTypeDict | tuple[~typing.Any, ~typing.Any] = None, masked: bool = True, cached_read: bool = False) → ReferencedRaster

Read mask as array regardless of source data type (raster or vector).

mapchete_eo.io.assets.should_be_converted(path: MPath, resolution: None | float | int = None, profile: Profile | None = None) → bool

Decide whether a raster file should be converted or not.

mapchete_eo.io.items module

mapchete_eo.io.items.expand_params(param: Any, length: int) → List[Any]

Expand parameters if they are not a list.

mapchete_eo.io.items.get_item_property(item: Item, property: str | Tuple[str, ...], default: Any = None) → Any

Return item property.

A valid property can be a special property like “year” from the items datetime property or any key in the item properties or extra_fields.

Search order of properties is based on the pystac LayoutTemplate search order:

https://pystac.readthedocs.io/en/stable/_modules/pystac/layout.html#LayoutTemplate - The object’s attributes - Keys in the properties attribute, if it exists. - Keys in the extra_fields attribute, if it exists.

Some special keys can be used in template variables:

Template variable	Meaning
year	The year of an Item’s datetime, or start_datetime if datetime is null
month	The month of an Item’s datetime, or start_datetime if datetime is null
day	The day of an Item’s datetime, or start_datetime if datetime is null
date	The date (iso format) of an Item’s datetime, or start_datetime if datetime is null
collection	The collection ID of an Item’s collection.

mapchete_eo.io.items.item_fix_footprint(item: Item, bbox_width_threshold: float = 180.0) → Item

mapchete_eo.io.items.item_to_np_array(item: Item, band_locations: List[BandLocation], grid: GridProtocol | None = None, resampling: Resampling = Resampling.nearest, nodatavals: List[float | None] | float | None = None, raise_empty: bool = False, apply_offset: bool = True) → MaskedArray

Read window of STAC Item and merge into a 3D ma.MaskedArray.

mapchete_eo.io.levelled_cubes module

mapchete_eo.io.levelled_cubes.read_levelled_cube_to_np_array(products: ~typing.List[~mapchete_eo.protocols.EOProductProtocol], target_height: int, grid: ~mapchete.protocols.GridProtocol, assets: ~typing.List[str] | None = None, eo_bands: ~typing.List[str] | None = None, resampling: ~rasterio.enums.Resampling = Resampling.nearest, nodatavals: ~typing.List[float | None] | float | None = None, merge_products_by: str | None = None, merge_method: ~mapchete_eo.types.MergeMethod = MergeMethod.first, sort: ~mapchete_eo.sort.SortMethodConfig = TargetDateSort(func=<function sort_objects_by_target_date>, target_date=None, reverse=False), product_read_kwargs: dict = {}, raise_empty: bool = True, out_dtype: ~numpy.dtype[~typing.Any] | None | type[~typing.Any] | ~numpy._typing._dtype_like._SupportsDType[~numpy.dtype[~typing.Any]] | str | tuple[~typing.Any, int] | tuple[~typing.Any, ~typing.SupportsIndex | ~collections.abc.Sequence[~typing.SupportsIndex]] | list[~typing.Any] | ~numpy._typing._dtype_like._DTypeDict | tuple[~typing.Any, ~typing.Any] = <class 'numpy.uint16'>, out_fill_value: float | None = 0, read_mask: ~numpy.ndarray | None = None) → MaskedArray

Read products as slices into a cube by filling up nodata gaps with next slice.

If a read_mask is provided, only the pixels marked True are considered to be read.

mapchete_eo.io.levelled_cubes.read_levelled_cube_to_xarray(products: ~typing.List[~mapchete_eo.protocols.EOProductProtocol], target_height: int, assets: ~typing.List[str] | None = None, eo_bands: ~typing.List[str] | None = None, grid: ~mapchete.protocols.GridProtocol | None = None, resampling: ~rasterio.enums.Resampling = Resampling.nearest, nodatavals: ~typing.List[float | None] | float | None = None, merge_products_by: str | None = None, merge_method: ~mapchete_eo.types.MergeMethod = MergeMethod.first, sort: ~mapchete_eo.sort.SortMethodConfig = TargetDateSort(func=<function sort_objects_by_target_date>, target_date=None, reverse=False), product_read_kwargs: dict = {}, raise_empty: bool = True, slice_axis_name: str = 'layers', band_axis_name: str = 'bands', x_axis_name: str = 'x', y_axis_name: str = 'y', read_mask: ~numpy.ndarray | None = None) → Dataset

Read products as slices into a cube by filling up nodata gaps with next slice.

mapchete_eo.io.path module

class mapchete_eo.io.path.ProductPathGenerationMethod(*values)

Bases: str, Enum

Option to generate product cache path.

date_day_first = 'date_day_first'

date_year_first = 'date_year_first'

hash = 'hash'

product_id = 'product_id'

mapchete_eo.io.path.asset_mpath(item: Item, asset: str | Tuple[str, ...], fs: AbstractFileSystem = None, absolute_path: bool = True) → MPath

Return MPath instance with asset href.

mapchete_eo.io.path.cached_path(path: MPath, active: bool = True) → Generator[MPath, None, None]

If path is remote, download to temporary directory and return path.

mapchete_eo.io.path.get_product_cache_path(item: Item, basepath: MPath, path_generation_method: ProductPathGenerationMethod = ProductPathGenerationMethod.product_id) → MPath

Create product path with high cardinality prefixes optimized for S3.

product_path_generation option:

“product_id”: <cache_basepath>/<product-id>

“product_hash”: <cache_basepath>/<product-hash>

“date_day_first”: <cache_basepath>/<product-day>/<product-month>/<product-year>/<product-id>

“date_year_first”: <cache_basepath>/<product-year>/<product-month>/<product-day>/<product-id>

mapchete_eo.io.path.open_xml(path: MPath) → Element

Parse an XML file path into an etree root element.

mapchete_eo.io.path.path_in_paths(path, existing_paths) → bool

Check if path is contained in list of existing paths independent of path prefix.

mapchete_eo.io.products module

class mapchete_eo.io.products.Slice(name: Any, products: Sequence[EOProductProtocol])

Bases: object

Combine multiple products into one slice.

cached() → Generator[Slice, None, None]

Clear caches and run garbage collector when context manager is closed.

clear_cached_data()

datetime: datetime

get_property(property: str) → Any

Return merged property over all products.

If property values are the same over all products, it will be returned. Otherwise a ValueError is raised.

name: Any

products: Sequence[EOProductProtocol]

property properties: Dict[str, Any]

read(merge_method: MergeMethod = MergeMethod.first, product_read_kwargs: dict = {}, raise_empty: bool = True) → MaskedArray

mapchete_eo.io.products.generate_slice_dataarrays(products: List[EOProductProtocol], assets: List[str] | None = None, eo_bands: List[str] | None = None, grid: GridProtocol | None = None, resampling: Resampling = Resampling.nearest, nodatavals: List[float | None] | float | None = None, merge_products_by: str | None = None, merge_method: MergeMethod = MergeMethod.first, sort: SortMethodConfig | None = None, product_read_kwargs: dict = {}, raise_empty: bool = True, read_mask: ndarray | None = None) → Iterator[DataArray]

Yield products or merged products into slices as DataArrays.

mapchete_eo.io.products.merge_products(products: Sequence[EOProductProtocol], merge_method: MergeMethod = MergeMethod.first, product_read_kwargs: dict = {}, raise_empty: bool = True) → MaskedArray

Merge given products into one array.

mapchete_eo.io.products.products_to_np_array(products: List[EOProductProtocol], assets: List[str] | None = None, eo_bands: List[str] | None = None, grid: GridProtocol | None = None, resampling: Resampling = Resampling.nearest, nodatavals: List[float | None] | float | None = None, merge_products_by: str | None = None, merge_method: MergeMethod = MergeMethod.first, sort: SortMethodConfig | None = None, product_read_kwargs: dict = {}, raise_empty: bool = True, out_dtype: dtype[Any] | None | type[Any] | _SupportsDType[dtype[Any]] | str | tuple[Any, int] | tuple[Any, SupportsIndex | Sequence[SupportsIndex]] | list[Any] | _DTypeDict | tuple[Any, Any] = None, read_mask: ndarray | None = None) → MaskedArray

Read grid window of EOProducts and merge into a 4D xarray.

mapchete_eo.io.products.products_to_slices(products: List[EOProductProtocol], group_by_property: str | None = None, sort: SortMethodConfig | None = None) → List[Slice]

Group products per given property into Slice objects and optionally sort slices.

mapchete_eo.io.products.products_to_xarray(products: List[EOProductProtocol], assets: List[str] | None = None, eo_bands: List[str] | None = None, grid: GridProtocol | None = None, resampling: Resampling = Resampling.nearest, nodatavals: List[float | None] | float | None = None, slice_axis_name: str = 'time', band_axis_name: str = 'bands', x_axis_name: str = 'x', y_axis_name: str = 'y', merge_products_by: str | None = None, merge_method: MergeMethod = MergeMethod.first, sort: SortMethodConfig | None = None, raise_empty: bool = True, product_read_kwargs: dict = {}, read_mask: ndarray | None = None) → Dataset

Read grid window of EOProducts and merge into a 4D xarray.

mapchete_eo.io.profiles module

class mapchete_eo.io.profiles.COGDeflateProfile(data={}, **kwds)

Bases: Profile

Standard COG profile.

defaults = {'blockxsize': 512, 'blockysize': 512, 'compress': 'DEFLATE', 'driver': 'COG', 'tiled': True}

class mapchete_eo.io.profiles.JP2LosslessProfile(data={}, **kwds)

Bases: Profile

Lossless JP2 profile used for lower size data.

defaults = {'blockxsize': 512, 'blockysize': 512, 'driver': 'JP2OpenJPEG', 'quality': 100, 'reversible': True, 'tiled': True}

class mapchete_eo.io.profiles.JP2LossyProfile(data={}, **kwds)

Bases: Profile

Very lossy JP2 profile used for low size test data.

defaults = {'blockxsize': 512, 'blockysize': 512, 'driver': 'JP2OpenJPEG', 'quality': 50, 'tiled': True}

Module contents

mapchete_eo.io.convert_asset(item: Item, asset: str, dst_dir: MPath, src_fs: AbstractFileSystem = None, overwrite: bool = False, resolution: None | float | int = None, profile: Profile | None = None, item_href_in_dst_dir: bool = True, ignore_if_exists: bool = False) → Item

Convert asset to a different format.

mapchete_eo.io.copy_asset(item: Item, asset: str, dst_dir: MPath, src_fs: AbstractFileSystem = None, overwrite: bool = False, item_href_in_dst_dir: bool = True, ignore_if_exists: bool = False) → Item

Copy asset from one place to another.

mapchete_eo.io.get_assets(item: Item, assets: List[str], dst_dir: MPath, src_fs: AbstractFileSystem = None, overwrite: bool = False, resolution: None | float | int = None, convert_profile: Profile | None = None, item_href_in_dst_dir: bool = True, ignore_if_exists: bool = False) → Item

Copy or convert assets depending on settings.

Conversion is triggered if either resolution or convert_profile is provided.

mapchete_eo.io.get_item_property(item: Item, property: str | Tuple[str, ...], default: Any = None) → Any

Return item property.

A valid property can be a special property like “year” from the items datetime property or any key in the item properties or extra_fields.

Search order of properties is based on the pystac LayoutTemplate search order:

https://pystac.readthedocs.io/en/stable/_modules/pystac/layout.html#LayoutTemplate - The object’s attributes - Keys in the properties attribute, if it exists. - Keys in the extra_fields attribute, if it exists.

Some special keys can be used in template variables:

Template variable	Meaning
year	The year of an Item’s datetime, or start_datetime if datetime is null
month	The month of an Item’s datetime, or start_datetime if datetime is null
day	The day of an Item’s datetime, or start_datetime if datetime is null
date	The date (iso format) of an Item’s datetime, or start_datetime if datetime is null
collection	The collection ID of an Item’s collection.

mapchete_eo.io.get_product_cache_path(item: Item, basepath: MPath, path_generation_method: ProductPathGenerationMethod = ProductPathGenerationMethod.product_id) → MPath

Create product path with high cardinality prefixes optimized for S3.

product_path_generation option:

“product_id”: <cache_basepath>/<product-id>

“product_hash”: <cache_basepath>/<product-hash>

“date_day_first”: <cache_basepath>/<product-day>/<product-month>/<product-year>/<product-id>

“date_year_first”: <cache_basepath>/<product-year>/<product-month>/<product-day>/<product-id>

mapchete_eo.io.item_fix_footprint(item: Item, bbox_width_threshold: float = 180.0) → Item

mapchete_eo.io.item_to_np_array(item: Item, band_locations: List[BandLocation], grid: GridProtocol | None = None, resampling: Resampling = Resampling.nearest, nodatavals: List[float | None] | float | None = None, raise_empty: bool = False, apply_offset: bool = True) → MaskedArray

Read window of STAC Item and merge into a 3D ma.MaskedArray.

mapchete_eo.io.merge_products(products: Sequence[EOProductProtocol], merge_method: MergeMethod = MergeMethod.first, product_read_kwargs: dict = {}, raise_empty: bool = True) → MaskedArray

Merge given products into one array.

mapchete_eo.io.open_xml(path: MPath) → Element

Parse an XML file path into an etree root element.

mapchete_eo.io.path_in_paths(path, existing_paths) → bool

Check if path is contained in list of existing paths independent of path prefix.

mapchete_eo.io.products_to_np_array(products: List[EOProductProtocol], assets: List[str] | None = None, eo_bands: List[str] | None = None, grid: GridProtocol | None = None, resampling: Resampling = Resampling.nearest, nodatavals: List[float | None] | float | None = None, merge_products_by: str | None = None, merge_method: MergeMethod = MergeMethod.first, sort: SortMethodConfig | None = None, product_read_kwargs: dict = {}, raise_empty: bool = True, out_dtype: dtype[Any] | None | type[Any] | _SupportsDType[dtype[Any]] | str | tuple[Any, int] | tuple[Any, SupportsIndex | Sequence[SupportsIndex]] | list[Any] | _DTypeDict | tuple[Any, Any] = None, read_mask: ndarray | None = None) → MaskedArray

Read grid window of EOProducts and merge into a 4D xarray.

mapchete_eo.io.products_to_slices(products: List[EOProductProtocol], group_by_property: str | None = None, sort: SortMethodConfig | None = None) → List[Slice]

Group products per given property into Slice objects and optionally sort slices.

mapchete_eo.io.products_to_xarray(products: List[EOProductProtocol], assets: List[str] | None = None, eo_bands: List[str] | None = None, grid: GridProtocol | None = None, resampling: Resampling = Resampling.nearest, nodatavals: List[float | None] | float | None = None, slice_axis_name: str = 'time', band_axis_name: str = 'bands', x_axis_name: str = 'x', y_axis_name: str = 'y', merge_products_by: str | None = None, merge_method: MergeMethod = MergeMethod.first, sort: SortMethodConfig | None = None, raise_empty: bool = True, product_read_kwargs: dict = {}, read_mask: ndarray | None = None) → Dataset

Read grid window of EOProducts and merge into a 4D xarray.

mapchete_eo.io.read_levelled_cube_to_np_array(products: ~typing.List[~mapchete_eo.protocols.EOProductProtocol], target_height: int, grid: ~mapchete.protocols.GridProtocol, assets: ~typing.List[str] | None = None, eo_bands: ~typing.List[str] | None = None, resampling: ~rasterio.enums.Resampling = Resampling.nearest, nodatavals: ~typing.List[float | None] | float | None = None, merge_products_by: str | None = None, merge_method: ~mapchete_eo.types.MergeMethod = MergeMethod.first, sort: ~mapchete_eo.sort.SortMethodConfig = TargetDateSort(func=<function sort_objects_by_target_date>, target_date=None, reverse=False), product_read_kwargs: dict = {}, raise_empty: bool = True, out_dtype: ~numpy.dtype[~typing.Any] | None | type[~typing.Any] | ~numpy._typing._dtype_like._SupportsDType[~numpy.dtype[~typing.Any]] | str | tuple[~typing.Any, int] | tuple[~typing.Any, ~typing.SupportsIndex | ~collections.abc.Sequence[~typing.SupportsIndex]] | list[~typing.Any] | ~numpy._typing._dtype_like._DTypeDict | tuple[~typing.Any, ~typing.Any] = <class 'numpy.uint16'>, out_fill_value: float | None = 0, read_mask: ~numpy.ndarray | None = None) → MaskedArray

Read products as slices into a cube by filling up nodata gaps with next slice.

If a read_mask is provided, only the pixels marked True are considered to be read.

mapchete_eo.io.read_levelled_cube_to_xarray(products: ~typing.List[~mapchete_eo.protocols.EOProductProtocol], target_height: int, assets: ~typing.List[str] | None = None, eo_bands: ~typing.List[str] | None = None, grid: ~mapchete.protocols.GridProtocol | None = None, resampling: ~rasterio.enums.Resampling = Resampling.nearest, nodatavals: ~typing.List[float | None] | float | None = None, merge_products_by: str | None = None, merge_method: ~mapchete_eo.types.MergeMethod = MergeMethod.first, sort: ~mapchete_eo.sort.SortMethodConfig = TargetDateSort(func=<function sort_objects_by_target_date>, target_date=None, reverse=False), product_read_kwargs: dict = {}, raise_empty: bool = True, slice_axis_name: str = 'layers', band_axis_name: str = 'bands', x_axis_name: str = 'x', y_axis_name: str = 'y', read_mask: ~numpy.ndarray | None = None) → Dataset

Read products as slices into a cube by filling up nodata gaps with next slice.

mapchete_eo.io.read_mask_as_raster(path: ~mapchete.path.MPath, indexes: ~typing.List[int] | None = None, dst_grid: ~mapchete.protocols.GridProtocol | None = None, resampling: ~rasterio.enums.Resampling = Resampling.nearest, rasterize_value_func: ~typing.Callable = <function <lambda>>, rasterize_feature_filter: ~typing.Callable = <function <lambda>>, dtype: ~numpy.dtype[~typing.Any] | None | type[~typing.Any] | ~numpy._typing._dtype_like._SupportsDType[~numpy.dtype[~typing.Any]] | str | tuple[~typing.Any, int] | tuple[~typing.Any, ~typing.SupportsIndex | ~collections.abc.Sequence[~typing.SupportsIndex]] | list[~typing.Any] | ~numpy._typing._dtype_like._DTypeDict | tuple[~typing.Any, ~typing.Any] = None, masked: bool = True, cached_read: bool = False) → ReferencedRaster

Read mask as array regardless of source data type (raster or vector).