>> chunk
- array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
- 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
- 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
- 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
- 1, 1, 1, 1, 1, 1, 1, 1], dtype=int32)
-
-Modify the array attributes::
-
- >>> a.attrs['foo'] = 42
- >>> a.attrs['bar'] = 'apples'
- >>> a.attrs['baz'] = [1, 2, 3, 4]
- >>> sorted(os.listdir('data/example.zarr'))
- ['.zarray', '.zattrs', '0.0', '0.1', '1.0', '1.1']
- >>> print(open('data/example.zarr/.zattrs').read())
- {
- "bar": "apples",
- "baz": [
- 1,
- 2,
- 3,
- 4
- ],
- "foo": 42
- }
-
-Storing multiple arrays in a hierarchy
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Below is an example of storing multiple Zarr arrays organized into a group
-hierarchy, using a directory on the local file system as storage. This storage
-implementation maps logical paths onto directory paths on the file system,
-however this is an implementation choice and is not required.
-
-Setup the store::
-
- >>> import zarr
- >>> store = zarr.DirectoryStore('data/group.zarr')
-
-Create the root group::
-
- >>> root_grp = zarr.group(store, overwrite=True)
-
-The metadata resource for the root group has been created::
-
- >>> import os
- >>> sorted(os.listdir('data/group.zarr'))
- ['.zgroup']
-
-Inspect the group metadata::
-
- >>> print(open('data/group.zarr/.zgroup').read())
- {
- "zarr_format": 2
- }
-
-Create a sub-group::
-
- >>> sub_grp = root_grp.create_group('foo')
-
-What has been stored::
-
- >>> sorted(os.listdir('data/group.zarr'))
- ['.zgroup', 'foo']
- >>> sorted(os.listdir('data/group.zarr/foo'))
- ['.zgroup']
-
-Create an array within the sub-group::
-
- >>> a = sub_grp.create_dataset('bar', shape=(20, 20), chunks=(10, 10))
- >>> a[:] = 42
-
-Set a custom attributes::
-
- >>> a.attrs['comment'] = 'answer to life, the universe and everything'
-
-What has been stored::
-
- >>> sorted(os.listdir('data/group.zarr'))
- ['.zgroup', 'foo']
- >>> sorted(os.listdir('data/group.zarr/foo'))
- ['.zgroup', 'bar']
- >>> sorted(os.listdir('data/group.zarr/foo/bar'))
- ['.zarray', '.zattrs', '0.0', '0.1', '1.0', '1.1']
-
-Here is the same example using a Zip file as storage::
-
- >>> store = zarr.ZipStore('data/group.zip', mode='w')
- >>> root_grp = zarr.group(store)
- >>> sub_grp = root_grp.create_group('foo')
- >>> a = sub_grp.create_dataset('bar', shape=(20, 20), chunks=(10, 10))
- >>> a[:] = 42
- >>> a.attrs['comment'] = 'answer to life, the universe and everything'
- >>> store.close()
-
-What has been stored::
-
- >>> import zipfile
- >>> zf = zipfile.ZipFile('data/group.zip', mode='r')
- >>> for name in sorted(zf.namelist()):
- ... print(name)
- .zgroup
- foo/.zgroup
- foo/bar/.zarray
- foo/bar/.zattrs
- foo/bar/0.0
- foo/bar/0.1
- foo/bar/1.0
- foo/bar/1.1
-
-.. _spec_v2_changes:
-
-Changes
--------
-
-Version 2 clarifications
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following changes have been made to the version 2 specification since it was
-initially published to clarify ambiguities and add some missing information.
-
-* The specification now describes how bytes fill values should be encoded and
- decoded for arrays with a fixed-length byte string data type (:issue:`165`,
- :issue:`176`).
-
-* The specification now clarifies that units must be specified for datetime64 and
- timedelta64 data types (:issue:`85`, :issue:`215`).
-
-* The specification now clarifies that the '.zattrs' key does not have to be present for
- either arrays or groups, and if absent then custom attributes should be treated as
- empty.
-
-* The specification now describes how structured datatypes with
- subarray shapes and/or with nested structured data types are encoded
- in array metadata (:issue:`111`, :issue:`296`).
-
-* Clarified the key/value pairs of custom attributes as "arbitrary" rather than
- "simple".
-
-Changes from version 1 to version 2
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The following changes were made between version 1 and version 2 of this specification:
-
-* Added support for storing multiple arrays in the same store and organising
- arrays into hierarchies using groups.
-* Array metadata is now stored under the ".zarray" key instead of the "meta"
- key.
-* Custom attributes are now stored under the ".zattrs" key instead of the
- "attrs" key.
-* Added support for filters.
-* Changed encoding of "fill_value" field within array metadata.
-* Changed encoding of compressor information within array metadata to be
- consistent with representation of filter information.
diff --git a/docs/subprojects.md b/docs/subprojects.md
new file mode 100644
index 0000000000..9f7951e836
--- /dev/null
+++ b/docs/subprojects.md
@@ -0,0 +1,45 @@
+# Subprojects
+
+Alongside `zarr` itself, the
+[zarr-python repository](https://github.com/zarr-developers/zarr-python) hosts a
+small number of companion packages. Each one is developed in the same repository
+but versioned, released, and documented independently, so you can depend on it
+without taking on `zarr` as a dependency.
+
+
+
+- [:material-code-json:{ .lg .middle } __zarr-metadata__](https://zarr.readthedocs.io/projects/zarr-metadata/)
+
+ ---
+
+ Spec-defined metadata types, models, and validators for Zarr v2 and v3, with
+ minimal dependencies. Useful if your software reads or writes Zarr metadata
+ documents but does not need a full Zarr implementation.
+
+ ```bash
+ pip install zarr-metadata
+ ```
+
+- [:material-vector-polyline:{ .lg .middle } __zarr-indexing__](https://zarr.readthedocs.io/projects/zarr-indexing/)
+
+ ---
+
+ Composable, lazy coordinate transforms for Zarr array indexing. Makes the
+ mapping from requested coordinates to stored coordinates a first-class,
+ composable value, and resolves which chunks a selection touches.
+
+ ```bash
+ pip install zarr-indexing
+ ```
+
+- [:material-server:{ .lg .middle } __zarr-http-server__](https://zarr.readthedocs.io/projects/zarr-http-server/)
+
+ ---
+
+ HTTP server for Zarr stores, arrays, and groups.
+
+ ```bash
+ pip install zarr-http-server
+ ```
+
+
diff --git a/docs/talks/scipy2019/submission.rst b/docs/talks/scipy2019/submission.rst
deleted file mode 100644
index a7d531c91e..0000000000
--- a/docs/talks/scipy2019/submission.rst
+++ /dev/null
@@ -1,144 +0,0 @@
-Zarr - scalable storage of tensor data for use in parallel and distributed computing
-====================================================================================
-
-SciPy 2019 submission.
-
-
-Short summary
--------------
-
-Many scientific problems involve computing over large N-dimensional
-typed arrays of data, and reading or writing data is often the major
-bottleneck limiting speed or scalability. The Zarr project is
-developing a simple, scalable approach to storage of such data in a
-way that is compatible with a range of approaches to distributed and
-parallel computing. We describe the Zarr protocol and data storage
-format, and the current state of implementations for various
-programming languages including Python. We also describe current uses
-of Zarr in malaria genomics, the Human Cell Atlas, and the Pangeo
-project.
-
-
-Abstract
---------
-
-Background
-~~~~~~~~~~
-
-Across a broad range of scientific disciplines, data are naturally
-represented and stored as N-dimensional typed arrays, also known as
-tensors. The volume of data being generated is outstripping our
-ability to analyse it, and scientific communities are looking for ways
-to leverage modern multi-core CPUs and distributed computing
-platforms, including cloud computing. Retrieval and storage of data is
-often the major bottleneck, and new approaches to data storage are
-needed to accelerate distributed computations and enable them to scale
-on a variety of platforms.
-
-Methods
-~~~~~~~
-
-We have designed a new storage format and protocol for tensor data
-[1_], and have released an open source Python implementation [2_,
-3_]. Our approach builds on data storage concepts from HDF5 [4_],
-particularly chunking and compression, and hierarchical organisation
-of datasets. Key design goals include: a simple protocol and format
-that can be implemented in other programming languages; support for
-multiple concurrent readers or writers; support for a variety of
-parallel computing environments, from multi-threaded execution on a
-single CPU to multi-process execution across a multi-node cluster;
-pluggable storage subsystem with support for file systems, key-value
-databases and cloud object stores; pluggable encoding subsystem with
-support for a variety of modern compressors.
-
-Results
-~~~~~~~
-
-We illustrate the use of Zarr with examples from several scientific
-domains. Zarr is being used within the Pangeo project [5_], which is
-building a community platform for big data geoscience. The Pangeo
-community have converted a number of existing climate modelling and
-satellite observation datasets to Zarr [6_], and have demonstrated
-their use in computations using HPC and cloud computing
-environments. Within the MalariaGEN project [7_], Zarr is used to
-store genome variation data from next-generation sequencing of natural
-populations of malaria parasites and mosquitoes [8_] and these data
-are used as input to analyses of the evolution of these organisms in
-response to selective pressure from anti-malarial drugs and
-insecticides. Zarr is being used within the Human Cell Atlas (HCA)
-project [9_], which is building a reference atlas of healthy human
-cell types. This project hopes to leverage this information to better
-understand the dysregulation of cellular states that underly human
-disease. The Human Cell Atlas uses Zarr as the output data format
-because it enables the project to easily generate matrices containing
-user-selected subsets of cells.
-
-Conclusions
-~~~~~~~~~~~
-
-Zarr is generating interest across a range of scientific domains, and
-work is ongoing to establish a community process to support further
-development of the specifications and implementations in other
-programming languages [10_, 11_, 12_] and building interoperability
-with a similar project called N5 [13_]. Other packages within the
-PyData ecosystem, notably Dask [14_], Xarray [15_] and Intake [16_],
-have added capability to read and write Zarr, and together these
-packages provide a compelling solution for large scale data science
-using Python [17_]. Zarr has recently been presented in several
-venues, including a webinar for the ESIP Federation tech dive series
-[18_], and a talk at the AGU Fall Meeting 2018 [19_].
-
-
-References
-~~~~~~~~~~
-
-.. _1: https://zarr.readthedocs.io/en/stable/spec/v2.html
-.. _2: https://github.com/zarr-developers/zarr-python
-.. _3: https://github.com/zarr-developers/numcodecs
-.. _4: https://www.hdfgroup.org/solutions/hdf5/
-.. _5: https://pangeo.io/
-.. _6: https://pangeo.io/catalog.html
-.. _7: https://www.malariagen.net/
-.. _8: http://alimanfoo.github.io/2016/09/21/genotype-compression-benchmark.html
-.. _9: https://www.humancellatlas.org/
-.. _10: https://github.com/constantinpape/z5
-.. _11: https://github.com/lasersonlab/ndarray.scala
-.. _12: https://github.com/meggart/ZarrNative.jl
-.. _13: https://github.com/saalfeldlab/n5
-.. _14: http://docs.dask.org/en/latest/array-creation.html
-.. _15: http://xarray.pydata.org/en/stable/io.html
-.. _16: https://github.com/ContinuumIO/intake-xarray
-.. _17: http://matthewrocklin.com/blog/work/2018/01/22/pangeo-2
-.. _18: http://wiki.esipfed.org/index.php/Interoperability_and_Technology/Tech_Dive_Webinar_Series#8_March.2C_2018:_.22Zarr:_A_simple.2C_open.2C_scalable_solution_for_big_NetCDF.2FHDF_data_on_the_Cloud.22:_Alistair_Miles.2C_University_of_Oxford.
-.. _19: https://agu.confex.com/agu/fm18/meetingapp.cgi/Paper/390015
-
-
-Authors
--------
-
-Project contributors are listed in alphabetical order by surname.
-
-* `Ryan Abernathey `_, Columbia University
-* `Stephan Balmer `_, Meteotest
-* `Ambrose Carr `_, Chan Zuckerberg Initiative
-* `Tim Crone `_, Columbia University
-* `Martin Durant `_, Anaconda, inc.
-* `Jan Funke `_, HHMI Janelia
-* `Darren Gallagher `_, Satavia
-* `Fabian Gans `_, Max Planck Institute for Biogeochemistry
-* `Shikhar Goenka `_, Satavia
-* `Joe Hamman `_, NCAR
-* `Stephan Hoyer `_, Google
-* `Jerome Kelleher `_, University of Oxford
-* `John Kirkham `_, HHMI Janelia
-* `Alistair Miles `_, University of Oxford
-* `Josh Moore `_, University of Dundee
-* `Charles Noyes `_, University of Southern California
-* `Tarik Onalan `_
-* `Constantin Pape `_, University of Heidelberg
-* `Zain Patel `_, University of Cambridge
-* `Matthew Rocklin `_, NVIDIA
-* `Stephan Saafeld `_, HHMI Janelia
-* `Vincent Schut `_, Satelligence
-* `Justin Swaney `_, MIT
-* `Ryan Williams `_, Chan Zuckerberg Initiative
diff --git a/docs/tutorial.rst b/docs/tutorial.rst
deleted file mode 100644
index a3421608cc..0000000000
--- a/docs/tutorial.rst
+++ /dev/null
@@ -1,1511 +0,0 @@
-.. _tutorial:
-
-Tutorial
-========
-
-Zarr provides classes and functions for working with N-dimensional arrays that
-behave like NumPy arrays but whose data is divided into chunks and each chunk is
-compressed. If you are already familiar with HDF5 then Zarr arrays provide
-similar functionality, but with some additional flexibility.
-
-.. _tutorial_create:
-
-Creating an array
------------------
-
-Zarr has several functions for creating arrays. For example::
-
- >>> import zarr
- >>> z = zarr.zeros((10000, 10000), chunks=(1000, 1000), dtype='i4')
- >>> z
-
-
-The code above creates a 2-dimensional array of 32-bit integers with 10000 rows
-and 10000 columns, divided into chunks where each chunk has 1000 rows and 1000
-columns (and so there will be 100 chunks in total).
-
-For a complete list of array creation routines see the :mod:`zarr.creation`
-module documentation.
-
-.. _tutorial_array:
-
-Reading and writing data
-------------------------
-
-Zarr arrays support a similar interface to NumPy arrays for reading and writing
-data. For example, the entire array can be filled with a scalar value::
-
- >>> z[:] = 42
-
-Regions of the array can also be written to, e.g.::
-
- >>> import numpy as np
- >>> z[0, :] = np.arange(10000)
- >>> z[:, 0] = np.arange(10000)
-
-The contents of the array can be retrieved by slicing, which will load the
-requested region into memory as a NumPy array, e.g.::
-
- >>> z[0, 0]
- 0
- >>> z[-1, -1]
- 42
- >>> z[0, :]
- array([ 0, 1, 2, ..., 9997, 9998, 9999], dtype=int32)
- >>> z[:, 0]
- array([ 0, 1, 2, ..., 9997, 9998, 9999], dtype=int32)
- >>> z[:]
- array([[ 0, 1, 2, ..., 9997, 9998, 9999],
- [ 1, 42, 42, ..., 42, 42, 42],
- [ 2, 42, 42, ..., 42, 42, 42],
- ...,
- [9997, 42, 42, ..., 42, 42, 42],
- [9998, 42, 42, ..., 42, 42, 42],
- [9999, 42, 42, ..., 42, 42, 42]], dtype=int32)
-
-.. _tutorial_persist:
-
-Persistent arrays
------------------
-
-In the examples above, compressed data for each chunk of the array was stored in
-main memory. Zarr arrays can also be stored on a file system, enabling
-persistence of data between sessions. For example::
-
- >>> z1 = zarr.open('data/example.zarr', mode='w', shape=(10000, 10000),
- ... chunks=(1000, 1000), dtype='i4')
-
-The array above will store its configuration metadata and all compressed chunk
-data in a directory called 'data/example.zarr' relative to the current working
-directory. The :func:`zarr.convenience.open` function provides a convenient way
-to create a new persistent array or continue working with an existing
-array. Note that although the function is called "open", there is no need to
-close an array: data are automatically flushed to disk, and files are
-automatically closed whenever an array is modified.
-
-Persistent arrays support the same interface for reading and writing data,
-e.g.::
-
- >>> z1[:] = 42
- >>> z1[0, :] = np.arange(10000)
- >>> z1[:, 0] = np.arange(10000)
-
-Check that the data have been written and can be read again::
-
- >>> z2 = zarr.open('data/example.zarr', mode='r')
- >>> np.all(z1[:] == z2[:])
- True
-
-If you are just looking for a fast and convenient way to save NumPy arrays to
-disk then load back into memory later, the functions
-:func:`zarr.convenience.save` and :func:`zarr.convenience.load` may be
-useful. E.g.::
-
- >>> a = np.arange(10)
- >>> zarr.save('data/example.zarr', a)
- >>> zarr.load('data/example.zarr')
- array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
-
-Please note that there are a number of other options for persistent array
-storage, see the section on :ref:`tutorial_storage` below.
-
-.. _tutorial_resize:
-
-Resizing and appending
-----------------------
-
-A Zarr array can be resized, which means that any of its dimensions can be
-increased or decreased in length. For example::
-
- >>> z = zarr.zeros(shape=(10000, 10000), chunks=(1000, 1000))
- >>> z[:] = 42
- >>> z.resize(20000, 10000)
- >>> z.shape
- (20000, 10000)
-
-Note that when an array is resized, the underlying data are not rearranged in
-any way. If one or more dimensions are shrunk, any chunks falling outside the
-new array shape will be deleted from the underlying store.
-
-For convenience, Zarr arrays also provide an ``append()`` method, which can be
-used to append data to any axis. E.g.::
-
- >>> a = np.arange(10000000, dtype='i4').reshape(10000, 1000)
- >>> z = zarr.array(a, chunks=(1000, 100))
- >>> z.shape
- (10000, 1000)
- >>> z.append(a)
- (20000, 1000)
- >>> z.append(np.vstack([a, a]), axis=1)
- (20000, 2000)
- >>> z.shape
- (20000, 2000)
-
-.. _tutorial_compress:
-
-Compressors
------------
-
-A number of different compressors can be used with Zarr. A separate package
-called NumCodecs_ is available which provides a common interface to various
-compressor libraries including Blosc, Zstandard, LZ4, Zlib, BZ2 and
-LZMA. Different compressors can be provided via the ``compressor`` keyword
-argument accepted by all array creation functions. For example::
-
- >>> from numcodecs import Blosc
- >>> compressor = Blosc(cname='zstd', clevel=3, shuffle=Blosc.BITSHUFFLE)
- >>> data = np.arange(100000000, dtype='i4').reshape(10000, 10000)
- >>> z = zarr.array(data, chunks=(1000, 1000), compressor=compressor)
- >>> z.compressor
- Blosc(cname='zstd', clevel=3, shuffle=BITSHUFFLE, blocksize=0)
-
-This array above will use Blosc as the primary compressor, using the Zstandard
-algorithm (compression level 3) internally within Blosc, and with the
-bit-shuffle filter applied.
-
-When using a compressor, it can be useful to get some diagnostics on the
-compression ratio. Zarr arrays provide a ``info`` property which can be used to
-print some diagnostics, e.g.::
-
- >>> z.info
- Type : zarr.core.Array
- Data type : int32
- Shape : (10000, 10000)
- Chunk shape : (1000, 1000)
- Order : C
- Read-only : False
- Compressor : Blosc(cname='zstd', clevel=3, shuffle=BITSHUFFLE,
- : blocksize=0)
- Store type : builtins.dict
- No. bytes : 400000000 (381.5M)
- No. bytes stored : 3379344 (3.2M)
- Storage ratio : 118.4
- Chunks initialized : 100/100
-
-If you don't specify a compressor, by default Zarr uses the Blosc
-compressor. Blosc is generally very fast and can be configured in a variety of
-ways to improve the compression ratio for different types of data. Blosc is in
-fact a "meta-compressor", which means that it can use a number of different
-compression algorithms internally to compress the data. Blosc also provides
-highly optimized implementations of byte- and bit-shuffle filters, which can
-improve compression ratios for some data. A list of the internal compression
-libraries available within Blosc can be obtained via::
-
- >>> from numcodecs import blosc
- >>> blosc.list_compressors()
- ['blosclz', 'lz4', 'lz4hc', 'snappy', 'zlib', 'zstd']
-
-In addition to Blosc, other compression libraries can also be used. For example,
-here is an array using Zstandard compression, level 1::
-
- >>> from numcodecs import Zstd
- >>> z = zarr.array(np.arange(100000000, dtype='i4').reshape(10000, 10000),
- ... chunks=(1000, 1000), compressor=Zstd(level=1))
- >>> z.compressor
- Zstd(level=1)
-
-Here is an example using LZMA with a custom filter pipeline including LZMA's
-built-in delta filter::
-
- >>> import lzma
- >>> lzma_filters = [dict(id=lzma.FILTER_DELTA, dist=4),
- ... dict(id=lzma.FILTER_LZMA2, preset=1)]
- >>> from numcodecs import LZMA
- >>> compressor = LZMA(filters=lzma_filters)
- >>> z = zarr.array(np.arange(100000000, dtype='i4').reshape(10000, 10000),
- ... chunks=(1000, 1000), compressor=compressor)
- >>> z.compressor
- LZMA(format=1, check=-1, preset=None, filters=[{'dist': 4, 'id': 3}, {'id': 33, 'preset': 1}])
-
-The default compressor can be changed by setting the value of the
-``zarr.storage.default_compressor`` variable, e.g.::
-
- >>> import zarr.storage
- >>> from numcodecs import Zstd, Blosc
- >>> # switch to using Zstandard
- ... zarr.storage.default_compressor = Zstd(level=1)
- >>> z = zarr.zeros(100000000, chunks=1000000)
- >>> z.compressor
- Zstd(level=1)
- >>> # switch back to Blosc defaults
- ... zarr.storage.default_compressor = Blosc()
-
-To disable compression, set ``compressor=None`` when creating an array, e.g.::
-
- >>> z = zarr.zeros(100000000, chunks=1000000, compressor=None)
- >>> z.compressor is None
- True
-
-.. _tutorial_filters:
-
-Filters
--------
-
-In some cases, compression can be improved by transforming the data in some
-way. For example, if nearby values tend to be correlated, then shuffling the
-bytes within each numerical value or storing the difference between adjacent
-values may increase compression ratio. Some compressors provide built-in filters
-that apply transformations to the data prior to compression. For example, the
-Blosc compressor has built-in implementations of byte- and bit-shuffle filters,
-and the LZMA compressor has a built-in implementation of a delta
-filter. However, to provide additional flexibility for implementing and using
-filters in combination with different compressors, Zarr also provides a
-mechanism for configuring filters outside of the primary compressor.
-
-Here is an example using a delta filter with the Blosc compressor::
-
- >>> from numcodecs import Blosc, Delta
- >>> filters = [Delta(dtype='i4')]
- >>> compressor = Blosc(cname='zstd', clevel=1, shuffle=Blosc.SHUFFLE)
- >>> data = np.arange(100000000, dtype='i4').reshape(10000, 10000)
- >>> z = zarr.array(data, chunks=(1000, 1000), filters=filters, compressor=compressor)
- >>> z.info
- Type : zarr.core.Array
- Data type : int32
- Shape : (10000, 10000)
- Chunk shape : (1000, 1000)
- Order : C
- Read-only : False
- Filter [0] : Delta(dtype='`_ documentation.
-
-.. _tutorial_groups:
-
-Groups
-------
-
-Zarr supports hierarchical organization of arrays via groups. As with arrays,
-groups can be stored in memory, on disk, or via other storage systems that
-support a similar interface.
-
-To create a group, use the :func:`zarr.group` function::
-
- >>> root = zarr.group()
- >>> root
-
-
-Groups have a similar API to the Group class from `h5py
-`_. For example, groups can contain other groups::
-
- >>> foo = root.create_group('foo')
- >>> bar = foo.create_group('bar')
-
-Groups can also contain arrays, e.g.::
-
- >>> z1 = bar.zeros('baz', shape=(10000, 10000), chunks=(1000, 1000), dtype='i4')
- >>> z1
-
-
-Arrays are known as "datasets" in HDF5 terminology. For compatibility with h5py,
-Zarr groups also implement the ``create_dataset()`` and ``require_dataset()``
-methods, e.g.::
-
- >>> z = bar.create_dataset('quux', shape=(10000, 10000), chunks=(1000, 1000), dtype='i4')
- >>> z
-
-
-Members of a group can be accessed via the suffix notation, e.g.::
-
- >>> root['foo']
-
-
-The '/' character can be used to access multiple levels of the hierarchy in one
-call, e.g.::
-
- >>> root['foo/bar']
-
- >>> root['foo/bar/baz']
-
-
-The :func:`zarr.hierarchy.Group.tree` method can be used to print a tree
-representation of the hierarchy, e.g.::
-
- >>> root.tree()
- /
- └── foo
- └── bar
- ├── baz (10000, 10000) int32
- └── quux (10000, 10000) int32
-
-The :func:`zarr.convenience.open` function provides a convenient way to create or
-re-open a group stored in a directory on the file-system, with sub-groups stored in
-sub-directories, e.g.::
-
- >>> root = zarr.open('data/group.zarr', mode='w')
- >>> root
-
- >>> z = root.zeros('foo/bar/baz', shape=(10000, 10000), chunks=(1000, 1000), dtype='i4')
- >>> z
-
-
-Groups can be used as context managers (in a ``with`` statement).
-If the underlying store has a ``close`` method, it will be called on exit.
-
-For more information on groups see the :mod:`zarr.hierarchy` and
-:mod:`zarr.convenience` API docs.
-
-.. _tutorial_diagnostics:
-
-Array and group diagnostics
----------------------------
-
-Diagnostic information about arrays and groups is available via the ``info``
-property. E.g.::
-
- >>> root = zarr.group()
- >>> foo = root.create_group('foo')
- >>> bar = foo.zeros('bar', shape=1000000, chunks=100000, dtype='i8')
- >>> bar[:] = 42
- >>> baz = foo.zeros('baz', shape=(1000, 1000), chunks=(100, 100), dtype='f4')
- >>> baz[:] = 4.2
- >>> root.info
- Name : /
- Type : zarr.hierarchy.Group
- Read-only : False
- Store type : zarr.storage.MemoryStore
- No. members : 1
- No. arrays : 0
- No. groups : 1
- Groups : foo
-
- >>> foo.info
- Name : /foo
- Type : zarr.hierarchy.Group
- Read-only : False
- Store type : zarr.storage.MemoryStore
- No. members : 2
- No. arrays : 2
- No. groups : 0
- Arrays : bar, baz
-
- >>> bar.info
- Name : /foo/bar
- Type : zarr.core.Array
- Data type : int64
- Shape : (1000000,)
- Chunk shape : (100000,)
- Order : C
- Read-only : False
- Compressor : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0)
- Store type : zarr.storage.MemoryStore
- No. bytes : 8000000 (7.6M)
- No. bytes stored : 33240 (32.5K)
- Storage ratio : 240.7
- Chunks initialized : 10/10
-
- >>> baz.info
- Name : /foo/baz
- Type : zarr.core.Array
- Data type : float32
- Shape : (1000, 1000)
- Chunk shape : (100, 100)
- Order : C
- Read-only : False
- Compressor : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0)
- Store type : zarr.storage.MemoryStore
- No. bytes : 4000000 (3.8M)
- No. bytes stored : 23943 (23.4K)
- Storage ratio : 167.1
- Chunks initialized : 100/100
-
-Groups also have the :func:`zarr.hierarchy.Group.tree` method, e.g.::
-
- >>> root.tree()
- /
- └── foo
- ├── bar (1000000,) int64
- └── baz (1000, 1000) float32
-
-If you're using Zarr within a Jupyter notebook (requires
-`ipytree `_), calling ``tree()`` will generate an
-interactive tree representation, see the `repr_tree.ipynb notebook
-`_
-for more examples.
-
-.. _tutorial_attrs:
-
-User attributes
----------------
-
-Zarr arrays and groups support custom key/value attributes, which can be useful for
-storing application-specific metadata. For example::
-
- >>> root = zarr.group()
- >>> root.attrs['foo'] = 'bar'
- >>> z = root.zeros('zzz', shape=(10000, 10000))
- >>> z.attrs['baz'] = 42
- >>> z.attrs['qux'] = [1, 4, 7, 12]
- >>> sorted(root.attrs)
- ['foo']
- >>> 'foo' in root.attrs
- True
- >>> root.attrs['foo']
- 'bar'
- >>> sorted(z.attrs)
- ['baz', 'qux']
- >>> z.attrs['baz']
- 42
- >>> z.attrs['qux']
- [1, 4, 7, 12]
-
-Internally Zarr uses JSON to store array attributes, so attribute values must be
-JSON serializable.
-
-.. _tutorial_indexing:
-
-Advanced indexing
------------------
-
-As of version 2.2, Zarr arrays support several methods for advanced or "fancy"
-indexing, which enable a subset of data items to be extracted or updated in an
-array without loading the entire array into memory.
-
-Note that although this functionality is similar to some of the advanced
-indexing capabilities available on NumPy arrays and on h5py datasets, **the Zarr
-API for advanced indexing is different from both NumPy and h5py**, so please
-read this section carefully. For a complete description of the indexing API,
-see the documentation for the :class:`zarr.core.Array` class.
-
-Indexing with coordinate arrays
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Items from a Zarr array can be extracted by providing an integer array of
-coordinates. E.g.::
-
- >>> z = zarr.array(np.arange(10))
- >>> z[:]
- array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
- >>> z.get_coordinate_selection([1, 4])
- array([1, 4])
-
-Coordinate arrays can also be used to update data, e.g.::
-
- >>> z.set_coordinate_selection([1, 4], [-1, -2])
- >>> z[:]
- array([ 0, -1, 2, 3, -2, 5, 6, 7, 8, 9])
-
-For multidimensional arrays, coordinates must be provided for each dimension,
-e.g.::
-
- >>> z = zarr.array(np.arange(15).reshape(3, 5))
- >>> z[:]
- array([[ 0, 1, 2, 3, 4],
- [ 5, 6, 7, 8, 9],
- [10, 11, 12, 13, 14]])
- >>> z.get_coordinate_selection(([0, 2], [1, 3]))
- array([ 1, 13])
- >>> z.set_coordinate_selection(([0, 2], [1, 3]), [-1, -2])
- >>> z[:]
- array([[ 0, -1, 2, 3, 4],
- [ 5, 6, 7, 8, 9],
- [10, 11, 12, -2, 14]])
-
-For convenience, coordinate indexing is also available via the ``vindex``
-property, e.g.::
-
- >>> z.vindex[[0, 2], [1, 3]]
- array([-1, -2])
- >>> z.vindex[[0, 2], [1, 3]] = [-3, -4]
- >>> z[:]
- array([[ 0, -3, 2, 3, 4],
- [ 5, 6, 7, 8, 9],
- [10, 11, 12, -4, 14]])
-
-Indexing with a mask array
-~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Items can also be extracted by providing a Boolean mask. E.g.::
-
- >>> z = zarr.array(np.arange(10))
- >>> z[:]
- array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
- >>> sel = np.zeros_like(z, dtype=bool)
- >>> sel[1] = True
- >>> sel[4] = True
- >>> z.get_mask_selection(sel)
- array([1, 4])
- >>> z.set_mask_selection(sel, [-1, -2])
- >>> z[:]
- array([ 0, -1, 2, 3, -2, 5, 6, 7, 8, 9])
-
-Here's a multidimensional example::
-
- >>> z = zarr.array(np.arange(15).reshape(3, 5))
- >>> z[:]
- array([[ 0, 1, 2, 3, 4],
- [ 5, 6, 7, 8, 9],
- [10, 11, 12, 13, 14]])
- >>> sel = np.zeros_like(z, dtype=bool)
- >>> sel[0, 1] = True
- >>> sel[2, 3] = True
- >>> z.get_mask_selection(sel)
- array([ 1, 13])
- >>> z.set_mask_selection(sel, [-1, -2])
- >>> z[:]
- array([[ 0, -1, 2, 3, 4],
- [ 5, 6, 7, 8, 9],
- [10, 11, 12, -2, 14]])
-
-For convenience, mask indexing is also available via the ``vindex`` property,
-e.g.::
-
- >>> z.vindex[sel]
- array([-1, -2])
- >>> z.vindex[sel] = [-3, -4]
- >>> z[:]
- array([[ 0, -3, 2, 3, 4],
- [ 5, 6, 7, 8, 9],
- [10, 11, 12, -4, 14]])
-
-Mask indexing is conceptually the same as coordinate indexing, and is
-implemented internally via the same machinery. Both styles of indexing allow
-selecting arbitrary items from an array, also known as point selection.
-
-Orthogonal indexing
-~~~~~~~~~~~~~~~~~~~
-
-Zarr arrays also support methods for orthogonal indexing, which allows
-selections to be made along each dimension of an array independently. For
-example, this allows selecting a subset of rows and/or columns from a
-2-dimensional array. E.g.::
-
- >>> z = zarr.array(np.arange(15).reshape(3, 5))
- >>> z[:]
- array([[ 0, 1, 2, 3, 4],
- [ 5, 6, 7, 8, 9],
- [10, 11, 12, 13, 14]])
- >>> z.get_orthogonal_selection(([0, 2], slice(None))) # select first and third rows
- array([[ 0, 1, 2, 3, 4],
- [10, 11, 12, 13, 14]])
- >>> z.get_orthogonal_selection((slice(None), [1, 3])) # select second and fourth columns
- array([[ 1, 3],
- [ 6, 8],
- [11, 13]])
- >>> z.get_orthogonal_selection(([0, 2], [1, 3])) # select rows [0, 2] and columns [1, 4]
- array([[ 1, 3],
- [11, 13]])
-
-Data can also be modified, e.g.::
-
- >>> z.set_orthogonal_selection(([0, 2], [1, 3]), [[-1, -2], [-3, -4]])
- >>> z[:]
- array([[ 0, -1, 2, -2, 4],
- [ 5, 6, 7, 8, 9],
- [10, -3, 12, -4, 14]])
-
-For convenience, the orthogonal indexing functionality is also available via the
-``oindex`` property, e.g.::
-
- >>> z = zarr.array(np.arange(15).reshape(3, 5))
- >>> z.oindex[[0, 2], :] # select first and third rows
- array([[ 0, 1, 2, 3, 4],
- [10, 11, 12, 13, 14]])
- >>> z.oindex[:, [1, 3]] # select second and fourth columns
- array([[ 1, 3],
- [ 6, 8],
- [11, 13]])
- >>> z.oindex[[0, 2], [1, 3]] # select rows [0, 2] and columns [1, 4]
- array([[ 1, 3],
- [11, 13]])
- >>> z.oindex[[0, 2], [1, 3]] = [[-1, -2], [-3, -4]]
- >>> z[:]
- array([[ 0, -1, 2, -2, 4],
- [ 5, 6, 7, 8, 9],
- [10, -3, 12, -4, 14]])
-
-Any combination of integer, slice, 1D integer array and/or 1D Boolean array can
-be used for orthogonal indexing.
-
-Indexing fields in structured arrays
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-All selection methods support a ``fields`` parameter which allows retrieving or
-replacing data for a specific field in an array with a structured dtype. E.g.::
-
- >>> a = np.array([(b'aaa', 1, 4.2),
- ... (b'bbb', 2, 8.4),
- ... (b'ccc', 3, 12.6)],
- ... dtype=[('foo', 'S3'), ('bar', 'i4'), ('baz', 'f8')])
- >>> z = zarr.array(a)
- >>> z['foo']
- array([b'aaa', b'bbb', b'ccc'],
- dtype='|S3')
- >>> z['baz']
- array([ 4.2, 8.4, 12.6])
- >>> z.get_basic_selection(slice(0, 2), fields='bar')
- array([1, 2], dtype=int32)
- >>> z.get_coordinate_selection([0, 2], fields=['foo', 'baz'])
- array([(b'aaa', 4.2), (b'ccc', 12.6)],
- dtype=[('foo', 'S3'), ('baz', '>> z = zarr.open('data/example.zarr', mode='w', shape=1000000, dtype='i4')
-
-...is short-hand for::
-
- >>> store = zarr.DirectoryStore('data/example.zarr')
- >>> z = zarr.create(store=store, overwrite=True, shape=1000000, dtype='i4')
-
-...and the following code::
-
- >>> root = zarr.open('data/example.zarr', mode='w')
-
-...is short-hand for::
-
- >>> store = zarr.DirectoryStore('data/example.zarr')
- >>> root = zarr.group(store=store, overwrite=True)
-
-Any other compatible storage class could be used in place of
-:class:`zarr.storage.DirectoryStore` in the code examples above. For example,
-here is an array stored directly into a Zip file, via the
-:class:`zarr.storage.ZipStore` class::
-
- >>> store = zarr.ZipStore('data/example.zip', mode='w')
- >>> root = zarr.group(store=store)
- >>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
- >>> z[:] = 42
- >>> store.close()
-
-Re-open and check that data have been written::
-
- >>> store = zarr.ZipStore('data/example.zip', mode='r')
- >>> root = zarr.group(store=store)
- >>> z = root['foo/bar']
- >>> z[:]
- array([[42, 42, 42, ..., 42, 42, 42],
- [42, 42, 42, ..., 42, 42, 42],
- [42, 42, 42, ..., 42, 42, 42],
- ...,
- [42, 42, 42, ..., 42, 42, 42],
- [42, 42, 42, ..., 42, 42, 42],
- [42, 42, 42, ..., 42, 42, 42]], dtype=int32)
- >>> store.close()
-
-Note that there are some limitations on how Zip files can be used, because items
-within a Zip file cannot be updated in place. This means that data in the array
-should only be written once and write operations should be aligned with chunk
-boundaries. Note also that the ``close()`` method must be called after writing
-any data to the store, otherwise essential records will not be written to the
-underlying zip file.
-
-Another storage alternative is the :class:`zarr.storage.DBMStore` class, added
-in Zarr version 2.2. This class allows any DBM-style database to be used for
-storing an array or group. Here is an example using a Berkeley DB B-tree
-database for storage (requires `bsddb3
-`_ to be installed)::
-
- >>> import bsddb3
- >>> store = zarr.DBMStore('data/example.bdb', open=bsddb3.btopen)
- >>> root = zarr.group(store=store, overwrite=True)
- >>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
- >>> z[:] = 42
- >>> store.close()
-
-Also added in Zarr version 2.2 is the :class:`zarr.storage.LMDBStore` class which
-enables the lightning memory-mapped database (LMDB) to be used for storing an array or
-group (requires `lmdb `_ to be installed)::
-
- >>> store = zarr.LMDBStore('data/example.lmdb')
- >>> root = zarr.group(store=store, overwrite=True)
- >>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
- >>> z[:] = 42
- >>> store.close()
-
-In Zarr version 2.3 is the :class:`zarr.storage.SQLiteStore` class which
-enables the SQLite database to be used for storing an array or group (requires
-Python is built with SQLite support)::
-
- >>> store = zarr.SQLiteStore('data/example.sqldb')
- >>> root = zarr.group(store=store, overwrite=True)
- >>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
- >>> z[:] = 42
- >>> store.close()
-
-Also added in Zarr version 2.3 are two storage classes for interfacing with server-client
-databases. The :class:`zarr.storage.RedisStore` class interfaces `Redis `_
-(an in memory data structure store), and the :class:`zarr.storage.MongoDB` class interfaces
-with `MongoDB `_ (an object oriented NoSQL database). These stores
-respectively require the `redis-py `_ and
-`pymongo `_ packages to be installed.
-
-For compatibility with the `N5 `_ data format, Zarr also provides
-an N5 backend (this is currently an experimental feature). Similar to the zip storage class, an
-:class:`zarr.n5.N5Store` can be instantiated directly::
-
- >>> store = zarr.N5Store('data/example.n5')
- >>> root = zarr.group(store=store)
- >>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4')
- >>> z[:] = 42
-
-For convenience, the N5 backend will automatically be chosen when the filename
-ends with `.n5`::
-
- >>> root = zarr.open('data/example.n5', mode='w')
-
-Distributed/cloud storage
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-It is also possible to use distributed storage systems. The Dask project has
-implementations of the ``MutableMapping`` interface for Amazon S3 (`S3Map
-`_), Hadoop
-Distributed File System (`HDFSMap
-`_) and
-Google Cloud Storage (`GCSMap
-`_), which
-can be used with Zarr.
-
-Here is an example using S3Map to read an array created previously::
-
- >>> import s3fs
- >>> import zarr
- >>> s3 = s3fs.S3FileSystem(anon=True, client_kwargs=dict(region_name='eu-west-2'))
- >>> store = s3fs.S3Map(root='zarr-demo/store', s3=s3, check=False)
- >>> root = zarr.group(store=store)
- >>> z = root['foo/bar/baz']
- >>> z
-
- >>> z.info
- Name : /foo/bar/baz
- Type : zarr.core.Array
- Data type : |S1
- Shape : (21,)
- Chunk shape : (7,)
- Order : C
- Read-only : False
- Compressor : Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0)
- Store type : fsspec.mapping.FSMap
- No. bytes : 21
- Chunks initialized : 3/3
- >>> z[:]
- array([b'H', b'e', b'l', b'l', b'o', b' ', b'f', b'r', b'o', b'm', b' ',
- b't', b'h', b'e', b' ', b'c', b'l', b'o', b'u', b'd', b'!'],
- dtype='|S1')
- >>> z[:].tostring()
- b'Hello from the cloud!'
-
-Zarr now also has a builtin storage backend for Azure Blob Storage.
-The class is :class:`zarr.storage.ABSStore` (requires
-`azure-storage-blob `_
-to be installed)::
-
- >>> import azure.storage.blob
- >>> container_client = azure.storage.blob.ContainerClient(...) # doctest: +SKIP
- >>> store = zarr.ABSStore(client=container_client, prefix='zarr-testing') # doctest: +SKIP
- >>> root = zarr.group(store=store, overwrite=True) # doctest: +SKIP
- >>> z = root.zeros('foo/bar', shape=(1000, 1000), chunks=(100, 100), dtype='i4') # doctest: +SKIP
- >>> z[:] = 42 # doctest: +SKIP
-
-When using an actual storage account, provide ``account_name`` and
-``account_key`` arguments to :class:`zarr.storage.ABSStore`, the
-above client is just testing against the emulator. Please also note
-that this is an experimental feature.
-
-Note that retrieving data from a remote service via the network can be significantly
-slower than retrieving data from a local file system, and will depend on network latency
-and bandwidth between the client and server systems. If you are experiencing poor
-performance, there are several things you can try. One option is to increase the array
-chunk size, which will reduce the number of chunks and thus reduce the number of network
-round-trips required to retrieve data for an array (and thus reduce the impact of network
-latency). Another option is to try to increase the compression ratio by changing
-compression options or trying a different compressor (which will reduce the impact of
-limited network bandwidth).
-
-As of version 2.2, Zarr also provides the :class:`zarr.storage.LRUStoreCache`
-which can be used to implement a local in-memory cache layer over a remote
-store. E.g.::
-
- >>> s3 = s3fs.S3FileSystem(anon=True, client_kwargs=dict(region_name='eu-west-2'))
- >>> store = s3fs.S3Map(root='zarr-demo/store', s3=s3, check=False)
- >>> cache = zarr.LRUStoreCache(store, max_size=2**28)
- >>> root = zarr.group(store=cache)
- >>> z = root['foo/bar/baz']
- >>> from timeit import timeit
- >>> # first data access is relatively slow, retrieved from store
- ... timeit('print(z[:].tostring())', number=1, globals=globals()) # doctest: +SKIP
- b'Hello from the cloud!'
- 0.1081731989979744
- >>> # second data access is faster, uses cache
- ... timeit('print(z[:].tostring())', number=1, globals=globals()) # doctest: +SKIP
- b'Hello from the cloud!'
- 0.0009490990014455747
-
-If you are still experiencing poor performance with distributed/cloud storage,
-please raise an issue on the GitHub issue tracker with any profiling data you
-can provide, as there may be opportunities to optimise further either within
-Zarr or within the mapping interface to the storage.
-
-IO with ``fsspec``
-~~~~~~~~~~~~~~~~~~
-
-As of version 2.5, zarr supports passing URLs directly to `fsspec`_,
-and having it create the "mapping" instance automatically. This means, that
-for all of the backend storage implementations `supported by fsspec`_,
-you can skip importing and configuring the storage explicitly.
-For example::
-
- >>> g = zarr.open_group("s3://zarr-demo/store", storage_options={'anon': True}) # doctest: +SKIP
- >>> g['foo/bar/baz'][:].tobytes() # doctest: +SKIP
- b'Hello from the cloud!'
-
-The provision of the protocol specifier "s3://" will select the correct backend.
-Notice the kwargs ``storage_options``, used to pass parameters to that backend.
-
-As of version 2.6, write mode and complex URLs are also supported, such as::
-
- >>> g = zarr.open_group("simplecache::s3://zarr-demo/store",
- ... storage_options={"s3": {'anon': True}}) # doctest: +SKIP
- >>> g['foo/bar/baz'][:].tobytes() # downloads target file # doctest: +SKIP
- b'Hello from the cloud!'
- >>> g['foo/bar/baz'][:].tobytes() # uses cached file # doctest: +SKIP
- b'Hello from the cloud!'
-
-The second invocation here will be much faster. Note that the ``storage_options``
-have become more complex here, to account for the two parts of the supplied
-URL.
-
-.. _fsspec: https://filesystem-spec.readthedocs.io/en/latest/
-
-.. _supported by fsspec: https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations
-
-.. _tutorial_copy:
-
-Consolidating metadata
-~~~~~~~~~~~~~~~~~~~~~~
-
-Since there is a significant overhead for every connection to a cloud object
-store such as S3, the pattern described in the previous section may incur
-significant latency while scanning the metadata of the array hierarchy, even
-though each individual metadata object is small. For cases such as these, once
-the data are static and can be regarded as read-only, at least for the
-metadata/structure of the array hierarchy, the many metadata objects can be
-consolidated into a single one via
-:func:`zarr.convenience.consolidate_metadata`. Doing this can greatly increase
-the speed of reading the array metadata, e.g.::
-
- >>> zarr.consolidate_metadata(store) # doctest: +SKIP
-
-This creates a special key with a copy of all of the metadata from all of the
-metadata objects in the store.
-
-Later, to open a Zarr store with consolidated metadata, use
-:func:`zarr.convenience.open_consolidated`, e.g.::
-
- >>> root = zarr.open_consolidated(store) # doctest: +SKIP
-
-This uses the special key to read all of the metadata in a single call to the
-backend storage.
-
-Note that, the hierarchy could still be opened in the normal way and altered,
-causing the consolidated metadata to become out of sync with the real state of
-the array hierarchy. In this case,
-:func:`zarr.convenience.consolidate_metadata` would need to be called again.
-
-To protect against consolidated metadata accidentally getting out of sync, the
-root group returned by :func:`zarr.convenience.open_consolidated` is read-only
-for the metadata, meaning that no new groups or arrays can be created, and
-arrays cannot be resized. However, data values with arrays can still be updated.
-
-Copying/migrating data
-----------------------
-
-If you have some data in an HDF5 file and would like to copy some or all of it
-into a Zarr group, or vice-versa, the :func:`zarr.convenience.copy` and
-:func:`zarr.convenience.copy_all` functions can be used. Here's an example
-copying a group named 'foo' from an HDF5 file to a Zarr group::
-
- >>> import h5py
- >>> import zarr
- >>> import numpy as np
- >>> source = h5py.File('data/example.h5', mode='w')
- >>> foo = source.create_group('foo')
- >>> baz = foo.create_dataset('bar/baz', data=np.arange(100), chunks=(50,))
- >>> spam = source.create_dataset('spam', data=np.arange(100, 200), chunks=(30,))
- >>> zarr.tree(source)
- /
- ├── foo
- │ └── bar
- │ └── baz (100,) int64
- └── spam (100,) int64
- >>> dest = zarr.open_group('data/example.zarr', mode='w')
- >>> from sys import stdout
- >>> zarr.copy(source['foo'], dest, log=stdout)
- copy /foo
- copy /foo/bar
- copy /foo/bar/baz (100,) int64
- all done: 3 copied, 0 skipped, 800 bytes copied
- (3, 0, 800)
- >>> dest.tree() # N.B., no spam
- /
- └── foo
- └── bar
- └── baz (100,) int64
- >>> source.close()
-
-If rather than copying a single group or array you would like to copy all
-groups and arrays, use :func:`zarr.convenience.copy_all`, e.g.::
-
- >>> source = h5py.File('data/example.h5', mode='r')
- >>> dest = zarr.open_group('data/example2.zarr', mode='w')
- >>> zarr.copy_all(source, dest, log=stdout)
- copy /foo
- copy /foo/bar
- copy /foo/bar/baz (100,) int64
- copy /spam (100,) int64
- all done: 4 copied, 0 skipped, 1,600 bytes copied
- (4, 0, 1600)
- >>> dest.tree()
- /
- ├── foo
- │ └── bar
- │ └── baz (100,) int64
- └── spam (100,) int64
-
-If you need to copy data between two Zarr groups, the
-:func:`zarr.convenience.copy` and :func:`zarr.convenience.copy_all` functions can
-be used and provide the most flexibility. However, if you want to copy data
-in the most efficient way possible, without changing any configuration options,
-the :func:`zarr.convenience.copy_store` function can be used. This function
-copies data directly between the underlying stores, without any decompression or
-re-compression, and so should be faster. E.g.::
-
- >>> import zarr
- >>> import numpy as np
- >>> store1 = zarr.DirectoryStore('data/example.zarr')
- >>> root = zarr.group(store1, overwrite=True)
- >>> baz = root.create_dataset('foo/bar/baz', data=np.arange(100), chunks=(50,))
- >>> spam = root.create_dataset('spam', data=np.arange(100, 200), chunks=(30,))
- >>> root.tree()
- /
- ├── foo
- │ └── bar
- │ └── baz (100,) int64
- └── spam (100,) int64
- >>> from sys import stdout
- >>> store2 = zarr.ZipStore('data/example.zip', mode='w')
- >>> zarr.copy_store(store1, store2, log=stdout)
- copy .zgroup
- copy foo/.zgroup
- copy foo/bar/.zgroup
- copy foo/bar/baz/.zarray
- copy foo/bar/baz/0
- copy foo/bar/baz/1
- copy spam/.zarray
- copy spam/0
- copy spam/1
- copy spam/2
- copy spam/3
- all done: 11 copied, 0 skipped, 1,138 bytes copied
- (11, 0, 1138)
- >>> new_root = zarr.group(store2)
- >>> new_root.tree()
- /
- ├── foo
- │ └── bar
- │ └── baz (100,) int64
- └── spam (100,) int64
- >>> new_root['foo/bar/baz'][:]
- array([ 0, 1, 2, ..., 97, 98, 99])
- >>> store2.close() # zip stores need to be closed
-
-.. _tutorial_strings:
-
-String arrays
--------------
-
-There are several options for storing arrays of strings.
-
-If your strings are all ASCII strings, and you know the maximum length of the string in
-your array, then you can use an array with a fixed-length bytes dtype. E.g.::
-
- >>> z = zarr.zeros(10, dtype='S6')
- >>> z
-
- >>> z[0] = b'Hello'
- >>> z[1] = b'world!'
- >>> z[:]
- array([b'Hello', b'world!', b'', b'', b'', b'', b'', b'', b'', b''],
- dtype='|S6')
-
-A fixed-length unicode dtype is also available, e.g.::
-
- >>> greetings = ['¡Hola mundo!', 'Hej Världen!', 'Servus Woid!', 'Hei maailma!',
- ... 'Xin chào thế giới', 'Njatjeta Botë!', 'Γεια σου κόσμε!',
- ... 'こんにちは世界', '世界，你好！', 'Helló, világ!', 'Zdravo svete!',
- ... 'เฮลโลเวิลด์']
- >>> text_data = greetings * 10000
- >>> z = zarr.array(text_data, dtype='U20')
- >>> z
-
- >>> z[:]
- array(['¡Hola mundo!', 'Hej Världen!', 'Servus Woid!', ...,
- 'Helló, világ!', 'Zdravo svete!', 'เฮลโลเวิลด์'],
- dtype='>> import numcodecs
- >>> z = zarr.array(text_data, dtype=object, object_codec=numcodecs.VLenUTF8())
- >>> z
-
- >>> z.filters
- [VLenUTF8()]
- >>> z[:]
- array(['¡Hola mundo!', 'Hej Världen!', 'Servus Woid!', ...,
- 'Helló, világ!', 'Zdravo svete!', 'เฮลโลเวิลด์'], dtype=object)
-
-As a convenience, ``dtype=str`` (or ``dtype=unicode`` on Python 2.7) can be used, which
-is a short-hand for ``dtype=object, object_codec=numcodecs.VLenUTF8()``, e.g.::
-
- >>> z = zarr.array(text_data, dtype=str)
- >>> z
- 
-
- 
-
-
-
- 
-
-
-
- 
-
- 
-
- 
-
-