diff --git a/docs/src/user_manual/explanation/netcdf_io.rst b/docs/src/user_manual/explanation/netcdf_io.rst index f5f608f5c8..04227b9f28 100644 --- a/docs/src/user_manual/explanation/netcdf_io.rst +++ b/docs/src/user_manual/explanation/netcdf_io.rst @@ -133,6 +133,201 @@ Iris' optimisation all together, and will take its chunksizes from Dask's behavi (70, 37, 49) +Character and String datatypes +------------------------------ +Summary +^^^^^^^ + +* Iris currently *only* fully supports fixed-width 'char' type data in netCDF variables + + * the 'string' type (variable-width unicode strings) will be added in a future release + +* 'char' variable data is represented as numpy string arrays in Iris objects, such as + cubes and coordinates. + + * the numpy dtype is of the type "U", where is a character width. + * the character width relates to a string *dimension* of the netCDF variable, + which is not in the dimensions of the Iris object or its data array. + * the dtype 'width' controls the length of string dimensions created when saving + +* Iris also uses a variable ``_Encoding`` attribute to enable storage of non-ascii + characters in 'char' type arrays. + + * it appears as a regular attribute of the Iris object + * it is not needed for ascii-only data + * it is not needed to *read* 'utf-8' encoded data correctly + * it **is** required to *save* any non-ascii characters + +The following describes the nature of character and string data handling in : +netCDF itself; the CF conventions; the netCDF4 Python module and the Iris implementation. +In practice all these are connected. + +The details are generally much simpler when strings may contain only ASCII characters. +When strings may include non-ascii characters, this requires a specific encoding to be +adopted when translating to and from bytes, and rules for determining what the encoding +is or was. + +Another significant factor is the way in which all the relevant projects have changed +their features and methods over time, so that historic datasets may often use +non-standard approaches to record string data. + + +In the NetCDF file format +^^^^^^^^^^^^^^^^^^^^^^^^^ +In the NetCDF v4 implementation, there are three specific areas where the datatype and +storage characteristics of character data are relevant: + +* The **names** of file components (variables, dimensions, and attributes) are + natively unicode-capable strings of arbitrary (variable) length. + +* **Attributes** with string content likewise *appear* to be natively unicode. However, + the actual datatype of the attribute may vary, being either 'char' or 'string'. + +* The **content of variables** can be either 'char' or 'string'. + + * 'string' type variables contain a variable-length unicode string at each array element. + + * 'char' type variables contain one-byte characters, and generally have a fixed-length + "string dimension". If they contain *only* ascii character values, this is + uncomplicated, but they may also be used to contain non-ascii data (i.e. + including unicode characters). There is no universally defined agreement for + how to indicate that bytes are encoded non-ascii data, but many older datasets + have used a variable attribute ``_Encoding`` indicating the encoding name. + +The NetCDF documentation also mentions that an ``_Encoding`` attribute may be used to +represent non-ascii strings. However this is described as "reserved for future use", +and its valid values and effects are not explicitly defined. + +However, it is also notable that the standard ``ncgen`` and ``ncdump`` tools *do* +correctly interpret an ``_Encoding`` attribute in most cases, despite this not being an +"official" solution. + + +In the netCDF CF Conventions +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The `CF Conventions `_ define a subset of +"allowed" datatypes, and various types of data elements represented by variables +-- such as data variables, auxiliary coordinates, cell methods, etc. + +CF currently supports the use of either netcdf 'string' or 'char' arrays for any +variables. +However, *historically*, CF has had more limited support, and also "unofficial +conventions" have been used for string data encoded as bytes, which may be encountered +in older datasets. + +**Prior to v1.8**, CF required to use 'char' type only, and provided +**no official means** of representing non-ascii data. + +**Since v1.8**, CF has allowed the use of 'string' data in all variables. +However, up to v1.12 there was still no official way of encoding non-ascii data in +'char' arrays. + +**Since v1.12**, CF now mandates a *default* assumption of utf-8 encoding to store +non-ascii data in 'char' form. It does also note that some data in the past has used an +``_Encoding`` attribute -- though this was never an official CF usage. + +Characteristics +~~~~~~~~~~~~~~~ +Where strings are stored as 'char' type, which is the more common traditional approach, +the array must have a "string dimension", which is a normal file dimension. Thus, these +strings always have a *fixed byte width*. (However, that is not the same as a fixed +*character* width, since in most encodings non-ascii characters require more bytes to +store). + +Although the variable-length 'string' data is now supported in CF, the use of +fixed-width 'char' arrays is obviously more efficient for storage and access, and it is +still the most common approach in practice. + + +In the netCDF4 Python module +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* attributes with string content always appear as Python 'str' (i.e. unicode strings). + It is not possible to distinguish or control the 'char' and 'string' type in the file, + this is hidden from the user by the Python implementation. + +* variable data of type 'string' is presented (read and written) as numpy arrays of dtype + "U", where is a (maximum) string length. + +* variable data of type 'char' is presented (read and written) as numpy arrays of + dtype "S1" -- that is, an array of length-1 Python "bytes" objects. + + .. note:: + + The netCDF4 package can also automatically translate this to string arrays of + dtype "U", if the variable has an ``_Encoding`` attribute. **However,** + Iris turns this feature *off*, in order to implement its own wider-ranging + encoding support (described below). + + +In Iris +^^^^^^^ +.. note:: + + In Iris, **the 'string' data type is not supported at present**, though this is + planned for future releases. See the following section `Variable-length datatypes`_ + for an interim solution enabling you at least to *load* variable-length string data. + +Iris stores string data in arrays of dtype "U", where is a maximum character +width. However, this data is currently **only** read and written in netCDF files as +'char' type variables (i.e. byte arrays). + +Iris supports a specific set of valid encodings for non-ascii data : +"ascii", "utf8", "utf16" and "utf32". These (or aliases) will appear in the +``_Encoding`` attribute of a file variable, and likewise as an attribute of the +corresponding Iris component object (e.g. cube or coordinate). + +When loading +~~~~~~~~~~~~ +If there is a valid ``_Encoding`` attribute this is used to decode the +data, otherwise a default encoding of "utf8" is applied: This works transparently when +only ascii characters are present, and also allows the ``_Encoding`` attribute to be +omitted as long as utf8 was used. An invalid or unsupported encoding name will be +ignored, with a warning, but the attribute will still be added to the Iris component +object. + +When saving +~~~~~~~~~~~ +Any string data with only ascii characters does not require an ``_Encoding`` attribute. +However if there are any non-ascii characters, and no ``_Encoding`` +attribute, then an error will be raised. An invalid or unsupported encoding name will +be ignored, with a warning, but the attribute will still be stored to the file. + +So effectively, the **default encoding is 'utf8' for load and 'ascii' for save**. + +String width dimensions +~~~~~~~~~~~~~~~~~~~~~~~ +For each valid encoding there is a definite relation between the string dimension length +in the file (actually, the number of *bytes*), and the maximum character length, aka +string width, in the array dtype : i.e. the "" in the "U" dtype. + +The **lengths of string dimensions created on write** are calculated as follows: + +* ascii : n-bytes = n-characters +* utf8 : n-bytes = n-characters +* utf16 : n-bytes = 2 * (n-characters + 1) +* utf32 : n-bytes = 4 * (n-characters + 1) + +For reading, the inverse relations are applied to determine the '"U"' dtype in which +the data is presented. This will always round-trip correctly, i.e. the dimension length +is unchanged if data is read and then written back. + +For 'ascii' and 'utf32' this character-to-byte relationship is simple + fixed, but for +'utf8' and 'utf16', the number of encoded bytes depends on the actual characters present +**and can exceed the numbers given above**. If any string in the *actual* data encodes +to more bytes than the above-calculated string dimension, then Iris will raise an +:class:`iris.exceptions.TranslationError`. In this case, the user must **explicitly +specify** a longer string dimension, by converting the data to a longer "U" dtype : +for example, ``cube.data = cube.core_data().astype("U20")``. + +.. warning:: + + When processing string arrays, Numpy does not routinely preserve the "" width part + of "U" type data : instead, some operations will reduce it to the maximum width + occurring. So in these cases also, it may be necessary to explicitly re-assert the + desired "string width" before saving -- use ``.astype()``, as above. + + Variable-length datatypes ------------------------- diff --git a/lib/iris/fileformats/netcdf/__init__.py b/lib/iris/fileformats/netcdf/__init__.py index 60b4b5895a..795a9c5af0 100644 --- a/lib/iris/fileformats/netcdf/__init__.py +++ b/lib/iris/fileformats/netcdf/__init__.py @@ -48,7 +48,6 @@ "CFNameCoordMap", "CF_CONVENTIONS_VERSION", "DEBUG", - "DECODE_TO_STRINGS_ON_READ", "DEFAULT_READ_ENCODING", "DEFAULT_WRITE_ENCODING", "MESH_ELEMENTS",