encode

The encode grouping of functions (encode, encode_to, and encode_into) perform the task of doing bulk decoding from an input of code_points to the encoding’s code_unit type. They are also accompanied by encode_one variants (encode_one, encode_one_to, encode_one_into), which serve the same purpose as their bulk counterpoints but only do a single indivisible unit of work’s worth of work.

Named Groups

There are 3 named functions for this behavior, and each function comes with several function overloads. Each named function produces increasingly more information, letting you opt into just how much information and control you’d like over the algorithm and behavior. The first one simply returns a container with the transformation applied, discarding much of the operation’s result information. This is useful for quick, one-off conversions where you do not care about any errors and would rather let it be handled by the error handler. The second _to suffixed functions return a container within a result type that contains additional information. The final _into suffixed functions take an output range to write into, letting you explicitly control just how much space there is to write into as well as returning a detailed result type.

The return type for these function calls is one of:

encode(...)

This is the highest level bulk function.

This set of function overloads takes the provided input, encoding, handler and state and produces an output container type. The default container will either be a std::basic_string of the code_unit type, or a std::vector if it is not a known “character” type.

The container type can be specified by passing it as an explicit template parameter to this function, such as ztd::text::encode<std::vector<std::byte>>(U"bark", ztd::text::utf16_be{});. The output container is default constructed.

It will either call push_back/insert directly on the target container to fill it up, or serialize data to a temporary buffer (controlled by ZTD_TEXT_INTERMEDIATE_TRANSCODE_BUFFER_BYTE_SIZE) before then copying it into the desired output container through any available means (bulk .insert, repeated .push_back, or repeated single .insert with the .cend() iterator in that order).

This is the “fire and forget” version of the encode function, returning only the container and not returning any of the result or state information used to construct it.

encode_to(...)

This is the mid level bulk function.

This set of function overloads takes the provided input, encoding, handler and state and produces an output container type that is embedded within a ztd::text::encode_result, or a ztd::text::stateless_encode_result, depending on whether or not you called the version which takes a ztd::text::encode_state_t<Encoding> as a parameter or if it had to create one on the stack internally and discard it after the operation was finished.

The container type can be specified by passing it as an explicit template parameter to this function, such as ztd::text::encode_to<std::string>(U"meow", ascii{});. The output container is default constructed.

It will either call push_back/insert directly on the target container to fill it up, or serialize data to a temporary buffer (controlled by ZTD_TEXT_INTERMEDIATE_TRANSCODE_BUFFER_BYTE_SIZE) before then copying it into the desired output container through any available means (bulk .insert, repeated .push_back, or repeated single .insert with the .cend() iterator in that order).

If nothing goes wrong or the error handler lets the algorithm continue, .input on the result should be empty.

encode_into(...)

This is the lowest level bulk function.

This set of function overloads takes the provided input, encoding, output, handler, and state and writes data into the output range specified by output. The result is a ztd::text::encode_result, or a ztd::text::stateless_encode_result, depending on whether or not you called the version which takes a ztd::text::encode_state_t<Encoding> as a parameter or if it had to create one on the stack internally and discard it after the operation was finished.

It is up to the end-user to provide a suitably-sized output range for output, otherwise this operation may return with ztd::text::encoding_error::insufficient_output. for the result‘s error_code member. The amount of space consumed can be determined by checking the std::distance between the .begin() of the original output parameter and the .begin() of the returned .output member. The result also has error information and an .input member for checking how much input was consumed.

If nothing goes wrong or the error handler lets the algorithm continue, .input on the result should be empty.

For Everything

All named functions have 4 overloads. Each of the “higher level” functions, at the end of their overload call chain, will call the lower-level encode_into to perform the work. The final encode_into call uses the following ordering of extension points into calling the base implementation:

  • text_encode_into(input, encoding, output, handler, state)

  • An internal, implementation-defined customization point.

  • basic_encode_into(input, encoding, output, handler, state)

The final function call, basic_encode_into, simply performs the core encode loop using the Lucky 7 design. This design also means minimal stack space is used, keeping the core algorithm suitable for resource-constrained devices.

Note

👉 This means that if you implement none of the extension points whatsoever, implementing the basic encode_one function on your Encoding Object type will guarantee a proper, working implementation.

Note

👉 If you need to call the “basic” form of this function that takes no secret implementation shortcuts or user-defined extension points, then call basic_encode_into directly. This can be useful to stop infinity loops when your extension points cannot handle certain inputs and thereby needs to “delegate” to the basic case.


Bulk Functions

template<typename _Input, typename _Encoding, typename _Output, typename _ErrorHandler, typename _State>
constexpr auto basic_encode_into(_Input &&__input, _Encoding &&__encoding, _Output &&__output, _ErrorHandler &&__error_handler, _State &__state)

Converts the code points of the given __input view through the encoding to code units into the __output view.

Remark

This function does not attempt to call any extension points for encoding. It simply uses the encoding and attempts to encode by repeatedly calling the encoding’s required encode_one function.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

  • __output[in] An output_view to write code units to as the result of the encode operation from the intermediate code points.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

  • __state[inout] A reference to the associated state for the __encoding ‘s encode step.

Returns

A ztd::text::encode_result object that contains references to __state.

template<typename _Input, typename _Encoding, typename _Output, typename _ErrorHandler, typename _State>
constexpr auto encode_into(_Input &&__input, _Encoding &&__encoding, _Output &&__output, _ErrorHandler &&__error_handler, _State &__state)

Converts the code points of the given __input view through the encoding to code units into the __output view.

Remark

This function detects whether or not the ADL extension point text_encode can be called with the provided parameters. If so, it will use that ADL extension point over the default implementation. Otherwise, it will loop over the two encodings and attempt to encode by repeatedly calling the encoding’s required encode_one function.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

  • __output[in] An output_view to write code units to as the result of the encode operation from the intermediate code points.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

  • __state[inout] A reference to the associated state for the __encoding ‘s encode step.

Returns

A ztd::text::encode_result object that contains references to __state.

template<typename _Input, typename _Encoding, typename _Output, typename _ErrorHandler>
constexpr auto encode_into(_Input &&__input, _Encoding &&__encoding, _Output &&__output, _ErrorHandler &&__error_handler)

Converts the code points of the given __input view through the encoding to code units into the __output view.

Remark

Creates a default state using ztd::text::make_encode_state.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

  • __output[in] An output_view to write code units to as the result of the encode operation from the intermediate code points.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

Returns

A ztd::text::stateless_encode_result object that contains references to __state.

template<typename _Input, typename _Encoding, typename _Output>
constexpr auto encode_into(_Input &&__input, _Encoding &&__encoding, _Output &&__output)

Converts the code points of the given __input view through the encoding to code units into the __output view.

Remark

Creates a default error_handler that is similar to ztd::text::default_handler_t, but marked as careless.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

  • __output[in] An output_view to write code units to as the result of the encode operation from the intermediate code points.

Returns

A ztd::text::stateless_encode_result object that contains references to __state.

template<typename _Input, typename _Output>
constexpr auto encode_into(_Input &&__input, _Output &&__output)

Converts the code points of the given __input view through the encoding to code units into the __output view.

Remark

Creates a default encoding by figuring out the value_type of the __input, then passing that type into ztd::text::default_code_point_encoding_t. That encoding is that used to encode the input code points, by default.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __output[in] An output_view to write code units to as the result of the encode operation from the intermediate code points.

Returns

A ztd::text::stateless_encode_result object that contains references to __state.

template<typename _OutputContainer = void, typename _Input, typename _Encoding, typename _ErrorHandler, typename _State>
constexpr auto encode_to(_Input &&__input, _Encoding &&__encoding, _ErrorHandler &&__error_handler, _State &__state)

Converts the code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function detects creates a container of type _OutputContainer and uses a typical std::back_inserter or std::push_back_inserter to fill in elements as it is written to. The result is then returned, with the .output value put into the container.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

  • __state[inout] A reference to the associated state for the __encoding ‘s encode step.

Returns

A ztd::text::encode_result object that contains references to __state and an output of type _OutputContainer.

template<typename _OutputContainer = void, typename _Input, typename _Encoding, typename _ErrorHandler>
constexpr auto encode_to(_Input &&__input, _Encoding &&__encoding, _ErrorHandler &&__error_handler)

Converts the code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates a state using ztd::text::make_encode_state.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

Returns

A ztd::text::stateless_encode_result object whose output is of type _OutputContainer.

template<typename _OutputContainer = void, typename _Input, typename _Encoding>
constexpr auto encode_to(_Input &&__input, _Encoding &&__encoding)

Converts the code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates a handler using ztd::text::default_handler_t, but marks it as careless.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

Returns

A ztd::text::stateless_encode_result object whose output is of type _OutputContainer.

template<typename _OutputContainer = void, typename _Input>
constexpr auto encode_to(_Input &&__input)

Converts the code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates an encoding by using the value_type of the __input which is then passed through the ztd::text::default_code_point_encoding type to get the default desired encoding.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters

__input[in] An input_view to read code points from and use in the encode operation that will produce code units.

Returns

A ztd::text::stateless_encode_result object whose output is of type _OutputContainer.

template<typename _OutputContainer = void, typename _Input, typename _Encoding, typename _ErrorHandler, typename _State>
constexpr auto encode(_Input &&__input, _Encoding &&__encoding, _ErrorHandler &&__error_handler, _State &__state)

Converts the code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function detects creates a container of type _OutputContainer and uses a typical std::back_inserter or std::push_back_inserter to fill in elements as it is written to.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

  • __state[inout] A reference to the associated state for the __encoding ‘s encode step.

Returns

An object of type _OutputContainer .

template<typename _OutputContainer = void, typename _Input, typename _Encoding, typename _ErrorHandler>
constexpr auto encode(_Input &&__input, _Encoding &&__encoding, _ErrorHandler &&__error_handler)

Converts the code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates a state using ztd::text::make_encode_state.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

Returns

An object of type _OutputContainer .

template<typename _OutputContainer = void, typename _Input, typename _Encoding>
constexpr auto encode(_Input &&__input, _Encoding &&__encoding)

Converts the code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates a handler using ztd::text::default_handler_t, but marks it as careless.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode the input’s code points into output code units.

Returns

An object of type _OutputContainer .

template<typename _OutputContainer = void, typename _Input>
constexpr auto encode(_Input &&__input)

Converts the code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates an encoding by using the value_type of the __input which is then passed through the ztd::text::default_code_point_encoding type to get the default desired encoding.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters

__input[in] An input_view to read code points from and use in the encode operation that will produce code units.

Returns

An object of type _OutputContainer .

Single Functions

template<typename _Input, typename _Encoding, typename _Output, typename _ErrorHandler, typename _State>
constexpr auto encode_one_into(_Input &&__input, _Encoding &&__encoding, _Output &&__output, _ErrorHandler &&__error_handler, _State &__state)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units into the __output view.

Remark

This function takes care of performing some “normalizations” of the output and input range types (like turning them into a span or string_view if at all recognizable or preferable).

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

  • __output[in] An output_view to write code units to as the result of the encode_one operation from the intermediate code points.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

  • __state[inout] A reference to the associated state for the __encoding ‘s encode_one step.

Returns

A ztd::text::encode_one_result object that contains references to __state.

template<typename _Input, typename _Encoding, typename _Output, typename _ErrorHandler>
constexpr auto encode_one_into(_Input &&__input, _Encoding &&__encoding, _Output &&__output, _ErrorHandler &&__error_handler)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units into the __output view.

Remark

Creates a default state using ztd::text::make_encode_state.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

  • __output[in] An output_view to write code units to as the result of the encode_one operation from the intermediate code points.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

Returns

A ztd::text::stateless_encode_one_result object that contains references to __state.

template<typename _Input, typename _Encoding, typename _Output>
constexpr auto encode_one_into(_Input &&__input, _Encoding &&__encoding, _Output &&__output)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units into the __output view.

Remark

Creates a default error_handler that is similar to ztd::text::default_handler_t, but marked as careless.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

  • __output[in] An output_view to write code units to as the result of the encode_one operation from the intermediate code points.

Returns

A ztd::text::stateless_encode_one_result object that contains references to __state.

template<typename _Input, typename _Output>
constexpr auto encode_one_into(_Input &&__input, _Output &&__output)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units into the __output view.

Remark

Creates a default encoding by figuring out the value_type of the __input, then passing that type into ztd::text::default_code_point_encoding_t. That encoding is that used to encode_one the input code points, by default.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __output[in] An output_view to write code units to as the result of the encode_one operation from the intermediate code points.

Returns

A ztd::text::stateless_encode_one_result object that contains references to __state.

template<typename _OutputContainer, typename _Input, typename _Encoding, typename _ErrorHandler, typename _State>
constexpr auto encode_one_to(_Input &&__input, _Encoding &&__encoding, _ErrorHandler &&__error_handler, _State &__state)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function detects creates a container of type _OutputContainer and uses a typical std::back_inserter or std::push_back_inserter to fill in elements as it is written to. The result is then returned, with the .output value put into the container.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

  • __state[inout] A reference to the associated state for the __encoding ‘s encode_one step.

Returns

A ztd::text::encode_one_result object that contains references to __state and an output of type _OutputContainer.

template<typename _OutputContainer, typename _Input, typename _Encoding, typename _ErrorHandler>
constexpr auto encode_one_to(_Input &&__input, _Encoding &&__encoding, _ErrorHandler &&__error_handler)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates a state using ztd::text::make_encode_state.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

Returns

A ztd::text::stateless_encode_one_result object whose output is of type _OutputContainer.

template<typename _OutputContainer, typename _Input, typename _Encoding>
constexpr auto encode_one_to(_Input &&__input, _Encoding &&__encoding)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates a handler using ztd::text::default_handler_t, but marks it as careless.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

Returns

A ztd::text::stateless_encode_one_result object whose output is of type _OutputContainer.

template<typename _OutputContainer, typename _Input>
constexpr auto encode_one_to(_Input &&__input)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates an encoding by using the value_type of the __input which is then passed through the ztd::text::default_code_point_encoding type to get the default desired encoding.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters

__input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

Returns

A ztd::text::stateless_encode_one_result object whose output is of type _OutputContainer.

template<typename _OutputContainer = void, typename _Input, typename _Encoding, typename _ErrorHandler, typename _State>
constexpr auto encode_one(_Input &&__input, _Encoding &&__encoding, _ErrorHandler &&__error_handler, _State &__state)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function detects creates a container of type _OutputContainer and uses a typical std::back_inserter or std::push_back_inserter to fill in elements as it is written to.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

  • __state[inout] A reference to the associated state for the __encoding ‘s encode_one step.

Returns

An object of type _OutputContainer .

template<typename _OutputContainer = void, typename _Input, typename _Encoding, typename _ErrorHandler>
constexpr auto encode_one(_Input &&__input, _Encoding &&__encoding, _ErrorHandler &&__error_handler)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates a state using ztd::text::make_encode_state.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

  • __error_handler[in] The error handlers for the from and to encodings, respectively.

Returns

An object of type _OutputContainer .

template<typename _OutputContainer = void, typename _Input, typename _Encoding>
constexpr auto encode_one(_Input &&__input, _Encoding &&__encoding)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates a handler using ztd::text::default_handler_t, but marks it as careless.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters
  • __input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

  • __encoding[in] The encoding that will be used to encode_one the input’s code points into output code units.

Returns

An object of type _OutputContainer .

template<typename _OutputContainer = void, typename _Input>
constexpr auto encode_one(_Input &&__input)

Converts a single indivisible unit of work’s worth of code points of the given __input view through the encoding to code units in the specified _OutputContainer type.

Remark

This function creates an encoding by using the value_type of the __input which is then passed through the ztd::text::default_code_point_encoding type to get the default desired encoding.

Template Parameters

_OutputContainer – The container type to serialize data into.

Parameters

__input[in] An input_view to read code points from and use in the encode_one operation that will produce code units.

Returns

An object of type _OutputContainer .