Count as Encoded

Counting encodable data is the action of finding out how many code units will result from a given sequence of already decoded information, AKA a sequence of code points. Essentially, we run the encoding algorithm loop, but instead of giving the end user the encoded values, we instead simply provide the count for running that bulk operation.

Thusly, we use the algorithm as below to do the work. Given an input of code_units with an encoding, an initial count set at 0, and any necessary additional state, we can generically predict how many code units will result from a decoding operation by running the following loop:

  • ⏩ Is the input value empty? If so, is the state finished and have nothing to output? If both are true, return the current results with the the empty input, curent count, and state, everything is okay ✅!

  • ⏩ Otherwise,

    1. Set up an intermediate storage location of code_units, using the max_code_units of the input encoding, for the next operations.

    2. Do the encode_one step from input (using its begin() and end()) into the intermediate code_unit storage location, saving the returned intermediate_output from the encode_one call.

      • 🛑 If it failed, return with the current input (unmodified from before this iteration, if possible), current count, and states.

    3. Compute the difference between the begin(intermediate) from the original step, and the begin(intermediate_output) returned by encode_one; add that difference to the current count.

  • ⏩ Update input‘s begin() value to point to after what was read by the encode_one step.

  • ⤴️ Go back to the start.

This involves a single encoding type, and so does not need any cooperation to count the code_points. Note that this algorithm doesn’t show what the error handler does; if the error handler “erases” the failure by setting the result type’s .error_code == ztd::text::encoding_error::ok, then the algorithm will keep going. This is useful to, for example, detect the maximum size of an operation even if it errors and would result in replacement characters being inserted (e.g., from ztd::text::replacement_handler_t).

Check out the API documentation for ztd::text::count_as_encoded to learn more.