std.utf

Encode and decode UTF-8, UTF-16 and UTF-32 strings.

UTF character support is restricted to

'\u0000' <= character <= '\U0010FFFF'.

Copyright

License

Boost License 1.0.

Authors

Walter Bright and Jonathan M Davis

Source: std/utf.d

var replacementDchar

tmpl byUTF codeUnitLimit toUTFz

alias byChar byDchar byWchar UseReplacementDchar

class UTFException

fn _utfException byCodeUnit canSearchInCodeUnits codeLength count decode decodeBack decodeFront decodeImpl encode invalidUTFstrings isValidCodepoint isValidDchar stride strideBack strideImpl toUCSindex toUTF16 toUTF16z toUTF32 toUTF8 toUTFImpl toUTFindex toUTFzImpl validate

Types 5

classUTFException : UnicodeException

Exception thrown on errors in std.utf functions.

Fields

uint[4] sequence

size_t len

Methods

UTFException setSequence(scope uint[] data...) @safe pure nothrow @nogc return

string toString() constReturns: A `string` detailing the invalid UTF sequence.

Constructors

this(string msg,  string file =  __FILE__,  size_t line =  __LINE__,
          Throwable next =  null)

Standard exception constructors.

this(string msg,  size_t index,  string file =  __FILE__,
          size_t line =  __LINE__,  Throwable next =  null)

ditto

aliasUseReplacementDchar = Flag!"useReplacementDchar"

Whether or not to replace invalid UTF with replacementDchar

aliasbyChar = byUTF!char

Iterate an input range of characters by char, wchar, or dchar. These aliases simply forward to byUTF with the corresponding C argument.

Parameters

`r`	input range of characters, or array of characters

aliasbyWchar = byUTF!wchar

Ditto

aliasbyDchar = byUTF!dchar

Ditto

Functions 51

fnauto invalidUTFstrings(Char)() if (isSomeChar!Char) @safe pure @nogc nothrow

fnbool isValidDchar(dchar c) pure nothrow @safe @nogcCheck whether the given Unicode code point is valid.

fnbool isValidCodepoint(Char)(Char c) if (isSomeChar!Char)Checks if a single character forms a valid code point.

uint stride(S)(auto  ref  S str,  size_t index) if (is(S :  const  char[])  ||
    (isRandomAccessRange!S &&  is(immutable  ElementType!S ==  immutable  char)))

Calculate the length of the UTF sequence starting at `index` in `str`.

uint stride(S)(auto  ref  S str) if (is(S :  const  char[])  ||
    (isInputRange!S &&  is(immutable  ElementType!S ==  immutable  char)))

Ditto

uint stride(S)(auto  ref  S str,  size_t index) if (is(S :  const  wchar[])  ||
    (isRandomAccessRange!S &&  is(immutable  ElementType!S ==  immutable  wchar)))

Ditto

fnuint stride(S)(auto ref S str) if (is(S : const wchar[])) @safe pureDitto

uint stride(S)(auto  ref  S str) if (isInputRange!S &&  is(immutable  ElementType!S ==  immutable  wchar)  &&
    !is(S :  const  wchar[]))

Ditto

uint stride(S)(auto  ref  S str,  size_t index =  0) if (is(S :  const  dchar[])  ||
    (isInputRange!S &&  is(immutable  ElementEncodingType!S ==  immutable  dchar)))

Ditto

private fnuint strideImpl(char c, size_t index) @trusted pure

uint strideBack(S)(auto  ref  S str,  size_t index) if (is(S :  const  char[])  ||
    (isRandomAccessRange!S &&  is(immutable  ElementType!S ==  immutable  char)))

Calculate the length of the UTF sequence ending one code unit before `index` in `str`.

uint strideBack(S)(auto  ref  S str) if (is(S :  const  char[])  ||
    (isRandomAccessRange!S &&  hasLength!S &&  is(immutable  ElementType!S ==  immutable  char)))

Ditto

uint strideBack(S)(auto  ref  S str) if (isBidirectionalRange!S &&  is(immutable  ElementType!S ==  immutable  char)  && !isRandomAccessRange!S)

Ditto

uint strideBack(S)(auto  ref  S str,  size_t index) if (is(S :  const  wchar[])  ||
    (isRandomAccessRange!S &&  is(immutable  ElementType!S ==  immutable  wchar)))

Ditto

uint strideBack(S)(auto  ref  S str) if (is(S :  const  wchar[])  ||
    (isBidirectionalRange!S &&  is(immutable  ElementType!S ==  immutable  wchar)))

Ditto

uint strideBack(S)(auto  ref  S str,  size_t index) if (isRandomAccessRange!S &&  is(immutable  ElementEncodingType!S ==  immutable  dchar))

Ditto

uint strideBack(S)(auto  ref  S str) if (isBidirectionalRange!S &&  is(immutable  ElementEncodingType!S ==  immutable  dchar))

Ditto

fnsize_t toUCSindex(C)(const(C)[] str, size_t index) if (isSomeChar!C) @safe pureGiven `index` into `str` and assuming that `index` is at the start of a UTF sequence, `toUCSindex` determines the number of UCS characters up to `index`. So, `index` is the index of a code unit at ...

fnsize_t toUTFindex(C)(const(C)[] str, size_t n) if (isSomeChar!C) @safe pureGiven a UCS index `n` into `str`, returns the UTF index. So, `n` is how many code points into the string the code point is, and the array index of the code unit is returned.

dchar decode(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(auto  ref  S str,  ref  size_t index) if (!isSomeString!S &&
     isRandomAccessRange!S &&  hasSlicing!S &&  hasLength!S &&  isSomeChar!(ElementType!S))

Decodes and returns the code point starting at `str[index]`. `index` is advanced to one past the decoded code point. If the code point is not well-formed, then a `UTFException` is thrown and `index...

dchar decode(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(
 auto  ref  scope  S str,  ref  size_t index) if (isSomeString!S)

@trusted pureditto

dchar decodeFront(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(
 ref  S str,  out  size_t numCodeUnits) if (!isSomeString!S &&  isInputRange!S &&  isSomeChar!(ElementType!S))

`decodeFront` is a variant of decode which specifically decodes the first code point. Unlike decode, `decodeFront` accepts any input range of code units (rather than just a string or random access ...

dchar decodeFront(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(
 ref  scope  S str,  out  size_t numCodeUnits) if (isSomeString!S)

@trusted pureditto

dchar decodeFront(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(ref  S str) if (isInputRange!S &&  isSomeChar!(ElementType!S))

Ditto

dchar decodeBack(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(
     ref  S str,  out  size_t numCodeUnits) if (isSomeString!S)

`decodeBack` is a variant of decode which specifically decodes the last code point. Unlike decode, `decodeBack` accepts any bidirectional range of code units (rather than just a string or random ac...

dchar decodeBack(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(
     ref  S str,  out  size_t numCodeUnits) if (!isSomeString!S &&  isSomeChar!(ElementType!S)  &&  isBidirectionalRange!S
    && ((isRandomAccessRange!S &&  hasLength!S)  || !isRandomAccessRange!S))

Ditto

dchar decodeBack(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(ref  S str) if (isSomeString!S
    || (isRandomAccessRange!S &&  hasLength!S &&  isSomeChar!(ElementType!S))
     || (!isRandomAccessRange!S &&  isBidirectionalRange!S &&  isSomeChar!(ElementType!S)))

Ditto

private fn

dchar decodeImpl(bool  canIndex,  UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(
     auto  ref  S str,  ref  size_t index) if (
     is(S :  const  char[])  || (isInputRange!S &&  is(immutable  ElementEncodingType!S ==  immutable  char)))

private fn

dchar decodeImpl(bool  canIndex,  UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(auto  ref  S str,  ref  size_t index) if (is(S :  const  wchar[])  || (isInputRange!S &&  is(immutable  ElementEncodingType!S ==  immutable  wchar)))

private fn

dchar decodeImpl(bool  canIndex,  UseReplacementDchar useReplacementDchar =  No.useReplacementDchar,  S)(
     auto  ref  S str,  ref  size_t index) if (is(S :  const  dchar[])  || (isInputRange!S &&  is(immutable  ElementEncodingType!S ==  immutable  dchar)))

private fndchar _utfException(UseReplacementDchar useReplacementDchar)(string msg, dchar c)

size_t encode(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar)(
     out  char[4]  buf,  dchar  c)

@safe pureEncodes `c` into the static array, `buf`, and returns the actual length of the encoded character (a number between `1` and `4` for `char[4]` buffers and a number between `1` and `2` for `wchar[2]` ...

size_t encode(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar)(
     out  wchar[2]  buf,  dchar  c)

@safe pureDitto

size_t encode(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar)(
     out  dchar[1]  buf,  dchar  c)

@safe pureDitto

void encode(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar)(
     ref  scope  char[]  str,  dchar  c)

@safe pureEncodes `c` in `str`'s encoding and appends it to `str`.

void encode(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar)(
     ref  scope  wchar[]  str,  dchar  c)

@safe pureditto

void encode(UseReplacementDchar useReplacementDchar =  No.useReplacementDchar)(
     ref  scope  dchar[]  str,  dchar  c)

@safe pureditto

fnubyte codeLength(C)(dchar c) if (isSomeChar!C) @safe pure nothrow @nogcReturns the number of code units that are required to encode the code point `c` when `C` is the character type used to encode it.

fnsize_t codeLength(C, InputRange)(InputRange input) if (isSomeFiniteCharInputRange!InputRange)Returns the number of code units that are required to encode `str` in a string whose character type is `C`. This is particularly useful when slicing one string with the length of another and the tw...

fnbool canSearchInCodeUnits(C)(dchar c) if (isSomeChar!C)

fnvoid validate(S)(in S str) if (isSomeString!S) @safe pureChecks to see if `str` is well-formed unicode or not.

fnstring toUTF8(S)(S s) if (isSomeFiniteCharInputRange!S)Encodes the elements of `s` to UTF-8 and returns a newly allocated string of the elements.

fnwstring toUTF16(S)(S s) if (isSomeFiniteCharInputRange!S)Encodes the elements of `s` to UTF-16 and returns a newly GC allocated `wstring` of the elements.

fndstring toUTF32(S)(scope S s) if (isSomeFiniteCharInputRange!S)Encodes the elements of `s` to UTF-32 and returns a newly GC allocated `dstring` of the elements.

private fnT toUTFImpl(T, S)(scope S s)

private fnP toUTFzImpl(P, S)(return scope S str) if (is(immutable typeof(* P.init) == typeof(str[0]))) @safe pure

private fn

P toUTFzImpl(P,  S)(return  scope  S str) if (is(typeof(str[0])  C)  &&  is(immutable  typeof(* P.init)  ==  immutable  C)  && !is(C ==  immutable))

@safe pure

private fnP toUTFzImpl(P, S)(S str) if (!is(immutable typeof(* P.init) == immutable typeof(str[0]))) @safe pure

fnconst(wchar) * toUTF16z(C)(const(C)[] str) if (isSomeChar!C) @safe pure`toUTF16z` is a convenience function for `toUTFz!(const(wchar)*)`.

fnsize_t count(C)(const(C)[] str) if (isSomeChar!C) @safe pure nothrow @nogcReturns the total number of code points encoded in `str`.

auto byCodeUnit(R)(R r) if ((isConvertibleToString!R && !isStaticArray!R)  ||
    (isInputRange!R &&  isSomeChar!(ElementEncodingType!R)))

Iterate a range of char, wchar, or dchars by code unit.

Variables 1

enumvarreplacementDchar = '\uFFFD'

Inserted in place of invalid UTF sequences.

References:

http://en.wikipedia.org/wiki/Replacement_character#Replacement_character

Templates 3

tmplcodeUnitLimit(S) if (isSomeChar!(ElementEncodingType!S))

tmpltoUTFz(P) if (is(P == C *, C) && isSomeChar!C)

Returns a C-style zero-terminated string equivalent to str. str must not contain embedded '\0''s as any C function will treat the first '\0' that it sees as the end of the string. If str.empty is true, then a string containing only '\0' is returned.

toUTFz accepts any type of string and is templated on the type of character pointer that you wish to convert to. It will avoid allocating a new string if it can, but there's a decent chance that it will end up having to allocate a new string - particularly when dealing with character types other than char.

Warning 1: If the result of toUTFz equals str.ptr, then if

anything alters the character one past the end of str (which is the '\0' character terminating the string), then the string won't be zero-terminated anymore. The most likely scenarios for that are if you append to str and no reallocation takes place or when str is a slice of a larger array, and you alter the character in the larger array which is one character past the end of str. Another case where it could occur would be if you had a mutable character array immediately after str in memory (for example, if they're member variables in a user-defined type with one declared right after the other) and that character array happened to start with '\0'. Such scenarios will never occur if you immediately use the zero-terminated string after calling toUTFz and the C function using it doesn't keep a reference to it. Also, they are unlikely to occur even if you save the zero-terminated string (the cases above would be among the few examples of where it could happen). However, if you save the zero-terminate string and want to be absolutely certain that the string stays zero-terminated, then simply append a '\0' to the string and use its ptr property rather than calling toUTFz.

Warning 2: When passing a character pointer to a C function, and the

C function keeps it around for any reason, make sure that you keep a reference to it in your D code. Otherwise, it may go away during a garbage collection cycle and cause a nasty bug when the C code tries to use it.

Functions

P toUTFz(S)(S str) if (isSomeString!S)

tmplbyUTF(C, UseReplacementDchar useReplacementDchar = Yes.useReplacementDchar) if (isSomeChar!C)

Iterate an input range of characters by char type C by encoding the elements of the range.

UTF sequences that cannot be converted to the specified encoding are either replaced by U+FFFD per "5.22 Best Practice for U+FFFD Substitution" of the Unicode Standard 6.2 or result in a thrown UTFException. Hence byUTF is not symmetric. This algorithm is lazy, and does not allocate memory. @nogc, pure-ity, nothrow, and @safe-ty are inferred from the r parameter.

Parameters

`C`	`char`, `wchar`, or `dchar`
`useReplacementDchar`	UseReplacementDchar.yes means replace invalid UTF with `replacementDchar`, UseReplacementDchar.no means throw `UTFException` for invalid UTF

Throws

UTFException if invalid UTF sequence and useReplacementDchar is set to UseReplacementDchar.no

GC: Does not use GC if useReplacementDchar is set to UseReplacementDchar.yes

Returns

A bidirectional range if R is a bidirectional range and not auto-decodable,

as defined by isAutodecodableString.

A forward range if R is a forward range and not auto-decodable.

Or, if R is a range and it is auto-decodable and is(ElementEncodingType!typeof(r) == C), then the range is passed to byCodeUnit.

Otherwise, an input range of characters.

std.utf

See Also

Copyright

License

Authors

Types 5

Parameters

Functions 51

Variables 1

Templates 3

Parameters

Throws

Returns