An iterator which returns Unicode character values from a UTF-8 encoded string. More...

#include <unicode.h>

Public Types
typedef std::input_iterator_tag	iterator_category
	We implement the semantics of an STL input_iterator. More...

typedef unsigned	value_type

typedef size_t	difference_type

typedef value_type *	pointer

typedef value_type	reference

Public Member Functions
const char *	raw () const
	Return the raw const char* pointer for the current position. More...

size_t	left () const
	Return the number of bytes left in the iterator's buffer. More...

void	assign (const char *p_, size_t len)
	Assign a new string to the iterator. More...

void	assign (std::string_view s)
	Assign a new string to the iterator. More...

	Utf8Iterator (const char *p_, size_t len)
	Create an iterator given a pointer and a length. More...

	Utf8Iterator (std::string_view s)
	Create an iterator given a string. More...

	Utf8Iterator () noexcept
	Create an iterator which is at the end of its iteration. More...

unsigned	operator* () const noexcept
	Get the current Unicode character value pointed to by the iterator. More...

Utf8Iterator	operator++ (int)
	Move forward to the next Unicode character. More...

Utf8Iterator &	operator++ ()
	Move forward to the next Unicode character. More...

bool	operator== (const Utf8Iterator &other) const noexcept
	Test two Utf8Iterators for equality. More...

bool	operator!= (const Utf8Iterator &other) const noexcept
	Test two Utf8Iterators for inequality. More...

Private Member Functions
bool	calculate_sequence_length () const noexcept

unsigned	get_char () const

	Utf8Iterator (const unsigned char p_, const unsigned char end_, unsigned seqlen_)

unsigned	strict_deref () const noexcept

Private Attributes
const unsigned char *	p

const unsigned char *	end

unsigned	seqlen

Detailed Description

An iterator which returns Unicode character values from a UTF-8 encoded string.

Definition at line 39 of file unicode.h.

Member Typedef Documentation

◆ difference_type

typedef size_t Xapian::Utf8Iterator::difference_type

Definition at line 214 of file unicode.h.

◆ iterator_category

typedef std::input_iterator_tag Xapian::Utf8Iterator::iterator_category

We implement the semantics of an STL input_iterator.

Definition at line 212 of file unicode.h.

◆ pointer

typedef value_type* Xapian::Utf8Iterator::pointer

Definition at line 215 of file unicode.h.

◆ reference

typedef value_type Xapian::Utf8Iterator::reference

Definition at line 216 of file unicode.h.

◆ value_type

typedef unsigned Xapian::Utf8Iterator::value_type

Definition at line 213 of file unicode.h.

Constructor & Destructor Documentation

◆ Utf8Iterator() [1/4]

Xapian::Utf8Iterator::Utf8Iterator	(	const unsigned char *	p_,
		const unsigned char *	end_,
		unsigned	seqlen_
	)

inlineprivate

Definition at line 48 of file unicode.h.

◆ Utf8Iterator() [2/4]

Xapian::Utf8Iterator::Utf8Iterator	(	const char *	p_,
		size_t	len
	)

inline

Create an iterator given a pointer and a length.

The iterator will return characters from the start of the string when next called. The string is not copied into the iterator, so it must remain valid while the iteration is in progress.

Parameters

p_	A pointer to the start of the string to read.
len	The length of the string to read.

Definition at line 105 of file unicode.h.

◆ Utf8Iterator() [3/4]

Xapian::Utf8Iterator::Utf8Iterator ( std::string_view s )

inlineexplicit

Create an iterator given a string.

The iterator will return characters from the start of the string when next called. The string is not copied into the iterator, so it must remain valid while the iteration is in progress.

Parameters

s	The string to read. Must not be modified while the iteration is in progress.

This parameter is of type std::string_view, so you can pass in types which automatically convert to that such as std::string, or a const char* pointing to a nul-terminated string.

Definition at line 122 of file unicode.h.

◆ Utf8Iterator() [4/4]

Xapian::Utf8Iterator::Utf8Iterator ( )

inlinenoexcept

Create an iterator which is at the end of its iteration.

This can be compared to another iterator to check if the other iterator has reached its end.

Definition at line 129 of file unicode.h.

Member Function Documentation

◆ assign() [1/2]

void Xapian::Utf8Iterator::assign	(	const char *	p_,
		size_t	len
	)

inline

Assign a new string to the iterator.

The iterator will forget the string it was iterating through, and return characters from the start of the new string when next called. The string is not copied into the iterator, so it must remain valid while the iteration is in progress.

Parameters

p_	A pointer to the start of the string to read.
len	The length of the string to read.

Definition at line 73 of file unicode.h.

References p.

Referenced by Xapian::SnipPipe::drain().

◆ assign() [2/2]

void Xapian::Utf8Iterator::assign ( std::string_view s )

inline

Assign a new string to the iterator.

The iterator will forget the string it was iterating through, and return characters from the start of the new string when next called. The string is not copied into the iterator, so it must remain valid while the iteration is in progress.

Parameters

s	The string to read. Must not be modified while the iteration is in progress.

Definition at line 93 of file unicode.h.

References assign().

Referenced by assign().

◆ calculate_sequence_length()

bool Xapian::Utf8Iterator::calculate_sequence_length ( ) const

privatenoexcept

Definition at line 66 of file utf8itor.cc.

References bad_cont(), and p.

◆ get_char()

unsigned Xapian::Utf8Iterator::get_char ( ) const

private

◆ left()

size_t Xapian::Utf8Iterator::left ( ) const

inline

Return the number of bytes left in the iterator's buffer.

Definition at line 60 of file unicode.h.

References p.

Referenced by Xapian::break_words(), and Xapian::parse_terms().

◆ operator!=()

bool Xapian::Utf8Iterator::operator!= ( const Utf8Iterator & other ) const

inlinenoexcept

Test two Utf8Iterators for inequality.

Parameters

other The Utf8Iterator to compare this one with.

Returns: true iff the iterators do not point to the same position.

Definition at line 206 of file unicode.h.

References p.

◆ operator*()

unsigned Xapian::Utf8Iterator::operator* ( ) const

noexcept

Get the current Unicode character value pointed to by the iterator.

If an invalid UTF-8 sequence is encountered, then the byte values comprising it are returned until valid UTF-8 or the end of the input is reached.

This handling applies to invalid byte sequences, truncated UTF-8 sequences, overlong sequences and (since Xapian 2.0.0) surrogate pair codepoints encoded as UTF-8.

If you want to reject or otherwise discriminate invalid UTF-8 sequences then see the strict_deref() method.

Returns unsigned(-1) if the iterator has reached the end of its buffer.

Definition at line 109 of file utf8itor.cc.

References p.

◆ operator++() [1/2]

Utf8Iterator& Xapian::Utf8Iterator::operator++ ( )

inline

Move forward to the next Unicode character.

Returns: A reference to this object.

Definition at line 184 of file unicode.h.

References p.

◆ operator++() [2/2]

Utf8Iterator Xapian::Utf8Iterator::operator++ ( int )

inline

Move forward to the next Unicode character.

Returns: An iterator pointing to the position before the move.

Definition at line 169 of file unicode.h.

References p.

◆ operator==()

bool Xapian::Utf8Iterator::operator== ( const Utf8Iterator & other ) const

inlinenoexcept

Test two Utf8Iterators for equality.

Parameters

other The Utf8Iterator to compare this one with.

Returns: true iff the iterators point to the same position.

Definition at line 197 of file unicode.h.

References p.

◆ raw()

const char* Xapian::Utf8Iterator::raw ( ) const

inline

Return the raw const char* pointer for the current position.

Definition at line 55 of file unicode.h.

References p.

Referenced by Xapian::break_words(), Xapian::SnipPipe::drain(), and Xapian::QueryParser::Internal::parse_term().

◆ strict_deref()

unsigned Xapian::Utf8Iterator::strict_deref ( ) const

privatenoexcept

Get the current Unicode character value pointed to by the iterator.

If an invalid UTF-8 sequence is encountered, then the byte values comprising it are returned with the top bit set (so the caller can differentiate these from the same values arising from valid UTF-8) until valid UTF-8 or the end of the input is reached.

This handling applies to invalid byte sequences, truncated UTF-8 sequences, overlong sequences and (since Xapian 2.0.0) surrogate pair codepoints encoded as UTF-8.

Returns unsigned(-1) if the iterator has reached the end of its buffer.

Definition at line 122 of file utf8itor.cc.

References p.

Member Data Documentation

◆ end

const unsigned char* Xapian::Utf8Iterator::end

private

Definition at line 41 of file unicode.h.

◆ p

const unsigned char* Xapian::Utf8Iterator::p

private

Definition at line 40 of file unicode.h.

◆ seqlen

unsigned Xapian::Utf8Iterator::seqlen

mutableprivate

Definition at line 42 of file unicode.h.

The documentation for this class was generated from the following files:

include/xapian/unicode.h
unicode/utf8itor.cc

Public Types

Public Member Functions

Private Member Functions

Private Attributes

Detailed Description

Member Typedef Documentation

◆ difference_type

◆ iterator_category

◆ pointer

◆ reference

◆ value_type

Constructor & Destructor Documentation

◆ Utf8Iterator() [1/4]

◆ Utf8Iterator() [2/4]

◆ Utf8Iterator() [3/4]

◆ Utf8Iterator() [4/4]

Member Function Documentation

◆ assign() [1/2]

◆ assign() [2/2]

◆ calculate_sequence_length()

◆ get_char()

◆ left()

◆ operator!=()

◆ operator*()

◆ operator++() [1/2]

◆ operator++() [2/2]

◆ operator==()

◆ raw()

◆ strict_deref()

Member Data Documentation

◆ end

◆ p

◆ seqlen