Pack types into strings and unpack them again. More...

#include <cstring>
#include <string>
#include <type_traits>
#include "omassert.h"
#include "xapian/types.h"

Include dependency graph for pack.h:

This graph shows which files directly or indirectly include this file:

Functions
void	pack_bool (std::string &s, bool value)
	Append an encoded bool to a string. More...

bool	unpack_bool (const char *p, const char end, bool *result)
	Decode a bool from a string. More...

template<class U >
void	pack_uint_last (std::string &s, U value)
	Append an encoded unsigned integer to a string as the last item. More...

template<class U >
bool	unpack_uint_last (const char *p, const char end, U *result)
	Decode an unsigned integer as the last item in a string. More...

template<class U >
void	C_pack_uint_preserving_sort (std::string &s, U value)
	Append an encoded unsigned integer to a string, preserving the sort order. More...

template<class U >
bool	C_unpack_uint_preserving_sort (const char *p, const char end, U *result)
	Decode an "sort preserved" unsigned integer from a string. More...

template<class U >
void	pack_uint_preserving_sort (std::string &s, U value)
	Append an encoded unsigned integer to a string, preserving the sort order. More...

template<class U >
bool	unpack_uint_preserving_sort (const char *p, const char end, U *result)
	Decode a "sort preserved" unsigned integer from a string. More...

template<class U >
void	pack_uint (std::string &s, U value)
	Append an encoded unsigned integer to a string. More...

template<>
void	pack_uint (std::string &s, bool value)
	Append an encoded unsigned integer (bool type) to a string. More...

template<class U >
bool	unpack_uint (const char *p, const char end, U *result)
	Decode an unsigned integer from a string. More...

void	pack_string (std::string &s, const std::string &value)
	Append an encoded std::string to a string. More...

void	pack_string (std::string &s, const char *ptr)
	Append an encoded C-style string to a string. More...

bool	unpack_string (const char *p, const char end, std::string &result)
	Decode a std::string from a string. More...

void	pack_string_preserving_sort (std::string &s, const std::string &value, bool last=false)
	Append an encoded std::string to a string, preserving the sort order. More...

bool	unpack_string_preserving_sort (const char *p, const char end, std::string &result)
	Decode a "sort preserved" std::string from a string. More...

std::string	pack_chert_postlist_key (const std::string &term)

std::string	pack_chert_postlist_key (const std::string &term, Xapian::docid did)

std::string	pack_glass_postlist_key (const std::string &term)

std::string	pack_glass_postlist_key (const std::string &term, Xapian::docid did)

Variables
const unsigned int	SORTABLE_UINT_LOG2_MAX_BYTES = 2
	How many bits to store the length of a sortable uint in. More...

const unsigned int	SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES
	Calculated value used below. More...

const unsigned int	SORTABLE_UINT_1ST_BYTE_MASK
	Calculated value used below. More...

Detailed Description

Pack types into strings and unpack them again.

Definition in file pack.h.

Function Documentation

◆ C_pack_uint_preserving_sort()

template<class U >

void C_pack_uint_preserving_sort	(	std::string &	s,
		U	value
	)

inline

Append an encoded unsigned integer to a string, preserving the sort order.

[Chert variant]

The appended string data will sort in the same order as the unsigned integer being encoded.

Note that the first byte of the encoding will never be \xff, so it is safe to store the result of this function immediately after the result of pack_string_preserving_sort().

Parameters

s	The string to append to.
value	The unsigned integer to encode.

Definition at line 149 of file pack.h.

References Assert, SORTABLE_UINT_1ST_BYTE_MASK, SORTABLE_UINT_LOG2_MAX_BYTES, and SORTABLE_UINT_MAX_BYTES.

Referenced by make_key(), ChertTermListTable::make_key(), ChertPositionListTable::make_key(), make_slot_key(), make_valuechunk_key(), ChertCompact::merge_docid_keyed(), ChertCompact::PostlistCursor::next(), pack_chert_postlist_key(), and test_pack_uint_preserving_sort2().

◆ C_unpack_uint_preserving_sort()

template<class U >

bool C_unpack_uint_preserving_sort	(	const char **	p,
		const char *	end,
		U *	result
	)

inline

Decode an "sort preserved" unsigned integer from a string.

[Chert variant]

The unsigned integer must have been encoded with C_pack_uint_preserving_sort().

Parameters

p	Pointer to pointer to the current position in the string.
end	Pointer to the end of the string.
result	Where to store the result.

Definition at line 185 of file pack.h.

References Assert, rare, SORTABLE_UINT_1ST_BYTE_MASK, and SORTABLE_UINT_LOG2_MAX_BYTES.

Referenced by check_chert_table(), docid_from_key(), Chert::PostlistChunkWriter::flush(), ChertPostListTable::get_chunk(), ChertValueManager::get_chunk_containing_did(), ChertPostListTable::get_used_docid_range(), ChertCompact::merge_docid_keyed(), ChertPostList::move_to_chunk_containing(), ChertCompact::PostlistCursor::next(), ChertPostList::next_chunk(), and test_pack_uint_preserving_sort2().

◆ pack_bool()

void pack_bool	(	std::string &	s,
		bool	value
	)

inline

Append an encoded bool to a string.

Parameters

s	The string to append to.
value	The bool to encode.

Definition at line 57 of file pack.h.

Referenced by make_start_of_chunk().

◆ pack_chert_postlist_key() [1/2]

std::string pack_chert_postlist_key ( const std::string & term )

inline

Definition at line 585 of file pack.h.

References pack_string_preserving_sort().

Referenced by ChertPostListTable::get_used_docid_range(), ChertPostListTable::make_key(), ChertCompact::merge_postlists(), ChertAllTermsList::next(), and ChertAllTermsList::skip_to().

◆ pack_chert_postlist_key() [2/2]

std::string pack_chert_postlist_key	(	const std::string &	term,
		Xapian::docid	did
	)

inline

Definition at line 597 of file pack.h.

References C_pack_uint_preserving_sort(), and pack_string_preserving_sort().

◆ pack_glass_postlist_key() [1/2]

std::string pack_glass_postlist_key ( const std::string & term )

inline

Definition at line 613 of file pack.h.

References pack_string_preserving_sort().

Referenced by ChertPostListTable::get_used_docid_range(), GlassPostListTable::get_used_docid_range(), GlassPostListTable::make_key(), GlassCompact::merge_postlists(), GlassAllTermsList::next(), and GlassAllTermsList::skip_to().

◆ pack_glass_postlist_key() [2/2]

std::string pack_glass_postlist_key	(	const std::string &	term,
		Xapian::docid	did
	)

inline

Definition at line 625 of file pack.h.

References pack_string_preserving_sort(), and pack_uint_preserving_sort().

◆ pack_string() [1/2]

void pack_string	(	std::string &	s,
		const char *	ptr
	)

inline

Append an encoded C-style string to a string.

Parameters

s	The string to append to.
ptr	The C-style string to encode.

Definition at line 489 of file pack.h.

References Assert, and pack_uint().

◆ pack_string() [2/2]

void pack_string	(	std::string &	s,
		const std::string &	value
	)

inline

Append an encoded std::string to a string.

Parameters

s	The string to append to.
value	The std::string to encode.

Definition at line 477 of file pack.h.

References pack_uint().

Referenced by ValueUpdater::append_to_stream(), Glass::ValueUpdater::append_to_stream(), ChertCompact::encode_valuestats(), GlassCompact::encode_valuestats(), Glass::RootInfo::serialise(), ChertValueManager::set_value_stats(), GlassValueManager::set_value_stats(), ChertTable::write_changed_blocks(), and ChertTable_base::write_to_file().

◆ pack_string_preserving_sort()

void pack_string_preserving_sort	(	std::string &	s,
		const std::string &	value,
		bool	last = `false`
	)

inline

Append an encoded std::string to a string, preserving the sort order.

The byte which follows this encoded value must not be \xff, or the sort order won't be correct. You may need to store a padding byte (\0 say) to ensure this. Note that pack_uint_preserving_sort() can never produce \xff as its first byte so is safe to use immediately afterwards.

Parameters

s	The string to append to.
value	The std::string to encode.
last	If true, this is the last thing to be encoded in this string - see note below (default: false)

It doesn't make sense to use pack_string_preserving_sort() if nothing can ever follow, but if optional items can, you can set last=true in cases where nothing does and get a shorter encoding in those cases.

Definition at line 539 of file pack.h.

Referenced by GlassPositionListTable::make_key(), GlassCompact::PositionCursor::next(), pack_chert_postlist_key(), and pack_glass_postlist_key().

◆ pack_uint() [1/2]

template<>

void pack_uint	(	std::string &	s,
		bool	value
	)

inline

Append an encoded unsigned integer (bool type) to a string.

Parameters

s	The string to append to.
value	The unsigned integer to encode.

Definition at line 400 of file pack.h.

◆ pack_uint() [2/2]

template<class U >

void pack_uint	(	std::string &	s,
		U	value
	)

inline

Append an encoded unsigned integer to a string.

Parameters

s	The string to append to.
value	The unsigned integer to encode.

Definition at line 382 of file pack.h.

Referenced by ChertValueManager::add_document(), GlassValueManager::add_document(), Chert::PostlistChunkWriter::append(), Glass::PostlistChunkWriter::append(), ValueUpdater::append_to_stream(), Glass::ValueUpdater::append_to_stream(), ChertDatabaseReplicator::apply_changeset_from_conn(), GlassDatabaseReplicator::apply_changeset_from_conn(), ChertCompact::encode_valuestats(), GlassCompact::encode_valuestats(), ChertDatabase::get_revision_info(), GlassDatabase::get_revision_info(), make_start_of_chunk(), make_start_of_first_chunk(), make_valuechunk_key(), Glass::make_valuechunk_key(), ChertCompact::merge_postlists(), GlassCompact::merge_postlists(), ChertCompact::PostlistCursor::next(), GlassCompact::PostlistCursor::next(), GlassFLCursor::pack(), GlassFreeList::pack(), GlassPositionListTable::pack(), pack_string(), ChertDatabase::send_whole_database(), GlassDatabase::send_whole_database(), Glass::RootInfo::serialise(), GlassVersion::serialise_stats(), ChertPositionListTable::set_positionlist(), ChertDatabase::set_revision_number(), ChertTermListTable::set_termlist(), GlassTermListTable::set_termlist(), ChertValueManager::set_value_stats(), GlassValueManager::set_value_stats(), GlassChanges::start(), ChertDatabaseStats::write(), GlassVersion::write(), GlassTable::write_block(), ChertTable::write_changed_blocks(), ChertDatabase::write_changesets_to_fd(), GlassDatabase::write_changesets_to_fd(), and ChertTable_base::write_to_file().

◆ pack_uint_last()

template<class U >

void pack_uint_last	(	std::string &	s,
		U	value
	)

inline

Append an encoded unsigned integer to a string as the last item.

This encoding is only suitable when this is the last thing encoded as the encoding used doesn't contain its own length.

Parameters

s	The string to append to.
value	The unsigned integer to encode.

Definition at line 93 of file pack.h.

Referenced by make_valuestats_key(), ChertSpellingTable::merge_changes(), GlassSpellingTable::merge_changes(), ChertCompact::merge_postlists(), ChertCompact::merge_spellings(), GlassCompact::merge_spellings(), and ChertDatabaseStats::write().

◆ pack_uint_preserving_sort()

template<class U >

void pack_uint_preserving_sort	(	std::string &	s,
		U	value
	)

inline

Append an encoded unsigned integer to a string, preserving the sort order.

[Glass and newer variant]

The appended string data will sort in the same order as the unsigned integer being encoded.

Note that the first byte of the encoding will never be \xff, so it is safe to store the result of this function immediately after the result of pack_string_preserving_sort().

Parameters

s	The string to append to.
value	The unsigned integer to encode.

Definition at line 269 of file pack.h.

References Assert, and AssertRel.

Referenced by GlassDocDataTable::make_key(), GlassTermListTable::make_key(), GlassPositionListTable::make_key(), make_slot_key(), Glass::make_valuechunk_key(), GlassCompact::merge_docid_keyed(), GlassCompact::PostlistCursor::next(), GlassCompact::PositionCursor::next(), pack_glass_postlist_key(), and test_pack_uint_preserving_sort1().

◆ unpack_bool()

bool unpack_bool	(	const char **	p,
		const char *	end,
		bool *	result
	)

inline

Decode a bool from a string.

Parameters

p	Pointer to pointer to the current position in the string.
end	Pointer to the end of the string.
result	Where to store the result.

Definition at line 69 of file pack.h.

References Assert, and rare.

Referenced by check_chert_table(), check_glass_table(), and read_start_of_chunk().

◆ unpack_string()

bool unpack_string	(	const char **	p,
		const char *	end,
		std::string &	result
	)

inline

Decode a std::string from a string.

Parameters

p	Pointer to pointer to the current position in the string.
end	Pointer to the end of the string.
result	Where to store the result.

Definition at line 504 of file pack.h.

References rare, and unpack_uint().

Referenced by ChertDatabaseReplicator::apply_changeset_from_conn(), ValueChunkReader::assign(), Glass::ValueChunkReader::assign(), check_chert_table(), check_glass_table(), ChertValueManager::get_value_stats(), GlassValueManager::get_value_stats(), ChertCompact::merge_postlists(), GlassCompact::merge_postlists(), ValueChunkReader::next(), Glass::ValueChunkReader::next(), and Glass::RootInfo::unserialise().

◆ unpack_string_preserving_sort()

bool unpack_string_preserving_sort	(	const char **	p,
		const char *	end,
		std::string &	result
	)

inline

Decode a "sort preserved" std::string from a string.

The std::string must have been encoded with pack_string_preserving_sort().

Parameters

p	Pointer to pointer to the current position in the string.
end	Pointer to the end of the string.
result	Where to store the result.

Definition at line 562 of file pack.h.

References Assert, rare, and usual.

Referenced by check_chert_table(), check_glass_table(), get_tname_from_key(), ChertCompact::merge_postlists(), GlassCompact::merge_postlists(), ChertAllTermsList::next(), ChertCompact::PostlistCursor::next(), GlassAllTermsList::next(), GlassCompact::PostlistCursor::next(), GlassCompact::PositionCursor::next(), ChertAllTermsList::skip_to(), and GlassAllTermsList::skip_to().

◆ unpack_uint()

template<class U >

bool unpack_uint	(	const char **	p,
		const char *	end,
		U *	result
	)

inline

Decode an unsigned integer from a string.

Parameters

p	Pointer to pointer to the current position in the string.
end	Pointer to the end of the string.
result	Where to store the result (or NULL to just skip it).

Definition at line 413 of file pack.h.

References Assert, and rare.

Referenced by ChertDatabaseReplicator::apply_changeset_from_conn(), GlassDatabaseReplicator::apply_changeset_from_conn(), GlassChanges::check(), check_chert_table(), check_glass_table(), ChertDatabaseReplicator::check_revision_at_least(), GlassDatabaseReplicator::check_revision_at_least(), ChertTermList::ChertTermList(), ChertValueManager::delete_document(), GlassValueManager::delete_document(), do_unpack_uint(), docid_from_key(), Glass::docid_from_key(), ChertValueManager::get_all_values(), GlassValueManager::get_all_values(), ChertDatabase::get_changeset_revisions(), GlassDatabase::get_changeset_revisions(), ChertValueManager::get_chunk_containing_did(), GlassValueManager::get_chunk_containing_did(), GlassPostListTable::get_freqs(), Xapian::Database::get_revision(), ChertValueManager::get_value_stats(), GlassValueManager::get_value_stats(), GlassTermList::GlassTermList(), ChertCompact::merge_postlists(), GlassCompact::merge_postlists(), ChertCompact::PostlistCursor::next(), ChertTermList::next(), ValueChunkReader::next(), GlassCompact::PostlistCursor::next(), GlassTermList::next(), Glass::ValueChunkReader::next(), GlassPositionListTable::positionlist_count(), ChertPositionListTable::positionlist_count(), ChertDatabaseReplicator::process_changeset_chunk_base(), ChertDatabaseReplicator::process_changeset_chunk_blocks(), GlassDatabaseReplicator::process_changeset_chunk_blocks(), GlassDatabaseReplicator::process_changeset_chunk_version(), GlassVersion::read(), ChertDatabaseStats::read(), ChertTable_base::read(), ChertPositionList::read_data(), GlassPositionList::read_data(), read_did_increase(), ChertPostList::read_number_of_entries(), GlassPostList::read_number_of_entries(), read_start_of_chunk(), read_start_of_first_chunk(), read_wdf(), ValueChunkReader::skip_to(), Glass::ValueChunkReader::skip_to(), GlassFLCursor::unpack(), GlassFreeList::unpack(), unpack_string(), Glass::RootInfo::unserialise(), GlassVersion::unserialise_stats(), ChertDatabase::write_changesets_to_fd(), and GlassDatabase::write_changesets_to_fd().

◆ unpack_uint_last()

template<class U >

bool unpack_uint_last	(	const char **	p,
		const char *	end,
		U *	result
	)

inline

Decode an unsigned integer as the last item in a string.

Parameters

p	Pointer to pointer to the current position in the string.
end	Pointer to the end of the string.
result	Where to store the result.

Definition at line 111 of file pack.h.

References Assert, and rare.

Referenced by ChertSpellingTable::add_word(), GlassSpellingTable::add_word(), check_chert_table(), check_glass_table(), ChertSpellingWordsList::get_termfreq(), GlassSpellingWordsList::get_termfreq(), ChertSpellingTable::get_word_frequency(), GlassSpellingTable::get_word_frequency(), ChertCompact::merge_postlists(), ChertCompact::merge_spellings(), GlassCompact::merge_spellings(), ChertDatabaseStats::read(), ChertSpellingTable::remove_word(), and GlassSpellingTable::remove_word().

◆ unpack_uint_preserving_sort()

template<class U >

bool unpack_uint_preserving_sort	(	const char **	p,
		const char *	end,
		U *	result
	)

inline

Decode a "sort preserved" unsigned integer from a string.

[Glass and newer variant]

The unsigned integer must have been encoded with pack_uint_preserving_sort().

Parameters

p	Pointer to pointer to the current position in the string.
end	Pointer to the end of the string.
result	Where to store the result.

Definition at line 318 of file pack.h.

References Assert, and rare.

Referenced by check_glass_table(), Glass::docid_from_key(), Glass::PostlistChunkWriter::flush(), GlassPostListTable::get_chunk(), GlassValueManager::get_chunk_containing_did(), GlassPostListTable::get_used_docid_range(), GlassCompact::merge_docid_keyed(), GlassPostList::move_to_chunk_containing(), GlassCompact::PostlistCursor::next(), GlassCompact::PositionCursor::next(), GlassPostList::next_chunk(), and test_pack_uint_preserving_sort1().

Variable Documentation

◆ SORTABLE_UINT_1ST_BYTE_MASK

const unsigned int SORTABLE_UINT_1ST_BYTE_MASK

Initial value:

=

(0xffu >> SORTABLE_UINT_LOG2_MAX_BYTES)

SORTABLE_UINT_LOG2_MAX_BYTES

const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES

How many bits to store the length of a sortable uint in.

Definition: pack.h:42

Calculated value used below.

Definition at line 48 of file pack.h.

Referenced by C_pack_uint_preserving_sort(), and C_unpack_uint_preserving_sort().

◆ SORTABLE_UINT_LOG2_MAX_BYTES

const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES = 2

How many bits to store the length of a sortable uint in.

Setting this to 2 limits us to 2**32 documents in the database. If set to 3, then 2**64 documents are possible, but the database format isn't compatible.

Definition at line 42 of file pack.h.

Referenced by C_pack_uint_preserving_sort(), and C_unpack_uint_preserving_sort().

◆ SORTABLE_UINT_MAX_BYTES

const unsigned int SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES

Calculated value used below.

Definition at line 45 of file pack.h.

Referenced by C_pack_uint_preserving_sort().

Functions

Variables

Detailed Description

Function Documentation

◆ C_pack_uint_preserving_sort()

◆ C_unpack_uint_preserving_sort()

◆ pack_bool()

◆ pack_chert_postlist_key() [1/2]

◆ pack_chert_postlist_key() [2/2]

◆ pack_glass_postlist_key() [1/2]

◆ pack_glass_postlist_key() [2/2]

◆ pack_string() [1/2]

◆ pack_string() [2/2]

◆ pack_string_preserving_sort()

◆ pack_uint() [1/2]

◆ pack_uint() [2/2]

◆ pack_uint_last()

◆ pack_uint_preserving_sort()

◆ unpack_bool()

◆ unpack_string()

◆ unpack_string_preserving_sort()

◆ unpack_uint()

◆ unpack_uint_last()

◆ unpack_uint_preserving_sort()

Variable Documentation

◆ SORTABLE_UINT_1ST_BYTE_MASK

◆ SORTABLE_UINT_LOG2_MAX_BYTES

◆ SORTABLE_UINT_MAX_BYTES