common/pack.h File Reference

Pack types into strings and unpack them again. More...

#include <cstring>
#include <string>
#include "omassert.h"
#include "xapian/types.h"

Include dependency graph for pack.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Functions

void pack_bool (std::string &s, bool value)

Append an encoded bool to a string.

bool unpack_bool (const char **p, const char *end, bool *result)

Decode a bool from a string.

template<class U >

void pack_uint_last (std::string &s, U value)

Append an encoded unsigned integer to a string as the last item.

template<class U >

bool unpack_uint_last (const char **p, const char *end, U *result)

Decode an unsigned integer as the last item in a string.

template<class U >

void pack_uint_preserving_sort (std::string &s, U value)

Append an encoded unsigned integer to a string, preserving the sort order.

template<class U >

bool unpack_uint_preserving_sort (const char **p, const char *end, U *result)

Decode an "sort preserved" unsigned integer from a string.

template<class U >

void pack_uint (std::string &s, U value)

Append an encoded unsigned integer to a string.

template<class U >

bool unpack_uint (const char **p, const char *end, U *result)

Decode an unsigned integer from a string.

void pack_string (std::string &s, const std::string &value)

Append an encoded std::string to a string.

void pack_string (std::string &s, const char *ptr)

Append an encoded C-style string to a string.

bool unpack_string (const char **p, const char *end, std::string &result)

Decode a std::string from a string.

void pack_string_preserving_sort (std::string &s, const std::string &value, bool last=false)

Append an encoded std::string to a string, preserving the sort order.

bool unpack_string_preserving_sort (const char **p, const char *end, std::string &result)

Decode a "sort preserved" std::string from a string.

std::string pack_chert_postlist_key (const std::string &term)

std::string pack_chert_postlist_key (const std::string &term, Xapian::docid did)

std::string pack_brass_postlist_key (const std::string &term)

std::string pack_brass_postlist_key (const std::string &term, Xapian::docid did)

Variables

const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES = 2

How many bits to store the length of a sortable uint in.

const unsigned int SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES

Calculated value used below.

const unsigned int SORTABLE_UINT_1ST_BYTE_MASK

Calculated value used below.

Detailed Description

Pack types into strings and unpack them again.

Definition in file pack.h.

Function Documentation

void pack_bool	(	std::string &	s,
		bool	value
	)			`[inline]`

Append an encoded bool to a string.

Parameters:

	s	The string to append to.
	value	The bool to encode.

Definition at line 52 of file pack.h.

Referenced by make_start_of_chunk().

std::string pack_brass_postlist_key	(	const std::string &	term,
		Xapian::docid	did
	)			`[inline]`

Definition at line 445 of file pack.h.

References pack_chert_postlist_key().

std::string pack_brass_postlist_key ( const std::string & term ) [inline]

Definition at line 439 of file pack.h.

References pack_chert_postlist_key().

Referenced by BrassPostListTable::make_key(), BrassCompact::merge_postlists(), BrassAllTermsList::next(), and BrassAllTermsList::skip_to().

std::string pack_chert_postlist_key	(	const std::string &	term,
		Xapian::docid	did
	)			`[inline]`

Definition at line 423 of file pack.h.

References pack_string_preserving_sort(), and pack_uint_preserving_sort().

std::string pack_chert_postlist_key ( const std::string & term ) [inline]

Definition at line 411 of file pack.h.

References pack_string_preserving_sort().

Referenced by copy_postlist(), ChertPostListTable::make_key(), ChertCompact::merge_postlists(), ChertAllTermsList::next(), pack_brass_postlist_key(), and ChertAllTermsList::skip_to().

void pack_string	(	std::string &	s,
		const char *	ptr
	)			`[inline]`

Append an encoded C-style string to a string.

Parameters:

	s	The string to append to.
	ptr	The C-style string to encode.

Definition at line 315 of file pack.h.

References Assert, and pack_uint().

void pack_string	(	std::string &	s,
		const std::string &	value
	)			`[inline]`

Append an encoded std::string to a string.

Parameters:

	s	The string to append to.
	value	The std::string to encode.

Definition at line 303 of file pack.h.

References pack_uint().

Referenced by ValueUpdater::append_to_stream(), Brass::ValueUpdater::append_to_stream(), ChertCompact::encode_valuestats(), BrassCompact::encode_valuestats(), ChertValueManager::set_value_stats(), BrassValueManager::set_value_stats(), ChertTable::write_changed_blocks(), BrassTable::write_changed_blocks(), ChertTable_base::write_to_file(), and BrassTable_base::write_to_file().

void pack_string_preserving_sort	(	std::string &	s,
		const std::string &	value,
		bool	last = `false`
	)			`[inline]`

Append an encoded std::string to a string, preserving the sort order.

The byte which follows this encoded value *must not* be , or the sort order won't be correct. You may need to store a padding byte ( say) to ensure this. Note that pack_uint_preserving_sort() can never produce as its first byte so is safe to use immediately afterwards.

Parameters:

	s	The string to append to.
	value	The std::string to encode.
	last	If true, this is the last thing to be encoded in this string - see note below (default: false)

It doesn't make sense to use pack_string_preserving_sort() if nothing can ever follow, but if optional items can, you can set last=true in cases where nothing does and get a shorter encoding in those cases.

Definition at line 365 of file pack.h.

Referenced by pack_chert_postlist_key().

template<class U >

void pack_uint	(	std::string &	s,
		U	value
	)			`[inline]`

Append an encoded unsigned integer to a string.

Parameters:

	s	The string to append to.
	value	The unsigned integer to encode.

Definition at line 218 of file pack.h.

References STATIC_ASSERT_UNSIGNED_TYPE.

Referenced by ChertValueManager::add_document(), BrassValueManager::add_document(), Chert::PostlistChunkWriter::append(), Brass::PostlistChunkWriter::append(), ValueUpdater::append_to_stream(), Brass::ValueUpdater::append_to_stream(), ChertDatabaseReplicator::apply_changeset_from_conn(), BrassDatabaseReplicator::apply_changeset_from_conn(), ChertCompact::encode_valuestats(), BrassCompact::encode_valuestats(), ChertDatabase::get_revision_info(), BrassDatabase::get_revision_info(), make_start_of_chunk(), make_start_of_first_chunk(), make_valuechunk_key(), Brass::make_valuechunk_key(), ChertCompact::merge_postlists(), BrassCompact::merge_postlists(), ChertCompact::PostlistCursor::next(), BrassCompact::PostlistCursor::next(), pack_string(), ChertDatabase::send_whole_database(), BrassDatabase::send_whole_database(), ChertPositionListTable::set_positionlist(), BrassPositionListTable::set_positionlist(), ChertDatabase::set_revision_number(), BrassDatabase::set_revision_number(), ChertTermListTable::set_termlist(), BrassTermListTable::set_termlist(), ChertValueManager::set_value_stats(), BrassValueManager::set_value_stats(), ChertDatabaseStats::write(), BrassDatabaseStats::write(), ChertTable::write_changed_blocks(), BrassTable::write_changed_blocks(), ChertDatabase::write_changesets_to_fd(), BrassDatabase::write_changesets_to_fd(), ChertTable_base::write_to_file(), and BrassTable_base::write_to_file().

template<class U >

void pack_uint_last	(	std::string &	s,
		U	value
	)			`[inline]`

Append an encoded unsigned integer to a string as the last item.

This encoding is only suitable when this is the last thing encoded as the encoding used doesn't contain its own length.

Parameters:

	s	The string to append to.
	value	The unsigned integer to encode.

Definition at line 88 of file pack.h.

References STATIC_ASSERT_UNSIGNED_TYPE.

Referenced by make_valuestats_key(), ChertSpellingTable::merge_changes(), BrassSpellingTable::merge_changes(), ChertCompact::merge_postlists(), BrassCompact::merge_postlists(), ChertCompact::merge_spellings(), BrassCompact::merge_spellings(), ChertDatabaseStats::write(), and BrassDatabaseStats::write().

template<class U >

void pack_uint_preserving_sort	(	std::string &	s,
		U	value
	)			`[inline]`

Append an encoded unsigned integer to a string, preserving the sort order.

The appended string data will sort in the same order as the unsigned integer being encoded.

Note that the first byte of the encoding will never be , so it is safe to store the result of this function immediately after the result of pack_string_preserving_sort().

Parameters:

	s	The string to append to.
	value	The unsigned integer to encode.

Definition at line 144 of file pack.h.

References SORTABLE_UINT_1ST_BYTE_MASK, SORTABLE_UINT_LOG2_MAX_BYTES, SORTABLE_UINT_MAX_BYTES, STATIC_ASSERT, and STATIC_ASSERT_UNSIGNED_TYPE.

Referenced by copy_docid_keyed(), copy_position(), copy_postlist(), copy_termlist(), ChertTermListTable::make_key(), ChertPositionListTable::make_key(), BrassTermListTable::make_key(), make_key(), BrassPositionListTable::make_key(), make_slot_key(), make_valuechunk_key(), Brass::make_valuechunk_key(), ChertCompact::merge_docid_keyed(), BrassCompact::merge_docid_keyed(), ChertCompact::PostlistCursor::next(), BrassCompact::PostlistCursor::next(), pack_chert_postlist_key(), and test_pack_uint_preserving_sort1().

bool unpack_bool	(	const char **	p,
		const char *	end,
		bool *	result
	)			`[inline]`

Decode a bool from a string.

Parameters:

	p	Pointer to pointer to the current position in the string.
	end	Pointer to the end of the string.
	result	Where to store the result.

Definition at line 64 of file pack.h.

References Assert, and rare.

Referenced by check_brass_table(), check_chert_table(), and read_start_of_chunk().

bool unpack_string	(	const char **	p,
		const char *	end,
		std::string &	result
	)			`[inline]`

Decode a std::string from a string.

Parameters:

	p	Pointer to pointer to the current position in the string.
	end	Pointer to the end of the string.
	result	Where to store the result.

Definition at line 330 of file pack.h.

References rare, and unpack_uint().

Referenced by ChertDatabaseReplicator::apply_changeset_from_conn(), BrassDatabaseReplicator::apply_changeset_from_conn(), ValueChunkReader::assign(), Brass::ValueChunkReader::assign(), check_brass_table(), check_chert_table(), ChertValueManager::get_value_stats(), BrassValueManager::get_value_stats(), ChertCompact::merge_postlists(), BrassCompact::merge_postlists(), ValueChunkReader::next(), and Brass::ValueChunkReader::next().

bool unpack_string_preserving_sort	(	const char **	p,
		const char *	end,
		std::string &	result
	)			`[inline]`

Decode a "sort preserved" std::string from a string.

The std::string must have been encoded with pack_string_preserving_sort().

Parameters:

	p	Pointer to pointer to the current position in the string.
	end	Pointer to the end of the string.
	result	Where to store the result.

Definition at line 388 of file pack.h.

References Assert, rare, and usual.

Referenced by check_brass_table(), check_chert_table(), get_tname_from_key(), ChertCompact::merge_postlists(), BrassCompact::merge_postlists(), ChertCompact::PostlistCursor::next(), ChertAllTermsList::next(), BrassCompact::PostlistCursor::next(), BrassAllTermsList::next(), ChertAllTermsList::skip_to(), and BrassAllTermsList::skip_to().

template<class U >

bool unpack_uint	(	const char **	p,
		const char *	end,
		U *	result
	)			`[inline]`

Decode an unsigned integer from a string.

Parameters:

	p	Pointer to pointer to the current position in the string.
	end	Pointer to the end of the string.
	result	Where to store the result (or NULL to just skip it).

Definition at line 238 of file pack.h.

References Assert, rare, and STATIC_ASSERT_UNSIGNED_TYPE.

Referenced by ChertDatabaseReplicator::apply_changeset_from_conn(), BrassDatabaseReplicator::apply_changeset_from_conn(), BrassTermList::BrassTermList(), check_brass_table(), check_chert_table(), ChertDatabaseReplicator::check_revision_at_least(), BrassDatabaseReplicator::check_revision_at_least(), ChertTermList::ChertTermList(), copy_postlist(), ChertValueManager::delete_document(), BrassValueManager::delete_document(), do_unpack_uint(), docid_from_key(), Brass::docid_from_key(), ChertValueManager::get_all_values(), BrassValueManager::get_all_values(), ChertDatabase::get_changeset_revisions(), BrassDatabase::get_changeset_revisions(), ChertValueManager::get_chunk_containing_did(), BrassValueManager::get_chunk_containing_did(), ChertValueManager::get_value_stats(), BrassValueManager::get_value_stats(), ChertCompact::merge_postlists(), BrassCompact::merge_postlists(), ValueChunkReader::next(), ChertTermList::next(), ChertCompact::PostlistCursor::next(), Brass::ValueChunkReader::next(), BrassTermList::next(), BrassCompact::PostlistCursor::next(), ChertPositionListTable::positionlist_count(), BrassPositionListTable::positionlist_count(), ChertDatabaseReplicator::process_changeset_chunk_base(), BrassDatabaseReplicator::process_changeset_chunk_base(), ChertDatabaseReplicator::process_changeset_chunk_blocks(), BrassDatabaseReplicator::process_changeset_chunk_blocks(), ChertDatabaseStats::read(), ChertTable_base::read(), BrassDatabaseStats::read(), BrassTable_base::read(), ChertPositionList::read_data(), BrassPositionList::read_data(), read_did_increase(), ChertPostList::read_number_of_entries(), BrassPostList::read_number_of_entries(), read_start_of_chunk(), read_start_of_first_chunk(), read_wdf(), ValueChunkReader::skip_to(), Brass::ValueChunkReader::skip_to(), unpack_string(), ChertDatabase::write_changesets_to_fd(), and BrassDatabase::write_changesets_to_fd().

template<class U >

bool unpack_uint_last	(	const char **	p,
		const char *	end,
		U *	result
	)			`[inline]`

Decode an unsigned integer as the last item in a string.

Parameters:

	p	Pointer to pointer to the current position in the string.
	end	Pointer to the end of the string.
	result	Where to store the result.

Definition at line 107 of file pack.h.

References Assert, rare, and STATIC_ASSERT_UNSIGNED_TYPE.

Referenced by ChertSpellingTable::add_word(), BrassSpellingTable::add_word(), check_brass_table(), check_chert_table(), ChertSpellingWordsList::get_termfreq(), BrassSpellingWordsList::get_termfreq(), ChertSpellingTable::get_word_frequency(), BrassSpellingTable::get_word_frequency(), ChertCompact::merge_postlists(), BrassCompact::merge_postlists(), ChertCompact::merge_spellings(), BrassCompact::merge_spellings(), ChertDatabaseStats::read(), BrassDatabaseStats::read(), ChertSpellingTable::remove_word(), and BrassSpellingTable::remove_word().

template<class U >

bool unpack_uint_preserving_sort	(	const char **	p,
		const char *	end,
		U *	result
	)			`[inline]`

Decode an "sort preserved" unsigned integer from a string.

The unsigned integer must have been encoded with pack_uint_preserving_sort().

Parameters:

	p	Pointer to pointer to the current position in the string.
	end	Pointer to the end of the string.
	result	Where to store the result.

Definition at line 174 of file pack.h.

References Assert, rare, SORTABLE_UINT_1ST_BYTE_MASK, SORTABLE_UINT_LOG2_MAX_BYTES, STATIC_ASSERT, and STATIC_ASSERT_UNSIGNED_TYPE.

Referenced by check_brass_table(), check_chert_table(), docid_from_key(), Brass::docid_from_key(), Chert::PostlistChunkWriter::flush(), Brass::PostlistChunkWriter::flush(), ChertPostListTable::get_chunk(), BrassPostListTable::get_chunk(), ChertValueManager::get_chunk_containing_did(), BrassValueManager::get_chunk_containing_did(), ChertCompact::merge_docid_keyed(), BrassCompact::merge_docid_keyed(), ChertPostList::move_to_chunk_containing(), BrassPostList::move_to_chunk_containing(), ChertCompact::PostlistCursor::next(), BrassCompact::PostlistCursor::next(), ChertPostList::next_chunk(), BrassPostList::next_chunk(), and test_pack_uint_preserving_sort1().

Variable Documentation

const unsigned int SORTABLE_UINT_1ST_BYTE_MASK

Initial value:

        (0xffu >> SORTABLE_UINT_LOG2_MAX_BYTES)

Calculated value used below.

Definition at line 43 of file pack.h.

Referenced by pack_uint_preserving_sort(), and unpack_uint_preserving_sort().

const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES = 2

How many bits to store the length of a sortable uint in.

Setting this to 2 limits us to 2**32 documents in the database. If set to 3, then 2**64 documents are possible, but the database format isn't compatible.

Definition at line 37 of file pack.h.

Referenced by pack_uint_preserving_sort(), and unpack_uint_preserving_sort().

const unsigned int SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES

Calculated value used below.

Definition at line 40 of file pack.h.

Referenced by pack_uint_preserving_sort().


Functions
void	pack_bool (std::string &s, bool value)
	Append an encoded bool to a string.
bool	unpack_bool (const char *p, const char end, bool *result)
	Decode a bool from a string.
template<class U >
void	pack_uint_last (std::string &s, U value)
	Append an encoded unsigned integer to a string as the last item.
template<class U >
bool	unpack_uint_last (const char *p, const char end, U *result)
	Decode an unsigned integer as the last item in a string.
template<class U >
void	pack_uint_preserving_sort (std::string &s, U value)
	Append an encoded unsigned integer to a string, preserving the sort order.
template<class U >
bool	unpack_uint_preserving_sort (const char *p, const char end, U *result)
	Decode an "sort preserved" unsigned integer from a string.
template<class U >
void	pack_uint (std::string &s, U value)
	Append an encoded unsigned integer to a string.
template<class U >
bool	unpack_uint (const char *p, const char end, U *result)
	Decode an unsigned integer from a string.
void	pack_string (std::string &s, const std::string &value)
	Append an encoded std::string to a string.
void	pack_string (std::string &s, const char *ptr)
	Append an encoded C-style string to a string.
bool	unpack_string (const char *p, const char end, std::string &result)
	Decode a std::string from a string.
void	pack_string_preserving_sort (std::string &s, const std::string &value, bool last=false)
	Append an encoded std::string to a string, preserving the sort order.
bool	unpack_string_preserving_sort (const char *p, const char end, std::string &result)
	Decode a "sort preserved" std::string from a string.
std::string	pack_chert_postlist_key (const std::string &term)
std::string	pack_chert_postlist_key (const std::string &term, Xapian::docid did)
std::string	pack_brass_postlist_key (const std::string &term)
std::string	pack_brass_postlist_key (const std::string &term, Xapian::docid did)
Variables
const unsigned int	SORTABLE_UINT_LOG2_MAX_BYTES = 2
	How many bits to store the length of a sortable uint in.
const unsigned int	SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES
	Calculated value used below.
const unsigned int	SORTABLE_UINT_1ST_BYTE_MASK
	Calculated value used below.