xapian-core  2.0.0
Functions | Variables
pack.h File Reference

Pack types into strings and unpack them again. More...

#include <string>
#include <string_view>
#include <type_traits>
#include "omassert.h"
#include "xapian/types.h"
+ Include dependency graph for pack.h:

Go to the source code of this file.

Functions

void unpack_throw_serialisation_error (const char *p)
 Throw appropriate SerialisationError. More...
 
void pack_bool (std::string &s, bool value)
 Append an encoded bool to a string. More...
 
bool unpack_bool (const char **p, const char *end, bool *result)
 Decode a bool from a string. More...
 
template<class U >
void pack_uint_last (std::string &s, U value)
 Append an encoded unsigned integer to a string as the last item. More...
 
template<class U >
bool unpack_uint_last (const char **p, const char *end, U *result)
 Decode an unsigned integer as the last item in a string. More...
 
template<class U >
void pack_uint_preserving_sort (std::string &s, U value)
 Append an encoded unsigned integer to a string, preserving the sort order. More...
 
template<class U >
bool unpack_uint_preserving_sort (const char **p, const char *end, U *result)
 Decode a "sort preserved" unsigned integer from a string. More...
 
template<class U >
void pack_uint (std::string &s, U value)
 Append an encoded unsigned integer to a string. More...
 
template<>
void pack_uint (std::string &s, bool value)
 Append an encoded unsigned integer (bool type) to a string. More...
 
template<class U >
bool unpack_uint (const char **p, const char *end, U *result)
 Decode an unsigned integer from a string. More...
 
template<class U >
bool unpack_uint_backwards (const char **p, const char *start, U *result)
 Decode an unsigned integer from a string, going backwards. More...
 
void pack_string (std::string &s, std::string_view value)
 Append an encoded std::string to a string. More...
 
void pack_string_empty (std::string &s)
 Append an empty encoded std::string to a string. More...
 
bool unpack_string (const char **p, const char *end, std::string &result)
 Decode a std::string from a string. More...
 
bool unpack_string_append (const char **p, const char *end, std::string &result)
 Decode a std::string from a string and append. More...
 
void pack_string_preserving_sort (std::string &s, std::string_view value, bool last=false)
 Append an encoded std::string to a string, preserving the sort order. More...
 
bool unpack_string_preserving_sort (const char **p, const char *end, std::string &result)
 Decode a "sort preserved" std::string from a string. More...
 
std::string pack_glass_postlist_key (std::string_view term)
 
std::string pack_glass_postlist_key (std::string_view term, Xapian::docid did)
 
std::string pack_honey_postlist_key (std::string_view term)
 
std::string pack_honey_postlist_key (std::string_view term, Xapian::docid did)
 

Variables

const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES = 2
 How many bits to store the length of a sortable uint in. More...
 
const unsigned int SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES
 Calculated value used below. More...
 
const unsigned int SORTABLE_UINT_1ST_BYTE_MASK
 Calculated value used below. More...
 

Detailed Description

Pack types into strings and unpack them again.

Definition in file pack.h.

Function Documentation

◆ pack_bool()

void pack_bool ( std::string &  s,
bool  value 
)
inline

Append an encoded bool to a string.

Parameters
sThe string to append to.
valueThe bool to encode.

Definition at line 64 of file pack.h.

Referenced by make_start_of_chunk(), RemoteServer::msg_update(), serialise_stats(), and RemoteDatabase::set_query().

◆ pack_glass_postlist_key() [1/2]

std::string pack_glass_postlist_key ( std::string_view  term)
inline

◆ pack_glass_postlist_key() [2/2]

std::string pack_glass_postlist_key ( std::string_view  term,
Xapian::docid  did 
)
inline

Definition at line 586 of file pack.h.

References pack_string_preserving_sort(), pack_uint_preserving_sort(), and term.

◆ pack_honey_postlist_key() [1/2]

std::string pack_honey_postlist_key ( std::string_view  term)
inline

◆ pack_honey_postlist_key() [2/2]

std::string pack_honey_postlist_key ( std::string_view  term,
Xapian::docid  did 
)
inline

Definition at line 611 of file pack.h.

References Assert, pack_string_preserving_sort(), pack_uint_preserving_sort(), and term.

◆ pack_string()

void pack_string ( std::string &  s,
std::string_view  value 
)
inline

◆ pack_string_empty()

void pack_string_empty ( std::string &  s)
inline

Append an empty encoded std::string to a string.

This is equivalent to pack_string(s, ""sv) but is probably a bit more efficient.

Parameters
sThe string to append to.

Definition at line 456 of file pack.h.

Referenced by RemoteDatabase::send_global_stats(), Xapian::Internal::QueryValueLE::serialise(), and test_packstring2().

◆ pack_string_preserving_sort()

void pack_string_preserving_sort ( std::string &  s,
std::string_view  value,
bool  last = false 
)
inline

Append an encoded std::string to a string, preserving the sort order.

The byte which follows this encoded value must not be \xff, or the sort order won't be correct. You may need to store a padding byte (\0 say) to ensure this. Note that pack_uint_preserving_sort() can never produce \xff as its first byte so is safe to use immediately afterwards.

Parameters
sThe string to append to.
valueThe std::string to encode.
lastIf true, this is the last thing to be encoded in this string - see note below (default: false)

It doesn't make sense to use pack_string_preserving_sort() if nothing can ever follow, but if optional items can, you can set last=true in cases where nothing does and get a shorter encoding in those cases.

Definition at line 528 of file pack.h.

Referenced by GlassPositionListTable::make_key(), HoneyPositionTable::make_key(), Honey::make_postingchunk_key(), GlassCompact::PositionCursor::next(), HoneyCompact::PositionCursor< const GlassTable & >::next(), HoneyCompact::PositionCursor< const HoneyTable & >::next(), pack_glass_postlist_key(), and pack_honey_postlist_key().

◆ pack_uint() [1/2]

template<>
void pack_uint ( std::string &  s,
bool  value 
)
inline

Append an encoded unsigned integer (bool type) to a string.

Parameters
sThe string to append to.
valueThe unsigned integer to encode.

Definition at line 333 of file pack.h.

◆ pack_uint() [2/2]

template<class U >
void pack_uint ( std::string &  s,
value 
)
inline

Append an encoded unsigned integer to a string.

Parameters
sThe string to append to.
valueThe unsigned integer to encode.

Definition at line 315 of file pack.h.

Referenced by HoneyTable::add(), HoneyValueManager::add_document(), GlassValueManager::add_document(), RemoteDatabase::add_spelling(), Glass::PostlistChunkWriter::append(), Glass::ValueUpdater::append_to_stream(), Honey::ValueUpdater::append_to_stream(), GlassDatabaseReplicator::apply_changeset_from_conn(), encode_delta_chunk_header(), encode_delta_chunk_header_no_wdf(), encode_initial_chunk_header(), Honey::encode_valuestats(), GlassCompact::encode_valuestats(), Xapian::DatabaseReplica::Internal::get_revision_info(), make_start_of_chunk(), make_start_of_first_chunk(), Glass::make_valuechunk_key(), SSTIndex::maybe_add_entry(), HoneyCompact::merge_docid_keyed(), GlassCompact::merge_postlists(), HoneyCompact::merge_postlists(), RemoteServer::msg_allterms(), RemoteServer::msg_document(), RemoteServer::msg_freqs(), RemoteServer::msg_metadatakeylist(), RemoteServer::msg_positionlist(), RemoteServer::msg_postlist(), RemoteServer::msg_synonymkeylist(), RemoteServer::msg_synonymtermlist(), RemoteServer::msg_termlist(), RemoteServer::msg_update(), RemoteServer::msg_valuestats(), GlassCompact::PostlistCursor::next(), HoneyCompact::PostlistCursor< const GlassTable & >::next(), RemoteDatabase::open_position_list(), GlassFLCursor::pack(), GlassFreeList::pack(), HoneyFLCursor::pack(), HoneyFreeList::pack(), GlassPositionListTable::pack(), HoneyPositionTable::pack(), pack_string(), RemoteDatabase::positionlist_count(), RemoteDatabase::reconstruct_text(), RemoteDatabase::remove_spelling(), RemoteDatabase::replace_document(), RemoteDatabase::request_document(), RemoteConnection::send_file(), RemoteDatabase::send_global_stats(), RemoteConnection::send_message(), GlassDatabase::send_whole_database(), Xapian::MSet::Internal::serialise(), Xapian::LatLongDistancePostingSource::serialise(), Xapian::MultiValueKeyMaker::serialise(), Xapian::DecreasingValueWeightPostingSource::serialise(), Xapian::ValueMapPostingSource::serialise(), Xapian::Internal::QueryTerm::serialise(), Xapian::Internal::QueryValueRange::serialise(), Xapian::Internal::QueryValueLE::serialise(), Xapian::Internal::QueryValueGE::serialise(), Xapian::Internal::QueryWildcard::serialise(), Xapian::Internal::QueryEditDistance::serialise(), Glass::RootInfo::serialise(), Honey::RootInfo::serialise(), Xapian::Internal::QueryBranch::serialise_(), serialise_document(), Xapian::ValueCountMatchSpy::serialise_results(), serialise_rset(), GlassVersion::serialise_stats(), HoneyVersion::serialise_stats(), serialise_stats(), RemoteDatabase::set_query(), GlassTermListTable::set_termlist(), HoneyTermListTable::set_termlist(), GlassValueManager::set_value_stats(), GlassChanges::start(), test_packuint1(), GlassVersion::write(), HoneyVersion::write(), GlassTable::write_block(), and GlassDatabase::write_changesets_to_fd().

◆ pack_uint_last()

template<class U >
void pack_uint_last ( std::string &  s,
value 
)
inline

◆ pack_uint_preserving_sort()

template<class U >
void pack_uint_preserving_sort ( std::string &  s,
value 
)
inline

Append an encoded unsigned integer to a string, preserving the sort order.

The appended string data will sort in the same order as the unsigned integer being encoded. The encoding used supports types up to 64-bit wide.

The most significant of the first byte are a series of 0-7 consecutive set bits followed by a clear bit. The number of set bits is 2 less than the total number of bytes in the encoded form. The value can be got by zeroing these leading set bits and then interpreting the bytes as a big-endian value (so except for the 9 byte encoded form, the lower order bits of the first byte give the most significant set byte of the value).

Values are encoded in the shortest way possible:

[0x00000000, 0x00007fff] 0AAAAAAA BBBBBBBB [0x00008000, 0x003fffff] 10AAAAAA BBBBBBBB CCCCCCCC [0x00400000, 0x1fffffff] 110AAAAA BBBBBBBB CCCCCCCC DDDDDDDD [0x20000000,0x0fffffffff] 1110AAAA BBBBBBBB CCCCCCCC DDDDDDDD EEEEEEEE

and so on with a full-64bit value encoding as 0b11111110 (0xfe) followed by the 8-byte big-endian representation:

11111110 AAAAAAAA BBBBBBBB CCCCCCCC ... HHHHHHHH

Note that the first byte of the encoding will never be \xff, so it is safe to store the result of this function immediately after the result of pack_string_preserving_sort().

Parameters
sThe string to append to.
valueThe unsigned integer to encode.

Definition at line 204 of file pack.h.

References Assert, and AssertRel.

Referenced by GlassDocDataTable::make_key(), GlassTermListTable::make_key(), HoneyDocDataTable::make_key(), HoneyTermListTable::make_key(), GlassPositionListTable::make_key(), HoneyPositionTable::make_key(), Honey::make_postingchunk_key(), make_slot_key(), Glass::make_valuechunk_key(), Honey::make_valuechunk_key(), Honey::make_valuestats_key(), GlassCompact::merge_docid_keyed(), HoneyCompact::merge_docid_keyed(), GlassCompact::PostlistCursor::next(), GlassCompact::PositionCursor::next(), HoneyCompact::PositionCursor< const GlassTable & >::next(), HoneyCompact::PositionCursor< const HoneyTable & >::next(), pack_glass_postlist_key(), pack_honey_postlist_key(), and test_pack_uint_preserving_sort1().

◆ unpack_bool()

bool unpack_bool ( const char **  p,
const char *  end,
bool *  result 
)
inline

Decode a bool from a string.

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result.

Definition at line 76 of file pack.h.

References Assert, p, and rare.

Referenced by check_glass_table(), RemoteServer::msg_query(), read_start_of_chunk(), unserialise_stats(), and RemoteDatabase::update_stats().

◆ unpack_string()

bool unpack_string ( const char **  p,
const char *  end,
std::string &  result 
)
inline

◆ unpack_string_append()

bool unpack_string_append ( const char **  p,
const char *  end,
std::string &  result 
)
inline

Decode a std::string from a string and append.

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result.

Definition at line 493 of file pack.h.

References p, rare, and unpack_uint().

Referenced by RemoteAllTermsList::next(), RemoteKeyList::next(), RemoteTermList::next(), and unserialise_error().

◆ unpack_string_preserving_sort()

bool unpack_string_preserving_sort ( const char **  p,
const char *  end,
std::string &  result 
)
inline

◆ unpack_throw_serialisation_error()

void unpack_throw_serialisation_error ( const char *  p)

◆ unpack_uint()

template<class U >
bool unpack_uint ( const char **  p,
const char *  end,
U *  result 
)
inline

Decode an unsigned integer from a string.

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result (or NULL to just skip it).

Definition at line 346 of file pack.h.

References Assert, p, and rare.

Referenced by GlassDatabaseReplicator::apply_changeset_from_conn(), Honey::ValueChunkReader::assign(), GlassChanges::check(), check_glass_table(), GlassDatabaseReplicator::check_revision_at_least(), decode_delta_chunk_header(), decode_delta_chunk_header_no_wdf(), decode_initial_chunk_header(), decode_initial_chunk_header_freqs(), HoneyValueManager::delete_document(), GlassValueManager::delete_document(), HoneyCursor::do_find(), Glass::docid_from_key(), GlassValueManager::get_all_values(), HoneyValueManager::get_all_values(), GlassDatabase::get_changeset_revisions(), HoneyCompact::PostlistCursor< const GlassTable & >::DoclenEncoder::get_chunk(), GlassValueManager::get_chunk_containing_did(), HoneyTable::get_exact_entry(), RemoteDatabase::get_freqs(), GlassPostListTable::get_freqs(), RemoteConnection::get_message(), RemoteConnection::get_message_chunked(), GlassValueManager::get_value_stats(), HoneyValueManager::get_value_stats(), GlassTermList::GlassTermList(), HoneyTermList::HoneyTermList(), HoneyCompact::PostlistCursor< const GlassTable & >::DoclenEncoder::initialise(), HoneyCompact::merge_docid_keyed(), GlassCompact::merge_postlists(), HoneyCompact::merge_postlists(), Xapian::ValueCountMatchSpy::merge_results(), RemoteServer::msg_addspelling(), RemoteServer::msg_positionlist(), RemoteServer::msg_positionlistcount(), RemoteServer::msg_query(), RemoteServer::msg_reconstructtext(), RemoteServer::msg_removespelling(), RemoteServer::msg_replacedocument(), GlassCompact::PostlistCursor::next(), GlassTermList::next(), Glass::ValueChunkReader::next(), HoneyCompact::PostlistCursor< const GlassTable & >::next(), Honey::PostingChunkReader::next(), HoneyTermList::next(), Honey::ValueChunkReader::next(), RemoteAllTermsList::next(), RemoteTermList::next(), NetworkPostList::next(), HoneyCursor::next_from_index(), RemoteDatabase::open_document(), RemoteDatabase::open_position_list(), RemoteDatabase::open_term_list(), GlassPositionListTable::positionlist_count(), HoneyPositionTable::positionlist_count(), GlassDatabaseReplicator::process_changeset_chunk_blocks(), GlassDatabaseReplicator::process_changeset_chunk_version(), GlassVersion::read(), HoneyVersion::read(), read_did_increase(), GlassPostList::read_freqs(), HoneyTable::read_key(), read_start_of_chunk(), read_start_of_first_chunk(), RemoteDatabase::read_value_stats(), read_wdf(), HoneyBasePositionList::set_data(), GlassBasePositionList::set_data(), Glass::ValueChunkReader::skip_to(), Honey::PostingChunkReader::skip_to(), Honey::ValueChunkReader::skip_to(), test_packuint1(), GlassFLCursor::unpack(), HoneyFLCursor::unpack(), GlassFreeList::unpack(), HoneyFreeList::unpack(), unpack_string(), unpack_string_append(), unpack_uint_backwards(), Glass::RootInfo::unserialise(), Honey::RootInfo::unserialise(), Xapian::Query::Internal::unserialise(), Xapian::MSet::Internal::unserialise(), Xapian::DecreasingValueWeightPostingSource::unserialise(), Xapian::ValueMapPostingSource::unserialise(), Xapian::MultiValueKeyMaker::unserialise(), unserialise_document(), unserialise_rset(), GlassVersion::unserialise_stats(), HoneyVersion::unserialise_stats(), unserialise_stats(), Xapian::LatLongDistancePostingSource::unserialise_with_registry(), Honey::ValueUpdater::update(), RemoteDatabase::update_stats(), and GlassDatabase::write_changesets_to_fd().

◆ unpack_uint_backwards()

template<class U >
bool unpack_uint_backwards ( const char **  p,
const char *  start,
U *  result 
)
inline

Decode an unsigned integer from a string, going backwards.

Parameters
pPointer to pointer just after the position in the string.
startPointer to the start of the string.
resultWhere to store the result (or NULL to just skip it).

Definition at line 412 of file pack.h.

References Assert, p, rare, and unpack_uint().

Referenced by Honey::PostingChunkReader::skip_to().

◆ unpack_uint_last()

template<class U >
bool unpack_uint_last ( const char **  p,
const char *  end,
U *  result 
)
inline

◆ unpack_uint_preserving_sort()

template<class U >
bool unpack_uint_preserving_sort ( const char **  p,
const char *  end,
U *  result 
)
inline

Variable Documentation

◆ SORTABLE_UINT_1ST_BYTE_MASK

const unsigned int SORTABLE_UINT_1ST_BYTE_MASK
Initial value:
=
const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES
How many bits to store the length of a sortable uint in.
Definition: pack.h:42

Calculated value used below.

Definition at line 48 of file pack.h.

◆ SORTABLE_UINT_LOG2_MAX_BYTES

const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES = 2

How many bits to store the length of a sortable uint in.

Setting this to 2 limits us to 2**32 documents in the database. If set to 3, then 2**64 documents are possible, but the database format isn't compatible.

Definition at line 42 of file pack.h.

◆ SORTABLE_UINT_MAX_BYTES

const unsigned int SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES

Calculated value used below.

Definition at line 45 of file pack.h.