xapian-core  1.4.25
Functions | Variables
pack.h File Reference

Pack types into strings and unpack them again. More...

#include <cstring>
#include <string>
#include <type_traits>
#include "omassert.h"
#include "xapian/types.h"
+ Include dependency graph for pack.h:
+ This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Functions

void pack_bool (std::string &s, bool value)
 Append an encoded bool to a string. More...
 
bool unpack_bool (const char **p, const char *end, bool *result)
 Decode a bool from a string. More...
 
template<class U >
void pack_uint_last (std::string &s, U value)
 Append an encoded unsigned integer to a string as the last item. More...
 
template<class U >
bool unpack_uint_last (const char **p, const char *end, U *result)
 Decode an unsigned integer as the last item in a string. More...
 
template<class U >
void C_pack_uint_preserving_sort (std::string &s, U value)
 Append an encoded unsigned integer to a string, preserving the sort order. More...
 
template<class U >
bool C_unpack_uint_preserving_sort (const char **p, const char *end, U *result)
 Decode an "sort preserved" unsigned integer from a string. More...
 
template<class U >
void pack_uint_preserving_sort (std::string &s, U value)
 Append an encoded unsigned integer to a string, preserving the sort order. More...
 
template<class U >
bool unpack_uint_preserving_sort (const char **p, const char *end, U *result)
 Decode a "sort preserved" unsigned integer from a string. More...
 
template<class U >
void pack_uint (std::string &s, U value)
 Append an encoded unsigned integer to a string. More...
 
template<>
void pack_uint (std::string &s, bool value)
 Append an encoded unsigned integer (bool type) to a string. More...
 
template<class U >
bool unpack_uint (const char **p, const char *end, U *result)
 Decode an unsigned integer from a string. More...
 
void pack_string (std::string &s, const std::string &value)
 Append an encoded std::string to a string. More...
 
void pack_string (std::string &s, const char *ptr)
 Append an encoded C-style string to a string. More...
 
bool unpack_string (const char **p, const char *end, std::string &result)
 Decode a std::string from a string. More...
 
void pack_string_preserving_sort (std::string &s, const std::string &value, bool last=false)
 Append an encoded std::string to a string, preserving the sort order. More...
 
bool unpack_string_preserving_sort (const char **p, const char *end, std::string &result)
 Decode a "sort preserved" std::string from a string. More...
 
std::string pack_chert_postlist_key (const std::string &term)
 
std::string pack_chert_postlist_key (const std::string &term, Xapian::docid did)
 
std::string pack_glass_postlist_key (const std::string &term)
 
std::string pack_glass_postlist_key (const std::string &term, Xapian::docid did)
 

Variables

const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES = 2
 How many bits to store the length of a sortable uint in. More...
 
const unsigned int SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES
 Calculated value used below. More...
 
const unsigned int SORTABLE_UINT_1ST_BYTE_MASK
 Calculated value used below. More...
 

Detailed Description

Pack types into strings and unpack them again.

Definition in file pack.h.

Function Documentation

◆ C_pack_uint_preserving_sort()

template<class U >
void C_pack_uint_preserving_sort ( std::string &  s,
value 
)
inline

Append an encoded unsigned integer to a string, preserving the sort order.

[Chert variant]

The appended string data will sort in the same order as the unsigned integer being encoded.

Note that the first byte of the encoding will never be , so it is safe to store the result of this function immediately after the result of pack_string_preserving_sort().

Parameters
sThe string to append to.
valueThe unsigned integer to encode.

Definition at line 149 of file pack.h.

References Assert, SORTABLE_UINT_1ST_BYTE_MASK, SORTABLE_UINT_LOG2_MAX_BYTES, and SORTABLE_UINT_MAX_BYTES.

Referenced by make_key(), ChertTermListTable::make_key(), ChertPositionListTable::make_key(), make_slot_key(), make_valuechunk_key(), ChertCompact::merge_docid_keyed(), ChertCompact::PostlistCursor::next(), pack_chert_postlist_key(), and test_pack_uint_preserving_sort2().

◆ C_unpack_uint_preserving_sort()

template<class U >
bool C_unpack_uint_preserving_sort ( const char **  p,
const char *  end,
U *  result 
)
inline

Decode an "sort preserved" unsigned integer from a string.

[Chert variant]

The unsigned integer must have been encoded with C_pack_uint_preserving_sort().

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result.

Definition at line 185 of file pack.h.

References Assert, rare, SORTABLE_UINT_1ST_BYTE_MASK, and SORTABLE_UINT_LOG2_MAX_BYTES.

Referenced by check_chert_table(), docid_from_key(), Chert::PostlistChunkWriter::flush(), ChertPostListTable::get_chunk(), ChertValueManager::get_chunk_containing_did(), ChertPostListTable::get_used_docid_range(), ChertCompact::merge_docid_keyed(), ChertPostList::move_to_chunk_containing(), ChertCompact::PostlistCursor::next(), ChertPostList::next_chunk(), and test_pack_uint_preserving_sort2().

◆ pack_bool()

void pack_bool ( std::string &  s,
bool  value 
)
inline

Append an encoded bool to a string.

Parameters
sThe string to append to.
valueThe bool to encode.

Definition at line 57 of file pack.h.

Referenced by make_start_of_chunk().

◆ pack_chert_postlist_key() [1/2]

std::string pack_chert_postlist_key ( const std::string &  term)
inline

◆ pack_chert_postlist_key() [2/2]

std::string pack_chert_postlist_key ( const std::string &  term,
Xapian::docid  did 
)
inline

Definition at line 597 of file pack.h.

References C_pack_uint_preserving_sort(), and pack_string_preserving_sort().

◆ pack_glass_postlist_key() [1/2]

std::string pack_glass_postlist_key ( const std::string &  term)
inline

◆ pack_glass_postlist_key() [2/2]

std::string pack_glass_postlist_key ( const std::string &  term,
Xapian::docid  did 
)
inline

Definition at line 625 of file pack.h.

References pack_string_preserving_sort(), and pack_uint_preserving_sort().

◆ pack_string() [1/2]

void pack_string ( std::string &  s,
const std::string &  value 
)
inline

◆ pack_string() [2/2]

void pack_string ( std::string &  s,
const char *  ptr 
)
inline

Append an encoded C-style string to a string.

Parameters
sThe string to append to.
ptrThe C-style string to encode.

Definition at line 489 of file pack.h.

References Assert, and pack_uint().

◆ pack_string_preserving_sort()

void pack_string_preserving_sort ( std::string &  s,
const std::string &  value,
bool  last = false 
)
inline

Append an encoded std::string to a string, preserving the sort order.

The byte which follows this encoded value must not be , or the sort order won't be correct. You may need to store a padding byte (\0 say) to ensure this. Note that pack_uint_preserving_sort() can never produce as its first byte so is safe to use immediately afterwards.

Parameters
sThe string to append to.
valueThe std::string to encode.
lastIf true, this is the last thing to be encoded in this string - see note below (default: false)

It doesn't make sense to use pack_string_preserving_sort() if nothing can ever follow, but if optional items can, you can set last=true in cases where nothing does and get a shorter encoding in those cases.

Definition at line 539 of file pack.h.

Referenced by GlassPositionListTable::make_key(), GlassCompact::PositionCursor::next(), pack_chert_postlist_key(), and pack_glass_postlist_key().

◆ pack_uint() [1/2]

template<class U >
void pack_uint ( std::string &  s,
value 
)
inline

Append an encoded unsigned integer to a string.

Parameters
sThe string to append to.
valueThe unsigned integer to encode.

Definition at line 382 of file pack.h.

Referenced by ChertValueManager::add_document(), GlassValueManager::add_document(), Chert::PostlistChunkWriter::append(), Glass::PostlistChunkWriter::append(), ValueUpdater::append_to_stream(), Glass::ValueUpdater::append_to_stream(), ChertDatabaseReplicator::apply_changeset_from_conn(), GlassDatabaseReplicator::apply_changeset_from_conn(), GlassCompact::encode_valuestats(), ChertCompact::encode_valuestats(), GlassDatabase::get_revision_info(), ChertDatabase::get_revision_info(), make_start_of_chunk(), make_start_of_first_chunk(), make_valuechunk_key(), Glass::make_valuechunk_key(), GlassCompact::merge_postlists(), ChertCompact::merge_postlists(), GlassCompact::PostlistCursor::next(), ChertCompact::PostlistCursor::next(), GlassFLCursor::pack(), GlassPositionListTable::pack(), GlassFreeList::pack(), pack_string(), GlassDatabase::send_whole_database(), ChertDatabase::send_whole_database(), Glass::RootInfo::serialise(), GlassVersion::serialise_stats(), ChertPositionListTable::set_positionlist(), ChertDatabase::set_revision_number(), ChertTermListTable::set_termlist(), GlassTermListTable::set_termlist(), ChertValueManager::set_value_stats(), GlassValueManager::set_value_stats(), GlassChanges::start(), ChertDatabaseStats::write(), GlassVersion::write(), GlassTable::write_block(), ChertTable::write_changed_blocks(), GlassDatabase::write_changesets_to_fd(), ChertDatabase::write_changesets_to_fd(), and ChertTable_base::write_to_file().

◆ pack_uint() [2/2]

template<>
void pack_uint ( std::string &  s,
bool  value 
)
inline

Append an encoded unsigned integer (bool type) to a string.

Parameters
sThe string to append to.
valueThe unsigned integer to encode.

Definition at line 400 of file pack.h.

◆ pack_uint_last()

template<class U >
void pack_uint_last ( std::string &  s,
value 
)
inline

Append an encoded unsigned integer to a string as the last item.

This encoding is only suitable when this is the last thing encoded as the encoding used doesn't contain its own length.

Parameters
sThe string to append to.
valueThe unsigned integer to encode.

Definition at line 93 of file pack.h.

Referenced by make_valuestats_key(), ChertSpellingTable::merge_changes(), GlassSpellingTable::merge_changes(), ChertCompact::merge_postlists(), GlassCompact::merge_spellings(), ChertCompact::merge_spellings(), and ChertDatabaseStats::write().

◆ pack_uint_preserving_sort()

template<class U >
void pack_uint_preserving_sort ( std::string &  s,
value 
)
inline

Append an encoded unsigned integer to a string, preserving the sort order.

[Glass and newer variant]

The appended string data will sort in the same order as the unsigned integer being encoded.

Note that the first byte of the encoding will never be , so it is safe to store the result of this function immediately after the result of pack_string_preserving_sort().

Parameters
sThe string to append to.
valueThe unsigned integer to encode.

Definition at line 269 of file pack.h.

References Assert, and AssertRel.

Referenced by GlassDocDataTable::make_key(), GlassTermListTable::make_key(), GlassPositionListTable::make_key(), make_slot_key(), Glass::make_valuechunk_key(), GlassCompact::merge_docid_keyed(), GlassCompact::PostlistCursor::next(), GlassCompact::PositionCursor::next(), pack_glass_postlist_key(), and test_pack_uint_preserving_sort1().

◆ unpack_bool()

bool unpack_bool ( const char **  p,
const char *  end,
bool *  result 
)
inline

Decode a bool from a string.

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result.

Definition at line 69 of file pack.h.

References Assert, and rare.

Referenced by check_chert_table(), check_glass_table(), and read_start_of_chunk().

◆ unpack_string()

bool unpack_string ( const char **  p,
const char *  end,
std::string &  result 
)
inline

Decode a std::string from a string.

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result.

Definition at line 504 of file pack.h.

References rare, and unpack_uint().

Referenced by ChertDatabaseReplicator::apply_changeset_from_conn(), ValueChunkReader::assign(), Glass::ValueChunkReader::assign(), check_chert_table(), check_glass_table(), ChertValueManager::get_value_stats(), GlassValueManager::get_value_stats(), GlassCompact::merge_postlists(), ChertCompact::merge_postlists(), ValueChunkReader::next(), Glass::ValueChunkReader::next(), and Glass::RootInfo::unserialise().

◆ unpack_string_preserving_sort()

bool unpack_string_preserving_sort ( const char **  p,
const char *  end,
std::string &  result 
)
inline

Decode a "sort preserved" std::string from a string.

The std::string must have been encoded with pack_string_preserving_sort().

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result.

Definition at line 562 of file pack.h.

References Assert, rare, and usual.

Referenced by check_chert_table(), check_glass_table(), get_tname_from_key(), GlassCompact::merge_postlists(), ChertCompact::merge_postlists(), ChertAllTermsList::next(), GlassAllTermsList::next(), GlassCompact::PostlistCursor::next(), ChertCompact::PostlistCursor::next(), GlassCompact::PositionCursor::next(), ChertAllTermsList::skip_to(), and GlassAllTermsList::skip_to().

◆ unpack_uint()

template<class U >
bool unpack_uint ( const char **  p,
const char *  end,
U *  result 
)
inline

Decode an unsigned integer from a string.

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result (or NULL to just skip it).

Definition at line 413 of file pack.h.

References Assert, and rare.

Referenced by ChertDatabaseReplicator::apply_changeset_from_conn(), GlassDatabaseReplicator::apply_changeset_from_conn(), GlassChanges::check(), check_chert_table(), check_glass_table(), ChertDatabaseReplicator::check_revision_at_least(), GlassDatabaseReplicator::check_revision_at_least(), ChertTermList::ChertTermList(), ChertValueManager::delete_document(), GlassValueManager::delete_document(), do_unpack_uint(), docid_from_key(), Glass::docid_from_key(), ChertValueManager::get_all_values(), GlassValueManager::get_all_values(), GlassDatabase::get_changeset_revisions(), ChertDatabase::get_changeset_revisions(), ChertValueManager::get_chunk_containing_did(), GlassValueManager::get_chunk_containing_did(), GlassPostListTable::get_freqs(), Xapian::Database::get_revision(), ChertValueManager::get_value_stats(), GlassValueManager::get_value_stats(), GlassTermList::GlassTermList(), GlassCompact::merge_postlists(), ChertCompact::merge_postlists(), GlassCompact::PostlistCursor::next(), ChertCompact::PostlistCursor::next(), ChertTermList::next(), GlassTermList::next(), ValueChunkReader::next(), Glass::ValueChunkReader::next(), ChertPositionListTable::positionlist_count(), GlassPositionListTable::positionlist_count(), ChertDatabaseReplicator::process_changeset_chunk_base(), GlassDatabaseReplicator::process_changeset_chunk_blocks(), ChertDatabaseReplicator::process_changeset_chunk_blocks(), GlassDatabaseReplicator::process_changeset_chunk_version(), ChertTable_base::read(), ChertDatabaseStats::read(), GlassVersion::read(), ChertPositionList::read_data(), GlassPositionList::read_data(), read_did_increase(), ChertPostList::read_number_of_entries(), GlassPostList::read_number_of_entries(), read_start_of_chunk(), read_start_of_first_chunk(), read_wdf(), ValueChunkReader::skip_to(), Glass::ValueChunkReader::skip_to(), GlassFLCursor::unpack(), GlassFreeList::unpack(), unpack_string(), Glass::RootInfo::unserialise(), GlassVersion::unserialise_stats(), GlassDatabase::write_changesets_to_fd(), and ChertDatabase::write_changesets_to_fd().

◆ unpack_uint_last()

template<class U >
bool unpack_uint_last ( const char **  p,
const char *  end,
U *  result 
)
inline

◆ unpack_uint_preserving_sort()

template<class U >
bool unpack_uint_preserving_sort ( const char **  p,
const char *  end,
U *  result 
)
inline

Decode a "sort preserved" unsigned integer from a string.

[Glass and newer variant]

The unsigned integer must have been encoded with pack_uint_preserving_sort().

Parameters
pPointer to pointer to the current position in the string.
endPointer to the end of the string.
resultWhere to store the result.

Definition at line 318 of file pack.h.

References Assert, and rare.

Referenced by check_glass_table(), Glass::docid_from_key(), Glass::PostlistChunkWriter::flush(), GlassPostListTable::get_chunk(), GlassValueManager::get_chunk_containing_did(), GlassPostListTable::get_used_docid_range(), GlassCompact::merge_docid_keyed(), GlassPostList::move_to_chunk_containing(), GlassCompact::PostlistCursor::next(), GlassCompact::PositionCursor::next(), GlassPostList::next_chunk(), and test_pack_uint_preserving_sort1().

Variable Documentation

◆ SORTABLE_UINT_1ST_BYTE_MASK

const unsigned int SORTABLE_UINT_1ST_BYTE_MASK
Initial value:
=
const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES
How many bits to store the length of a sortable uint in.
Definition: pack.h:42

Calculated value used below.

Definition at line 48 of file pack.h.

Referenced by C_pack_uint_preserving_sort(), and C_unpack_uint_preserving_sort().

◆ SORTABLE_UINT_LOG2_MAX_BYTES

const unsigned int SORTABLE_UINT_LOG2_MAX_BYTES = 2

How many bits to store the length of a sortable uint in.

Setting this to 2 limits us to 2**32 documents in the database. If set to 3, then 2**64 documents are possible, but the database format isn't compatible.

Definition at line 42 of file pack.h.

Referenced by C_pack_uint_preserving_sort(), and C_unpack_uint_preserving_sort().

◆ SORTABLE_UINT_MAX_BYTES

const unsigned int SORTABLE_UINT_MAX_BYTES = 1 << SORTABLE_UINT_LOG2_MAX_BYTES

Calculated value used below.

Definition at line 45 of file pack.h.

Referenced by C_pack_uint_preserving_sort().