FlintDatabase Class Reference

A backend designed for efficient indexing and retrieval, using compressed posting lists and a btree storage scheme. More...

#include <flint_database.h>

Inheritance diagram for FlintDatabase:

Inheritance graph
[legend]
Collaboration diagram for FlintDatabase:

Collaboration graph
[legend]

List of all members.

Public Member Functions

 FlintDatabase (const string &db_dir_, int action=XAPIAN_DB_READONLY, unsigned int block_size=0u)
 Create and open a flint database.
 ~FlintDatabase ()
Xapian::doccount get_doccount () const
 Virtual methods of Database::Internal.
Xapian::docid get_lastdocid () const
 Virtual methods of Database::Internal.
totlen_t get_total_length () const
 Virtual methods of Database::Internal.
Xapian::doclength get_avlength () const
 Virtual methods of Database::Internal.
Xapian::termcount get_doclength (Xapian::docid did) const
 Virtual methods of Database::Internal.
Xapian::doccount get_termfreq (const string &tname) const
 Virtual methods of Database::Internal.
Xapian::termcount get_collection_freq (const string &tname) const
 Virtual methods of Database::Internal.
bool term_exists (const string &tname) const
 Virtual methods of Database::Internal.
bool has_positions () const
 Virtual methods of Database::Internal.
LeafPostListopen_post_list (const string &tname) const
 Virtual methods of Database::Internal.
Xapian::Document::Internalopen_document (Xapian::docid did, bool lazy) const
 Virtual methods of Database::Internal.
PositionListopen_position_list (Xapian::docid did, const string &term) const
 Virtual methods of Database::Internal.
TermListopen_term_list (Xapian::docid did) const
 Virtual methods of Database::Internal.
TermListopen_allterms (const string &prefix) const
 Virtual methods of Database::Internal.
TermListopen_spelling_termlist (const string &word) const
 Virtual methods of Database::Internal.
TermListopen_spelling_wordlist () const
 Virtual methods of Database::Internal.
Xapian::doccount get_spelling_frequency (const string &word) const
 Virtual methods of Database::Internal.
TermListopen_synonym_termlist (const string &term) const
 Virtual methods of Database::Internal.
TermListopen_synonym_keylist (const string &prefix) const
 Virtual methods of Database::Internal.
string get_metadata (const string &key) const
 Virtual methods of Database::Internal.
TermListopen_metadata_keylist (const std::string &prefix) const
 Virtual methods of Database::Internal.
void write_changesets_to_fd (int fd, const string &start_revision, bool need_whole_db, Xapian::ReplicationInfo *info)
 Virtual methods of Database::Internal.
string get_revision_info () const
 Virtual methods of Database::Internal.
string get_uuid () const
 Virtual methods of Database::Internal.

Private Member Functions

void read_metainfo ()
 Read lastdocid and total_length from the postlist table.
bool database_exists ()
 Return true if a database exists at the path specified for this database.
void create_and_open_tables (unsigned int blocksize)
 Create new tables, and open them.
void open_tables_consistent ()
 Open all tables at most recent consistent revision.
void get_database_write_lock (bool creating)
 Get a write lock on the database, or throw an Xapian::DatabaseLockError if failure.
void open_tables (flint_revision_number_t revision)
 Open tables at specified revision number.
flint_revision_number_t get_revision_number () const
 Get an object holding the revision number which the tables are opened at.
flint_revision_number_t get_next_revision_number () const
 Get an object holding the next revision number which should be used in the tables.
void set_revision_number (flint_revision_number_t new_revision)
 Set the revision number in the tables.
void reopen ()
 Re-open tables to recover from an overwritten condition, or just get most up-to-date version.
void close ()
 Close all the tables permanently.
void modifications_failed (flint_revision_number_t old_revision, flint_revision_number_t new_revision, const std::string &msg)
 Called if a modifications fail.
void apply ()
 Apply any outstanding changes to the tables.
void cancel ()
 Cancel any outstanding changes to the tables.
void send_whole_database (RemoteConnection &conn, double end_time)
 Send a set of messages which transfer the whole database.
void get_changeset_revisions (const string &path, flint_revision_number_t *startrev, flint_revision_number_t *endrev) const
 Get the revision stored in a changeset.

Private Attributes

std::string db_dir
 Directory to store databases in.
bool readonly
 Whether the database is readonly.
FlintVersion version_file
 The file describing the Flint database.
FlintPostListTable postlist_table
 Table storing posting lists.
FlintPositionListTable position_table
 Table storing position lists.
FlintTermListTable termlist_table
 Table storing term lists.
FlintValueTable value_table
 Table storing values.
FlintSynonymTable synonym_table
 Table storing synonym data.
FlintSpellingTable spelling_table
 Table storing spelling correction data.
FlintRecordTable record_table
 Table storing records.
FlintLock lock
 Lock object.
totlen_t total_length
 Total length of all documents including unflushed modifications.
Xapian::docid lastdocid
 Highest document ID ever allocated by this database.
unsigned int max_changesets
 The maximum number of changesets which should be kept in the database.

Friends

class FlintWritableDatabase
class FlintTermList
class FlintPostList
class FlintAllTermsList
class FlintAllDocsPostList


Detailed Description

A backend designed for efficient indexing and retrieval, using compressed posting lists and a btree storage scheme.

Definition at line 49 of file flint_database.h.


Constructor & Destructor Documentation

FlintDatabase::FlintDatabase ( const string &  db_dir_,
int  action = XAPIAN_DB_READONLY,
unsigned int  block_size = 0u 
)

Create and open a flint database.

Exceptions:
Xapian::DatabaseCorruptError is thrown if there is no consistent revision available.
Xapian::DatabaseOpeningError thrown if database can't be opened.
Xapian::DatabaseVersionError thrown if database is in an unsupported format. This implies that the database was created by an older or newer version of Xapian.
Parameters:
dbdir directory holding flint tables
block_size Block size, in bytes, to use when creating tables. This is only important, and has the correct value, when the database is being created.

Definition at line 101 of file flint_database.cc.

References create_and_open_tables(), database_exists(), Xapian::DB_CREATE, Xapian::DB_CREATE_OR_OVERWRITE, db_dir, Xapian::DB_OPEN, get_database_write_lock(), FlintTable::get_latest_revision_number(), get_next_revision_number(), FlintTable::get_open_revision_number(), LOGCALL_CTOR, mkdir(), open_tables_consistent(), postlist_table, record_table, S_ISDIR, set_revision_number(), stat(), and XAPIAN_DB_READONLY.

FlintDatabase::~FlintDatabase (  ) 

Definition at line 173 of file flint_database.cc.

References LOGCALL_DTOR.


Member Function Documentation

void FlintDatabase::apply (  )  [private]

Apply any outstanding changes to the tables.

If an error occurs during this operation, this will be signalled by an exception being thrown. In this case the contents of the tables on disk will be left in an unmodified state (though possibly with increased revision numbers), and the outstanding changes will be lost.

Definition at line 756 of file flint_database.cc.

References Xapian::Error::get_description(), get_next_revision_number(), get_revision_number(), FlintSpellingTable::is_modified(), FlintSynonymTable::is_modified(), FlintTable::is_modified(), LOGCALL_VOID, modifications_failed(), position_table, postlist_table, record_table, set_revision_number(), spelling_table, synonym_table, termlist_table, and value_table.

Referenced by FlintWritableDatabase::add_document_(), FlintWritableDatabase::commit(), FlintWritableDatabase::delete_document(), and FlintWritableDatabase::replace_document().

void FlintDatabase::cancel (  )  [private, virtual]

void FlintDatabase::close (  )  [private, virtual]

void FlintDatabase::create_and_open_tables ( unsigned int  blocksize  )  [private]

bool FlintDatabase::database_exists (  )  [private]

Return true if a database exists at the path specified for this database.

Definition at line 199 of file flint_database.cc.

References FlintTable::exists(), LOGCALL, postlist_table, record_table, RETURN, and termlist_table.

Referenced by create_and_open_tables(), FlintDatabase(), and get_database_write_lock().

Xapian::doclength FlintDatabase::get_avlength (  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Definition at line 818 of file flint_database.cc.

References FlintRecordTable::get_doccount(), LOGCALL, record_table, RETURN, and total_length.

void FlintDatabase::get_changeset_revisions ( const string &  path,
flint_revision_number_t startrev,
flint_revision_number_t endrev 
) const [private]

Xapian::termcount FlintDatabase::get_collection_freq ( const string &  tname  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 846 of file flint_database.cc.

References Assert, FlintPostListTable::get_collection_freq(), LOGCALL, postlist_table, and RETURN.

void FlintDatabase::get_database_write_lock ( bool  creating  )  [private]

Get a write lock on the database, or throw an Xapian::DatabaseLockError if failure.

Parameters:
creating true if the database is in the process of being created - if false, will throw a DatabaseOpening error if the lock can't be acquired and the database doesn't exist.

Definition at line 537 of file flint_database.cc.

References database_exists(), db_dir, FlintLock::lock(), lock, LOGCALL_VOID, FlintLock::SUCCESS, FlintLock::throw_databaselockerror(), and FlintLock::UNKNOWN.

Referenced by FlintDatabase().

Xapian::doccount FlintDatabase::get_doccount (  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Definition at line 797 of file flint_database.cc.

References FlintRecordTable::get_doccount(), LOGCALL, record_table, and RETURN.

Referenced by FlintWritableDatabase::open_post_list(), and open_post_list().

Xapian::termcount FlintDatabase::get_doclength ( Xapian::docid  did  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 830 of file flint_database.cc.

References Assert, FlintTermListTable::get_doclength(), LOGCALL, RETURN, and termlist_table.

Referenced by FlintWritableDatabase::get_doclength().

Xapian::docid FlintDatabase::get_lastdocid (  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Definition at line 804 of file flint_database.cc.

References lastdocid, LOGCALL, and RETURN.

string FlintDatabase::get_metadata ( const string &  key  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Definition at line 967 of file flint_database.cc.

References FlintTable::get_exact_entry(), LOGCALL, postlist_table, and RETURN.

flint_revision_number_t FlintDatabase::get_next_revision_number (  )  const [private]

Get an object holding the next revision number which should be used in the tables.

Returns:
the next revision number.

Definition at line 353 of file flint_database.cc.

References FlintTable::get_latest_revision_number(), LOGCALL, postlist_table, and RETURN.

Referenced by apply(), and FlintDatabase().

string FlintDatabase::get_revision_info (  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Definition at line 988 of file flint_database.cc.

References F_pack_uint(), get_revision_number(), LOGCALL, and RETURN.

flint_revision_number_t FlintDatabase::get_revision_number (  )  const [private]

Get an object holding the revision number which the tables are opened at.

Returns:
the current revision number.

Definition at line 345 of file flint_database.cc.

References FlintTable::get_open_revision_number(), LOGCALL, postlist_table, and RETURN.

Referenced by apply(), get_revision_info(), send_whole_database(), set_revision_number(), and write_changesets_to_fd().

Xapian::doccount FlintDatabase::get_spelling_frequency ( const string &  word  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Definition at line 946 of file flint_database.cc.

References FlintSpellingTable::get_word_frequency(), and spelling_table.

Xapian::doccount FlintDatabase::get_termfreq ( const string &  tname  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 838 of file flint_database.cc.

References Assert, FlintPostListTable::get_termfreq(), LOGCALL, postlist_table, and RETURN.

totlen_t FlintDatabase::get_total_length (  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Definition at line 811 of file flint_database.cc.

References LOGCALL, RETURN, and total_length.

string FlintDatabase::get_uuid (  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Definition at line 997 of file flint_database.cc.

References FlintVersion::get_uuid_string(), LOGCALL, RETURN, and version_file.

Referenced by send_whole_database(), and write_changesets_to_fd().

bool FlintDatabase::has_positions (  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Definition at line 862 of file flint_database.cc.

References FlintTable::empty(), and position_table.

void FlintDatabase::modifications_failed ( flint_revision_number_t  old_revision,
flint_revision_number_t  new_revision,
const std::string &  msg 
) [private]

Called if a modifications fail.

Parameters:
msg is a string description of the exception that was raised when the modifications failed.

Definition at line 728 of file flint_database.cc.

References cancel(), close(), Xapian::Error::get_msg(), open_tables(), and set_revision_number().

Referenced by apply().

TermList * FlintDatabase::open_allterms ( const string &  prefix  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 923 of file flint_database.cc.

References FlintAllTermsList, LOGCALL, and RETURN.

Referenced by FlintWritableDatabase::open_allterms().

Xapian::Document::Internal * FlintDatabase::open_document ( Xapian::docid  did,
bool  lazy 
) const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 895 of file flint_database.cc.

References Assert, LOGCALL, record_table, RETURN, and value_table.

TermList * FlintDatabase::open_metadata_keylist ( const std::string &  prefix  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Definition at line 978 of file flint_database.cc.

References FlintTable::cursor_get(), LOGCALL, and postlist_table.

PositionList * FlintDatabase::open_position_list ( Xapian::docid  did,
const string &  term 
) const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Definition at line 908 of file flint_database.cc.

References Assert, and position_table.

LeafPostList * FlintDatabase::open_post_list ( const string &  tname  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 868 of file flint_database.cc.

References FlintAllDocsPostList, FlintPostList, get_doccount(), lastdocid, LOGCALL, and RETURN.

TermList * FlintDatabase::open_spelling_termlist ( const string &  word  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Definition at line 931 of file flint_database.cc.

References FlintSpellingTable::open_termlist(), and spelling_table.

TermList * FlintDatabase::open_spelling_wordlist (  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 937 of file flint_database.cc.

References FlintTable::cursor_get(), and spelling_table.

TermList * FlintDatabase::open_synonym_keylist ( const string &  prefix  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 958 of file flint_database.cc.

References FlintTable::cursor_get(), and synonym_table.

TermList * FlintDatabase::open_synonym_termlist ( const string &  term  )  const [virtual]

Virtual methods of Database::Internal.

Reimplemented from Xapian::Database::Internal.

Definition at line 952 of file flint_database.cc.

References FlintSynonymTable::open_termlist(), and synonym_table.

void FlintDatabase::open_tables ( flint_revision_number_t  revision  )  [private]

Open tables at specified revision number.

Exceptions:
Xapian::InvalidArgumentError is thrown if the specified revision is not available.

Definition at line 322 of file flint_database.cc.

References FlintTable::get_block_size(), LOGCALL_VOID, FlintTable::open(), position_table, postlist_table, FlintVersion::read_and_check(), readonly, record_table, FlintTable::set_block_size(), spelling_table, synonym_table, termlist_table, value_table, and version_file.

Referenced by modifications_failed().

void FlintDatabase::open_tables_consistent (  )  [private]

TermList * FlintDatabase::open_term_list ( Xapian::docid  did  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Definition at line 885 of file flint_database.cc.

References Assert, FlintTermList, LOGCALL, and RETURN.

void FlintDatabase::read_metainfo (  )  [private]

Read lastdocid and total_length from the postlist table.

Definition at line 179 of file flint_database.cc.

References F_unpack_uint(), F_unpack_uint_last(), FlintTable::get_exact_entry(), lastdocid, LOGCALL_VOID, METAINFO_KEY(), postlist_table, and total_length.

Referenced by FlintWritableDatabase::cancel(), and open_tables_consistent().

void FlintDatabase::reopen (  )  [private, virtual]

Re-open tables to recover from an overwritten condition, or just get most up-to-date version.

Reimplemented from Xapian::Database::Internal.

Definition at line 514 of file flint_database.cc.

References LOGCALL_VOID, open_tables_consistent(), and readonly.

Referenced by write_changesets_to_fd().

void FlintDatabase::send_whole_database ( RemoteConnection conn,
double  end_time 
) [private]

void FlintDatabase::set_revision_number ( flint_revision_number_t  new_revision  )  [private]

Set the revision number in the tables.

This updates the disk tables so that the currently open revision becomes the specified revision number.

Parameters:
new_revision The new revision number to store. This must be greater than the latest revision number (see get_latest_revision_number()), or undefined behaviour will result.

Definition at line 417 of file flint_database.cc.

References CHANGES_MAGIC_STRING, CHANGES_VERSION, FlintTable::commit(), create_changeset_file(), db_dir, F_pack_uint(), FlintSpellingTable::flush_db(), FlintSynonymTable::flush_db(), FlintTable::flush_db(), get_revision_number(), io_unlink(), io_write(), LOGCALL_VOID, max_changesets, position_table, postlist_table, record_table, spelling_table, Xapian::Internal::str(), synonym_table, termlist_table, value_table, and FlintTable::write_changed_blocks().

Referenced by apply(), FlintDatabase(), and modifications_failed().

bool FlintDatabase::term_exists ( const string &  tname  )  const [virtual]

Virtual methods of Database::Internal.

Implements Xapian::Database::Internal.

Reimplemented in FlintWritableDatabase.

Definition at line 854 of file flint_database.cc.

References Assert, LOGCALL, postlist_table, and FlintPostListTable::term_exists().

void FlintDatabase::write_changesets_to_fd ( int  fd,
const string &  start_revision,
bool  need_whole_db,
Xapian::ReplicationInfo info 
) [virtual]


Friends And Related Function Documentation

friend class FlintAllDocsPostList [friend]

Definition at line 54 of file flint_database.h.

Referenced by FlintWritableDatabase::open_post_list(), and open_post_list().

friend class FlintAllTermsList [friend]

Definition at line 53 of file flint_database.h.

Referenced by open_allterms().

friend class FlintPostList [friend]

Definition at line 52 of file flint_database.h.

Referenced by FlintWritableDatabase::open_post_list(), and open_post_list().

friend class FlintTermList [friend]

Definition at line 51 of file flint_database.h.

Referenced by open_term_list().

friend class FlintWritableDatabase [friend]

Definition at line 50 of file flint_database.h.


Member Data Documentation

std::string FlintDatabase::db_dir [private]

Directory to store databases in.

Definition at line 58 of file flint_database.h.

Referenced by FlintDatabase(), get_database_write_lock(), send_whole_database(), set_revision_number(), and write_changesets_to_fd().

Lock object.

Definition at line 109 of file flint_database.h.

Referenced by close(), and get_database_write_lock().

unsigned int FlintDatabase::max_changesets [private]

The maximum number of changesets which should be kept in the database.

Definition at line 119 of file flint_database.h.

Referenced by set_revision_number().

Table storing posting lists.

Whenever an update is performed, this table is the first to be updated: therefore, its most recent revision number is the most recent anywhere in the database.

Definition at line 76 of file flint_database.h.

Referenced by apply(), cancel(), close(), create_and_open_tables(), database_exists(), FlintDatabase(), FlintWritableDatabase::flush_postlist_changes(), get_collection_freq(), get_metadata(), get_next_revision_number(), get_revision_number(), get_termfreq(), open_metadata_keylist(), open_tables(), open_tables_consistent(), read_metainfo(), FlintWritableDatabase::set_metadata(), set_revision_number(), and term_exists().

bool FlintDatabase::readonly [private]

Whether the database is readonly.

Definition at line 62 of file flint_database.h.

Referenced by open_tables(), open_tables_consistent(), and reopen().

Table storing records.

Whenever an update is performed, this table is the last to be updated: therefore, its most recent revision number is the most recent consistent revision available. If this table's most recent revision number is not available for all tables, there is no consistent revision available, and the database is corrupt.

Definition at line 106 of file flint_database.h.

Referenced by FlintWritableDatabase::add_document_(), apply(), cancel(), close(), create_and_open_tables(), database_exists(), FlintWritableDatabase::delete_document(), FlintDatabase(), get_avlength(), get_doccount(), open_document(), open_tables(), open_tables_consistent(), FlintWritableDatabase::replace_document(), and set_revision_number().

The file describing the Flint database.

This file has information about the format of the database which can't easily be stored in any of the individual tables.

Definition at line 68 of file flint_database.h.

Referenced by create_and_open_tables(), get_uuid(), open_tables(), and open_tables_consistent().


The documentation for this class was generated from the following files:

Documentation for Xapian (version 1.2.8).
Generated on 14 Dec 2011 by Doxygen 1.5.9.