xapian-core
1.4.26
|
Abstract base class for weighting schemes. More...
#include <weight.h>
Classes | |
class | Internal |
Class to hold statistics for a given collection. More... | |
Public Types | |
enum | type_smoothing { TWO_STAGE_SMOOTHING = 1, DIRICHLET_SMOOTHING = 2, ABSOLUTE_DISCOUNT_SMOOTHING = 3, JELINEK_MERCER_SMOOTHING = 4, DIRICHLET_PLUS_SMOOTHING = 5 } |
Type of smoothing to use with the Language Model Weighting scheme. More... | |
Public Member Functions | |
Weight () | |
Default constructor, needed by subclass constructors. More... | |
virtual | ~Weight () |
Virtual destructor, because we have virtual methods. More... | |
virtual Weight * | clone () const =0 |
Clone this object. More... | |
virtual std::string | name () const |
Return the name of this weighting scheme. More... | |
virtual std::string | serialise () const |
Return this object's parameters serialised as a single string. More... | |
virtual Weight * | unserialise (const std::string &serialised) const |
Unserialise parameters. More... | |
virtual double | get_sumpart (Xapian::termcount wdf, Xapian::termcount doclen, Xapian::termcount uniqterms) const =0 |
Calculate the weight contribution for this object's term to a document. More... | |
virtual double | get_maxpart () const =0 |
Return an upper bound on what get_sumpart() can return for any document. More... | |
virtual double | get_sumextra (Xapian::termcount doclen, Xapian::termcount uniqterms) const =0 |
Calculate the term-independent weight component for a document. More... | |
virtual double | get_maxextra () const =0 |
Return an upper bound on what get_sumextra() can return for any document. More... | |
Protected Types | |
enum | stat_flags { COLLECTION_SIZE = 1, RSET_SIZE = 2, AVERAGE_LENGTH = 4, TERMFREQ = 8, RELTERMFREQ = 16, QUERY_LENGTH = 32, WQF = 64, WDF = 128, DOC_LENGTH = 256, DOC_LENGTH_MIN = 512, DOC_LENGTH_MAX = 1024, WDF_MAX = 2048, COLLECTION_FREQ = 4096, UNIQUE_TERMS = 8192, TOTAL_LENGTH = COLLECTION_SIZE | AVERAGE_LENGTH } |
Stats which the weighting scheme can use (see need_stat()). More... | |
Protected Member Functions | |
void | need_stat (stat_flags flag) |
Tell Xapian that your subclass will want a particular statistic. More... | |
virtual void | init (double factor)=0 |
Allow the subclass to perform any initialisation it needs to. More... | |
Weight (const Weight &) | |
Don't allow copying. More... | |
Xapian::doccount | get_collection_size () const |
The number of documents in the collection. More... | |
Xapian::doccount | get_rset_size () const |
The number of documents marked as relevant. More... | |
Xapian::doclength | get_average_length () const |
The average length of a document in the collection. More... | |
Xapian::doccount | get_termfreq () const |
The number of documents which this term indexes. More... | |
Xapian::doccount | get_reltermfreq () const |
The number of relevant documents which this term indexes. More... | |
Xapian::termcount | get_collection_freq () const |
The collection frequency of the term. More... | |
Xapian::termcount | get_query_length () const |
The length of the query. More... | |
Xapian::termcount | get_wqf () const |
The within-query-frequency of this term. More... | |
Xapian::termcount | get_doclength_upper_bound () const |
An upper bound on the maximum length of any document in the shard. More... | |
Xapian::termcount | get_doclength_lower_bound () const |
A lower bound on the minimum length of any document in the shard. More... | |
Xapian::termcount | get_wdf_upper_bound () const |
An upper bound on the wdf of this term in the shard. More... | |
Xapian::totallength | get_total_length () const |
Total length of all documents in the collection. More... | |
Private Member Functions | |
void | operator= (const Weight &) |
Don't allow assignment. More... | |
void | init_ (const Internal &stats, Xapian::termcount query_len_, const std::string &term, Xapian::termcount wqf_, double factor) |
void | init_ (const Internal &stats, Xapian::termcount query_len_, const std::string &term, Xapian::termcount wqf_, double factor, void *postlist) |
void | init_ (const Internal &stats, Xapian::termcount query_len_, double factor, Xapian::doccount termfreq, Xapian::doccount reltermfreq, Xapian::termcount collection_freq) |
void | init_ (const Internal &stats, Xapian::termcount query_len_) |
bool | get_sumpart_needs_doclength_ () const |
bool | get_sumpart_needs_wdf_ () const |
bool | get_sumpart_needs_uniqueterms_ () const |
bool | is_bool_weight_ () const |
Private Attributes | |
stat_flags | stats_needed |
A bitmask of the statistics this weighting scheme needs. More... | |
Xapian::doccount | collection_size_ |
The number of documents in the collection. More... | |
Xapian::doccount | rset_size_ |
The number of documents marked as relevant. More... | |
Xapian::doclength | average_length_ |
The average length of a document in the collection. More... | |
Xapian::doccount | termfreq_ |
The number of documents which this term indexes. More... | |
Xapian::termcount | collectionfreq_ |
Xapian::doccount | reltermfreq_ |
The number of relevant documents which this term indexes. More... | |
Xapian::termcount | query_length_ |
The length of the query. More... | |
Xapian::termcount | wqf_ |
The within-query-frequency of this term. More... | |
Xapian::termcount | doclength_lower_bound_ |
A lower bound on the minimum length of any document in the shard. More... | |
Xapian::termcount | doclength_upper_bound_ |
An upper bound on the maximum length of any document in the shard. More... | |
Xapian::termcount | wdf_upper_bound_ |
An upper bound on the wdf of this term in the shard. More... | |
|
protected |
Stats which the weighting scheme can use (see need_stat()).
Enumerator | |
---|---|
COLLECTION_SIZE | Number of documents in the collection. |
RSET_SIZE | Number of documents in the RSet. |
AVERAGE_LENGTH | Average length of documents in the collection. |
TERMFREQ | How many documents the current term is in. |
RELTERMFREQ | How many documents in the RSet the current term is in. |
QUERY_LENGTH | Sum of wqf for terms in the query. |
WQF | Within-query-frequency of the current term. |
WDF | Within-document-frequency of the current term in the current document. |
DOC_LENGTH | Length of the current document (sum wdf). |
DOC_LENGTH_MIN | Lower bound on (non-zero) document lengths. This bound is for the current shard and is suitable for using to calculate upper bounds to return from get_maxpart() and get_maxextra(). |
DOC_LENGTH_MAX | Upper bound on document lengths. This bound is for the current shard and is suitable for using to calculate upper bounds to return from get_maxpart() and get_maxextra(). |
WDF_MAX | Upper bound on wdf. This bound is for the current shard and is suitable for using to calculate upper bounds to return from get_maxpart() and get_maxextra(). |
COLLECTION_FREQ | Sum of wdf over the whole collection for the current term. |
UNIQUE_TERMS | Number of unique terms in the current document. |
TOTAL_LENGTH | Sum of lengths of all documents in the collection. This gives the total number of term occurrences. |
|
inline |
|
virtual |
|
protected |
Don't allow copying.
This would ideally be private, but that causes a compilation error with GCC 4.1 (which appears to be a bug).
|
pure virtual |
Clone this object.
This method allocates and returns a copy of the object it is called on.
If your subclass is called FooWeight and has parameters a and b, then you would implement FooWeight::clone() like so:
FooWeight * FooWeight::clone() const { return new FooWeight(a, b); }
Note that the returned object will be deallocated by Xapian after use with "delete". If you want to handle the deletion in a special way (for example when wrapping the Xapian API for use from another language) then you can define a static operator delete
method in your subclass as shown here: https://trac.xapian.org/ticket/554#comment:1
Implemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, Xapian::InL2Weight, CheckStatsWeight, Xapian::TradWeight, CheckInitWeight, Xapian::BM25PlusWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, ExceptionalWeight, ZWeight, and LazyWeight.
Referenced by LocalSubMatch::open_post_list(), and Xapian::Enquire::set_weighting_scheme().
|
inlineprotected |
The average length of a document in the collection.
Definition at line 380 of file weight.h.
Referenced by Xapian::InL2Weight::init(), Xapian::IfB2Weight::init(), Xapian::IneB2Weight::init(), Xapian::BB2Weight::init(), Xapian::PL2Weight::init(), and Xapian::PL2PlusWeight::init().
|
inlineprotected |
The collection frequency of the term.
Definition at line 389 of file weight.h.
Referenced by Xapian::BB2Weight::get_sumpart(), Xapian::IfB2Weight::init(), Xapian::IneB2Weight::init(), Xapian::BB2Weight::init(), Xapian::PL2Weight::init(), and Xapian::PL2PlusWeight::init().
|
inlineprotected |
The number of documents in the collection.
Definition at line 374 of file weight.h.
Referenced by Xapian::TfIdfWeight::get_idfn(), Xapian::BB2Weight::get_sumpart(), Xapian::InL2Weight::init(), Xapian::IfB2Weight::init(), Xapian::IneB2Weight::init(), Xapian::BB2Weight::init(), Xapian::PL2Weight::init(), and Xapian::PL2PlusWeight::init().
|
inlineprotected |
A lower bound on the minimum length of any document in the shard.
This bound does not include any zero-length documents.
This should only be used by get_maxpart() and get_maxextra().
Definition at line 411 of file weight.h.
Referenced by Xapian::TfIdfWeight::get_maxpart(), Xapian::InL2Weight::init(), Xapian::IfB2Weight::init(), Xapian::IneB2Weight::init(), Xapian::BB2Weight::init(), Xapian::PL2Weight::init(), and Xapian::PL2PlusWeight::init().
|
inlineprotected |
An upper bound on the maximum length of any document in the shard.
This should only be used by get_maxpart() and get_maxextra().
Definition at line 401 of file weight.h.
Referenced by Xapian::BB2Weight::init(), Xapian::PL2Weight::init(), and Xapian::PL2PlusWeight::init().
|
pure virtual |
Return an upper bound on what get_sumextra() can return for any document.
This information is used by the matcher to perform various optimisations, so strive to make the bound as tight as possible.
Implemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, CheckStatsWeight, Xapian::InL2Weight, Xapian::TradWeight, Xapian::BM25PlusWeight, CheckInitWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, ExceptionalWeight, ZWeight, and LazyWeight.
|
pure virtual |
Return an upper bound on what get_sumpart() can return for any document.
This information is used by the matcher to perform various optimisations, so strive to make the bound as tight as possible.
Implemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, CheckStatsWeight, Xapian::InL2Weight, Xapian::TradWeight, Xapian::BM25PlusWeight, CheckInitWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, ExceptionalWeight, ZWeight, and LazyWeight.
Referenced by SynonymPostList::get_maxweight(), SynonymPostList::get_weight(), LocalSubMatch::open_post_list(), and LeafPostList::resolve_lazy_termweight().
|
inlineprotected |
|
inlineprotected |
|
inlineprotected |
|
pure virtual |
Calculate the term-independent weight component for a document.
The parameter gives information about the document which may be used in the calculations:
doclen | The document's length (unnormalised). |
uniqterms | The number of unique terms in the document. |
Implemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, CheckStatsWeight, Xapian::InL2Weight, Xapian::TradWeight, Xapian::BM25PlusWeight, CheckInitWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, ExceptionalWeight, ZWeight, and LazyWeight.
Referenced by ExtraWeightPostList::get_weight().
|
pure virtual |
Calculate the weight contribution for this object's term to a document.
The parameters give information about the document which may be used in the calculations:
wdf | The within document frequency of the term in the document. |
doclen | The document's length (unnormalised). |
uniqterms | Number of unique terms in the document (used for absolute smoothing). |
Implemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, Xapian::InL2Weight, CheckStatsWeight, Xapian::TradWeight, Xapian::BM25PlusWeight, CheckInitWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, ExceptionalWeight, ZWeight, and LazyWeight.
Referenced by SynonymPostList::get_weight().
|
inlineprivate |
Return true if the document length is needed.
If this method returns true, then the document length will be fetched and passed to get_sumpart(). Otherwise 0 may be passed for the document length.
Definition at line 333 of file weight.h.
Referenced by LeafPostList::resolve_lazy_termweight(), and SynonymPostList::set_weight().
|
inlineprivate |
Return true if the number of unique terms is needed.
If this method returns true, then the number of unique terms will be fetched and passed to get_sumpart(). Otherwise 0 may be passed for the number of unique terms.
Definition at line 352 of file weight.h.
Referenced by SynonymPostList::set_weight().
|
inlineprivate |
Return true if the WDF is needed.
If this method returns true, then the WDF will be fetched and passed to get_sumpart(). Otherwise 0 may be passed for the wdf.
Definition at line 342 of file weight.h.
Referenced by SynonymPostList::set_weight().
|
inlineprotected |
The number of documents which this term indexes.
Definition at line 383 of file weight.h.
Referenced by Xapian::TfIdfWeight::get_maxpart(), Xapian::TfIdfWeight::get_sumpart(), Xapian::InL2Weight::init(), Xapian::IfB2Weight::init(), Xapian::IneB2Weight::init(), and Xapian::BB2Weight::init().
|
inlineprotected |
|
inlineprotected |
An upper bound on the wdf of this term in the shard.
This should only be used by get_maxpart() and get_maxextra().
Definition at line 419 of file weight.h.
Referenced by Xapian::TfIdfWeight::get_maxpart(), Xapian::InL2Weight::init(), Xapian::IfB2Weight::init(), Xapian::IneB2Weight::init(), Xapian::BB2Weight::init(), Xapian::PL2Weight::init(), and Xapian::PL2PlusWeight::init().
|
inlineprotected |
The within-query-frequency of this term.
Definition at line 395 of file weight.h.
Referenced by Xapian::TfIdfWeight::init(), Xapian::InL2Weight::init(), Xapian::IfB2Weight::init(), Xapian::IneB2Weight::init(), Xapian::BB2Weight::init(), Xapian::PL2Weight::init(), and Xapian::PL2PlusWeight::init().
|
protectedpure virtual |
Allow the subclass to perform any initialisation it needs to.
factor | Any scaling factor (e.g. from OP_SCALE_WEIGHT). If the Weight object is for the term-independent weight supplied by get_sumextra()/get_maxextra(), then init(0.0) is called (starting from Xapian 1.2.11 and 1.3.1 - earlier versions failed to call init() for such Weight objects). |
Implemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, Xapian::InL2Weight, CheckStatsWeight, Xapian::TradWeight, CheckInitWeight, Xapian::BM25PlusWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, ExceptionalWeight, ZWeight, and LazyWeight.
|
private |
Initialise this object to calculate weights for term term.
Old version of method, as used by 1.4.18 and earlier. This should only be referenced from inside the library and 1.4.19 and later will call the new version instead. We continue to provide it mainly to avoid triggering ABI checking tools.
stats | Source of statistics. |
query_len_ | Query length. |
term | The term for the new object. |
wqf_ | The within-query-frequency of term. |
factor | Any scaling factor (e.g. from OP_SCALE_WEIGHT). |
Definition at line 93 of file weight.cc.
Referenced by LocalSubMatch::open_post_list(), and LeafPostList::resolve_lazy_termweight().
|
private |
Initialise this object to calculate weights for term term.
stats | Source of statistics. |
query_len_ | Query length. |
term | The term for the new object. |
wqf_ | The within-query-frequency of term. |
factor | Any scaling factor (e.g. from OP_SCALE_WEIGHT). |
postlist | Pointer to a LeafPostList for the term (cast to void* to avoid needing to forward declare class LeafPostList in public API headers) which can be used to get wdf upper bound |
Definition at line 60 of file weight.cc.
References Assert, Xapian::Weight::Internal::collection_size, Xapian::Weight::Internal::db, Xapian::Weight::Internal::get_average_length(), Xapian::Database::get_doclength_lower_bound(), Xapian::Database::get_doclength_upper_bound(), Xapian::Weight::Internal::get_stats(), LeafPostList::get_wdf_upper_bound(), Xapian::Database::get_wdf_upper_bound(), LOGCALL_VOID, Xapian::Weight::Internal::rset_size, and usual.
|
private |
Initialise this object to calculate weights for a synonym.
stats | Source of statistics. |
query_len_ | Query length. |
factor | Any scaling factor (e.g. from OP_SCALE_WEIGHT). |
termfreq | The termfreq to use. |
reltermfreq | The reltermfreq to use. |
collection_freq | The collection frequency to use. |
Definition at line 100 of file weight.cc.
References Xapian::Weight::Internal::collection_size, Xapian::Weight::Internal::db, Xapian::Weight::Internal::get_average_length(), Xapian::Database::get_doclength_lower_bound(), Xapian::Database::get_doclength_upper_bound(), LOGCALL_VOID, and Xapian::Weight::Internal::rset_size.
|
private |
Initialise this object to calculate the extra weight component.
stats | Source of statistics. |
query_len_ | Query length. |
Definition at line 39 of file weight.cc.
References Xapian::Weight::Internal::collection_size, Xapian::Weight::Internal::db, Xapian::Weight::Internal::get_average_length(), Xapian::Database::get_doclength_lower_bound(), Xapian::Database::get_doclength_upper_bound(), LOGCALL_VOID, and Xapian::Weight::Internal::rset_size.
|
inlineprivate |
|
virtual |
Return the name of this weighting scheme.
This name is used by the remote backend. It is passed along with the serialised parameters to the remote server so that it knows which class to create.
Return the full namespace-qualified name of your class here - if your class is called FooWeight, return "FooWeight" from this method (Xapian::BM25Weight returns "Xapian::BM25Weight" here).
If you don't want to support the remote backend, you can use the default implementation which simply returns an empty string.
Reimplemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, Xapian::InL2Weight, Xapian::TradWeight, Xapian::BM25PlusWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, ExceptionalWeight, and LazyWeight.
Definition at line 135 of file weight.cc.
Referenced by Xapian::Registry::Internal::add_defaults(), DEFINE_TESTCASE(), Xapian::Registry::register_weighting_scheme(), and RemoteDatabase::set_query().
|
inlineprotected |
Tell Xapian that your subclass will want a particular statistic.
Some of the statistics can be costly to fetch or calculate, so Xapian needs to know which are actually going to be used. You should call need_stat() from your constructor for each such statistic.
flag | The stat_flags value for a required statistic. |
Definition at line 94 of file weight.h.
Referenced by Xapian::BB2Weight::BB2Weight(), Xapian::IfB2Weight::IfB2Weight(), Xapian::IneB2Weight::IneB2Weight(), Xapian::InL2Weight::InL2Weight(), Xapian::PL2PlusWeight::PL2PlusWeight(), Xapian::PL2Weight::PL2Weight(), and Xapian::TfIdfWeight::TfIdfWeight().
|
private |
Don't allow assignment.
|
virtual |
Return this object's parameters serialised as a single string.
If you don't want to support the remote backend, you can use the default implementation which simply throws Xapian::UnimplementedError.
Reimplemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, Xapian::InL2Weight, Xapian::TradWeight, Xapian::BM25PlusWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, and LazyWeight.
Definition at line 141 of file weight.cc.
Referenced by RemoteDatabase::set_query(), and test_weight_class().
|
virtual |
Unserialise parameters.
This method unserialises parameters serialised by the serialise() method and allocates and returns a new object initialised with them.
If you don't want to support the remote backend, you can use the default implementation which simply throws Xapian::UnimplementedError.
Note that the returned object will be deallocated by Xapian after use with "delete". If you want to handle the deletion in a special way (for example when wrapping the Xapian API for use from another language) then you can define a static operator delete
method in your subclass as shown here: https://trac.xapian.org/ticket/554#comment:1
serialised | A string containing the serialised parameters. |
Reimplemented in MyWeight, Xapian::CoordWeight, Xapian::LMWeight, Xapian::DPHWeight, Xapian::PL2PlusWeight, Xapian::PL2Weight, Xapian::DLHWeight, Xapian::BB2Weight, Xapian::IneB2Weight, Xapian::IfB2Weight, Xapian::InL2Weight, Xapian::TradWeight, Xapian::BM25PlusWeight, Xapian::BM25Weight, Xapian::TfIdfWeight, Xapian::BoolWeight, and LazyWeight.
Definition at line 147 of file weight.cc.
Referenced by RemoteServer::msg_query().
|
private |
|
private |
|
private |
|
private |
|
private |
|
private |
|
private |
|
private |
|
private |
|
private |
|
private |
|
private |