Language Model weighting with Dirichlet or Dir+ smoothing. More...

#include <weight.h>

Inheritance diagram for Xapian::LMDirichletWeight:

Collaboration diagram for Xapian::LMDirichletWeight:

Public Member Functions
	LMDirichletWeight (double mu=2000.0, double delta=0.05)
	Construct a LMDirichletWeight. More...

double	get_sumpart (Xapian::termcount wdf, Xapian::termcount doclen, Xapian::termcount uniqterm, Xapian::termcount wdfdocmax) const
	Calculate the weight contribution for this object's term to a document. More...

double	get_maxpart () const
	Return an upper bound on what get_sumpart() can return for any document. More...

double	get_sumextra (Xapian::termcount doclen, Xapian::termcount, Xapian::termcount) const
	Calculate the term-independent weight component for a document. More...

double	get_maxextra () const
	Return an upper bound on what get_sumextra() can return for any document. More...

std::string	name () const
	Return the name of this weighting scheme, e.g. More...

std::string	serialise () const
	Return this object's parameters serialised as a single string. More...

LMDirichletWeight *	unserialise (const std::string &serialised) const
	Unserialise parameters. More...

LMDirichletWeight *	create_from_parameters (const char *params) const
	Create from a human-readable parameter string. More...

Public Member Functions inherited from Xapian::Weight
	Weight ()
	Default constructor, needed by subclass constructors. More...

virtual	~Weight ()
	Virtual destructor, because we have virtual methods. More...

Private Member Functions
LMDirichletWeight *	clone () const
	Clone this object. More...

void	init (double factor_)
	Allow the subclass to perform any initialisation it needs to. More...

Private Attributes
double	factor
	The factor to multiply weights by. More...

double	param_mu
	Parameter controlling the smoothing. More...

double	param_delta
	A pseudo TF value to control the scale of the TF lower bound. More...

double	multiplier
	Precalculated multiplier for use in weight calculations. More...

double	extra_offset
	Precalculated offset to add to every sumextra. More...

Additional Inherited Members
Static Public Member Functions inherited from Xapian::Weight
static const Weight *	create (const std::string &scheme, const Registry &reg=Registry())
	Return the appropriate weighting scheme object. More...

Protected Types inherited from Xapian::Weight
enum	stat_flags { COLLECTION_SIZE = 0 , RSET_SIZE = 0 , AVERAGE_LENGTH = 4 , TERMFREQ = 1 , RELTERMFREQ = 1 , QUERY_LENGTH = 0 , WQF = 0 , WDF = 2 , DOC_LENGTH = 8 , DOC_LENGTH_MIN = 16 , DOC_LENGTH_MAX = 32 , WDF_MAX = 64 , COLLECTION_FREQ = 1 , UNIQUE_TERMS = 128 , TOTAL_LENGTH = 256 , WDF_DOC_MAX = 512 , UNIQUE_TERMS_MIN = 1024 , UNIQUE_TERMS_MAX = 2048 , DB_DOC_LENGTH_MIN = 4096 , DB_DOC_LENGTH_MAX = 8192 , DB_UNIQUE_TERMS_MIN = 16384 , DB_UNIQUE_TERMS_MAX = 32768 , DB_WDF_MAX = 65536 , IS_BOOLWEIGHT_ = static_cast<int>(0x80000000) }
	Stats which the weighting scheme can use (see need_stat()). More...

Protected Member Functions inherited from Xapian::Weight
void	need_stat (stat_flags flag)
	Tell Xapian that your subclass will want a particular statistic. More...

	Weight (const Weight &)
	Don't allow copying. More...

Xapian::doccount	get_collection_size () const
	The number of documents in the collection. More...

Xapian::doccount	get_rset_size () const
	The number of documents marked as relevant. More...

Xapian::doclength	get_average_length () const
	The average length of a document in the collection. More...

Xapian::doccount	get_termfreq () const
	The number of documents which this term indexes. More...

Xapian::doccount	get_reltermfreq () const
	The number of relevant documents which this term indexes. More...

Xapian::termcount	get_collection_freq () const
	The collection frequency of the term. More...

Xapian::termcount	get_query_length () const
	The length of the query. More...

Xapian::termcount	get_wqf () const
	The within-query-frequency of this term. More...

Xapian::termcount	get_doclength_upper_bound () const
	An upper bound on the maximum length of any document in the shard. More...

Xapian::termcount	get_doclength_lower_bound () const
	A lower bound on the minimum length of any document in the shard. More...

Xapian::termcount	get_wdf_upper_bound () const
	An upper bound on the wdf of this term in the shard. More...

Xapian::totallength	get_total_length () const
	Total length of all documents in the collection. More...

Xapian::termcount	get_unique_terms_upper_bound () const
	A lower bound on the number of unique terms in any document in the shard. More...

Xapian::termcount	get_unique_terms_lower_bound () const
	An upper bound on the number of unique terms in any document in the shard. More...

Xapian::termcount	get_db_doclength_upper_bound () const
	An upper bound on the maximum length of any document in the database. More...

Xapian::termcount	get_db_doclength_lower_bound () const
	A lower bound on the minimum length of any document in the database. More...

Xapian::termcount	get_db_unique_terms_upper_bound () const
	A lower bound on the number of unique terms in any document in the database. More...

Xapian::termcount	get_db_unique_terms_lower_bound () const
	An upper bound on the number of unique terms in any document in the database. More...

Xapian::termcount	get_db_wdf_upper_bound () const
	An upper bound on the wdf of this term in the database. More...

Detailed Description

Language Model weighting with Dirichlet or Dir+ smoothing.

Dirichlet smoothing is as described in:

Zhai, C., & Lafferty, J.D. (2004). A study of smoothing methods for language models applied to information retrieval. ACM Trans. Inf. Syst., 22, 179-214.

Dir+ is described in:

Lv, Y., & Zhai, C. (2011). Lower-bounding term frequency normalization. International Conference on Information and Knowledge Management.

Since: 2.0.0

Definition at line 1948 of file weight.h.

Constructor & Destructor Documentation

◆ LMDirichletWeight()

Xapian::LMDirichletWeight::LMDirichletWeight	(	double	mu = `2000.0`,
		double	delta = `0.05`
	)

inlineexplicit

Construct a LMDirichletWeight.

Parameters

mu	A parameter which is > 0. Default: 2000
delta	A parameter which is >= 0, which is "a pseudo [wdf] value to control the scale of the [wdf] lower bound". If this parameter is > 0, then the smoothing is Dir+; if it's zero, it's Dirichlet. Default: 0.05

Definition at line 1981 of file weight.h.

Member Function Documentation

◆ clone()

LMDirichletWeight * Xapian::LMDirichletWeight::clone ( ) const

privatevirtual

Clone this object.

This method allocates and returns a copy of the object it is called on.

If your subclass is called FooWeight and has parameters a and b, then you would implement FooWeight::clone() like so:

FooWeight * FooWeight::clone() const { return new FooWeight(a, b); }

Note that the returned object will be deallocated by Xapian after use with "delete". If you want to handle the deletion in a special way (for example when wrapping the Xapian API for use from another language) then you can define a static operator delete method in your subclass as shown here: https://trac.xapian.org/ticket/554#comment:1

Implements Xapian::Weight.

Definition at line 244 of file lmweight.cc.

◆ create_from_parameters()

LMDirichletWeight * Xapian::LMDirichletWeight::create_from_parameters ( const char * params ) const

virtual

Create from a human-readable parameter string.

Parameters

params string containing weighting scheme parameter values.

Since: 2.0.0

Reimplemented from Xapian::Weight.

Definition at line 277 of file lmweight.cc.

References Xapian::Weight::Internal::double_param(), p, and parameter_error().

◆ get_maxextra()

double Xapian::LMDirichletWeight::get_maxextra ( ) const

virtual

Return an upper bound on what get_sumextra() can return for any document.

The default implementation always returns 0 (in Xapian < 2.0.0 this was a pure virtual method).

This information is used by the matcher to perform various optimisations, so strive to make the bound as tight as possible.

Reimplemented from Xapian::Weight.

Definition at line 237 of file lmweight.cc.

◆ get_maxpart()

double Xapian::LMDirichletWeight::get_maxpart ( ) const

virtual

Return an upper bound on what get_sumpart() can return for any document.

This information is used by the matcher to perform various optimisations, so strive to make the bound as tight as possible.

Implements Xapian::Weight.

Definition at line 201 of file lmweight.cc.

◆ get_sumextra()

double Xapian::LMDirichletWeight::get_sumextra	(	Xapian::termcount	doclen,
		Xapian::termcount	uniqterms,
		Xapian::termcount	wdfdocmax
	)		const

virtual

Calculate the term-independent weight component for a document.

The default implementation always returns 0 (in Xapian < 2.0.0 this was a pure virtual method).

The parameter gives information about the document which may be used in the calculations:

Parameters

doclen	The document's length (unnormalised). You need to call need_stat(DOC_LENGTH) if you use this value.
uniqterms	Number of unique terms in the document. You need to call need_stat(UNIQUE_TERMS) if you use this value.
wdfdocmax	Maximum wdf value in the document. You need to call need_stat(WDF_DOC_MAX) if you use this value.

Since: The wdfdocmax parameter was added in Xapian 2.0.0.

Reimplemented from Xapian::Weight.

Definition at line 208 of file lmweight.cc.

◆ get_sumpart()

double Xapian::LMDirichletWeight::get_sumpart	(	Xapian::termcount	wdf,
		Xapian::termcount	doclen,
		Xapian::termcount	uniqterms,
		Xapian::termcount	wdfdocmax
	)		const

virtual

Calculate the weight contribution for this object's term to a document.

The parameters give information about the document which may be used in the calculations:

Parameters

wdf	The within document frequency of the term in the document. You need to call need_stat(WDF) if you use this value.
doclen	The document's length (unnormalised). You need to call need_stat(DOC_LENGTH) if you use this value.
uniqterms	Number of unique terms in the document. You need to call need_stat(UNIQUE_TERMS) if you use this value.
wdfdocmax	Maximum wdf value in the document. You need to call need_stat(WDF_DOC_MAX) if you use this value.

You can rely of wdf <= doclen if you call both need_stat(WDF) and need_stat(DOC_LENGTH) - this is trivially true for terms, but Xapian also ensure it's true for OP_SYNONYM, where the wdf is approximated.

Since: The wdfdocmax parameter was added in Xapian 2.0.0.

Implements Xapian::Weight.

Definition at line 194 of file lmweight.cc.

◆ init()

void Xapian::LMDirichletWeight::init ( double factor )

privatevirtual

Allow the subclass to perform any initialisation it needs to.

Parameters

factor Any scaling factor (e.g. from OP_SCALE_WEIGHT). If the Weight object is for the term-independent weight supplied by get_sumextra()/get_maxextra(), then init(0.0) is called (starting from Xapian 1.2.11 and 1.3.1 - earlier versions failed to call init() for such Weight objects).

Implements Xapian::Weight.

Definition at line 161 of file lmweight.cc.

References rare.

◆ name()

string Xapian::LMDirichletWeight::name ( ) const

virtual

Return the name of this weighting scheme, e.g.

"bm25+".

This is the name that the weighting scheme gets registered under when passed to Xapian:Registry::register_weighting_scheme().

As a result:

this is the name that needs to be used in Weight::create() to create a Weight object from a human-readable string description.
it is also used by the remote backend where it is sent (along with the serialised parameters) to the remote server so that it knows which class to create.

For 1.4.x and earlier we recommended returning the full namespace-qualified name of your class here, but now we recommend returning a just the name in lower case, e.g. "foo" instead of "FooWeight", "bm25+" instead of "Xapian::BM25PlusWeight".

If you don't want to support creation via Weight::create() or the remote backend, you can use the default implementation which simply returns an empty string.

Reimplemented from Xapian::Weight.

Definition at line 249 of file lmweight.cc.

◆ serialise()

string Xapian::LMDirichletWeight::serialise ( ) const

virtual

Return this object's parameters serialised as a single string.

If you don't want to support the remote backend, you can use the default implementation which simply throws Xapian::UnimplementedError.

Reimplemented from Xapian::Weight.

Definition at line 255 of file lmweight.cc.

References serialise_double().

◆ unserialise()

LMDirichletWeight * Xapian::LMDirichletWeight::unserialise ( const std::string & serialised ) const

virtual

Unserialise parameters.

This method unserialises parameters serialised by the serialise() method and allocates and returns a new object initialised with them.

If you don't want to support the remote backend, you can use the default implementation which simply throws Xapian::UnimplementedError.

Note that the returned object will be deallocated by Xapian after use with "delete". If you want to handle the deletion in a special way (for example when wrapping the Xapian API for use from another language) then you can define a static operator delete method in your subclass as shown here: https://trac.xapian.org/ticket/554#comment:1

Parameters

serialised A string containing the serialised parameters.

Reimplemented from Xapian::Weight.

Definition at line 263 of file lmweight.cc.

References rare, and unserialise_double().

Member Data Documentation

◆ extra_offset

double Xapian::LMDirichletWeight::extra_offset

private

Precalculated offset to add to every sumextra.

This is needed because the formula can return a negative term-independent weight.

Definition at line 1966 of file weight.h.

◆ factor

double Xapian::LMDirichletWeight::factor

private

The factor to multiply weights by.

Definition at line 1950 of file weight.h.

◆ multiplier

double Xapian::LMDirichletWeight::multiplier

private

Precalculated multiplier for use in weight calculations.

Definition at line 1959 of file weight.h.

◆ param_delta

double Xapian::LMDirichletWeight::param_delta

private

A pseudo TF value to control the scale of the TF lower bound.

Definition at line 1956 of file weight.h.

◆ param_mu

double Xapian::LMDirichletWeight::param_mu

private

Parameter controlling the smoothing.

Definition at line 1953 of file weight.h.

The documentation for this class was generated from the following files:

include/xapian/weight.h
weight/lmweight.cc

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

◆ LMDirichletWeight()

Member Function Documentation

◆ clone()

◆ create_from_parameters()

◆ get_maxextra()

◆ get_maxpart()

◆ get_sumextra()

◆ get_sumpart()

◆ init()

◆ name()

◆ serialise()

◆ unserialise()

Member Data Documentation

◆ extra_offset

◆ factor

◆ multiplier

◆ param_delta

◆ param_mu