Handle text without explicit word breaks. More...
#include <config.h>#include "word-breaker.h"#include "omassert.h"#include "xapian/unicode.h"#include <cstdlib>#include <string>
Include dependency graph for word-breaker.cc:
Go to the source code of this file.
Functions | |
| bool | is_ngram_enabled () |
| Should we use the n-gram code? More... | |
| bool | is_unbroken_script (unsigned p) |
| void | get_unbroken (Xapian::Utf8Iterator &it) |
Detailed Description
Handle text without explicit word breaks.
Definition in file word-breaker.cc.
Function Documentation
◆ get_unbroken()
| void get_unbroken | ( | Xapian::Utf8Iterator & | it | ) |
Definition at line 86 of file word-breaker.cc.
References is_unbroken_script(), and Xapian::Unicode::is_wordchar().
Referenced by Xapian::QueryParser::Internal::parse_term().
◆ is_ngram_enabled()
| bool is_ngram_enabled | ( | ) |
Should we use the n-gram code?
The first time this is called it reads the environment variable XAPIAN_CJK_NGRAM and returns true if it is set to a non-empty value. Subsequent calls cache and return the same value.
Definition at line 41 of file word-breaker.cc.
Referenced by Xapian::TermGenerator::Internal::index_text(), and Xapian::QueryParser::Internal::parse_query().
◆ is_unbroken_script()
| bool is_unbroken_script | ( | unsigned | p | ) |
Definition at line 71 of file word-breaker.cc.
Referenced by get_unbroken(), NgramIterator::init(), NgramIterator::operator++(), Xapian::QueryParser::Internal::parse_term(), and Xapian::parse_terms().