Lucene in Action

...Terms enclosed in double quotes create a PhraseQuery. The text between the quotes is analyzed; thus the resultant PhraseQuery may not be exactly the phrase originally specified. This process has been the subject of some confusion. For example, the query "This is Some Phrase*", when analyzed ... *\"", "field", new StandardAnalyzer()); assertEquals("analyzed", "\"some phrase\"", q.toString("field ... phrase\"", "field", analyzer); assertEquals("zero slop", "\"exact phrase\"", q.toString("field... [Full sample chapter]

... and the slop factor in section 3.5.6. Phrases are analyzed by the analyzer passed to the QueryParser ... contained the phrase "the quick brown fox jumped over the lazy dog". Without knowing the exact phrase ... words; but in this case we only want documents that have phrases where the words are either exactly ... 4.8.3. reconstruct the phrase in order. Let's take the phrase just mentioned and see how the slop ... a phrase query matched the test document: public class PhraseQueryTest extends TestCase { private... [Full sample chapter]

.... Two phrases are analyzed, each by four of the built-in analyzers. The phrases are "The quick brown ... Before we get into the gory details of what lurks inside an analyzer, let's look at how an analyzer ... . In the following two sections, we detail how an analyzer is used in these scenarios. Before we begin ... ! Listing 4.1 Visualizing analyzer effects Analyzing "The quick brown fox jumped over the lazy dogs ... ] Analyzing "XY&Z Corporation - " WhitespaceAnalyzer: [XY&Z] [Corporation...

... to demonstrate the Nutch analysis trickery. Listing 4.10 first analyzes the phrase "The quick brown..." using ... of the Nutch architecture. There is another interesting facet to Nutch: how it analyzes text. Nutch does ... common terms are used in a query, but not within a phrase, such as the quick brown with no other ... words as a single token) technique with a query-time optimization of phrases. This results in a far ... public class NutchExample { Custom analyzer public static void main(String[] args) throws IOException...

... during searching. PhraseQuery and QueryParser are the two troublemakers. Exact phrase matches now fail ... ()); } } As shown, an exact phrase query didn't match. This is disturbing, of course. Unlike the synonym analyzer situation, using a different analyzer won't solve the problem. The difficulty lies ... in a PhraseQuery must be side by side, and in our test case, the phrase it's searching for is "over lazi" (stop ... showing a phrase query working. Setting the slop to 1 allows the query to effectively ignore the gap...

.... AnalyzerDemo includes two predefined phrases and an array of the four analyzers we're focusing on in this section. Each phrase is analyzed by all the analyzers, with bracketed output to indicate ... into different types based on a grammar. Analyzing the phrase "I'll e-mail you at ... It's important to understand what various analyzers do with your text. Seeing the effect of an analyzer is a powerful and immediate aid to this understanding. List- ing 4.2 provides a quick and easy...

... analyzers Handling foreign languages 102 Analysis, in Lucene, is the process of converting field ... terms in sequence. An analyzer is an encapsula- tion of the analysis process. An analyzer tokenizes ... that text. In order for Lucene to know what "words" are, it analyzes the text during indexing, extracting ... analyzer is a crucial development decision with Lucene. One size doesn't fit all when it comes to choosing an analyzer. Language is one factor in choosing an analyzer, because each has its own unique...

... stream analyzing this phrase with the SimpleAnalyzer. Figure 4.1 Token stream with positional ... designated for tokenization are processed with the specified ana- lyzer, and each token is written ... what forms a Token; we'll come back to how that translates into a term. For example, let's analyze ... into terms After text is analyzed during indexing, each token is posted to the index as a term ... and successive position in the field. Position increments factor directly into perform- ing phrase queries...

... without the slop signature (slop makes sense only on multiterm phrase query); parsing a quoted phrase ... the analyzer.4 Given enough slop, PhraseQuery will match terms out of order in the original text ... a PhraseQuery with an ordered SpanNearQuery: protected Query getFieldQuery( String field, Analyzer analyzer, String queryText, int slop) throws ParseException { Query orig = super.getFieldQuery(field, analyzer, queryText); b Delegate to QueryParser's if (! (orig instanceof PhraseQuery...

... analyzers on the phrase "XY&Z Corporation - ". StandardAnalyzer is the only one ... StandardAnalyzer holds the honor as the most generally useful built-in analyzer. A JavaCC-based2 grammar underlies it, tokenizing with cleverness for the following lexical types: alphanumerics, acronyms, company names, e-mail addresses, com- puter host names, numbers, words with an interior ... choice. Using StandardAnalyzer is no different than using any of the other analyzers, as you can see...

... analyzes phrases, and each of these reduces to "one enough" and matches the terms indexed. There is a "hole ... Query- Parser along with StopAnalyzer, this document would match phrase queries for "one enough", "one ... , only the tokens emitted from the analyzer (or indexed as Field.Keyword) are available for searching....

... b QueryParser analyzes each term and phrase of the query expression. Both Q36 and SPACE are analyzed ... (), true); Field not Document doc = new Document(); analyzed doc.add(Field.Keyword("partnum ... ("partnum:Q36 AND SPACE", "description", new SimpleAnalyzer()); b QueryParser analyzes each Hits hits = searcher.search(query); term and phrase assertEquals("note Q36 -> q", "+partnum:q +space", query ... q. But at indexing time, Q36 was left as is. Notice, also, that this is the same analyzer used during indexing...

... to force ordered or phrase matches. getFieldQuery(String field, Analyzer analyzer, String queryText ... Although QueryParser has some quirks, such as the interactions with an analyzer, it does have ... ? getFieldQuery(String field, These methods are responsible for the construction of either a Analyzer analyzer, TermQuery or a PhraseQuery. If special analysis is needed, or a String queryText) unique type ... string handed to this method doesn't include the trailing asterisk and isn't analyzed. Override...

... is the primary purpose of the QueryParser. QueryParser requires an analyzer to break pieces ... be parsed into something at a lower level that the database server can understand directly. analyzer. Querying through the API using TermQuery and the others discussed in section 3.4 doesn't use an analyzer ... signature is static public Query parse(String query, String field, Analyzer analyzer) throws ... on this in section 3.5.4). The final argument is an Analyzer instance. (We discuss analyzers in detail... [Full sample chapter]

... we've discussed thus far, it isn't possible to formulate such a query. Phrase queries could get close with something like "quick fox" AND "lazy dog", but these phrases may be too distant from one another ... methods to make our later code clearer, as shown in listing 5.3. We index two similar phrases in a field ... cat; private Analyzer analyzer; protected void setUp() throws Exception { directory = new RAMDirectory(); analyzer = new WhitespaceAnalyzer(); IndexWriter writer = new IndexWriter(directory, analyzer...

... to view what an analyzer is doing with text. Using our Analyzer- Utils, two phrases that sound similar ... , we'll implement an analyzer to convert words to a phonetic root using an implementation ... () throws Exception { RAMDirectory directory = new RAMDirectory(); Analyzer analyzer = new MetaphoneReplacementAnalyzer(); IndexWriter writer = new IndexWriter(directory, analyzer, true); Document doc ... = QueryParser.parse("kool kat", User typed in "contents", hip query analyzer); Hits hits = searcher.search...

....length()); Analyzer can't find document using phrase query = QueryParser.parse("\"fox jumps\"", from ... mean the same Our next custom analyzer injects synonyms of words into the outgoing token stream ... [] tokens = AnalyzerUtils.tokensFromAnalysis(synonymAnalyzer4, "jumps"); Analyze one word AnalyzerUtils ... Analyzer { private SynonymEngine engine; public SynonymAnalyzer(SynonymEngine engine) { this.engine ... (reader))), StandardAnalyzer.STOP_WORDS), engine ); return result; } } Once again, the analyzer code...

... Analyzer analyzer; private RAMDirectory directory; protected void setUp() throws Exception { super.setUp(); analyzer = new WhitespaceAnalyzer(); directory = new RAMDirectory(); IndexWriter writer = new IndexWriter(directory, analyzer, true); for (int i = 1; i <= 500; i++) { Document doc = new ... 346 phrased as id:[37 TO 346] won't work as expected with the 3 Lucene stores term information ... the getRangeQuery() method: protected Query getRangeQuery(String field, Analyzer analyzer, String part...

... that matching documents must contain the word.8 For example, given a search phrase of "Mother and child ... , had the 8 Authors' note: There are enough odd interactions between analyzers and QueryParser...

... and zero-slop default phrase queries. NOTE Generally speaking, querying on multiple fields isn't the best ... + " " + subjects)); We used a space (" ") between author and subjects to separate words for the analyzer ... , be sure your queries are fabricated appropriately using the QueryParser and Analyzer diagnostic...