Lucene in Action

... during search. Lucene offers four different types of fields from which you can choose: Keyword--Isn't analyzed, but is indexed and stored in the index verbatim. This type is suitable for fields whose ... value is stored in the index as is. This type is suitable for fields that you need to display ... . Since the original value of a field of this type is stored in the index, this type isn't suitable ... of UnIndexed. This field type is analyzed and indexed but isn't stored in the index. It's suitable... [Full sample chapter]

...TermQuery is the most basic type of query supported by Lucene, and it's one of the primitive query types. It's used for matching documents that contain fields with specific values, as you've seen in the last few paragraphs.... [Full sample chapter]

... to be able to sort results by a Field value, you must add it as a Field that is indexed but not tokenized (for example, Field.Keyword). Fields used for sorting must be convertible to Integers, Floats, or Strings: Field.Keyword("size", "4096"); Field.Keyword("price", "10.99"); Field.Keyword("author", "Arthur C. Clark"); Although we've indexed numeric values as Strings, you can specify the correct Field type (such as Integer or Long) at sort time, as described in section 5.1.7. NOTE Fields used...

...) When an entry is returned as part of a search, the title, link, date, type, and description fields ... All jGuru Lucene databases have the same form for consistency, although some fields are unused depending on the indexed entity type. For example, the for- eign search database stores a site ID, but it is unused in the regular jGuru Lucene database. Some fields are used for display, and some are used for searching. The complete list of fields is shown in table 10.1. Table 10.1 jGuru Lucene index...

...A Document represents a collection of fields. You can think of it as a virtual docu- ment--a chunk ... at a later time. Fields of a document represent the document or meta-data associated ... modified, and so on, are indexed and stored separately as fields of a document. NOTE When we refer to a document in this book, we mean a Microsoft Word, RTF, PDF, or other type of a document; we aren ... . Although various types of documents can be indexed and made searchable, processing them... [Full sample chapter]

...Sorting comes at the expense of resources. More memory is needed to keep the fields used for sorting available. For numeric types, each field being sorted for each document in the index requires that four bytes be cached. For String types, each unique term is also cached for each document. Only the actual fields used for sorting are cached in this manner. Plan your system resources accordingly ... type in terms of resources....

...A Term is the basic unit for searching. Similar to the Field object, it consists of a pair of string elements: the name of the field and the value of that field. Note that Term objects are also involved in the indexing process. However, they're cre- ated by Lucene's internals, so you typically don ... in a field named contents. Because the TermQuery object is derived from the abstract parent class Query, you can use the Query type on the left side of the statement.... [Full sample chapter]

... of text representing each field. Chapter 7 covers several specific document types and provides ... An important point about analyzers is that they're used internally for fields flagged ... , this meta-data should be separated and indexed as separate fields. Analyzers are used to analyze a spe- cific field at a time and break things into tokens only within that field; creating new fields isn't possible within an analyzer. Analyzers don't help in field separation because their scope is to deal...

... library. Each tokenized field of each document indexed with the IndexWriter instance uses the analyzer specified. Two special Field types are designated to be tokenized: Text and UnStored. NOTE Field.Text(String, String) creates a tokenized and stored field. Rest assured the original String value ... demonstrates indexing of a document with these two field types: Document doc = new Document(); doc.add(Field.Text("title", "This is the title")); doc.add(Field.UnStored("contents", "...document...

... shortcuts to creating the SortField array. A SortField holds the field name, a field type, and the reverse order flag. SortField contains constants for several field types, including SCORE, DOC, AUTO ... ID. AUTO is the type used by each of our other examples, which sort by a field name. The type of field ... . If you're using strings that may appear as numeric in some fields, be sure to specify the type ... Implicitly we've been sorting by multiple fields, since the Sort object appends a sort by document...

...When you're sorting on a SortField.STRING type, order is determined under the covers using String.compareTo by default. However, if you need a different colla- tion order, SortField lets you specify ... con- structors for use when you need to specify locale: public SortField (String field, Locale locale) public SortField (String field, Locale locale, boolean reverse) Both of these constructors imply the SortField.STRING type because locale applies only to string-type sorting, not to numerics....

... text, and we then index two simple Documents, each containing all four types of Fields: Keyword ... ; i++) { Document doc = new Document(); doc.add(Field.Keyword("id", keywords[i])); doc.add(Field.UnIndexed("country", unindexed[i])); doc.add(Field.UnStored("contents", unstored[i])); doc.add(Field.Text("city ... override the get- Analyzer() method to return a different type of Analyzer. Heterogeneous Documents One handy feature of Lucene is that it allows Documents with different sets of Fields to coexist...

..., which allowed us to slurp their content and use it to populate Field instances. However, things aren ... documents and use that extracted data to create Lucene Documents and their Fields. If you look back at table 1.2, page 21, you'll see that Field methods always take String values and, in some cases, Date and Reader values. No methods would accept a PDF Java type, even if such a type existed. You face...

...By search time, the fields that can be sorted on and their corresponding types are already set. Indexing time is when the decision about sorting capabilities should be made; however, custom sorting ... . Sorting by a numeric type consumes fewer memory resources than by STRING; section 5.1.9 discusses ... to facili- tate sorting on those fields, not to constrain a search on a range of values. The numeric range ... indexing and searching in order to use numeric fields for searching. All terms in an index are Strings...

.... Now we will discuss the index building process and the design choices that we made. What fields should make up our index? We wanted to create a fairly generic set of fields that our index would contain. We ended up with the fields shown in table 10.7. Table 10.7 TheServerSide index field structure Field Lucene Type Description title Field.Text A short title of the content. summary Field.Text ... of the article?). category Field.Keyword The type of this content (is it a news item? an article?). path...

...Once you've prepared the data for indexing and created Lucene Documents pop- ulated with Fields, you can call IndexWriter's addDocument(Document) method and hand your data off to Lucene to index. When you do that, Lucene first ana- lyzes the data to make it more suitable for indexing. To do so, it splits the textual data into chunks, or tokens, and performs a number of optional operations on them ... in chapter 4. For now, think of this step as a type of a filter....

... equipped with a Field.Keyword(String, Date) method, as well as a DateField class, which make date ... .add(Field.Keyword("indexDate", new Date())); Internally, Lucene uses the DateField class to convert ... problems for certain types of queries. In practice, you rarely need dates that are precise down ... Field values are eventually turned into text, you may very well index dates as Strings. For instance, if you can round the date to a day, index dates as YYYYMMDD Strings using the Field.Keyword...

... require a form field argument that specifies which field to populate or validate. To escape a query, call doEscapeQuery with the form field or a text string (it detects the type); the escaped query ... 8.10) the JavaScript files and referring to them in the section in this manner: