Lucene in Action

...IndexWriter is the central component of the indexing process. This class creates a new index and adds documents to an existing index. You can think of Index- Writer as an object that gives you write access to the index but doesn't let you read or search it. Despite its name, IndexWriter isn't the only class that's used to modify an index; section 2.2 describes how to use the Lucene API to modify an index.... [Full sample chapter]

... instances of IndexWriter or IndexReader isn't allowed, as shown in table 2.2, both of these classes ... close the IndexWriter instance that opened that same index before. The concurrency matrix in the table ... that a single instance of IndexWriter or a single instance of IndexReader is used. Note that we don't list Figure 2.7 A single IndexWriter or IndexReader can be shared by multiple threads. updating ... , as you saw in section 2.2.4 Table 2.3 Concurrency matrix when the same instance of IndexWriter...

...Let's discuss one final, fairly unknown Lucene feature (if we may so call it). If you ever need to debug Lucene's index-writing process, remember that you can get Lucene to output information about its indexing operations by setting Index- Writer's public instance variable infoStream to one of the OutputStreams, such as System.out: IndexWriter writer = new IndexWriter(dir, new SimpleAnalyzer(), true); writer.infoStream = System.out; ... This reveals information about segment merges, as shown...

...As you saw in our Indexer class, you need the following classes to perform the simplest indexing procedure: IndexWriter Directory Analyzer Document Field What follows is a brief overview of these classes, to give you a rough idea about their role in Lucene. We'll use these classes throughout this book. 3 Neal Stephenson details this process nicely in "In the Beginning Was the Command Line": http:// www.cryptonomicon.com/beginning.html.... [Full sample chapter]

... by IndexWriter when IndexWriter is instantiated and kept until it's closed. The same lock file is also ... and read all the referenced segments. IndexWriter also obtains the commit.lock right before ... IndexWriter Constructor close() Lock released when IndexWriter is closed write.lock IndexReader delete(int ... , String, byte) IndexReader is closed commit.lock IndexWriter Constructor Constructor Lock released as soon as segment information is read or written commit.lock IndexWriter addIndexes addIndexes Lock...

...IndexSearcher is to searching what IndexWriter is to indexing: the central link to the index that exposes several search methods. You can think of IndexSearcher as a class that opens an index in a read-only mode. It offers a number of search methods, some of which are implemented in its abstract parent class Searcher; the simplest takes a single Query object as a parameter and returns a Hits object. A typical use of this method looks like this: IndexSearcher is = new IndexSearcher( FSDirectory... [Full sample chapter]

...In this section, we cover three closely related topics: concurrent index access, thread-safety of IndexReader and IndexWriter, and the locking mechanism that Lucene uses to prevent index corruption. These issues are often misunderstood by users new to Lucene. Understanding these topics is important, because it will eliminate surprises that can result when your indexing application starts serving multiple users simultaneously or when it has to deal with a sudden need to scale by parallelizing...

... index-modifying operation may execute at a time. An index should be opened by a single IndexWriter or a single ... or updating documents in the same index using multiple instances Disallowed of IndexWriter Failing ... a new IndexWriter to add more documents to the same index Failing to close the IndexWriter...

...During indexing, an Analyzer instance is handed to the IndexWriter in this manner: Analyzer analyzer = new StandardAnalyzer(); IndexWriter writer = new IndexWriter(directory, analyzer, true ... library. Each tokenized field of each document indexed with the IndexWriter instance uses ... instance provided to the IndexWriter. However, if an individual document has special analysis needs ... ); During indexing, the granularity of analyzer choice is at the IndexWriter or per- Document level...

..., you may want to index only the first 200 words of each document. Lucene's IndexWriter exposes ... IOException { IndexWriter writer = new IndexWriter(dir, new SimpleAnalyzer(), true); writer.maxFieldLength ... how we can limit the number of Document terms we index: b f First we instruct IndexWriter to index ... ". d We reindex this Document, instructing IndexWriter to index only the first term. e Now...

... indexDirectory = new RAMDirectory(); IndexWriter indexWriter = new IndexWriter(indexDirectory,new ... ,false,false)); indexWriter.addDocument(doc); } indexWriter.optimize(); indexWriter.close(); return...

...The Directory class represents the location of a Lucene index. It's an abstract class that allows its subclasses (two of which are included in Lucene) to store the index as they see fit. In our Indexer example, we used a path to an actual file system directory to obtain an instance of Directory, which we passed to IndexWriter's constructor. IndexWriter then used one of the concrete Directory implementa- tions, FSDirectory, and created our index in a directory in the file system. In your... [Full sample chapter]

...(Directory dir) throws IOException { IndexWriter writer = new IndexWriter(dir, new SimpleAnalyzer(), true ... IndexWriter's mergeFactor, maxMergeDocs, and minMergeDocs proves insuf- ficient. You have the option ... (); IndexWriter fsWriter = IndexWriter(fsDir, new SimpleAnalyzer(), true); IndexWriter ramWriter = new IndexWriter(ramDir, new SimpleAnalyzer(), true); while (there are documents to index ... (); with on-disk FSDirectory ramWriter = new IndexWriter(ramDir, new SimpleAnalyzer(), true); Create new...

...Before text is indexed, it's passed through an Analyzer. The Analyzer, specified in the IndexWriter constructor, is in charge of extracting tokens out of text to be indexed and eliminating the rest. If the content to be indexed isn't plain text, it should first be converted to it, as depicted in figure 2.1. Chapter 7 shows how to extract text from the most common rich-media document formats. Analyzer is an abstract class, but Lucene comes with several implementations of it. Some of them deal... [Full sample chapter]

...Once you've prepared the data for indexing and created Lucene Documents pop- ulated with Fields, you can call IndexWriter's addDocument(Document) method and hand your data off to Lucene to index. When you do that, Lucene first ana- lyzes the data to make it more suitable for indexing. To do so, it splits the textual data into chunks, or tokens, and performs a number of optional operations on them. For instance, the tokens could be lowercased before indexing, to make searches case-insensitive...

...", "Amsterdam")); Verify Document removal IndexWriter writer = new IndexWriter(dir, getAnalyzer(), false ... you need to delete. 3 Close IndexReader. 4 Open IndexWriter. 5 Add all the Documents you need to add. 6 Close IndexWriter. This is important to remember: Batching Document deletion and indexing...

... or is not a directory"); } IndexWriter writer = new IndexWriter(indexDir, b Create new StandardAnalyzer ... (IndexWriter writer, File dir) throws IOException { File[] files = dir.listFiles(); for (int ... to actually index a file using Lucene private static void indexFile(IndexWriter writer, File ... of the IndexWriter (b) and four lines in the indexFile method (d, e, f) of Indexer involve the Lucene API--effect... [Full sample chapter]

... = new RAMDirectory(); IndexWriter aTOmWriter = new IndexWriter(aTOmDirectory, analyzer, true); IndexWriter nTOzWriter = new IndexWriter(nTOzDirectory, analyzer, true); for (int i=0; i < animals...

... an Index- Writer.optimize() call every time we added a document. When we relaxed that to run less...

...()); assertEquals(2, reader.numDocs()); reader.delete(1); reader.close(); IndexWriter writer = new IndexWriter(dir, getAnalyzer(), false); writer.optimize(); writer.close(); reader = IndexReader.open(dir ... performs Document deletion from IndexReader and not IndexWriter instances. That question is asked ... names. Lucene users often think that IndexWriter is the only class that can modify an index and that IndexReader accesses an index in a read-only fashion. In reality, IndexWriter touches only the list...