Major Concepts

This topic covers the following subjects:

Introduction to Oracle interMedia Text

The text management component (in previous releases called the Oracle ConText® Cartridge) of the Oracle interMedia product provides a text-search system that allows text to be searched and managed within the Oracle database as quickly and easily as any other data. Oracle interMedia Text provides sophisticated text search techniques, making text a standard datatype in the Oracle server. Using Oracle8i and Oracle interMedia Text, new text-based developments or extensions to existing applications are easy and cost-effective to build using standard SQL tools.

Application developers can take advantage of Oracle interMedia Text searches in any Oracle database application that uses text, from search-enabling a comments field in an existing application to implementing large-scale document management systems dealing with multiple document formats and complex search criteria. Oracle interMedia Text supports basic full-text search in most languages supported by the Oracle database.

Configuring Indexes

Documents are stored in many different schemes, formats, and languages. Consequently, each Oracle interMedia Text index has many options that need to be set to configure it for a particular situation.

When creating an index, Oracle interMedia Text can use several default values, but most situations require users to configure the indexes by specifying preferences.

The many options for each index are organized into functional groups, called "classes", which each concentrate on a particular aspect of the configuration. You can think of them as asking questions about the document database:

Each class has a number of pre-defined behaviors called objects. Each object is a possible answer to the class question. For example, the Data Store class contains the following objects:

You can customize preferences to suit your needs. For example, assume that your documents are stored in the directory /doc (ctx_ddl.set_attribute('MYPREF','PATH','/doc'). The preference MYPREF specifies that documents are stored as external files in the directory /doc. To use this preference in index creation, identify it as the default data store for the desired schema.

A complete list of objects and attributes is contained in the product documentation.

Two groups of options are not configured as preferences:

Default Preferences

When creating an index, Oracle interMedia Text looks for a preference for each class (data store, filter, section group, and so on). If no preference is specified by the user, Oracle interMedia Text consults system parameters. Each system parameter contains the name of the preference to be used as default.

There are system parameters for each class:

At database creation time Oracle interMedia Text sets these system parameters to appropriate default preferences. Users, connecting as CTXSYS, can reset these system parameters.

How interMedia Text Performs Indexing

The following are the main logic steps that interMedia Text uses when it indexes a document:

  1. The data store logic loops over the rows of the table and reads the data out of the column. Usually, this is simply the column data, but some data stores use the column data as a pointer to the document data. For example, URL_DATASTORE uses the column data as a URL.

  2. The filter takes the document data and transforms it to a text representation. This is needed when storing binary documents such as Word or Acrobat files. The output of the filter does not have to be plain text -- it can be a text format such as XML or HTML.

  3. The sectioner takes the output from the filter, and converts it to plain text. There are different sectioners for different text formats, including XML and HTML. Conversion to plain text includes detecting important section tags, removing "invisible" information, and reformatting the text.

  4. The lexer takes the plain text from the sectioner and splits it into discrete tokens. There lexers for whitespace-delimited languages, as well as specialized lexers for Asian languages, where segmentation is complex.

  5. The engine takes all the tokens from the lexer, the section offsets from the sectioner, and a list of low-information words called stop words, and builds an inverted index. The inverted index stores tokens and the documents in which these tokens occur.

See also: Main topics (outline)