MULTI_LEXER Usage Information

A MULTI_LEXER lexer allows documents of different languages to be stored in a single table. This type of lexer chooses a language-specific lexer for each row, based on a language column. This topic explains how to use this type of lexer, with examples from the interMedia Text API.

The first step in using the MULTI_LEXER feature is the creation of language-specific lexer preferences for each language in the multi-lingual table. The following API example creates English, French, and German lexers:

ctx_ddl.create_preference('english_lexer', 'basic_lexer');

ctx_ddl.set_attribute('english_lexer','index_themes','yes');

ctx_ddl.create_preference('french_lexer', 'basic_lexer');

ctx_ddl.set_attribute('french_lexer','base_letter','yes');

ctx_ddl.create_preference('german_lexer', 'basic_lexer');

ctx_ddl.set_attribute('german_lexer','composite','german');

ctx_ddl.set_attribute('german_lexer','alternate_spelling','german');

ctx_ddl.set_attribute('german_lexer','mixed_case','yes');

Once the language-specific lexer preferences are created, they need to be gathered together under a single MULTI lexer preference. First, create the MULTI lexer preference, using the MULTI_LEXER object:

ctx_ddl.create_preference('global_lexer','multi_lexer');

Add the language-specific lexers to the multi lexer preference using the add_sub_lexer call:

ctx_ddl.add_sub_lexer(<multi_lexer>, <language>, <lang-specific lexer>)

For example:

ctx_ddl.add_sub_lexer('global_lexer', 'german', 'german_lexer');

ctx_ddl.add_sub_lexer('global_lexer', 'french', 'french_lexer');

ctx_ddl.add_sub_lexer('global_lexer', 'default','english_lexer');

This designates the german_lexer preference to handle German documents, the french_lexer preference to handle French documents, and the english_lexer preference to handle everything else, using 'DEFAULT' as the language.

The MULTI lexer decides which lexer to use for each row based on a language column. This is a character column in the table which stores the language of the document in the text column. For example, the table may be defined as follows:

create table globaldoc (

doc_id number primary key,

lang varchar2(10),

text clob

);

insert into globaldoc values (1, 'german', 'Ich bin Berliner');

insert into globaldoc values (2, 'french', 'Je suis Parisian');

insert into globaldoc values (3, 'english', 'I''m from New York');

To index this table, you must use the MULTI lexer preference, but you must also specify the name of the language column:

create index globalx on globaldoc(text)

indextype is ctxsys.context

parameters ('lexer global_lexer

language column lang');

The MULTI_LEXER lexer then reads the value of the language column for each row, and uses the associated language-specific lexer for that language.

The language column is case-independent, and can contain either the NLS name or abbreviation of the language. If the table uses some other value for the language, you need to specify that alternate value as the fourth argument when adding the sub lexers. For example, to use the ISO 639-2 string for German instead:

ctx_ddl.add_sub_lexer('global_lexer', 'german', 'german_lexer', 'ger');

Note that these alternate values are in addition to the NLS names and abbreviations, so "D" and "GERMAN" in the language column still select the German lexer. This means that alternate values cannot be NLS language names or abbreviations.

The DEFAULT lexer is used when the language of the document is unknown or is not mapped to a language-specific lexer. The DEFAULT lexer is also used to process stopwords. A DEFAULT lexer MUST be specified, or you will get an error when using the MULTI lexer preference in create index. You cannot specify an alternate value for the DEFAULT sub lexer.

At query time, the MULTI lexer chooses a language-specific lexer to process the query tokens. This is based on the NLS_LANG setting for the query session. Thus, a query session in the FRENCH language will use the lexer for FRENCH.

See also: Main topics (outline), Getting started with interMedia Text