| content
| - %META:TOPICPARENT{name="VirtTipsAndTricksGuide"}%
---+ Normalization of UNICODE3 accented characters for Virtuoso free-text indexing
Normalization of UNICODE3 accented characters in a free-text index can be controlled by setting
the <b><code><nowiki>XAnyNormalization</nowiki></code></b> configuration parameter in the
<b><code>[I18N]</code></b> section of the Virtuoso configuration file, <code>virtuoso.ini</code>.
This parameter controls whether accented UNICODE characters should be converted to their non-accented
base variants when creating a free-text index or when parsing a free-text query string. The parameter's
value is a bitmask integer, currently with only 2 bits in use:
| *XAnyNormalization value* | *bit equivalent* | *Description* |
| <code>0</code> | <code>00</code> | Default. Nothing is normalized, so "Jose" and "Jos?" are two distinct words. |
| <code>1</code> | <code>01</code> | <i>ToBeDone</i> |
| <code>2</code> | <code>10</code> | Any "combining character sequence" (a combination of a base character and one or more combining characters) is converted to its (smallest known) base. For example, "?" will lose its accent, and become a plain ASCII "e". |
| <code>3</code> | <code>11</code> | This combines <code>1</code> and <code>2</code>, and so causes both conversions. Any pair of base character and combining character loses the second character, and characters with accents lose their accents. |
So the fragment of <code>virtuoso.ini</code> would look like:
<verbatim>
...
[I18N]
XAnyNormalization = 3
...
</verbatim>
* <code><nowiki>XAnyNormalization = 3</nowiki></code> is recommended for most scenarios requiring
such normalization. In some rare cases, <code><nowiki>XAnyNormalization = 1</nowiki></code> may be
more appropriate.
* The parameter should generally be set before creating a database, and must be set identically
for all instances in a cluster configuration. If changed on an existing database, you should rebuild
all free-text indexes that may contain non-ASCII data by running the following procedure from isql --
<verbatim>
VT_INDEX_DB_DBA_RDF_OBJ(0)
</verbatim>
* On a typical system, the parameter affects all text columns, XML columns, RDF literals, and
queries. (Strictly speaking, it only affects items that use default "<code>x-any</code>" language,
or a language derived from <code>x-any</code> such as "<code>en</code>" or "<code>en-US</code>". If
you haven't tried writing new C plug-ins for custom languages, you need not look so deep.)
* <i><b>Note:</b> We have had requests for a database function that normalizes characters in
strings, as the free-text engine does with <code><nowiki>XAnyNormalization=3</nowiki></code>. This function will be
provided as a separate patch/update, and will depend on <code><nowiki>XAnyNormalization</nowiki></code>.</i>
---++ Example
With <b><code><nowiki>XAnyNormalization=3</nowiki></code></b>, one can get the following:
<verbatim>
SQL> SPARQL
INSERT
IN <http://InternationalNSMs/>
{
<s> <sp> "?ndio Jo?o Macap? J?nior T?rres Lu?s Ara?jo Jos?" ;
<ru> "?? ??????? ????????, ??????? ? ???????? ???????? ?? ?????"
}
;
INSERT INTO <http://InternationalNSMs/>, 2 (or less) triples -- done
SQL> DB.DBA.RDF_OBJ_FT_RULE_ADD (NULL, NULL, 'InternationalNSMs.wb');
Done. -- 0 msec.
SQL> VT_INDEX_DB_DBA_RDF_OBJ(0);
Done. -- 26 msec.
SQL> SPARQL
SELECT *
FROM <http://InternationalNSMs/>
WHERE
{
?s ?p ?o
}
ORDER BY ASC (str(?o))
;
s sp ?ndio Jo?o Macap? J?nior T?rres Lu?s Ara?jo Jos?
s ru ?? ??????? ????????, ??????? ? ???????? ???????? ?? ?????
2 Rows. -- 2 msec.
SQL> SPARQL
SELECT *
FROM <http://InternationalNSMs/>
WHERE
{
?s ?p ?o .
?o bif:contains "'?ndio Jo?o Macap? J?nior T?rres Lu?s Ara?jo Jos?'"
}
;
s sp ?ndio Jo?o Macap? J?nior T?rres Lu?s Ara?jo Jos?
1 Rows. -- 2 msec.
SQL> SPARQL
SELECT *
FROM <http://InternationalNSMs/>
WHERE
{
?s ?p ?o .
?o bif:contains "'Indio Joao Macapa Junior Torres Luis Araujo Jose'"
}
;
s sp ?ndio Jo?o Macap? J?nior T?rres Lu?s Ara?jo Jos?
1 Rows. -- 1 msec.
SQL> SPARQL
SELECT *
FROM <http://InternationalNSMs/>
WHERE
{
?s ?p ?o .
?o bif:contains "'???????? ???????? ?? ?????'"
}
;
s ru ?? ??????? ????????, ??????? ? ???????? ???????? ?? ?????
</verbatim>
---++ Related
* [[http://docs.openlinksw.com/virtuoso/databaseadmsrv.html#ini_I18N][Virtuoso ini I18N section]]
|
![[RDF Data]](/fct/images/sw-rdf-blue.png)
![[cxml]](/fct/images/cxml_doc.png)
![[csv]](/fct/images/csv_doc.png)
![[text]](/fct/images/ntriples_doc.png)
![[turtle]](/fct/images/n3turtle_doc.png)
![[ld+json]](/fct/images/jsonld_doc.png)
![[rdf+json]](/fct/images/json_doc.png)
![[rdf+xml]](/fct/images/xml_doc.png)
![[atom+xml]](/fct/images/atom_doc.png)
![[html]](/fct/images/html_doc.png)