nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

As we transition to the new Hyland Community, some post links may be temporarily disrupted as we finalize redirects. We apologize for the inconvenience and are working to resolve this as quickly as possible. Details.

Lucene search with accented characters

pchoe

I have a situation where I have two folders with almost the same name, except for the accent mark.

Hernández-Monrreal, Juan Gabriel
Hernandez-Monrreal, Juan Gabriel

I am trying to search for folders using the following query:
+@cm\:name:"cm:Hernández-Monrreal_x002c__x0020_Juan_x0020_Gabriel" +TYPE:"cm:folder" +PATH:"/app:company_home/st:sites/cm:CLCHURCHLIFE/cm:documentLibrary/cm:G_x0020_-_x0020_H/cm:H/*"

I would expect to just get back the folder with exact match including the accent mark, but I get back both the folders, with and without the accent mark. It seems lucene will just ignore the accent mark and return matches based on the letters.

Is there a way to make lucene do a search with the accent marks?

Find more posts tagged with

Comments

andy

Hi

The default analysis is to strip accents.

The analysers are configurable.
You would need to change the analyser for d:text and then reindex.

Change the setting in alfresco/model/dataTypeAnalyzers.properties to:
d_dictionary.datatype.d_text.analyzer=your.analyzer.class

or, copy alfresco/model/dataTypeAnalyzers.properties and related files to a new location and changing the definition of the bean that loads this file – “dictionaryBootstrap” – currently defined in core-services-context.xml.

Andy

pchoe

I put the following custom code for the analyzer:

import java.io.Reader;
import java.util.Set;

import org.alfresco.repo.search.impl.lucene.analysis.AlfrescoStandardFilter;
import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.ISOLatin1AccentFilter;
import org.apache.lucene.analysis.LowerCaseFilter;
import org.apache.lucene.analysis.StopAnalyzer;
import org.apache.lucene.analysis.StopFilter;
import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.standard.StandardFilter;
import org.apache.lucene.analysis.standard.StandardTokenizer;

/**
 * Custom Lucene analyzer that doesn't implement ISOLatin1AccentFilter.
 * 
 * @author pchoe
 *
 */
public class MSICustomStrictAnalyzer extends Analyzer {
    private Set stopSet;
    public static final String STOP_WORDS[];

    static 
    {
        STOP_WORDS = StopAnalyzer.ENGLISH_STOP_WORDS;
    }


    public MSICustomStrictAnalyzer()
    {
        this(STOP_WORDS);
    }

    public MSICustomStrictAnalyzer(String stopWords[])
    {
        stopSet = StopFilter.makeStopSet(stopWords);
    }

    /**
     * 
     * @see org.apache.lucene.analysis.Analyzer#tokenStream(java.lang.String, java.io.Reader)
     */
    public TokenStream tokenStream(String fieldName, Reader reader)
    {
        TokenStream result = new StandardTokenizer(reader);
        result = new StandardFilter(result);
        result = new LowerCaseFilter(result);
        result = new StopFilter(result, stopSet);
        //result = new ISOLatin1AccentFilter(result);
        return result;
    }

}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

and modified the dataTypeAnalyzers.properties to

# Data Type Index Analyzers

d_dictionary.datatype.d_any.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
#d_dictionary.datatype.d_text.analyzer=org.alfresco.repo.search.impl.lucene.analysis.AlfrescoStandardAnalyser
#d_dictionary.datatype.d_content.analyzer=org.alfresco.repo.search.impl.lucene.analysis.AlfrescoStandardAnalyser
d_dictionary.datatype.d_int.analyzer=org.alfresco.repo.search.impl.lucene.analysis.IntegerAnalyser
d_dictionary.datatype.d_long.analyzer=org.alfresco.repo.search.impl.lucene.analysis.LongAnalyser
d_dictionary.datatype.d_float.analyzer=org.alfresco.repo.search.impl.lucene.analysis.FloatAnalyser
d_dictionary.datatype.d_double.analyzer=org.alfresco.repo.search.impl.lucene.analysis.DoubleAnalyser
d_dictionary.datatype.d_date.analyzer=org.alfresco.repo.search.impl.lucene.analysis.DateAnalyser
d_dictionary.datatype.d_datetime.analyzer=org.alfresco.repo.search.impl.lucene.analysis.DateTimeAnalyser
d_dictionary.datatype.d_boolean.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
d_dictionary.datatype.d_qname.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
d_dictionary.datatype.d_guid.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
d_dictionary.datatype.d_category.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
d_dictionary.datatype.d_noderef.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
d_dictionary.datatype.d_path.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
d_dictionary.datatype.d_locale.analyzer=org.alfresco.repo.search.impl.lucene.analysis.LowerCaseVerbatimAnalyser
d_dictionary.datatype.d_text.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
d_dictionary.datatype.d_content.analyzer=com.microstrat.alfresco.lucene.MSICustomStrictAnalyzer
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

so that the custom analyzer would work. When I run it in the debug mode from eclipse, I see that it will go into the custom analyzer. But when I do a lucene search even after reindexing, I still get the result as from the default analyzer.

Am I missing a configuration somewhere?

andy

Hi

Where have you made the changes?
Sounds like your changes are lost post deployment (probvably over-written in the tomcat expanded view??)

It is best to add an extension to wire up the changes to avoid this.

Andy