nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

As we transition to the new Hyland Community, some post links may be temporarily disrupted as we finalize redirects. We apologize for the inconvenience and are working to resolve this as quickly as possible. Details.

URGENT:Sorting Lucene result on tokenized property possible?

serverok

Hi all,
we have the problem with sorting of lucene search result by some properties which are indexed and tokenised.

What is the best/fastest method to correctly sort lucene results by properties that contain multiple tokens (http://lucene.apache.org/java/2_3_2/api/org/apache/lucene/search/Sort.html)? :

<index enabled="true">
   <atomic>true</atomic> 
   <stored>false</stored> 
   <tokenised>true</tokenised> 
</index>‍‍‍‍‍

Is the only choice to sort "manually" in the web script after running the lucene query without sorting?

I tried use JS sort function:

var result = search.luceneSearch(query);
result.sort(Sorting);
….
function Sorting(a, b) {
   var x = a.properties['cm:title'];
   var y = b.properties['cm:title'];
   return ((x > y) ? 1 : -1);
}‍‍‍‍‍‍‍‍

but if we have a big list of content in lucene result (100, 200 and more) - then this section

var x = a.properties['cm:title'];
var y = b.properties['cm:title'];‍‍

very long working.

Also we have around 50 properties to each content (custom Model) - maybe this is the reason why we have so long going on reading the value of properties?

I would be very grateful for help in solving this problem.

Thank's,
Oleg Koval

Find more posts tagged with

Comments

serverok

Additional information:

this is the simple code of our webscript:

// this query produces 458 rows on our fresh alfresco test installation with dummy content:
var query = "TYPE:\"{http://www.alfresco.org/model/content/1.0}content\"";

if( query != null && query.length > 0 ) {
   var result = search.luceneSearch(query); // fast

   // now to sort manually on tokenized values I'll need to look at each attribute:
   // I took out the sorting for simplicity: just saving to a temp variable shows slowness!
   var temp;
   for(i=0;i<result.length;i++) {
      temp = result[i].properties['cm:title'];  // very slow!
   }
}‍‍‍‍‍‍‍‍‍‍‍‍‍

you can see that we have one piece where we got bottleneck:

temp = result[i].properties['cm:title'];  // very slow!‍

I tested sorting and got this results:
* I have fresh alfresco installation with default content model
* for timing I used "YSlow" addon for FireFox
** I have fresh alfresco installation ( added 1 category and 1 subcategory, created 400 )
** the timing on your server may be slightly different taking into account the difference in hardware configurations
*** the time interval include: the time of working of webscript, reading (by cURL) the results by PHP script (a JSON data) and generation of webpage with the data
10 nodes - ~2-3s
100 nodes - ~ 9-10s
200 nodes - ~19s
270 nodes - ~ 25s

90% of this time is simply examining one property of each node!

Also I have three question to all:
1) do you use the sorting for search result?
2) how you sort the search result's by tokenised fields?
3) if you use the standart JS "sort" function - what is timing for sorting around 200-300 results?

Thanks,
Oleg Koval

dwilson

I also see the same problem.

Interestingly it seems some attributes of the ScriptNode object are retrieved very quickly, but some are retrieved very slowly. Why is this, and is there no way to programmatically look at individual properties on a node array without waiting almost 100 milliseconds for EACH property examined? Looking at the results below, to view a node property it takes a full two orders of magnitude longer than viewing say, the node type.

(For testing I've used a sample URL executable script placed in Data Dictionary/Scripts

// In my test case, this query returns 394 nodes.
var query = "TYPE:\"{http://www.alfresco.org/model/content/1.0}content\"";
var total = "";
var out = "";

if( query != null && query.length > 0 ) {
&nbsp;&nbsp;&nbsp;var result = search.luceneSearch(query); // fast
&nbsp;&nbsp;&nbsp;total = result.length;
&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;for (i=0;i<result.length;i++) {

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// let's take a look at how long the script takes on various ScriptNode object attributes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// I uncommented each of these one by one and measured how long it took to print out that
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;// individual ScriptNode attribute:

&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].id +"<br>";&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // ~&nbsp; 0.3 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].nodeRef +"<br>";&nbsp;&nbsp;&nbsp;&nbsp; // ~&nbsp; 0.3 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].displayPath +"<br>"; // ~&nbsp; 0.6 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].qnamePath +"<br>";&nbsp;&nbsp; // ~&nbsp; 0.4 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].isLocked() +"<br>";&nbsp; // ~&nbsp; 0.3 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].type +"<br>";&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // ~&nbsp; 0.4 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].parent.id +"<br>";&nbsp;&nbsp; // ~&nbsp; 0.3 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].isCategory +"<br>";&nbsp; // ~&nbsp; 0.2 seconds on 394 nodes.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].aspects +"<br>";&nbsp;&nbsp;&nbsp;&nbsp; // ~&nbsp; 0.4 seconds on 394 nodes. !!! fast.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].size +"<br>";&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // ~ 27.4 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].url +"<br>";&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // ~ 28.1 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].downloadUrl +"<br>"; // ~ 28.2 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;//out += result[i].name +"<br>";&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // ~ 27.8 seconds on 394 nodes.
&nbsp;&nbsp;&nbsp;}
}

out += "<br><br>";
out += "<b>query:</b> "+query+"<br>";
out += "<b>total:</b> "+total+"<br>";
out;‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It seems like the properties array just isn't loaded up into lucene result list of nodes.

kevinr

The Alfresco Lucene API can perform some sorting for you, and we provide access to that through the scripting API:

Array luceneSearch(string query, string sortColumn, boolean asc)
Returns an array of ScriptNode satisfying the search criteria sorted by the specified sortColumn (the property name to sort on) and asc (true => ascending order, false => descending order). For example var nodes = search.luceneSearch("TEXT:alfresco", "@cm:modified", false);

If you need to sort by more than one column then the Script API does not yet provide this. But i can easily add it for Alfresco 3.2 since it has now been requested

The reason some properties take longer than others to retrieve is that some properties are easily resolvable from data already cached on the ScriptNode instance - and some data must be retrieved directly from the repository (and then cached). Accessing individual properties of 1000's of nodes via the ScriptNode API is not going to be as fast as writing some Java code to do it - as the ScriptNode API calls must always pass through all levels of Permissions and Public Service Interceptors etc. for each call (Java code does not always need to do this…)

Thanks,

Kevin

dwilson

Thanks for the reply, Kevin!

The Alfresco Lucene API can perform some sorting for you, and we provide access to that through the scripting API:
Array luceneSearch(string query, string sortColumn, boolean asc)
Returns an array of ScriptNode satisfying the search criteria sorted by the specified sortColumn (the property name to sort on) and asc (true => ascending order, false => descending order). For example var nodes = search.luceneSearch("TEXT:alfresco", "@cm:modified", false);

Ah, this was used at first but was not returning correct results with tokenized fields, e.g. cm:title because each token was being considered separately.

If you need to sort by more than one column then the Script API does not yet provide this. But i can easily add it for Alfresco 3.2 since it has now been requested

That's great- thanks Kevin- Though while we're prioritizing features for Alfresco 3.2, I'd place these above multi-column sort as they are likely even more common needs:

Sorting correctly on a tokenized field
Returning only a specified page worth of all result data (e.g. results 21 through 30)
Perhaps with the now small set of data thanks to the above paging, all properties of those 10 nodes can be fetched & cached? (Although 100ms X 10 isn't quite as terrible.)

The reason some properties take longer than others to retrieve is that some properties are easily resolvable from data already cached on the ScriptNode instance - and some data must be retrieved directly from the repository (and then cached). Accessing individual properties of 1000's of nodes via the ScriptNode API is not going to be as fast as writing some Java code to do it - as the ScriptNode API calls must always pass through all levels of Permissions and Public Service Interceptors etc. for each call (Java code does not always need to do this…)

In order to tap into the power of the Java API, from the webscript would we call the Java API like this? (Or is there another more standard way?)
http://wiki.alfresco.com/wiki/3.0_JavaScript_API#Native_Java_API_Access

Thanks for your help!
Dave

kevinr

Hi,

I'll talk to our Lucene guy today to see what is the issue around sorting on tokenized fields. If we can fix that or find you a solution then hopefully you won't have to go the java route. Yes that is the right link for integrating Java calls into secure WebScripts - but it's not a nice solution if you can avoid it then I would.

FYI "Returning only a specified page worth of all result data" - we need the repo to support paged resultsets for this to work - but currently it does not, so it just ends up re-querying and walking to page N - which is not much use. Paged ResultSet support is coming soon though…

Kev

serverok

Thank you for the reply, Kevin.

I'll talk to our Lucene guy today to see what is the issue around sorting on tokenized fields. If we can fix that or find you a solution then hopefully you won't have to go the java route. Yes that is the right link for integrating Java calls into secure WebScripts - but it's not a nice solution if you can avoid it then I would.

If we will use "external" sorting functions - then we need read values of the property which we use for sorting - then we get the situation above (view values some node property is very slow). The sorting by nontokenised and tokenised field in lucene will be VERY GOOD solution.

Thank you for your help.

Oleg.

kevinr

FYI a new JavaScript Search API has been added to Alfresco 3.2 (should be in the next nightly build or now in HEAD SVN)

Supports multi-column sorting, paging (once added to the underlying search API - before 3.2 final) and the new alfresco-fts search language:
http://wiki.alfresco.com/wiki/Full_Text_Search_Query_Syntax

- Query, language (lucene, xpath, jcr-path and alfresco-fts etc), store (workspace or avm), multi-column sorting and paging all supported via search definition object

- A query definition object with a number of parameter objects can be simple to use as:

   var results = search.query({query: "TEXT:alfresco"});

- Or as richly defined as:

   var sort1 =
   {
      column: "@{http://www.alfresco.org/model/content/1.0}modified",
      ascending: false
   };
   var sort2 =
   {
      column: "@{http://www.alfresco.org/model/content/1.0}created",
      ascending: false
   };
   var paging =
   {
      maxItems: 100,
      skipCount: 0
   };
   var def =
   {
      query: "cm:name:test*",
      store: "workspace://SpacesStore",
      language: "fts-alfresco",
      sort: [sort1, sort2],
      page: paging
   };
   var results = search.query(def);
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Kev

dwilson

Kevin - That sounds perfect, I can't wait!!

samuel_penn

According to the wiki docs, in 3.2 the sort parameter is only available on search.luceneSearch(). If I need to search on an AVM store, then store.luceneSearch() doesn't have this option. Is this an oversight in the docs, or are these new snazzy options missing from AVM searches? If the latter, could they be added?

Thanks,
Sam.

kevinr

See the docs at the bottom of this section:
http://wiki.alfresco.com/wiki/3.2_JavaScript_API#Search_API

You can specify whatever store you like (including AVM), whatever language you like (say Lucene) and specify sorting also all together with the new API.

Kev

samuel_penn

Ah, right, so the search method on the AVM store itself is now effectively redundant. That works.

Thanks,
Sam.

samuel_penn

I've been playing a bit with 3.2 Preview, and the sort argument doesn't always seem to work. Sorting on name or modified time works fine, however sorting on title seems to sort by name.

I'm calling search.luceneSearch(query, "@cm:title", true);

Items are returned in cm:name order. Replacing "@cm:title" with "@cm:modified" returns them correctly in modified date order.

Has anyone else seen this?

Sam.

dwilson

Sam, perhaps there is a correlation with attribute tokenization?

samuel_penn

Sam, perhaps there is a correlation with attribute tokenization?

I don't think so, at least not unless tokenisation works a lot differently to my understanding of it. For example, the document with name "aidan-kingdom.yags" and title "Kingdom of Aidan" appears under the a's when documents are listed alphabetically.

Sam.

andy

Hi

Name is available tokenized and as an id, So sort works as you expect.
Title is only tokenised. So
"Kingdom of Aidan"
will most likely be two tokens
"Kingdom" and "Aidan"
It will sort by the lowest token.
It has nothing to do with name.

Change the default model and reindex if you do nto want the default behaviour.
Title is ML text and so will order by locale - depending on what does have properties in the locale …

Andy

samuel_penn

Okay, that makes sense from a technical perspective. I'd have to say though, I can't think of any situation where anyone would want sorting to actually behave like this (at least, not on an attribute such as cm:title).

I've changed the code to sort the results in Javascript, which seems safer than changing the default model.

Thanks,
Sam.

kbootz

so has there been any resolution to this issue? From what i've read and seen so far, if the sorted column contains multiple tokens then the result is inconsistent and a manual javascript exposed java backed sorting approach seems to be recommened(?)

OR

has this been corrected in the new 3.2 enterprise release so q query with a sorted multiple token field is sorted?

thanks

PS. we're using 3.2 community build 2039, schema 2019 but have purchased the enterprise 3.2 awaiting installation