[solved]The more search queries the slower lucene gets

As we transition to the new Hyland Community, some post links may be temporarily disrupted as we finalize redirects. We apologize for the inconvenience and are working to resolve this as quickly as possible. Details.

sebp

I want to search my repository for duplicate documents. Therefore, in a first step, I go through it recursively and give each content node a hash sum property. In a second step, to know which document has how many duplicates I have to go through the repository again. For each node it is checked if there are other nodes with the same hash sum. So for each node the following method is called:

   /**
    * Sets the number of duplicates on each duplicate node with the given
    * hashsum.
    * 
    * @param storeRef
    * @param hashSum
    * @param deletedNode
    *            the number will not be set on this node. If null it is
    *            ignored.
    */
   void setOthersForHashsum(StoreRef storeRef, String hashSum,
         NodeRef deletedNode) {
      ResultSet rs = searchService.query(storeRef, "lucene",
            "@dup\\:hashsum:" + hashSum);

      if (logger.isDebugEnabled())
         logger.debug("Found " + rs.length() + " equals for hashsum " + hashSum);

      List<NodeRef> nodeRefs = rs.getNodeRefs();
      if (deletedNode != null) {
         int l = nodeRefs.size();
         for (int i = 0; i < l; i++)
            if (nodeRefs.get(i).getId().equals(deletedNode.getId())) {
               nodeRefs.remove(i);
               break;
            }
      }
      
      int duplicateCount = nodeRefs.size() - 1;
      for (NodeRef current : nodeRefs)
         if (nodeService.exists(current))
            nodeService.setProperty(current, PROP_COUNT, duplicateCount);
      
      rs.close();
      rs = null;
   }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

My problem is that this method gets slower and slower the more nodes I process. In the beginning it takes 100ms to process a node but after 10000 nodes it takes 1s to process a single node with this method. My repository contains ~55000 nodes in ~2000 folders. With my current implementation it would take days to process it completely.
In the beginning I forgot to close the search ResultSet. This had a huge impact on the performance. Now, since I close the ResultSet the method is processed much faster but still the more nodes I process the slower it gets.

Btw: The whole project can be found here http://forge.alfresco.com/projects/duplicates/

rogier_oudshoor

In Alfresco, most processing is done inside of a transaction. What you're doing, is add more and more information inside a single transaction - increasing the transaction overheid. You can try to break down your batch into multiple transactions and see if it speeds up.