nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

As we transition to the new Hyland Community, some post links may be temporarily disrupted as we finalize redirects. We apologize for the inconvenience and are working to resolve this as quickly as possible. Details.

OutOfMemory errors while uploading files through web service

sabsurendran

We are evaluating the performance of Alfresco for our project, and are trying to test the performance under heavy loads.
We have 3 different java clients which use the Alfresco Webservice API to upload 100 files of 4 MB each to the same space. We frequently get OutOfMemory exceptions on the Java heap space, and tweaking the Java memory params has not helped.

This is what we have in alfresco.bat:
-Xms518m -Xmx1024m -Xss96k -XX:MaxPermSize=128m -server

The machine where Alfresco repository is hosted has 2 GB RAM. Does anyone know if we need to tweak something else?

Find more posts tagged with

Comments

derek

Hi,
With WebServices, there is a very definite wrong and right way to upload files. You need to tweak your code to upload the content direct to the server over http, otherwise WebServices (and this is just how it is) will be sucking the binary data into memory before pushing it into Alfresco. Alfresco itself does not pull the binaries into memory first.
Regards

sisao

since i've had problems of java heap space overflows that i've solved increasing the Xmx paramter for the alfresco lauch configuration i'd like you to be more precise about these "tweaks" and how can i understand if i'm going the right or the wrong way.
For example this is the upload method i use in my applications:

UpdateResult[] result = WebServiceFactory.getRepositoryService().update(cml);     
         Reference content = result[0].getDestination();
         byte[] bytes = ContentUtils.convertToByteArray(stream);
         ContentFormat contentFormat = new ContentFormat(tipo, "UTF-8");
         WebServiceFactory.getContentService().write(content, Constants.PROP_CONTENT, bytes, contentFormat);
         System.out.println("DONE");
         uuid=content.getUuid();‍‍‍‍‍‍‍

stream is the InputStream of the file i'm going to send to the repository which i guess take some space in the java heap and you cannot avoid this memory usage.
Regards.

derek

You need to tweak your code to upload the content direct to the server over http

sisao

Of course i've carefully read your previous post but i cant see a different way of uploading content to the repository via WS different from the one i've posted which is the result of the alfresco SDK examples…maybe i'm not getting the point

Regards.

kbonnet

Derek,

The Alfresco Release Script for Kofax is also using the webservices. One of our client is using this setup, but ends up with OutOfMemoryErrors when posting large documents.

Could this have to do with the fact that the release script doesnt make the document available over htttp first, but instead wrapping it in the soap request?

I also read somewhere that Axis is only able to handle 32mb documents in soap requests. Are you familiar with such a boundary?

Thanks for your time.

Regards,

Koen Bonnet

sisao

I also read somewhere that Axis is only able to handle 32mb documents in soap requests. Are you familiar with such a boundary?

I use WS APIs and i also send documents of 1GB of size without any java heap issues.
Check your application server startup parameters and alfresco's.

Regards.

kbonnet

Hi Sisao,

Thanks for your reply. Are you writing your own webservices client? Are you pushing the document to alfresco first, as suggested by Derek as being the right way?

We have to deal with a webservice client which is part of kind of standard software, the release script. I dont see a way of altering this to push the document first.

Thanks for your help.

Koen Bonnet

luca

Hi all,
does anyone found a solution?

I'm getting the same problem, but I can't figure it out!

Derek, can you give us some more explanation please?

Thanks,
Luca

lakshya

Did anyone find the solution??

I am using SOAP API to upload files.

   private String writeContentInSpace(String contentName, String title,
         String description, String spaceName, String mimeType,
         String encoding, byte[] content, String author, String isbn,
         int pageCount, String classification, String clientName,
         String clientMail, String imageId) {
      try {
         // Start the session
         AuthenticationUtils.startSession(userName, password);
      } catch (AuthenticationFault fault) {
         fault.printStackTrace();
         return "Exception occured";
      }
      try {

         // Searching for the space UUID
         // Get a reference to the repository web service
         RepositoryServiceSoapBindingStub repositoryService;
         repositoryService = WebServiceFactory.getRepositoryService();
         // Get a reference to the space we have named
         Store storeRef = new Store(Constants.WORKSPACE_STORE, "SpacesStore");
         Node[] nodes = null;
         Reference reference = null;
         Predicate predicate = null;
         try {
            reference = new Reference(storeRef, null,
                  "/app:company_home/cm:" + spaceName);

            predicate = new Predicate(new Reference[] { reference }, null,
                  null);
            nodes = repositoryService.get(predicate);
         } catch (Exception ex) {
            System.out.println("Exception occured " + ex.getMessage());
            ex.printStackTrace();
         }
         // Searching for the UUID out of the result

         String spaceUUID = "";

         if (nodes[0] != null) {
            NamedValue[] properties = nodes[0].getProperties();
            for (int j = 0; j < properties.length; j++) {
               NamedValue nv = properties[j];
               if ("{http://www.alfresco.org/model/content/1.0}name"
                     .equals(nv.getName())
                     && nv.getValue().equals(spaceName)) {
                  System.out.println("In Space : " + nv.getValue());
               }
               if ("{http://www.alfresco.org/model/system/1.0}node-uuid"
                     .equals(nv.getName())) {
                  System.out.println("Space UUID: " + nv.getValue());
                  spaceUUID = nv.getValue();
               }
            }
         }

         // Create a reference to the parent where we want to insert content
         Reference referenceWrite = new Reference(storeRef, spaceUUID, null);

         ParentReference parentReference = new ParentReference(
               referenceWrite.getStore(), referenceWrite.getUuid(), null,
               Constants.ASSOC_CONTAINS, Constants.ASSOC_CONTAINS);
         // Create content
         String page_count_str = Integer.toString(pageCount);
         NamedValue[] newProperties = new NamedValue[] {
               Utils.createNamedValue(Constants.PROP_NAME, contentName),
               Utils.createNamedValue(Constants.PROP_DESCRIPTION,
                     description),
               Utils.createNamedValue(Constants.PROP_TITLE, title),
               Utils.createNamedValue(Constants.PROP_AUTHOR, author),
               Utils.createNamedValue(Constants.PROP_ISBN, isbn),
               Utils.createNamedValue(Constants.PROP_CLIENT_MAIL,
                     clientMail),
               Utils.createNamedValue(Constants.PROP_CLIENT_NAME,
                     clientName),
               Utils.createNamedValue(Constants.PROP_PAGE_COUNT,
                     page_count_str),
               Utils.createNamedValue(Constants.PROP_CLASSIFICATION,
                     classification),
         /* Utils.createNamedValue("IMAGE_ID", imageId), */
         };

         CMLCreate create = new CMLCreate("1", parentReference, null, null,
               null, Constants.TYPE_CONTENT, newProperties);

         CMLAddAspect addAspect = new CMLAddAspect();
         addAspect.setAspect(Constants.ASPECT_VERSIONABLE);
         addAspect.setProperty(newProperties);

         addAspect.setWhere(new Predicate(
               new Reference[] { referenceWrite }, null, null));
         addAspect.setWhere_id("1");

         CML cml = new CML();
         cml.setCreate(new CMLCreate[] { create });
         cml.setAddAspect(new CMLAddAspect[] { addAspect });

         // update
         UpdateResult[] results;
         results = WebServiceFactory.getRepositoryService().update(cml);
         // Set content
         ContentFormat format = new ContentFormat(mimeType.toString(),
               encoding);
         Content newContent;
         newContent = WebServiceFactory.getContentService().write(
               results[0].getDestination(), Constants.PROP_CONTENT,
               content, format);
         // System.out.println("UUID of uploaded content : " +
         // newContent.getNode().getUuid());
         System.out.println("File Uploaded.");

         // versioning(storeRef);

         return newContent.getNode().getUuid();

      } catch (Exception e) {
         // e.printStackTrace();
         return "Exception occured";
      } finally {
         // End the session
         AuthenticationUtils.endSession();
      }
}   
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

kebbs

Here is what worked for me…

            ContentFormat format = new ContentFormat(properties.getString("contentType"), "UTF-8");
            DataHandler attachmentFile = new DataHandler(new FileDataSource(file));
            // commented out since this results in an exception
            //_contentService._setProperty(Call.ATTACHMENT_ENCAPSULATION_FORMAT, Call.ATTACHMENT_ENCAPSULATION_FORMAT_DIME);
            _contentService.addAttachment(attachmentFile);
            _contentService.writeWithAttachment(obj.getNode().getReference(), Constants.PROP_CONTENT, format);
‍‍‍‍‍‍‍‍

junieboy

Hi,
With WebServices, there is a very definite wrong and right way to upload files. You need to tweak your code to upload the content direct to the server over http, otherwise WebServices (and this is just how it is) will be sucking the binary data into memory before pushing it into Alfresco. Alfresco itself does not pull the binaries into memory first.
Regards

Hi Derek,

Good evening. It is very probable by this quarter that we start our development for Alfresco for our Document Management, and our communication would be through web services (more importantly "both saving of files and metadata"). We will be saving files from our system to Alfresco through the Alfresco web service API and basically, we'll develop a custom content model and all the metadata will be coming from our system will be attached to files and these metadata will be used in searching for these files in Alfresco.

My question is how reliable will this Alfresco web services API approach be since I'm seeing in this post that there's an issue saving files through web service? Also, I was having some question since I am seeing relatively limited Javadoc/API documentation regarding web services.

Also, could I ask you to please further elaborate the solution? How do we tweak the code such that we upload direct to the server over http? What Java API should be used to do this? Please share some examples or diagrams for our reference. Thanks a lot!

Best Regards,
Junie

derek

Hi,
If you're starting from scratch, you should consider using CMIS as your starting point. There are servers available for testing directly e.g. http://cmis.alfresco.com/cmis.
Regards

junieboy

Hi,
If you're starting from scratch, you should consider using CMIS as your starting point. There are servers available for testing directly e.g. http://cmis.alfresco.com/cmis.
Regards

Hi Derek,

Unfortunately we are using Alfresco 3.0 (the instance is already existing for sometime in our infrastructure and is already being used by our domains but with no web services integration) and I believe CMIS is only available at 3.3. So this is the first time we're doing it. Could I ask for more insights re my questions? Thanks a lot.

Regards.

akoudrin

Could you please post all your includes to make the above work, please?