As we transition to the new Hyland Community, some post links may be temporarily disrupted as we finalize redirects. We apologize for the inconvenience and are working to resolve this as quickly as possible. Details.
Hi,
I installed ACS 6.0 in cluster (2 nodes) under two windows 2016 server failover cluster (in vmware). The server didn't took the load (users sessions) althought i gave them a lot of ram (96g) and CPU (16). Tomcat jvm has 64g. Did someone installed an ACS 6 cluster that is running fine and support about 200-300 users? What is your solution?
Ubuntu?
Thanks
Do you mean 200-300 concurrent users or 200-300 users?
Anyway, in both cases, 64 GB and 16 CPUs is over-sized...
Hi:
If you are using virtual machines, you may be sure that your RAM or CPU resources are really (or near) dedicated to your cluster (at least de RAM ones). If you have 200-300 potential users it seems oversized as commented by Angel initially, but having a huge number of documents in your repo, SOLR setup may need a lot JVM and CPU resources. Anyway, some basic profiling may help you to diagnose it (Support Tools or jvisualvm may help to see your JVM use and CPU load).
You should consider to have SOLR Search Services in a dedicated virtual machine and having SSD disks for SOLR indices, if you do not do it yet. Appropiate storage and database resources helps too, and normally the second ones are more relevant. Finally, I use Ubuntu or Linux based OS (when I can) in my setups, but this probably is not as important if you are a Windows administrator.
Regards.
--C.
It looks like from my actual installation (4.2), it would reach 100 concurrent users. And it works fine. No problem at all. My actual installation is on a physical server with windows 2012 R2 (128g ram, 32 cpu cores, 64g for tomcat). Nb of documents is about 2 500 000. We don't use share interface, only alfresco explorer. The new installation is for an upgrade to 6.0.1 with the same load. But after 30 concurrents sessions, alfresco became non responsive and slow.
I don't understand what could cause that.
You need to perform some insights on your 6.0.1 configuration, it should work fine with your current resources.
I'm asking that question, because i never heard or read on someone that used my kind of architecture. I mean two windows 2016 failover cluster with a CSV drive (Cluster Shared Volume) for the contentstore. I don't know if i've go to suspect my alfresco installation or my OS solution.
What about db.pool.max parameter in alfresco-global.properties ?
If you're talking about cluster the Alfresco support team may be a better audience 😉
100 concurrent users shouldn't a problem depending on the usage scenario, number of nodes and the customizations. We have single instance (no cluster) customers with >500 concurrent, active working users with > 20 mio doc nodes in a 3 vm (alfersco, solr, postgres) installation having 3x 8 Cores and it is very fast.
You should know that a cluster will always cost performance and will never increase it (except some very uncommon read only scenarios).
Anyway: Independant from using a cluster or not: You should focus on common tuning and archtecture best practices which could not be covered by a forum thread.
In short:
* more RAM and (v)CPUs are not always better. More RAM may cause more CPU load and may just hide the real problems
* Run and tune db, repo/share tier, solr on separate systems
* In most cases the transformation, the db and especially their storage are the bottlenecks. Given you have the knowledge and resocurces making the db very fast may be the easier part. Avoid shared SAN LUNs and try to run at least the db on an exclusive cluster aware SSD backed storage. Same for the Solr index.
* it's easier to tune and find bottlenecks in linux instead of in windows
* never ever run docker on windows in production. It will not scale. Docker in production should only an option if you run it directly on bare metal installed linux systems / not VMs or maybe if you know what your're doing only one tier (alfresco, db, solr) on 1 VM. Our customers don't have the required teams for that. For the large systems mentioned before we run even 6.0 / 6.2 in esx VMs without any docker involved.
db.pool.max=100 alfresco tell me that they recommended 275.
I will try it but i doubt that it is the only concern.
Thnks
Marc
Heiko,
I really apreciate your feedback. It will help us re-think the solution.
Regards
BTW, Heiko, what are the version of alfresco and solr you are running? On which OS?
Marc,
most of our customers run on Alfresco 5.1/5.2, Solr 4 but from my experience the versions 5-6 differ only marginally in terms of performance and scalability.
Solr is a memory beast and requires special attention. Solr 6 has more features and better tools to avoid OOM exceptions but you need to handle them. Some recommend to split your index using sharding to overcome hardware limits.
90 % of unplanned downtimes we have seen in the last 10 years were related to an effect which we call thread escalation: If the Alfresco repository waits on or gets exceptions from other components (database, transformation, solr query) it will create new threads until the whole system has eaten all resources. The retrying transaction concept even accelerates the thread escalation. So
After many tests, it looks like my problem is a combination of cluster feature + High volume documents. When i test a 2 nodes cluster (2,5 million docs) with 10 concurrent sessions, it take tomcat about 30 min to top his memory usage and CPU. At that moment, alfresco become very slow, almost non-responsive. And most important, we could have more than 40 actives connexion to Oracle. Same test, same server with cluster feature disable, the server runs fine and the database connexion stay low (1-3). On the other end, i've got another setup (identical to the previous cluster) that is for dev, so with less documents (170 000). When i did the same test in cluster, alfresco stay very responsive, no balloon for tomcat, low active connexion to Oracle. So it looks like the volume has an influence when in cluster mode. Did someone experience this kind of situation?
Hi Marc, volumetry has always a big influence (but not only for a cluster). The term sizing speaks about this too. It is not the same an Alfresco alf_node_properties table with thousands of records, than one with several hundreds of millions. It is not the same a database size of 10Gb than a database 100 times bigger. It is not same the indices size in SOLR for some thousands of documents, than the size for several millions. The number of documents, the number of metadata properties and ACLs have a big impact on the resources needed by Alfresco and its components such as as SOLR or relational database. For example, the needed JVM for a SOLR instance depends directly on the number of documents of the repository.
https://docs.alfresco.com/5.2/concepts/solrnodes-memory.html
In your cluster, do you have SOLR in a dedicated machine or do you deploy SOLR in both nodes ?
Kind regards.
Hi Cesar,
I did the calculation for SOLR and validate with alfresco, so i hope that it is ok. My alf_node_properties table is about 50 millions records. Solr engine is deploy in both nodes. It really looks like there is a problem when i activate the cluster feature on my server with high volume database. Without cluster activated, the same server react normally. What overhead, cluster is causing?
Regards,
I can't say what is causing this overhead.
But I would point or try with SOLR first. I mean, one of the recommendations/best practices for reducing performance problems (in a cluster or not) is to separate SOLR from the repository. The memory calculation and CPU consumption is only for SOLR. You have a competition on machine resources between SOLR and Alfresco repository each every moment, not only in CPU and JVM memory, also in IO (SOLR indexing/writing and searching/reading) while Alfresco is ingesting data in contentstore. So you can try to:
