Search stops due to solr4 not working properly

As we transition to the new Hyland Community, some post links may be temporarily disrupted as we finalize redirects. We apologize for the inconvenience and are working to resolve this as quickly as possible. Details.

hardik_thakkar

Hi Team,

I have restarted server and solr4 stops working:
Below are the error logs from solr.log file:

2020-10-09 02:18:32,330 ERROR [org.alfresco.solr.tracker.AbstractTracker] Tracking failed
org.alfresco.solr.TrackerStateException: 09090000 The trackers work was rolled back by another tracker error
	at org.alfresco.solr.SolrInformationServer.canUpdate(SolrInformationServer.java:3238)
	at org.alfresco.solr.SolrInformationServer.commit(SolrInformationServer.java:773)
	at org.alfresco.solr.tracker.ContentTracker.doTrack(ContentTracker.java:111)
	at org.alfresco.solr.tracker.AbstractTracker.track(AbstractTracker.java:190)
	at org.alfresco.solr.tracker.TrackerJob.execute(TrackerJob.java:54)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
2020-10-09 03:34:45,363 ERROR [org.alfresco.solr.tracker.AbstractTracker] Tracking failed
java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:210)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)

Below are the logs from alfresco.log file and based on the logs I can see solr backup job runs everyday at  2 AM and that is failing, I have verified that solr backup was done 3 days back.

2020-10-09 02:00:41,110 ERROR [org.quartz.core.JobRunShell] [SearchScheduler_Worker-2] Job DEFAULT.search.alfrescoCoreBackupJobDetail threw an unhandled Exception: 
org.alfresco.error.AlfrescoRuntimeException: 09091799 Backup for core alfresco failed .... 
	at org.alfresco.repo.search.impl.solr.SolrBackupClient.executeImpl(SolrBackupClient.java:203)
	at org.alfresco.repo.search.impl.solr.SolrBackupClient.execute(SolrBackupClient.java:140)
	at org.alfresco.repo.search.impl.solr.SolrBackupJob.execute(SolrBackupJob.java:65)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.apache.solr.client.solrj.SolrServerException: java.net.SocketTimeoutException: Read timed out
	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:483)
	at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:244)
	at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:89)
	at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
	at org.alfresco.repo.solr.SOLRAdminClient.query(SOLRAdminClient.java:203)
	at org.alfresco.repo.search.impl.solr.SolrBackupClient.executeImpl(SolrBackupClient.java:192)
	... 4 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
	at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.flushRequestOutputStream(MultiThreadedHttpConnectionManager.java:1565)
	at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)

For the socket time out exception, I have added below configuration in custom-slingshot-application-context.xml (/tomcat/shared/classes/alfresco/web-extension)

<bean id="connector.remoteclient" parent="connector.remoteclient.abstract" 
  class="org.alfresco.web.scripts.SlingshotRemoteClient" scope="prototype" > 
<!-- the http.connection.timeout value in milliseconds to apply to HTTP connections  --> 
    <property name="connectTimeout"><value>420000</value></property>
<!-- the http.socket.timeout value in milliseconds to apply to HTTP connections    -->
	<property name="readTimeout"><value>420000</value></property> 
</bean> 

In the alfresco.log, I can see read timed out error frequently.

What is the maximum timeout values we can set?
I have Postgres DB on other node, does that creates any issue?

I have set solr.suggester.enabled=false in solrcore.properties file before restart, does that creates any issue?

Need help/guidance to troubleshoot this issue.

Thanks,

Hardik

hardik_thakkar

Does anyone has faced this type of issue?