nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

As we transition to the new Hyland Community, some post links may be temporarily disrupted as we finalize redirects. We apologize for the inconvenience and are working to resolve this as quickly as possible. Details.

Jobexecutor does not cope with simultaneousness

sazzadul

Hi,

I wanted to share an experience with you and hopefully find a solution for it too.

It seems like Jobexecutor simply goes into hibernation when it gets to much to do.

Here is the scenario…
I have a process with a few servicetasks(async) and a user task with timerboundaryevent which was a duedate say 15 minutes ahead.
Now when I start a number of processes (say 100) they all enter the user task and waits for the duedate to expire.
So far so good, but as soon as the duedate expires the jobexecutor kicks in and it begins to work, I can see in the RU_JOB table that it updates the lock column for a few rows but thats basically it. The jobexecutor hereafter doesn't do a single thing, it totally stops. What is even worse it stops processing the new processes as well.

I have tried to increase the maxpoolsize, waitTimeInMillis without any luck. I have even tried a customjobexecutor(as suggested in this http://forums.activiti.org/en/viewtopic.php?f=6&t=3523&hilit=jobexecutor) which uses commonj but the result is same.

I am using jetty with Oracle, have also tried postgres and mssql2008 the result is same.
There is no exception whatsoever being thrown from activiti so I have no clue where the problem lies.

Is there any bug in the jobexecutor ? I hope I have made myself clear enough to reproduce this error.

Thanks.
Sazzadul

Jobexecutor does not seem to tackle simultaneousness at all.

Find more posts tagged with

Comments

meyerd

Hi sazzadul,

Sounds strange.

Do you see any exceptions - either in the log or in the database?

bheurter

Hi,

I experienced the same issue.
When Activiti Starts, It reloads running processes and starts remaining async tasks. 1 times on 3, It hangs. No explanations, no messages.

Any solutions ?

sazzadul

Hi,

I experienced the same issue.
When Activiti Starts, It reloads running processed and starts remaining async tasks. 1 times on 3, It hangs. No explanations, no messages.

Any solutions ?

I would recommend to run only one instance of the JobExecutor and check the db configuration as it may have major impact. Here is the configuration I am using and it's working like a charm. I have tested with 500 simultenouse tasks with no issues. I would also recommend use of an another id generator for instance the
StrongUuidGenerator which comes with Activiti. Let me know if this configuration helps.

<br>&nbsp;&nbsp;&nbsp; <bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <property name="driverClassName" value="${jdbc.driver}" /><br>&nbsp;&nbsp;&nbsp; <property name="url" value="${jdbc.url}" /><br>&nbsp;&nbsp;&nbsp; <property name="username" value="${jdbc.username}" /><br>&nbsp;&nbsp;&nbsp; <property name="password" value="${jdbc.password}" /><br>&nbsp;&nbsp;&nbsp; </bean><br><br><bean id="entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <property name="persistenceUnitManager"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <bean class="org.springframework.orm.jpa.persistenceunit.DefaultPersistenceUnitManager"<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; p:defaultDataSource-ref="dataSource"/><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </property><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <property name="jpaProperties"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <props><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <prop key="hibernate.dialect">${hibernate.dialect}</prop><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <prop key="hibernate.hbm2ddl.auto">update</prop><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </props><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </property><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <property name="jpaVendorAdapter"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <property name="showSql" value="false"/><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </bean><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </property><br>&nbsp;&nbsp;&nbsp; </bean><br> <br>&nbsp; <bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <property name="entityManagerFactory" ref="entityManagerFactory"/><br> </bean><br><br> <bean id="processEngineConfiguration" class="org.activiti.spring.SpringProcessEngineConfiguration"><br>&nbsp; <property name="databaseType" value="oracle" /><br>&nbsp; <property name="dataSource" ref="dataSource" /><br>&nbsp; <property name="transactionManager" ref="transactionManager" /><br>&nbsp; <property name="databaseSchemaUpdate" value="true" /><br>&nbsp; <property name="jobExecutorActivate" value="true" /><br>&nbsp; <property name="history" value="activity" /><br>&nbsp; <property name="idGenerator"><br>&nbsp;&nbsp; <bean class="org.activiti.engine.impl.persistence.StrongUuidGenerator" /><br>&nbsp; </property><br>&nbsp; <property name="idBlockSize" value="2000" /><br> </bean><br>

demotdi

Hi,

I am facing an issue with job executor. Sometimes the jobs update the lock exp time and owner in the table and the job executor does not respond. It doesn't throw any exceptions. The application hangs and I have to restart it.

Is there a way to see the exception and clear out the locks when the application is running? I would appreciate any help.

Thanks.

trademak

Hi,

Did you try to run Activiti with a lower log level? Because the job executor then logs a message every couple of seconds that it's querying the database.
It's really strange if the job executor doesn't respond. Can it be that the job is executed 3 times and failed every time?
After 3 times, the job is not retried anymore by default.

Best regards,

mmaker1234

Hello Activiti developers,

We also suffered by shutting JobExecutor on Activiti Engine 5.11.

The environment is:
- A modified job retry command that actually does not retry;
- A process definition with asynchronous ServiceTask activities.
- Engine configuration, which allows only one thread in the pool:

&nbsp; <bean id="processEngineConfiguration" class="org.activiti.engine.impl.cfg.JtaProcessEngineConfiguration"><br>&nbsp;&nbsp;&nbsp; &hellip;<br>&nbsp;&nbsp;&nbsp; <property name="jobExecutorActivate" value="true" /><br>&nbsp;&nbsp;&nbsp; <property name="jobExecutor"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <bean class="org.activiti.engine.impl.jobexecutor.DefaultJobExecutor"><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <property name="corePoolSize" value="1"/><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <property name="maxPoolSize" value="1"/><br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </bean><br>&nbsp;&nbsp;&nbsp; </property>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <br>&nbsp;&nbsp;&nbsp; &hellip;<br>

My observations are the following:
- We have about ten instances of the process definition;
- We have a bug in the service task, which throws an exception
- The job exececutor creates a new thread after each failure:

pool-1-thread-1
pool-1-thread-2
pool-1-thread-3
pool-1-thread-4
pool-1-thread-5

- There are no further traces from the JobExecutor.

It seems the JobExecutor "refuses" to serve after the fifth failure. As you can see above we have more than five process instances, i.e. there are more "failures" to proceed.

We survived the problem by suspending these faulty processes, that is why I claim that there is no other reason to look for. As the processes are suspended, we can easily resume them and see if they will break again the JobExecutor, if you need such a proof (the server is a test one and its manipulation is somewhat difficult).

I hope this information could help you to track the problem.

trademak

Hi,

Would it be possible to create a unit test that shows the issue with the job executor?

Best regards,

mmaker1234

Hello Tijs,

Here is the unit test: https://docs.google.com/file/d/0B0LoSQX-9OwXRVVVNzhzTHJDWUE/edit?usp=sharing.

I hope it is understandable but I'm ready to answer any questions you might have regarding the test case.

stevennguyen

Hi Tijs & mmaker1234,

Is it's true that "The job exececutor creates a new thread after each failure" ?

We're facing the same issue wit Job Executor. We have 16 manage servers in 4 boxes dedicate for Job executors. If one hang and all Job Executors hang. I couldn't find which jobs cause Jo executors to hang. I've check the lower level log and found not exception.

Below are the end if the log:

013-08-13 18:23:12,222 insertTimer DEBUG - - ==> Parameters: 93061860(String), null, null, true(Boolean), 93061859(String), 85901776(String), null, 3(Integer), null, null, 2013-08-13 18:24:12.146(Timestamp), R5/2013-08-13T18:23:12/PT1M(String), timer-intermediate-transition(String), timerintermediatecatchevent1(String)
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG - - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG - - ==> Preparing: insert into ACT_HI_ACTINST ( ID_, PROC_DEF_ID_, PROC_INST_ID_, EXECUTION_ID_, ACT_ID_, TASK_ID_, CALL_PROC_INST_ID_, ACT_NAME_, ACT_TYPE_, ASSIGNEE_, START_TIME_, END_TIME_, DURATION_ ) values ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ? )
2013-08-13 18:23:12,224 insertHistoricActivityInstance DEBUG - - ==> Parameters: 93061861(String), ProvAddItem:1774:85456592(String), 85901776(String), 93061859(String), timerintermediatecatchevent1(String), null, null, TimerCatchEvent(String), intermediateTimer(String), null, 2013-08-13 18:23:12.147(Timestamp), null, null
2013-08-13 18:23:12,227 updateByteArray DEBUG - - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,227 updateByteArray DEBUG - - ==> Preparing: update ACT_GE_BYTEARRAY set REV_ = ?, BYTES_ = ? where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,227 updateByteArray DEBUG - - ==> Parameters: 2(Integer), java.io.ByteArrayInputStream@7ceaeb21(ByteArrayInputStream), 85901780(String), 1(Integer)
2013-08-13 18:23:12,241 updateExecution DEBUG - - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,241 updateExecution DEBUG - - ==> Preparing: update ACT_RU_EXECUTION set REV_ = ?, PROC_DEF_ID_ = ?, ACT_ID_ = ?, IS_ACTIVE_ = ?, IS_CONCURRENT_ = ?, IS_SCOPE_ = ?, IS_EVENT_SCOPE_ = ?, PARENT_ID_ = ?, SUPER_EXEC_ = ?, SUSPENSION_STATE_ = ?, CACHED_ENT_STATE_ = ? where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,241 updateExecution DEBUG - - ==> Parameters: 4(Integer), ProvAddItem:1774:85456592(String), null, false(Boolean), false(Boolean), true(Boolean), false(Boolean), null, null, 1(Integer), 0(Integer), 85901776(String), 3(Integer)
2013-08-13 18:23:12,243 deleteMessage DEBUG - - ooo Using Connection [weblogic.jdbc.wrapper.PoolConnection_oracle_jdbc_driver_T4CConnection@2df5]
2013-08-13 18:23:12,243 deleteMessage DEBUG - - ==> Preparing: delete from ACT_RU_JOB where ID_ = ? and REV_ = ?
2013-08-13 18:23:12,243 deleteMessage DEBUG - - ==> Parameters: 93037099(String), 2(Integer)

trademak

Hi,

The job executor is a thread pool and polls for jobs to be processed.
When a job fails it is placed back in the jobs table and the retry count is increased by 1.
When a job executor polls for jobs again the failed job will be processed again until the same job has been retried 3 times, then the job is placed in deadlock mode and it will be be retried again.

What's the exact behavior you are experiencing? How many jobs are there in the ACT_RU_JOB table and what are the job executor threads doing?

Best regards,

stevennguyen

Hi Tijs,

We have serveral job executors. When job executor hang, we restart it. After it pickup a few jobs, it hang again. All job executors will be hung within a few minutes. The number of jobs can be from 200 to 20,000 when we have the issue. We did a threaddump, but not able to find job executor thread.

I've seperated out different type of jobs, by trucate ACT_RU_JOB , manually insert jobs back and have only jobexecutors running to process them. Certain type of job, job executors can proccess 20K within 10 mins without an issue. We found that the job that causes job executor to hang, it requires external webservice call and take longer to process.

I see the last printout from the lob with Server error. Could this causes the job executor crash or hung?
Last printout from Job Executor:
Aug 16, 2013 8:20:47 PM org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable run
INFO: JobExecutor[org.activiti.engine.impl.jobexecutor.DefaultJobExecutor] starting to acquire jobs
INFO: Error while closing command context
org.activiti.engine.JobNotFoundException: No job found with id '94593698'.
        at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:59)
        at org.activiti.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
        at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:60)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32)
        at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:46)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
Aug 16, 2013 8:21:56 PM org.activiti.engine.impl.interceptor.CommandContext close
SEVERE: Error while closing command context
org.activiti.engine.ActivitiOptimisticLockingException: HistoricVariableInstanceEntity[103316416] was updated by another transaction concurrently
        at org.activiti.engine.impl.db.DbSqlSession.flushUpdates(DbSqlSession.java:652)
        at org.activiti.engine.impl.db.DbSqlSession.flush(DbSqlSession.java:460)
        at org.activiti.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:167)
        at org.activiti.engine.impl.interceptor.CommandContext.close(CommandContext.java:114)
        at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:69)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:32)
        at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:46)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)

Last few lines in the log:

Aug 16, 2013 8:25:53 PM weblogic.wsee.jaxws.framework.policy.advertisementimpl.AdvertisementHelperImpl registerExtension
WARNING: Registering oracle.j2ee.ws.wsdl.extensions.addressing.AddressingExtensionRegistry extension failed; java.lang.ClassNotFoundException: oracle.j2ee.ws.wsdl.extensions.addressing.AddressingExtensionRegistry
Aug 16, 2013 8:25:53 PM weblogic.wsee.jaxws.spi.WLSProvider createServiceDelegate
WARNING: Could not read WSDL Definition from URL wsdlDocumentLocation: 2 counts of InaccessibleWSDLException.

Aug 16, 2013 8:25:53 PM weblogic.wsee.jaxws.framework.policy.advertisementimpl.AdvertisementHelperImpl registerExtension
WARNING: Registering oracle.j2ee.ws.wsdl.extensions.addressing.AddressingExtensionRegistry extension failed; java.lang.ClassNotFoundException: oracle.j2ee.ws.wsdl.extensions.addressing.AddressingExtensionRegistry
Aug 16, 2013 8:25:53 PM weblogic.wsee.jaxws.spi.WLSServiceDelegate addWsdlDefinitionFeature
SEVERE: Failed to create WsdlDefinitionFeature for wsdl location: zip:/apps/opt/weblogic/config/BPS/servers/fduscpw8_bps01/tmp/_WL_user/CMPRealtime/shqoub/APP-INF/lib/ITWBAgent.jar!/META-INF/wsdl/heartbeat.wsdl, error: com.sun.xml.ws.wsdl.parser.InaccessibleWSDLException, message: 2 counts of InaccessibleWSDLException.

trademak

Hi,

From the last part of your log I see error messages for executing the web service call. This would mean that the job is retried 3 times and fails 3 times. After that the job will not be processes again. If this happens for every job, eventually the job executor will stop because all jobs are in failed state. Could that be the case?

Best regards,

stevennguyen

Tijs,

Found the root of this issue. We're using Weblogic version 11g and it's using older version of joda jar than Activiti's joda jar. In some scenario, we're getting "ERROR" and it's being not catched. This cause AcquireJobsRunnable to die. Below is the Error that I caught and print in AcquireJobsRunnable.

org.joda.time.DateTime.parse(Ljava/lang/String

Lorg/joda/time/DateTime;
Sep 2, 2013 9:32:54 PM org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable run
INFO: 232513==>org.activiti.engine.impl.calendar.DurationHelper.parseDate(DurationHelper.java:119)
org.activiti.engine.impl.calendar.DurationHelper.<init>(DurationHelper.java:64)
org.activiti.engine.impl.calendar.CycleBusinessCalendar.resolveDuedate(CycleBusinessCalendar.java:30)
org.activiti.engine.impl.persistence.entity.TimerEntity.calculateRepeat(TimerEntity.java:90)
org.activiti.engine.impl.persistence.entity.TimerEntity.execute(TimerEntity.java:72)
org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:71)
org.activiti.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:60)
org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:38)
org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:50)
org.activiti.engine.impl.jobexecutor.CallerRunsRejectedJobsHandler.jobsRejected(CallerRunsRejectedJobsHandler.java:26)
org.activiti.engine.impl.jobexecutor.DefaultJobExecutor.executeJobs(DefaultJobExecutor.java:82)
org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:67)
java.lang.Thread.run(Thread.java:662)

trademak

Okay, but are you saying that there are new jobs for the job executor to process and it will not execute the new ones anymore, because of this error?

Best regards,

mmaker1234

Hello Tijs,

At least in our case "<i>…there are new jobs for the job executor to process and it will not execute the new ones anymore, because of this error</i>" is exactly what we observed. I suppose Steven Nguyen observes the same.

trademak

Okay thanks, we'll try to reproduce it.

Best regards,

pollop

- Wrong post -

frederikherema1

This has been fixed in b10dfc9ab6170641030beb9e84151ed4a949ef03 on master, will be part of the 5.14 release

mmaker1234

Thank you very much, Frederik! 🙂

I'll check when 5.14 is released.

houssam

still facing the same issue with 5.14 version, any suggestion?

trademak

Could you explain what exactly you are facing? Please explain with some background about how you are using jobs etc.

Best regards,

erny18031

Hi.
Just a guess: Is there are remote possibility that the job executor threads are not returned to the thread pool after job execution errors, or that the returned threads are not functional anymore?

Regards.

trademak

It's implemented in a way that this shouldn't happen, but there can always be a bug of course.
That's why I'm asking for more information to be able to dive into the exact issue in more details.

Best regards,

huco

Hello.

We are experiencing similar issues with job executor. We are running 10 service tasks at most at the same time and sometimes the transition from completed script task (which precedes service task) to service task takes more than an hour and a half. We are using activiti 5.12 in our web app and after the script task was completed, the following exception was thrown:
<code>
org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. Could not load org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable. The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
at org.activiti.engine.impl.jobexecutor.DefaultJobExecutor.executeJobs(DefaultJobExecutor.java:82)
at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:62)
at java.lang.Thread.run(Thread.java:662)
Exception in thread "Thread-156" java.lang.NoClassDefFoundError: org/activiti/engine/impl/jobexecutor/ExecuteJobsRunnable
at org.activiti.engine.impl.jobexecutor.DefaultJobExecutor.executeJobs(DefaultJobExecutor.java:82)
at org.activiti.engine.impl.jobexecutor.AcquireJobsRunnable.run(AcquireJobsRunnable.java:62)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
… 3 more
</code>

Hour and a half later service task started it's execution and completed successfully. We are not using any timers in our process.

Data recored in act_hi_actinst for that particular case:
<code>
"Script Task";"scriptTask";"";"2014-01-22 00:53:34.307";"2014-01-22 00:53:34.318";11
"Unlock";"serviceTask";"";"2014-01-22 02:33:38.991";"2014-01-22 02:33:39.048";57
"Exclusive Gateway";"exclusiveGateway";"";"2014-01-22 02:33:39.048";"2014-01-22 02:33:39.049";1
"Receive Task";"receiveTask";"";"2014-01-22 02:33:39.049";"2014-01-22 02:33:41.951";2902
"Exclusive Gateway";"exclusiveGateway";"";"2014-01-22 02:33:41.951";"2014-01-22 02:33:41.952";1
"Exclusive Gateway";"exclusiveGateway";"";"2014-01-22 02:33:41.952";"2014-01-22 02:33:41.952";0
"End";"endEvent";"";"2014-01-22 02:33:41.952";"2014-01-22 02:33:41.952";0
</code>

In a few cases the transition never happend and service task never actually started. Such service tasks are present in act_ru_execution table but nothing is happening.
<code>
"servicetask3";t;f;t;f;1;4
</code>

Any idea what's causing these issues?

Best regards

frederikherema1

That's really strange… In case there is a "finished" entry in ACT_HI_ACTINST, the service-task did run and the transaction was comitted. How do you know the service-task wasn't run?

Seems like the error occurs when jobs are still being executed, but the server is going down.

huco

Hi.

Right at the start of service task we are printing progress lines to logger and catalina and nothing gets recorded.

We've upgraded to 5.14. version but it looks like the problem is still present. When time came to start the Service Task, the following exception was thrown:
<code>
org.activiti.engine.ActivitiException: Job 166561 failed
        at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:83)
        at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
        at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:37)
        at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
        at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
        at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:46)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.activiti.engine.ActivitiException: couldn't instantiate class com.activiti.UnlockServiceTask
        at org.activiti.engine.impl.util.ReflectUtil.instantiate(ReflectUtil.java:132)
        at org.activiti.engine.impl.bpmn.helper.ClassDelegate.instantiateDelegate(ClassDelegate.java:163)
        at org.activiti.engine.impl.bpmn.helper.ClassDelegate.getActivityBehaviorInstance(ClassDelegate.java:136)
        at org.activiti.engine.impl.bpmn.helper.ClassDelegate.execute(ClassDelegate.java:113)
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationActivityExecute.execute(AtomicOperationActivityExecute.java:45)
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:88)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527)
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerStart.java:52)
        at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:88)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527)
        at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:49)
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:88)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:532)
        at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:527)
        at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionCreateScope.execute(AtomicOperationTransitionCreateScope.java:49)
        at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:88)
        at org.activiti.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:36)
        at org.activiti.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:79)
        at org.activiti.engine.impl.persistence.entity.MessageEntity.execute(MessageEntity.java:29)
        at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:71)
        … 9 more
Caused by: org.activiti.engine.ActivitiClassLoadingException: Class not found: com.activiti.UnlockServiceTask
        at org.activiti.engine.impl.util.ReflectUtil.loadClass(ReflectUtil.java:82)
        at org.activiti.engine.impl.util.ReflectUtil.instantiate(ReflectUtil.java:129)
        … 31 more
Caused by: java.lang.ClassNotFoundException: com.activiti.UnlockServiceTask
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1714)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.activiti.engine.impl.util.ReflectUtil.loadClass(ReflectUtil.java:63)
        … 32 more
</code>
There were 80 other process instances which successfully completed that service task.

Any clue why this one failed?

Thanks

jbarrez

Absolutely no clue … if 80 instances can find the class … there is no reason why the 81st should fail