spring - Dynamic chunk size based on step execution duration - Stack Overflow

admin管理员组

文章数量:1277888

I have a very computation based Spring Batch job that is executed with multiple threads. The job might be executed on different computers with different hardware.

I currently configured a chunk size of 1.000.000 but depending on the processor speed and the number of processors, the processing of one chunk might take from 30 seconds up to 1 hour.

Is there any pragmatic way, to adapt the chunk size dynamically based on the runtime of the last StepExecution?

I have something like that in mind:

• Initial chunk size 1000; target runtime per step 5 minutes
• First chunk is processed and the real runtime was 1 minute --> increase chunk size by factor 5
• Set new chunk size is 5000
• Next chunk is processed within 5 minutes; no adaptation of chunk size necessary

Optionally scaling down:

• High workload on the system; real runtime was 10 minutes --> decrease of chunk size by factor 1/2 again
• ...

Is some mechanism like that possible to implement?

I have a very computation based Spring Batch job that is executed with multiple threads. The job might be executed on different computers with different hardware.

I currently configured a chunk size of 1.000.000 but depending on the processor speed and the number of processors, the processing of one chunk might take from 30 seconds up to 1 hour.

Is there any pragmatic way, to adapt the chunk size dynamically based on the runtime of the last StepExecution?

I have something like that in mind:

• Initial chunk size 1000; target runtime per step 5 minutes
• First chunk is processed and the real runtime was 1 minute --> increase chunk size by factor 5
• Set new chunk size is 5000
• Next chunk is processed within 5 minutes; no adaptation of chunk size necessary

Optionally scaling down:

• High workload on the system; real runtime was 10 minutes --> decrease of chunk size by factor 1/2 again
• ...

Is some mechanism like that possible to implement?

• spring
• spring-batch

Share Improve this question Follow edited Feb 25 at 8:24 jonrsharpe 122k3030 gold badges267267 silver badges474474 bronze badges asked Feb 25 at 8:22 OliverOliver 8311 silver badge77 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

It is not possible to re-configure a step dynamically while it is running. You can definitely set the chunk-size dynamically at runtime before starting the step, but not once the step is started and is running. The chunk size will remain the same for all chunks.

本文标签： springDynamic chunk size based on step execution durationStack Overflow