Today, while batch processing data with Laravel, I used chunkById to handle over 2 million records and found the processing speed was way too slow. After all, Laravel with chunkById for large datasets is often inefficient when running single-threaded, especially with over a million records.

This article will introduce how to use Java multi-threading technology to efficiently process a product table with over 2 million records in a Spring Boot project, along with key considerations.

Background

We have a products table containing a large amount of product information. To execute complex business logic — such as updating product statuses, calculating statistics, etc. — we need to iterate through and process this data.

Laravel Implementation

First, let’s look at how Laravel implements this:

    public function handle(): void
    {
        $i = 0;
        // Process in batches of 500
        Product::chunkById(500, function ($products) use (&$i) {
            /** @var Product $product */
            foreach ($products as $product) {
                $i ++;
                $this->info('i=' . $i . ', product id: ' . $product->id);
                try {
                    // your code here
                }catch (\Throwable $e){
                    Log::error('product id: ' . $product->id . ' error: ' . $e->getMessage());
                }
            }
        });
    }

Java Multi-threaded Implementation

I’m using the Spring Boot + MyBatis Plus framework here, so I’ll just show the key code:

    public void batchProcessProducts() {
        Long lastId = null;

        while (true) {
            List<ProductPO> productList = fetchNextBatch(lastId);
            if (productList.isEmpty()) {
                break;
            }

            List<? extends Future<?>> futures = productList.stream()
                    .map(product -> executorService.submit(() -> processProduct(product)))
                    .toList();

            // Wait for all threads to complete
            futures.forEach(future -> {
                try {
                    future.get();
                } catch (Exception e) {
                    log.debug(e.getMessage());
                }
            });

            // Update lastId to the ID of the last product in the current batch
            lastId = Long.valueOf(productList.get(productList.size() - 1).getId());
        }
    }

    private List<ProductPO> fetchNextBatch(Long lastId) {
        // If lastId exists, query data after that ID. Otherwise, query the first BATCH_SIZE records.
        if (lastId != null) {
            return productMapper.selectByLastId(lastId, BATCH_SIZE);
        } else {
            return productMapper.selectFirstBatch(BATCH_SIZE);
        }
    }

This uses the framework’s default thread pool — ExecutorService manages the lifecycle of multiple threads. We divide each batch of data into multiple tasks and submit them to the thread pool. By waiting for each Future result with future.get(), we ensure all tasks have completed.

Note: We didn’t use MyBatis Plus pagination for grouping. For the reason, see:

Why MySQL Pagination Slows Down When Offset Exceeds 1000

Using the default ExecutorService or ThreadPoolTaskExecutor usually meets most needs, but a custom thread pool provides finer control, such as:

• Thread count: You can customize the pool size based on system load and resource limits.
• Thread priority: In certain scenarios, you may want specific tasks to have higher priority.
• Thread naming: Custom thread naming helps trace thread activity more easily in logs.
• Rejection policy: When both the thread pool and queue are full, custom rejection policies determine how to handle new task requests.
• Lifecycle management: Custom thread pools allow better control over thread lifecycles, such as gracefully shutting down the pool when the application stops.

Results

Using Java multi-threading to process over 1 million records (updating just one field) took only 3 minutes and 25 seconds.

The same work in Laravel took 1 hour — countless times slower.