Performance

Shopware import: 33 hours to 3 hours for 530K SKUs

Cut Shopware 6 import time by 91% for a German retailer. 530K+ SKUs in under 3 hours using Redis, Symfony Messenger, and dynamic batch sizing.

Industry:: E-commerce / Retail
Project Type:: Performance Optimization
Duration:: 3 months
Location:: Germany
Published:: March 10, 2024

Outcome

91%

Faster imports

530K+

SKUs

2.8h

Import time (was 33h)

<30m

Delta sync

The Challenge

A German retailer with 530,000 SKUs was running an import that took 33 hours. Daily catalogue updates were impossible. The team was running it as a weekend job and praying nothing crashed at hour 18.

Critical Pain Points

A 33-hour import that blocked daily catalogue work
Inventory drift led to overselling between weekly syncs
Server resources buckled mid-run, killing performance for live shoppers
No way to push partial updates: every change meant a full reimport
Database locks during peak hours forced importers off business time

Weekly updates were the ceiling. Competitors with daily price feeds were eating their lunch on long-tail SKUs.

The Solution

I rebuilt the import pipeline so it stops doing the same lookups twice and runs in parallel.

Dynamic batch sizing

Batch size adjusts to payload weight on the fly. Light products move in big chunks, heavy ones in small ones, and memory never tips over.

Composite indexes where it counts

Profiled the import to find lookup hot spots, then added composite indexes that cut validation queries from milliseconds to microseconds.

Symfony Messenger queue

Multiple workers process import streams in parallel without stepping on each other. The pipeline scales horizontally if the catalogue grows.

Redis cache for the boring lookups

Category mappings and attribute IDs live in Redis. The import stops asking the database the same question 530,000 times.

Results & Business Impact

33 hours to 2.8 hours

Same catalogue, same data, 91% less time. Same-day updates are now routine instead of a weekend job.

Daily syncs unlocked

Inventory and pricing refresh every morning. Overselling tickets dropped to almost zero.

Server load down 60%

Imports run during business hours without any slowdown for live shoppers.

Delta imports under 30 minutes

Routine updates only touch what changed. The 3-hour full import is now the worst-case scenario.

Technologies Used

Shopware 6
PHP 8
Symfony
REST API
MySQL
Redis
Messenger
Doctrine

Imports eating your weekend?

If your Shopware import runs longer than your team's coffee break, there's usually a big speedup hiding in the pipeline. Let's find it.