A logistics data platform that answers in seconds

The challenge

A drayage logistics operator needed dispatch and billing data to reach people inside a working shift. There was no warehouse to start from, the pipeline cycle took 11 to 16 minutes, and the Python dispatch engine at the centre of the work carried accumulated defects.

The approach

1. Built the warehouse and its pipeline from scratch, in layers, so raw data, cleaned data and reporting tables each have one job.
2. Timed every stage of the cycle and cut it in rounds; at the 50-second stage, 30 seconds of each cycle was repeated metadata checks rather than real work, and a cached refresh log removed that overhead in code.
3. Read the query plans for the container lookup, which is searched on nearly every keystroke, and added a trigram index for the long patterns people actually type.
4. Restructured the dispatch engine in a single commit, from 22 modules to 60, taking its type errors from 36 to 0.
5. Measured the repository with a code-health tool before and after, on the same settings, so the improvement is a reading and not an opinion.

The result

Over several rounds of measured fixes, the pipeline cycle went from 11 to 16 minutes down to 6.8 to 17.3 seconds, under 20 seconds. Indexed lookups on the container table, searched on nearly every keystroke, return in about a millisecond: 1.19 to 1.59 ms across three runs, where the same query took 31.4 ms before. Repository code health moved from 7.91 to 9.53, and the worst file from 1.0 to 6.05.