System operations over seven centuries
Published by Martin Kleppmann on 12 Aug 2013.
On a walk in the Alps last
week we came across a wonderful piece of engineering, more successful than most software systems
could claim to be. It is the system of Waale,
an ancient irrigation system in the Vinschgau, South
The climate in the Vinschgau is sunny, dry and windy. Without irrigation, agriculture would barely
be possible, but if water from mountain streams is channelled to the fields,
apple trees and meadows can flourish. The area has been inhabited at least
since the Bronze Age, and it is likely that artificial irrigation started early. The oldest
documents on the Waal system date from the 12th century, and some Waale built in the 14th century
are still in use today.
The pictures in this post show the Leitenwaal and the
Berkwaal near the village of
Schluderns in South Tyrol, northern Italy. These two
conduits carry water from a mountain stream (the Saldurbach) to the fields and meadows around
Schluderns. Along their combined length of about six kilometers, they overcome many obstacles:
twisting along the face of steep mountainsides, crossing aqueducts over deep ravines, tunnelling
underneath boulders, before they finally arrive at the fields they supply.
Some sections look almost like a natural stream – except that they flow across the mountainside,
not down, because they are designed to cover the greatest possible distance with the smallest
possible loss in altitude. Other sections are more obviously artificial, where the furrow has been
lined with flat stones or blanks of wood.
This system was originally built almost 700 years ago, using the technology available at the time:
spade, axe, hammer and chisel. Of course, nowadays, electric pumps can take water from the river
at the valley floor, and sprinkle it on the fields on the slopes above. But for many centuries, the
only feasible option was to take water from a stream at high altitude, and let it flow down from
Here a feed of water is taken from a stream, and carried along a wooden gulley: the input to the
irrigation system. Along the way, gates regulate the flow of water in the direction of various
farms. For centuries, the details of water distribution – how much water shall be directed towards
which farm at which time – have been governed by detailed agreements, and led to many disputes
If the system were to fail for too long, crops would wither, so it was important that the system was
always well-maintained and operational. And of course, parts of the system would fail from time to
time – erosion, landslides, decay, accidents or any number of other faults could occur. When a part
of the system broke, it was replaced using whatever technology was available at the time.
Thus, the system is now a patchwork of different water-carrying technologies from different ages.
The oldest “pipes” were made from hollowed-out tree trunks, and some of them are still in use (water
flows through tree trunks across a ravine in the left picture below). Later replacements have been
made with concrete, steel or plastic pipes – whatever is believed to be the most reliable solution
in the long term.
Perhaps the most impressive aspect of this system are its operability features, i.e. the things
that help the operator of the Waal in his job of keeping the system running smoothly. For example,
at regular intervals, the water flows through gratings which filter out twigs or other objects
before they can cause blockages in pipes. The gratings are cleaned regularly, and tools for clearing
out pipes are kept near the Waal. Routine inspections help detect problems early, before they
escalate and cause further damage.
After heavy rainfall or melting of snow, the influx of water may exceed the Waal’s capacity. This is
problematic: if the Waal bursts its banks, those banks would be damaged by erosion or washed away,
making the problem much worse. Thus, the system includes overflow points at which water is
channelled back into the natural stream if the Waal is over capacity (left photo below).
There is even an ingenious monitoring system (right photo below). A waterwheel is placed in the
stream, and a cowbell is attached so that it rings on each rotation of the wheel
(video). Thus, the operator can tell the rate of water
flow from a distance, simply by listening for the rhythm of the bell.
The Waaler, the operator in charge of maintenance of the Waal, is an important and
highly-regarded member of the local community. Traditionally, this role is elected every year on the
first Sunday of Lent. The operator can be re-elected by the community if they were satisfied with
his work in the previous year.
Looking at the lessons from this ancient irrigation system, and adapting them to software systems,
my take-aways are:
- Good interface design can survive through multiple generations of technology. A stream of water,
flowing downhill, is a simple interface that can be implemented in stone-lined furrows,
hollowed-out tree trunks, concrete, steel and plastic pipes, and more.
- When replacing obsolete technology with new technology, some work is required to join them up –
two pieces of standardised plastic piping may fit snugly, but you can’t expect the same from a
hollow tree trunk interfacing with a plastic pipe.
- New technology is not necessarily better than old technology. Hollow tree trunks are still used
to feed water into 21st-century sprinkler irrigation systems.
- API rate limits are not a new thing.
- Continuously monitor the health of your system, and detect problems early.
- Operations doesn’t just happen; it has to be someone’s job.
- If a system solves an important problem, is well-engineered and well-operated, it can stick around
for a very, very long time.