The Guardian - Scottish independence referendum live dashboard

Three mobile screens showing the national gauge, interactive council map with Dumfries and Galloway selected, and the sortable results table
The responsive mobile views: the national vote gauge, the council map and search panel, and the sortable results table.

In September 2014, more than 3.6 million people (an 84.6% turnout) voted on the question: "Should Scotland be an independent country?" Working in The Guardian's Interactive team alongside another frontend developer, I built the live results dashboard in D3.js and SVG, as well as the Node.js and AWS S3 data pipeline that powered our coverage through the night.

The Challenge

As the polls closed on 18 September, all 32 Scottish council areas counted ballots overnight and declared their results one by one into the early hours of the morning. Our dashboard had to give readers an up-to-the-minute picture of both the national race to the 50% threshold and the local breakdown across every council.

We had two main engineering challenges:

The Stack

The Solution

Data Ingestion Pipeline and CDN Caching

Before referendum night, I worked with our external election data supplier to integrate their live results feed. They published updates as compressed .zip archives containing CSV files to an SFTP server. Because dozens of media organisations were polling the same server all night, connections frequently timed out.

I built a Node.js service on an AWS EC2 instance that ran on a timer to:

  1. Fetch & sort: Connect to the SFTP server with timeout and retry logic, check the directory listing, and sort drops by timestamp so we only downloaded the newest archive.
  2. Decompress & transform: Unzip the archive, parse the CSV rows, and merge them with our newsroom's Google Spreadsheet into a single normalised JSON format.
  3. Publish to S3: Upload the JSON to an Amazon S3 bucket behind our CDN with a short cache TTL. The CDN absorbed the massive read traffic from millions of open browsers, while still delivering new council declarations within seconds.

On the client side, the dashboard polled the CDN JSON endpoint every 30 seconds. To avoid unnecessary DOM updates and SVG redraws when no new councils had declared, the data layer deep-compared each incoming payload against the previous state in memory and only triggered a re-render when the numbers changed. It also sanitised messy inputs (stripping formatting, guarding against spreadsheet formula errors, handling recount states, and retrying failed fetches) with a manual JSON fallback ready in case of an emergency.

Synchronised D3.js and SVG Views

Desktop view of The Guardian's Scottish independence referendum dashboard showing the national half-donut gauge, electorate progress bar, and choropleth council map
The desktop dashboard showing the national vote share, electorate-weighted declaration progress, and interactive council map.

Each visual component subscribed to a central pub/sub event bus, allowing my teammate and me to build and test the views as independent modules:

Referendum Night in the Newsroom

In 2014, Guardian interactives still needed to work reliably in Internet Explorer 8 and 9 alongside modern mobile and desktop browsers. During testing, we worked around legacy IE bugs, including writing a custom XML-to-DOM node importer when IE9 failed to import the external SVG map, and adding cross-domain request fallbacks for polling S3.

On the night of the vote, I worked the overnight shift in the newsroom until 5am alongside the journalists and the Head of Interactive, monitoring the EC2 ingestion service and live dashboard. As councils declared through the early hours, watching the real-time traffic graphs climb higher and higher while hundreds of thousands of readers had the page open with the auto-refresh quietly ticking away was one of the highlights of my time at The Guardian. While the dashboard itself was built specifically for the Scottish vote, the data pipeline and caching approach shaped how we delivered subsequent live election interactives.

Technical Contributions