iSCAPE Citizen Science Workshops Air Quality Data

Category: Air Quality · Size: 147.1 MB · Format: CSV, YAML License: CC0-1.0 · Zenodo record · Data sheet on the CSDH

Air-quality sensor data (temperature, humidity, PM, noise, light) collected in citizen science workshops across 6 European cities: Vantaa, Dublin, Bologna, Bottrop, Hasselt and Guildford.

The data is mounted read-only at /srv/data/iscape-air-quality/. Save anything you produce in your personal folder (~/).

What's in the dataset

The iSCAPE citizen-science workshops handed low-cost Smart Citizen sensor kits to volunteers in six European cities. Each kit logged temperature, humidity, light, noise and — on most kits — particulate matter (PM) roughly once a minute.

The folder holds two kinds of file:

• <id>.csv — one file per sensor kit (the number is the device id). Columns vary between kits, but most share Time, EXT_PM_1/10/25 (PM in µg/m³), NOISE_A (dB(A)), TEMP, HUM, PRESS, LIGHT.
• test_description_*.yaml — one file per city, listing which device ids belong to that city and the local timezone (Europe/Dublin, Europe/Rome, …).

So there is no single "cities" column: we have to join the kits to their city using the YAML files. Let's build that map first.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import yaml
from pathlib import Path

sns.set_theme(style="whitegrid")
DATA = Path('/srv/data/iscape-air-quality')

# Build device_id -> (city, timezone) from the per-city YAML descriptions
device_city = {}
device_tz   = {}
for y in sorted(DATA.glob('test_description_*.yaml')):
    desc = yaml.safe_load(y.open())
    city = y.stem.split('WORKSHOP_')[1].title()   # e.g. 'Dublin'
    for dev_id, meta in desc.get('devices', {}).items():
        device_city[dev_id] = city
        device_tz[dev_id]   = meta.get('location', 'UTC')

cities = sorted(set(device_city.values()))
print(f"{len(device_city)} sensor kits across {len(cities)} cities: {', '.join(cities)}")
pd.Series(device_city).value_counts().rename('kits').to_frame()

91 sensor kits across 6 cities: Bologna, Bottrop, Dublin, Hasselt, Surrey, Vantaa

	kits
Bologna	38
Bottrop	15
Hasselt	11
Vantaa	11
Surrey	9
Dublin	7

Load only what we need, across every kit

The full dataset is ~147 MB spread over ~90 files (the biggest kit alone has >500k rows). To stay well inside the memory budget we read only the columns we care about and stack all kits into one tidy long table, tagging each row with its city and local hour.

Two details matter for air-quality work:

• Timestamps carry a timezone offset. We parse them as UTC, then convert to each city's local time — otherwise "rush hour" would be smeared across timezones.
• Only kits that actually recorded a sensor contribute that column (a few kits have no PM).

WANT = {'Time', 'EXT_PM_25', 'NOISE_A', 'TEMP'}

frames = []
for csv in sorted(DATA.glob('*.csv')):
    dev = csv.stem
    if dev not in device_city:
        continue
    # usecols callable keeps whichever of WANT exist in this particular kit
    d = pd.read_csv(csv, usecols=lambda c: c in WANT)
    if 'Time' not in d.columns:
        continue
    # parse as UTC (handles mixed DST offsets) then move to the city's wall-clock time
    t = pd.to_datetime(d['Time'], utc=True, errors='coerce').dt.tz_convert(device_tz[dev])
    d = d.drop(columns='Time')
    d['city'] = device_city[dev]
    d['device'] = dev
    d['hour'] = t.dt.hour
    d['date'] = t.dt.date
    frames.append(d)

df = pd.concat(frames, ignore_index=True)
# force numeric, drop rows with no usable timestamp
df = df[df['hour'].notna()].copy()
for c in ('EXT_PM_25', 'NOISE_A', 'TEMP'):
    df[c] = pd.to_numeric(df.get(c), errors='coerce')

print(f"{len(df):,} sensor readings loaded")
df.head()

1,964,666 sensor readings loaded

	EXT_PM_25	NOISE_A	TEMP	city	device	hour	date
0	17.0	48.84	18.83	Dublin	10361	20	2019-10-19
1	11.0	54.19	18.90	Dublin	10361	20	2019-10-19
2	4.0	51.72	21.09	Dublin	10361	12	2019-10-20
3	2.0	48.45	21.12	Dublin	10361	12	2019-10-20
4	2.0	50.48	21.13	Dublin	10361	12	2019-10-20

How much data does each city have?

Before drawing conclusions it helps to see the raw coverage: some cities ran many kits for weeks, others only a handful of workshop days. We also do a light sanity clean — PM and noise sensors occasionally spit out impossible values.

# Light physical-plausibility filter (values outside these ranges are sensor glitches)
df.loc[(df['EXT_PM_25'] < 0) | (df['EXT_PM_25'] > 1000), 'EXT_PM_25'] = np.nan
df.loc[(df['NOISE_A']   < 20) | (df['NOISE_A']   > 130), 'NOISE_A']   = np.nan

coverage = df.groupby('city').agg(
    kits=('device', 'nunique'),
    readings=('hour', 'size'),
    pm25_readings=('EXT_PM_25', 'count'),
    noise_readings=('NOISE_A', 'count'),
    median_pm25=('EXT_PM_25', 'median'),
    median_noise=('NOISE_A', 'median'),
).round(1)
coverage

	kits	readings	pm25_readings	noise_readings	median_pm25	median_noise
city
Bologna	38	55022	54658	55022	20.0	51.5
Bottrop	15	620738	618808	615146	4.0	48.5
Dublin	7	124575	122672	123652	7.0	51.3
Hasselt	11	18929	18328	18929	14.5	48.7
Surrey	9	216650	214972	216650	11.0	53.0
Vantaa	11	928752	928002	882109	4.0	54.4

Does PM2.5 follow a daily rhythm — and is it shared across cities?

Urban PM2.5 is driven partly by traffic, cooking and heating, so we expect a daily cycle. We average PM2.5 by local hour of day for each city and overlay the profiles. A shared morning/evening bump would point to a common (traffic-like) driver; flat or offset curves point to local factors.

The dashed line marks the WHO 24-hour guideline of 15 µg/m³ for reference.

hourly_pm = df.dropna(subset=['EXT_PM_25']).groupby(['city', 'hour'])['EXT_PM_25'].mean().unstack('city')

fig, ax = plt.subplots(figsize=(10, 5))
hourly_pm.plot(ax=ax, marker='o', ms=3, linewidth=1.8)
ax.axhline(15, ls='--', color='crimson', alpha=.7, label='WHO 24h guideline (15)')
ax.set_xlabel('Hour of day (local time)')
ax.set_ylabel('Mean PM2.5  (µg/m³)')
ax.set_title('Daily PM2.5 profile by city')
ax.set_xticks(range(0, 24, 2))
ax.legend(title='', fontsize=8, ncol=2)
plt.tight_layout()
plt.show()

The noise rush-hour, seen as a heatmap

Noise is a more direct proxy for street activity than PM. A city-by-hour heatmap of mean NOISE_A makes any shared daytime-loud / night-quiet pattern jump out, and lets us spot cities that break the rule.

hourly_noise = df.dropna(subset=['NOISE_A']).groupby(['city', 'hour'])['NOISE_A'].mean().unstack('hour')

fig, ax = plt.subplots(figsize=(11, 4))
sns.heatmap(hourly_noise, cmap='rocket_r', annot=False,
            cbar_kws={'label': 'Mean noise dB(A)'}, ax=ax)
ax.set_xlabel('Hour of day (local time)')
ax.set_ylabel('')
ax.set_title('Hourly noise level by city')
plt.tight_layout()
plt.show()

How different are the cities overall?

Finally, a distribution view. The boxplot compares the spread of PM2.5 across cities (sorted by median), which is what you would use to say whether one city's air was cleaner than another during its workshop — bearing in mind the coverage differences we saw above.

order = df.groupby('city')['EXT_PM_25'].median().sort_values().index
fig, ax = plt.subplots(figsize=(10, 5))
sns.boxplot(data=df, x='city', y='EXT_PM_25', order=order,
            showfliers=False, hue='city', hue_order=order, legend=False,
            palette='crest', ax=ax)
ax.axhline(15, ls='--', color='crimson', alpha=.7)
ax.set_xlabel('')
ax.set_ylabel('PM2.5  (µg/m³)')
ax.set_title('PM2.5 distribution by city  (outliers hidden; dashed = WHO 15 µg/m³)')
plt.tight_layout()
plt.show()

Working with data larger than memory

Your session has a 4 GB RAM limit, but you can analyse files of 10 GB or more without loading them whole:

• Read only the columns you need: pd.read_csv(f, usecols=[...]) / pd.read_parquet(f, columns=[...]).
• Process in chunks and keep only the result:

  total = 0
  for chunk in pd.read_csv(file, chunksize=1_000_000):
      total += len(chunk)
  

• Query with SQL without loading anything — DuckDB (already installed) reads CSV and Parquet straight from disk and only brings the result into memory:

  import duckdb
  duckdb.sql("SELECT column, count(*) FROM '/srv/data/.../file.parquet' GROUP BY column").df()
  

Your turn

This is just the starting point. Some ideas:

• Check the dataset challenge on its CSDH data sheet.
• Work on a copy: right-click the file → Duplicate (or Save Notebook As…). Your changes only live in your Hub space — they're never pushed to GitHub.
• Edited this notebook and want the original back? Use the Restore cell below (or the restore.ipynb notebook).
• Questions and results: on the platform forum.

Attribution: data from iSCAPE Citizen Science Workshops Air Quality Data, license CC0-1.0. Notebook from the Citizen Science Data Hub (CSDH) — Fundación Ibercivis.

# ⚠️ RESTORE: this DISCARDS YOUR CHANGES to this notebook and resets it to the original.
# 1. Uncomment the line below (remove the #)   2. Run this cell
# 3. Then: menu File → Reload Notebook from Disk

# !git -C ~/citizen-science-data fetch -q origin && git -C ~/citizen-science-data checkout origin/main -- iscape-air-quality.ipynb && echo "Restored. Now: File → Reload Notebook from Disk"