Pan-European Common Bird Monitoring Scheme (PECBMS)¶
Category: Ornithology · Size: 1.3 MB · Format: CSV, XLSX License: CC-BY-4.0 (Use open subset only; ES/CY 2016-17 restricted) · Zenodo record · Data sheet on the CSDH
Population indices for 170 breeding bird species across 28 European countries, produced by ~15,000 volunteers counting birds with standardised protocols every year.
The data is mounted read-only at /srv/data/pecbms-birds/.
Save anything you produce in your personal folder (~/).
The question: farmland birds vs forest birds¶
The Pan-European Common Bird Monitoring Scheme (PECBMS) turns millions of standardised counts by ~15,000 volunteers into a yearly population index for each species (the index is set to 100 in a baseline year, so 150 means "50% more birds than at the start").
A long-running conservation story is that birds of farmland have declined sharply as agriculture intensified, while birds of forest have held up much better. In this notebook we test that story directly with the open PECBMS tables: we build a composite trend for a group of farmland species and a group of forest species and compare them.
What's in the folder¶
Six small tables. We will mainly use two of them:
indices2017.csv— the supranational yearly index per species (this is the headline European trend we want).trends2017.csv— a one-line overall trend and classification per species (Stable,Moderate decline, ...).
from pathlib import Path
import pandas as pd
import numpy as np
DATA = Path('/srv/data/pecbms-birds')
for f in sorted(DATA.iterdir()):
print(f"{f.name:28s} {f.stat().st_size/1e3:7.1f} kB")
indices2017.csv 200.5 kB monitoring_schemes.xlsx 18.6 kB national_indices2017.csv 974.6 kB species_country.csv 13.0 kB trends2017.csv 13.1 kB trends_short2017.csv 32.8 kB
Load the two tables we need¶
indices2017.csv is long format: one row per species per year, with the index value
and its standard error. trends2017.csv is one row per species.
indices = pd.read_csv(DATA / 'indices2017.csv') # species, euring_code, year, index, se
trends = pd.read_csv(DATA / 'trends2017.csv') # species, ..., trend, class, note
print('indices:', indices.shape, '| years',
indices.year.min(), '-', indices.year.max(),
'|', indices.species.nunique(), 'species')
indices.head()
indices: (5628, 5) | years 1980 - 2017 | 170 species
| species | euring_code | year | index | se | |
|---|---|---|---|---|---|
| 0 | Tachybaptus ruficollis | 70 | 1990 | 100 | 0.0 |
| 1 | Tachybaptus ruficollis | 70 | 1991 | 77 | 18.0 |
| 2 | Tachybaptus ruficollis | 70 | 1992 | 106 | 26.0 |
| 3 | Tachybaptus ruficollis | 70 | 1993 | 103 | 21.0 |
| 4 | Tachybaptus ruficollis | 70 | 1994 | 128 | 27.0 |
# The classification column already summarises each species' fate across Europe.
trends['class'].value_counts()
class Moderate decline 66 Moderate increase 51 Stable 45 Uncertain 6 Steep decline 1 Strong increase 1 Name: count, dtype: int64
Assigning species to a habitat group¶
The tables have no habitat column, so we group species ourselves using the well-known PECBMS indicator lists (a subset that is present in this dataset). Farmland specialists live in open agricultural land (skylarks, partridges, buntings...); forest specialists depend on woodland (tits, treecreepers, woodpeckers...). Latin names must match the dataset's spelling exactly, so we filter to the ones actually present.
farmland = [
'Alauda arvensis', 'Anthus pratensis', 'Emberiza citrinella', 'Emberiza calandra',
'Linaria cannabina', 'Sturnus vulgaris', 'Vanellus vanellus', 'Perdix perdix',
'Passer montanus', 'Saxicola rubetra', 'Motacilla flava', 'Sylvia communis',
'Streptopelia turtur', 'Hirundo rustica', 'Lanius collurio', 'Galerida cristata',
]
forest = [
'Parus major', 'Cyanistes caeruleus', 'Fringilla coelebs', 'Sitta europaea',
'Certhia familiaris', 'Dendrocopos major', 'Regulus regulus', 'Phylloscopus collybita',
'Turdus philomelos', 'Erithacus rubecula', 'Coccothraustes coccothraustes',
'Poecile palustris', 'Periparus ater', 'Garrulus glandarius', 'Sylvia atricapilla',
'Phylloscopus sibilatrix', 'Ficedula hypoleuca',
]
present = set(indices.species)
farmland = [s for s in farmland if s in present]
forest = [s for s in forest if s in present]
group_of = {s: 'Farmland' for s in farmland} | {s: 'Forest' for s in forest}
print(f'{len(farmland)} farmland species, {len(forest)} forest species used')
16 farmland species, 17 forest species used
Building a composite index per group¶
Each species has its own baseline year, so we cannot just average raw index values. The standard PECBMS approach is a geometric mean of the species indices, which we then rebase to 100 in a common start year.
To keep every year comparable, we restrict to species that have a complete series from a shared start year onward, then take the geometric mean across the species in each group.
sel = indices[indices.species.isin(group_of)].copy()
sel['group'] = sel.species.map(group_of)
# Pick a common start year covered by all selected species, then keep full series only.
start_year = sel.groupby('species').year.min().max()
full = (sel[sel.year >= start_year]
.pivot_table(index='year', columns='species', values='index'))
full = full.dropna(axis=1) # keep species with a gap-free series
print('Common window:', start_year, '-', full.columns.size, 'species,',
full.index.min(), '-', full.index.max())
def composite(species_list):
cols = [s for s in species_list if s in full.columns]
sub = full[cols]
geo = np.exp(np.log(sub).mean(axis=1)) # geometric mean across species
return geo / geo.iloc[0] * 100 # rebase to 100 at start year
comp = pd.DataFrame({
'Farmland': composite(farmland),
'Forest': composite(forest),
})
comp.tail()
Common window: 1982 - 33 species, 1982 - 2017
| Farmland | Forest | |
|---|---|---|
| year | ||
| 2013 | 37.627878 | 111.998299 |
| 2014 | 37.145462 | 116.757777 |
| 2015 | 37.355516 | 116.276189 |
| 2016 | 37.749244 | 115.372108 |
| 2017 | 37.083841 | 119.863659 |
Chart 1 — the two composite trends¶
This is the headline comparison the challenge asks for.
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style='whitegrid')
fig, ax = plt.subplots(figsize=(9, 5))
ax.plot(comp.index, comp['Farmland'], marker='o', ms=3, lw=2, color='#d1495b', label='Farmland birds')
ax.plot(comp.index, comp['Forest'], marker='o', ms=3, lw=2, color='#2e7d32', label='Forest birds')
ax.axhline(100, color='grey', ls='--', lw=1)
ax.set_title(f'European common bird indices, {int(comp.index.min())} = 100')
ax.set_xlabel('Year'); ax.set_ylabel('Composite index (geometric mean)')
ax.legend()
plt.tight_layout(); plt.show()
change = (comp.iloc[-1] / comp.iloc[0] - 1) * 100
print(f"Net change {int(comp.index.min())}-{int(comp.index.max())}: "
f"Farmland {change['Farmland']:+.0f}% Forest {change['Forest']:+.0f}%")
Net change 1982-2017: Farmland -63% Forest +20%
The farmland composite sits clearly below the forest composite for most of the period —
consistent with the widely reported "farmland bird decline". Next we cross-check this with
the independent trend classification in trends2017.csv.
tsel = trends[trends.species.isin(group_of)].copy()
tsel['group'] = tsel.species.map(group_of)
order = ['Steep decline', 'Moderate decline', 'Uncertain', 'Stable',
'Moderate increase', 'Strong increase']
ct = (tsel.groupby(['group', 'class']).size()
.unstack(fill_value=0)
.reindex(columns=[c for c in order if c in tsel['class'].unique()]))
ct
| class | Steep decline | Moderate decline | Stable | Moderate increase |
|---|---|---|---|---|
| group | ||||
| Farmland | 1 | 12 | 2 | 1 |
| Forest | 0 | 4 | 4 | 9 |
Chart 2 — how many species decline vs increase, per group¶
share = ct.div(ct.sum(axis=1), axis=0) * 100 # % of species in each class
colors = ['#8b0000', '#d1495b', '#bbbbbb', '#f0c419', '#7bb661', '#2e7d32']
colors = [c for c, name in zip(colors, order) if name in share.columns]
ax = share.plot(kind='barh', stacked=True, figsize=(9, 3.2), color=colors)
ax.set_xlabel('% of species in the group'); ax.set_ylabel('')
ax.set_title('Trend classification by habitat group')
ax.legend(bbox_to_anchor=(1.01, 1), loc='upper left', fontsize=8)
plt.tight_layout(); plt.show()
Chart 3 — the individual species behind the averages¶
A composite can hide winners and losers. Here every species' overall trend (from
trends2017.csv, values <1 mean decline per year) is shown, coloured by group and sorted.
ind = tsel.sort_values('trend')
palette = ind['group'].map({'Farmland': '#d1495b', 'Forest': '#2e7d32'})
fig, ax = plt.subplots(figsize=(9, 8))
ax.barh(ind.species, ind.trend - 1, color=palette)
ax.axvline(0, color='k', lw=0.8)
ax.set_xlabel('Annual population trend (0 = stable, <0 = declining)')
ax.set_title('Per-species trends, farmland (red) vs forest (green)')
handles = [plt.Rectangle((0, 0), 1, 1, color=c) for c in ['#d1495b', '#2e7d32']]
ax.legend(handles, ['Farmland', 'Forest'], loc='lower right')
plt.tight_layout(); plt.show()
print('Median annual trend -> Farmland: %.4f Forest: %.4f'
% (ind[ind.group=='Farmland'].trend.median(),
ind[ind.group=='Forest'].trend.median()))
Median annual trend -> Farmland: 0.9752 Forest: 1.0044
Building a composite index per group¶
Each species has its own baseline year, so we cannot just average raw index values. The standard PECBMS approach is a geometric mean of the species indices, which we then rebase to 100 in a common start year.
To keep every year comparable, we restrict to species that have a complete series from a shared start year onward, then take the geometric mean across the species in each group.
sel = indices[indices.species.isin(group_of)].copy()
sel['group'] = sel.species.map(group_of)
# Pick a common start year covered by all selected species, then keep full series only.
start_year = sel.groupby('species').year.min().max()
full = (sel[sel.year >= start_year]
.pivot_table(index='year', columns='species', values='index'))
full = full.dropna(axis=1) # keep species with a gap-free series
print('Common window:', start_year, '-', full.columns.size, 'species,',
full.index.min(), '-', full.index.max())
def composite(species_list):
cols = [s for s in species_list if s in full.columns]
sub = full[cols]
geo = np.exp(np.log(sub).mean(axis=1)) # geometric mean across species
return geo / geo.iloc[0] * 100 # rebase to 100 at start year
comp = pd.DataFrame({
'Farmland': composite(farmland),
'Forest': composite(forest),
})
comp.tail()
Common window: 1982 - 33 species, 1982 - 2017
| Farmland | Forest | |
|---|---|---|
| year | ||
| 2013 | 37.627878 | 111.998299 |
| 2014 | 37.145462 | 116.757777 |
| 2015 | 37.355516 | 116.276189 |
| 2016 | 37.749244 | 115.372108 |
| 2017 | 37.083841 | 119.863659 |
Chart 1 — the two composite trends¶
This is the headline comparison the challenge asks for.