Bird Identification Accuracy in eBird Citizen Science Data (Argentina)¶
Category: Ornithology · Size: 2.7 MB · Format: HTML, RAR, XLSX License: CC-BY-4.0 · Zenodo record · Data sheet on the CSDH
Supplementary materials to quantify the accuracy of photographic bird identifications in eBird Argentina, with per-species ranking and network analysis.
The data is mounted read-only at /srv/data/ebird-id-accuracy/.
Save anything you produce in your personal folder (~/).
What's in the dataset¶
This is the supplementary material from a study that measured how accurately eBird volunteers identify birds from photographs in Argentina. The archive holds three files:
Appendix_S2.xlsx— the per-species ranking table (the core of this notebook).Appendix_S3.html— an interactive confusion network (which species get mistaken for which). We'll pull the raw nodes/edges straight out of the HTML.Appendix_S1.rar— the raw records, compressed (we won't need it here).
Everything is small (< 3 MB), so it loads comfortably in memory.
from pathlib import Path
DATA = Path('/srv/data/ebird-id-accuracy')
for f in sorted(DATA.rglob('*')):
if f.is_file():
print(f"{f.relative_to(DATA)} ({f.stat().st_size/1e6:,.2f} MB)")
Appendix_S1.rar (1.76 MB) Appendix_S2.xlsx (0.04 MB) Appendix_S3.html (0.93 MB)
Load the per-species ranking¶
Appendix_S2.xlsx has one sheet, species_rank, with one row per species. For each
species the authors report how many photo-records were checked (samples) and how the
crowd's identifications scored against expert review:
true_positives/false_positives/false_negatives— the confusion counts.precision= of the photos called this species, how many really were (100 = no false alarms).recall= of the photos that really were this species, how many the crowd caught (100 = none missed).min (precision, recall)— a single conservative accuracy score (the worse of the two).quality_group— the authors' banding into high / moderate / low quality.hard_to_id— a flag the authors set for species with known look-alikes.
import pandas as pd
df = pd.read_excel(DATA / 'Appendix_S2.xlsx', sheet_name='species_rank')
df = df.rename(columns={'min (precision, recall)': 'accuracy'})
print(df.shape, 'species')
df.head()
(377, 11) species
| rank | quality_group | scientific_name | hard_to_id | samples | true_positives | false_positives | false_negatives | precision | recall | accuracy | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 1 | high-quality | Polioptila dumicola | False | 773 | 773 | 0 | 0 | 100.0 | 100.0 | 100.0 |
| 1 | 2 | high-quality | Turdus chiguanco | False | 458 | 458 | 0 | 0 | 100.0 | 100.0 | 100.0 |
| 2 | 3 | high-quality | Amblyramphus holosericeus | False | 443 | 443 | 0 | 0 | 100.0 | 100.0 | 100.0 |
| 3 | 4 | high-quality | Coryphistera alaudina | False | 385 | 385 | 0 | 0 | 100.0 | 100.0 | 100.0 |
| 4 | 5 | high-quality | Tachuris rubrigastra | False | 287 | 287 | 0 | 0 | 100.0 | 100.0 | 100.0 |
df.info()
df[['samples', 'precision', 'recall', 'accuracy']].describe().round(1)
<class 'pandas.core.frame.DataFrame'> RangeIndex: 377 entries, 0 to 376 Data columns (total 11 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 rank 377 non-null int64 1 quality_group 377 non-null object 2 scientific_name 377 non-null object 3 hard_to_id 377 non-null bool 4 samples 377 non-null int64 5 true_positives 377 non-null int64 6 false_positives 377 non-null int64 7 false_negatives 377 non-null int64 8 precision 377 non-null float64 9 recall 377 non-null float64 10 accuracy 377 non-null float64 dtypes: bool(1), float64(3), int64(5), object(2) memory usage: 29.9+ KB
| samples | precision | recall | accuracy | |
|---|---|---|---|---|
| count | 377.0 | 377.0 | 377.0 | 377.0 |
| mean | 183.3 | 97.5 | 98.0 | 96.6 |
| std | 277.6 | 4.8 | 3.9 | 5.4 |
| min | 6.0 | 64.3 | 63.6 | 63.6 |
| 25% | 32.0 | 97.4 | 97.6 | 95.6 |
| 50% | 78.0 | 99.6 | 100.0 | 98.9 |
| 75% | 210.0 | 100.0 | 100.0 | 100.0 |
| max | 1878.0 | 100.0 | 100.0 | 100.0 |
A quick sanity check: most species are identified almost perfectly (the median accuracy is very high), but there's a long tail of problem species. And the crowd is graded on two very different sample sizes — some species have thousands of vetted photos, others only a handful.
print('Median accuracy: {:.1f}%'.format(df['accuracy'].median()))
print('Species scoring a perfect 100:', (df['accuracy'] == 100).sum(), 'of', len(df))
print('\nquality_group counts:')
print(df['quality_group'].value_counts())
print('\nhard_to_id (has look-alikes) counts:')
print(df['hard_to_id'].value_counts())
Median accuracy: 98.9% Species scoring a perfect 100: 122 of 377 quality_group counts: quality_group high-quality 291 moderate-quality 46 low-quality 40 Name: count, dtype: int64 hard_to_id (has look-alikes) counts: hard_to_id False 292 True 85 Name: count, dtype: int64
Which species are hardest to identify?¶
The scientific question is: is it the rare species (few photos) that trip volunteers up,
or the look-alikes (species with confusable relatives)? Let's start by simply ranking the
20 worst species by accuracy, and colour each bar by whether the authors flagged it as a
look-alike (hard_to_id).
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style='whitegrid')
worst = df.sort_values('accuracy').head(20).iloc[::-1]
colors = worst['hard_to_id'].map({True: '#d1495b', False: '#4a7ba6'})
fig, ax = plt.subplots(figsize=(9, 7))
ax.barh(worst['scientific_name'], worst['accuracy'], color=colors)
ax.set_xlabel('Accuracy — min(precision, recall) %')
ax.set_xlim(60, 100)
ax.set_title('The 20 hardest-to-identify species in eBird Argentina')
for y, (acc, n) in enumerate(zip(worst['accuracy'], worst['samples'])):
ax.text(acc + 0.3, y, f'n={n}', va='center', fontsize=8, color='#444')
handles = [plt.Rectangle((0,0),1,1,color='#d1495b'),
plt.Rectangle((0,0),1,1,color='#4a7ba6')]
ax.legend(handles, ['flagged look-alike (hard_to_id)', 'not flagged'], loc='lower right')
plt.tight_layout()
plt.show()
Most of the worst performers are red — species the authors already flagged as having
look-alikes. The n= labels also show these aren't only rare species; several have dozens
of vetted photos. Let's test both ideas directly.
Rarity vs. look-alikes: which explains the errors?¶
If rarity were the driver, accuracy would rise steeply with sample size. If look-alikes
were the driver, the flagged (hard_to_id) species would score much lower regardless of how
many photos they have. We plot accuracy against sample size (log scale) and split by the flag.
import numpy as np
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(13, 5))
# Left: accuracy vs sample size
for flag, sub in df.groupby('hard_to_id'):
ax1.scatter(sub['samples'], sub['accuracy'], s=22, alpha=0.6,
color='#d1495b' if flag else '#4a7ba6',
label='look-alike' if flag else 'not flagged')
ax1.set_xscale('log')
ax1.set_xlabel('Number of vetted photos (log scale)')
ax1.set_ylabel('Accuracy %')
ax1.set_title('Accuracy vs. how common the species is')
ax1.legend()
r = np.corrcoef(np.log10(df['samples']), df['accuracy'])[0, 1]
ax1.text(0.05, 0.06, f'corr(log samples, accuracy) = {r:.2f}',
transform=ax1.transAxes, fontsize=9, color='#444')
# Right: accuracy distribution by flag
sns.boxplot(data=df, x='hard_to_id', y='accuracy', hue='hard_to_id',
palette=['#4a7ba6', '#d1495b'], legend=False, ax=ax2)
ax2.set_xticks([0, 1]); ax2.set_xticklabels(['not flagged', 'look-alike (hard_to_id)'])
ax2.set_xlabel('')
ax2.set_ylabel('Accuracy %')
ax2.set_title('Accuracy by look-alike flag')
plt.tight_layout()
plt.show()
print('Mean accuracy — look-alikes: {:.1f}% others: {:.1f}%'.format(
df.loc[df.hard_to_id, 'accuracy'].mean(),
df.loc[~df.hard_to_id, 'accuracy'].mean()))
Mean accuracy — look-alikes: 90.9% others: 98.2%
The story is clear. Sample size barely matters (weak correlation ~0.2), so rarity is not the main culprit. But the flagged look-alikes sit a full ~7 points lower on average and fill almost the entire low-accuracy tail. Confusable relatives — not rarity — drive the errors.
Who gets confused with whom?¶
Appendix_S3.html is an interactive network the authors built (an R visNetwork widget).
We don't need a browser: the node and edge data are embedded as JSON inside the file. Each
edge is a directed "species A was misidentified as species B", weighted by how often it
happened. Let's extract it and see whether confusions stay within the same bird family
(the tell-tale signature of look-alike errors).
import re, json
html = (DATA / 'Appendix_S3.html').read_text(encoding='utf-8')
payload = re.search(r'<script type="application/json"[^>]*>(.*?)</script>', html, re.S)
x = json.loads(payload.group(1))['x']
nodes = pd.DataFrame({'id': x['nodes']['id'],
'species': x['nodes']['label'],
'family': x['nodes']['family']})
edges = pd.DataFrame({'from': x['edges']['from'],
'to': x['edges']['to'],
'weight': x['edges']['weight']})
fam = dict(zip(nodes['id'], nodes['family']))
lab = dict(zip(nodes['id'], nodes['species']))
edges['same_family'] = edges['from'].map(fam) == edges['to'].map(fam)
print(f"{len(nodes)} species, {len(edges)} confusion links across {nodes['family'].nunique()} families")
wframe = edges.groupby('same_family')['weight'].sum()
print(f"Share of confusions that stay within the same family: "
f"{wframe[True] / wframe.sum():.0%}")
272 species, 450 confusion links across 27 families Share of confusions that stay within the same family: 88%
Almost 9 in 10 misidentification events happen between members of the same family — overwhelming evidence that look-alikes, not rare oddities, cause the mistakes. Finally, let's see which individual species are the biggest confusion hubs by summing every link that touches them.
deg = (edges.groupby('to')['weight'].sum()
.add(edges.groupby('from')['weight'].sum(), fill_value=0)
.sort_values(ascending=False))
deg.index = deg.index.map(lab)
top = deg.head(15).iloc[::-1]
fig, ax = plt.subplots(figsize=(9, 6))
ax.barh(top.index, top.values, color='#e08a3c')
ax.set_xlabel('Total confusion weight (times involved in a mix-up)')
ax.set_title('Biggest confusion hubs in eBird Argentina')
plt.tight_layout()
plt.show()
top_pair = edges.sort_values('weight', ascending=False).head(5).copy()
top_pair['from'] = top_pair['from'].map(lab); top_pair['to'] = top_pair['to'].map(lab)
print('Most frequent single confusions (A mistaken for B):')
for _, row in top_pair.iterrows():
print(f" {row['from']} -> {row['to']} (x{int(row['weight'])})")
Most frequent single confusions (A mistaken for B): Molothrus bonariensis -> Molothrus rufoaxillaris (x26) Anthus correndera -> Anthus hellmayri (x22) Synallaxis frontalis -> Synallaxis albescens (x16) Molothrus rufoaxillaris -> Molothrus bonariensis (x15) Cinclodes atacamensis -> Cinclodes albiventris (x15)
The hubs read like a field-guide's list of infamous confusion pairs: the two Molothrus cowbirds, the Anthus pipits, the Asthenes canasteros. These are exactly the "little brown jobs" and near-identical congeners that challenge even experienced birders — confirming that eBird's photo-ID errors concentrate where nature itself makes species hard to tell apart.
Your turn¶
This is just the starting point. Some ideas:
- Check the dataset challenge on its CSDH data sheet.
- Work on a copy: right-click the file → Duplicate (or Save Notebook As…). Your changes only live in your Hub space — they're never pushed to GitHub.
- Edited this notebook and want the original back? Use the Restore cell
below (or the
restore.ipynbnotebook). - Questions and results: on the platform forum.
Attribution: data from Bird Identification Accuracy in eBird Citizen Science Data (Argentina), license CC-BY-4.0. Notebook from the Citizen Science Data Hub (CSDH) — Fundación Ibercivis.
# ⚠️ RESTORE: this DISCARDS YOUR CHANGES to this notebook and resets it to the original.
# 1. Uncomment the line below (remove the #) 2. Run this cell
# 3. Then: menu File → Reload Notebook from Disk
# !git -C ~/citizen-science-data fetch -q origin && git -C ~/citizen-science-data checkout origin/main -- ebird-id-accuracy.ipynb && echo "Restored. Now: File → Reload Notebook from Disk"