To Do
=====
tomtit cert 70
+_ species (ie no calltype)
Tui/Bellbird
silvereye/pipipi
rifleman/tomtit+contact/Tieke
Go through birdnet categories sample and try to work out what they are
Look at kiwi dataset
New Dataset
test database line update with index+fk v fk only
Read audio tool (pointless atm as most models can't use it)
Bounding Box script.py
to one hot encoded csv for opensoundscape (because python is so slow, and I would have to convert to raven selection.txt first)
day -> civil sunrise to !!civil sunset!!
claude --resume "reject-reserved-key-bindings"
multi label in tui. How?? also cli
Clip from wav when no .data file—skraak save image????
find morepork mewing sound for dataset
segment unstructured import into batches of 10000 files to keep within buffer limits, structured imports should be fine as we are talking 1 sd card (24/7 its 16000 max)
ingest my training datasets
buy a drive to backup mac ~
Update tools could allow setting active to false?? Currently do not
Make freebird to .data tool
npm install -g @mariozechner/pi-coding-agent
SKILLS
======
project/.claude/skills for most then link to project/.agents/skills for pi with:
find .claude/skills -type f -exec bash -c 'mkdir -p "$(dirname ".pi/skills/${1#.claude/skills/}")" && ln -s "$PWD/$1" "$PWD/.pi/skills/${1#.claude/skills/}"' _ {} \;
pi-specific are in ~ somewhere (ok because keeps them seperate) if installed with eg: $pi install npm:@tmustier/pi-ralph-wiggum
call-library: currently have a hard copy in .claude symlinked to .pi
Labels in opensoundscape multi-species model
=====================
ausbit1 Australasian Bittern
bluduc1 Blue Duck
comcha Common Chaffinch
comred Redpoll (Common)
dunnoc1 Dunnock
eurbla Eurasian Blackbird
eursta European Starling
fernbi1 New Zealand Fernbird
grskiw1 Great Spotted Kiwi/Roroa
gryger1 Gray Gerygone/Grey Warbler
kea1 Kea
liskiw1 Little Spotted Kiwi/Kiwi pukupuku
lotkoe1 Long-tailed Koel/Cuckoo
morepo2 Morepork
nezbel1 New Zealand Bellbird
nezfan1 New Zealand Fantail/Piwakawaka
nezkak1 New Zealand Kaka
nezpig2 New Zealand Pigeon/Kereru
nezrob3 South Island Robin/Kakaruai
nibkiw1 North Island Brown Kiwi/Kiwi-nui
okbkiw1 Okarito Brown Kiwi/Rowi
parake parakeet sp./Kakariki
pipipi1 Pipipi/Brown Creeper
riflem1 Rifleman
saddle3 South Island Saddleback?Tieke
shbcuc1 Shining Bronze-Cuckoo
silver3 Silvereye
sobkiw2 Southern Brown Kiwi (South I.)/Tokoeka
soioys1 South Island Oystercatcher
soiwre1 South Island Wren
sonthr1 Song Thrush
spocra2 Spotless Crake
tomtit1 Tomtit/Miromiro
tui1 Tui
varoys1 Variable Oystercatcher
weka1 Weka
yellow2 Yellowhammer
weta Weta (not bird)
cangoo1 Canada Goose
# Active DB Labels ebird_code
------------------ ----------
Australasian Bittern ausbit1 x
Bellbird nezbel1 x
Chaffinch comcha x
Crake_Spotless spocra2 x
Cuckoo_Shining shbcuc1 x
Duck_Blue_Whio bluduc1 x
Dunnock_Hedge_Sparrow dunnoc1 x
Eurasian Blackbird eurbla x
European Starling eursta x
Fantail nezfan1 x
Fernbird fernbi1 x
Haast Tokoeka sobkiw2 x
Kaka nezkak1 x
Kea kea1 x
Kereru nezpig2 x
Kiwi pukupuku liskiw1 x
Kiwi_Nth_Is_Brown nibkiw1 x
Long-tailed Koel lotkoe1 x
Morepork morepo2 x
Oystercatcher_Variable varoys1 x
Parakeet parake x
Pipipi pipipi1 x Brown Creeper
Redpoll comred x
Rifleman riflem1 x
Robin_Sth_Is nezrob3 x
Roroa grskiw1 x
Rowi okbkiw1 x
S. Fiordland Tokoeka sobkiw1 x
Saddleback_Sth_Is saddle3 x
Silvereye silver3 x
South Island Oystercatcher soioys1 x
South Island Wren soiwre1 X
Thrush_Song sonthr1 x
Tomtit tomtit1 x
Tui tui1 x
Warbler_Grey gryger1 x
Weka_spp weka1 x
Yellowhammer yellow2 x
Check
Don't Know
Fake Kiwi
Korero Gecko x
Question
Weta x
Noise
Keybindings
===========
see ~/.skraak/config.json
TUI cmd
=======
skraak calls classify --folder . --filter opensoundscape-multi-1.0 --species comcha
David's Kiwi Workflow
=====================
- cp data to main drives
- backup audio
- skraak import bulk to get files into db
- Run opensoundscape models on audio
- skraak calls from-preds to make .data files
- Run julia DFMN model (also LSK model for Inge)
- skraak calls classify TUI for kiwi on 1 model
- use minimax to check "Don't Know"
- skraak calls propogate on other models
- use minimax on cert 70 Kiwi and maybe Don't Know
- skraak calls classify on remaining cert 70 Kiwi
- skraak calls classify --sample 10 on cert 90 Kiwi
- skraak calls push-certainty on remaining cert 90 Kiwi if all good
- use minimax skill /detect-anomalies to correct problems
- skraak calls classify to resolve certainty mismatches
- skraak calls summarise
- run skill /data-mapping
- run skill /import-segments
Code stuff
==========
time ./skraak calls from-preds --csv /media/david/SSD4/Twenty_Four_Seven/R620/2024-05-06/preds9_opensoundscape-multi-1.0_2025-07-22.csv > /media/david/SSD4/Twenty_Four_Seven/R620/2024-05-06/preds9_opensoundscape-multi-1.0_2025-07-22.json
for item in a # **options selected for kiwi work**
try
jsonfile = replace(item, ".csv" => ".json")
run(pipeline(`skraak calls from-preds --csv $item --gap-multiplier 3 --min-detections 1`, jsonfile))
catch e
@error "skraak failed on $item" exception=(e, catch_backtrace())
end
end
model = "/media/david/SSD2/Secondary_Models/DFMN_Inge/model_DFMN1-5_CPU_epoch-9-0.9737-2024-10-25.jld2"
labels = Dict(1 => "Duet", 2 => "Female", 3 => "Male", 4 => "Don't Know")
## Check this logic in the code
predict(a, model, labels)
model = "/media/david/SSD2/Secondary_Models/LSK/model_GSK_LSK_DFM_FT_IngeDFMN_1-5_1-0_CPU_epoch-9-0.9745-2025-01-13.jld2"
labels = Dict(1 => "GSK", 2 => "GSK", 3 => "GSK", 4 => "LSK", 5 => "LSK", 6 => "LSK")
## Needed to change the logic
predict(a, model, labels)
model = "/media/david/SSD2/Secondary_Models/DFMN_Pomona/model_DFMN1-5_Pomona3_CPU_epoch-18-0.9785-2025-03-02.jld2"
labels = Dict(1 => "Duet", 2 => "Female", 3 => "Gecko", 4 => "Male", 5 => "Don't Know")
## Check this logic in the code
predict(a, model, labels)
## Change the date
for item in x
try
jsonfile = "$item/segment_summary_2026-05-08.json"
run(pipeline(`skraak calls summarise --folder $item`, jsonfile))
catch e
@error "skraak failed on $item" exception=(e, catch_backtrace())
end
end
skraak calls summarise --folder ./recordings --brief
# print brief summary to repl
for item in a
try
run(pipeline(`skraak calls summarise --folder $item --brief`))
catch e
@error "skraak failed on $item" exception=(e, catch_backtrace())
end
end
# save brief summary to ~
open("/home/david/summary_2026-05-08.jsonl", "w") do f
for item in x
try
run(pipeline(`skraak calls summarise --folder $item`, `jq 'del(.segments)'`, f))
catch e
@error "skraak failed on $item" exception=(e, catch_backtrace())
end
end
end
OLLAMA
======
ollama run gemma4:e4b
ollama launch pi --model gemma4:e4b # don't do this, it alters pi config
ollama run qwen3.5:9b # uninstalled
ollama list
ollama rm <model-name>
ollama rm qwen3.5:9b
R620/2024-05-06 only
Run Through Gemma
Opensoundscape Hand Classified BirdNET Hand Classified
============== =============== ======= ===============
comcha X X X
eurbla X X X
gryger1 X X none? X White-throated Sparrow (auto), Gray Gerygone
nezfan1 X X NZ Fantail
tomtit1 V. Bad garbage X
nezrob1 X X SI Robin (no types)
kereru
rifleman
silvereye
bellbird
tui
nezkak1 V. Bad(gecko, wing) V Bad, ongoing bellbird
weka1 V. Bad(noise) none
morepo2 many Gecko Also Gecko
lotkoe1 X X X
┌──────┬───────────────────────────┬───────┐
│ Rank │ Species │ Count │
├──────┼───────────────────────────┼───────┤
│ 1 │ White-throated Sparrow │ 5163 │ Gryger
├──────┼───────────────────────────┼───────┤
│ 2 │ New Zealand Bellbird │ 3812 │
├──────┼───────────────────────────┼───────┤
│ 3 │ Superb Lyrebird │ 3645 │ nezbel1+territorial
├──────┼───────────────────────────┼───────┤
│ 4 │ Common Crossbill │ 3247 │
├──────┼───────────────────────────┼───────┤
│ 5 │ Javan Shortwing │ 2824 │
├──────┼───────────────────────────┼───────┤
│ 6 │ Grey Gerygone │ 2286 │ Gryger
├──────┼───────────────────────────┼───────┤
│ 7 │ Yellow-bellied Flycatcher │ 1018 │
├──────┼───────────────────────────┼───────┤
│ 8 │ Tui │ 1004 │
├──────┼───────────────────────────┼───────┤
│ 9 │ Common Redpoll │ 949 │
├──────┼───────────────────────────┼───────┤
│ 10 │ Winter Wren │ 932 │
├──────┼───────────────────────────┼───────┤
│ 11 │ Blue-backed Manakin │ 784 │
├──────┼───────────────────────────┼───────┤
│ 12 │ Hermit Thrush │ 762 │
├──────┼───────────────────────────┼───────┤
│ 13 │ Blue Whistling-Thrush │ 728 │
├──────┼───────────────────────────┼───────┤
│ 14 │ Eastern Wood-Pewee │ 712 │
├──────┼───────────────────────────┼───────┤
│ 15 │ Common Nightingale │ 678 │
├──────┼───────────────────────────┼───────┤
│ 16 │ Red-breasted Flycatcher │ 678 │
├──────┼───────────────────────────┼───────┤
│ 17 │ New Zealand Kaka │ 639 │
├──────┼───────────────────────────┼───────┤
│ 18 │ Common Firecrest │ 608 │
├──────┼───────────────────────────┼───────┤
│ 19 │ New Zealand Fantail │ 583 │ X
├──────┼───────────────────────────┼───────┤
│ 20 │ Tomtit │ 570 │ X
├──────┼───────────────────────────┼───────┤
│ 21 │ Eurasian Golden Oriole │ 548 │
├──────┼───────────────────────────┼───────┤
│ 22 │ Musician Wren │ 526 │
├──────┼───────────────────────────┼───────┤
│ 23 │ White-browed Warbler │ 497 │
├──────┼───────────────────────────┼───────┤
│ 24 │ Cedar Waxwing │ 487 │
├──────┼───────────────────────────┼───────┤
│ 25 │ Iberian Chiffchaff │ 473 │
├──────┼───────────────────────────┼───────┤
│ 26 │ Common Redstart │ 461 │
├──────┼───────────────────────────┼───────┤
│ 27 │ European Greenfinch │ 454 │
├──────┼───────────────────────────┼───────┤
│ 28 │ Wood Thrush │ 432 │
├──────┼───────────────────────────┼───────┤
│ 29 │ Pheasant Cuckoo │ 427 │
├──────┼───────────────────────────┼───────┤
│ 30 │ Western Wood-Pewee │ 399 │
└──────┴───────────────────────────┴───────┘
skraak calls summarise --folder . > call_summary.json
# mapping.json for my big kiwi dataset
{
"Kiwi": {"species": "Kiwi"},
"Geese": {"species": "__NEGATIVE__"},
"Kaka": {"species": "__NEGATIVE__"},
"Kea": {"species": "__NEGATIVE__"},
"LTC": {"species": "__NEGATIVE__"},
"Morepork": {"species": "__NEGATIVE__"},
"Not": {"species": "__NEGATIVE__"},
"Plover": {"species": "__NEGATIVE__"}
}
# make csv to use for training big kiwi dataset
skraak calls clip-labels --folder . --mapping ./mapping.json \
--clip-duration 5 --clip-overlap 0 --min-label-overlap 0.25 --final-clip full \
--output ./clip_labels.csv
Lets manually execute this loop once, when we are happy we will design a ralph loop together to loop through remaining BirdNET classes /grill-me
# Retrieve BirdNET List from folder /media/david/SSD4/Twenty_Four_Seven/R620/2024-05-06/
skraak calls summarise --folder /media/david/SSD4/Twenty_Four_Seven/R620/2024-05-06/ --brief --filter BirdNET 2>/dev/null | jq -r '.filters.BirdNET.species | to_entries | map(select(.key | test("^[A-Z]"))) | sort_by(.value) | .[] | "\(.value)\t\(.key)"'
Start from the top of the BirdNET List and attempt to label a BirdNET class with one of the classes below using skill /call-classification, /call-classification-ollama, /call-library
While there is only a few segments in the BirdNET class, attempt to do this yourself reading data from /call-classification, /call-library. It is your role to work out what this class actually is (BirdNET mislabels many New Zealand Birds), and to assign correct labels wherever possible
When there are many segments in a BirdNET class use skill /call-classification-ollama. choose your reference images carefully. It is your role to work out what this class actually is (BirdNET mislabels many New Zealand Birds), then to use gemma to do the heavy lifting. If Gemma does a poor job of it, it is likely you have chosen the wrong species class.
Keep a .md document with your mappings, BirdNET => code as below
Common Bird List for R620
=========================
comcha Chaffinch
eurbla Blackbird
gryger1 Grey Warbler
kea1 Kea
lotkoe1 Long-tailed Cuckoo
morepo2 Morepork
nezbel1 Bellbird
nezfan1 Fantail
nezkak1 Kaka
nezpig2 Kereru
nezrob3 Kakaruai
pipipi1 Pipipi
riflem1 Rifleman
saddle3 Tieke
silver3 Silvereye
sobkiw2 Fiordland Tokoeka
soioys1 Pied Oystercatcher
tomtit1 Tomtit
tui1 Tui
yefpar3 Kakariki
yellow2 Yellowhammer
weta Weta
gecko Korero Gecko
You have access to skills /pi-ralph-wiggum to loop through the BirdNET list, and pi-heartbeat, to set a timer.
Lets manually execute this loop once, when we are happy we will design a ralph loop together to loop through remaining BirdNET classes /grill-me
Category A - Direct/Obvious Mappings
┌──────────────────────┬────────┬─────────┬───────────────────────────────────────┐
│ BirdNET │ Count │ Code │ Notes │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ New Zealand Bellbird │ 3,812 │ nezbel1 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Grey Gerygone │ 2,286 │ gryger1 │ BirdNET's name for Grey Warbler │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Tui │ 1,004 │ tui1 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ New Zealand Kaka │ 603 │ nezkak1 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Morepork │ 287 │ morepo2 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Silvereye │ 248 │ silver3 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Pipipi │ 79 │ pipipi1 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Long-tailed Koel │ 47 │ lotkoe1 │ BirdNET's name for Long-tailed Cuckoo │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Eurasian Blackbird │ 27 │ eurbla │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ North Island Robin │ 132 │ nezrob3 │ Robin = Kakaruai │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ European Robin │ 124 │ nezrob3 │ Same species │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Dunnock │ 89 │ dunnoc1 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Song Thrush │ 173 │ sonthr1 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Common Redpoll │ 949 │ comred │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Common Starling │ 1 │ eursta │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Yellowhammer │ 4 │ yellow2 │ Exact match │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ House Sparrow │ 36 │ — │ House Sparrow not on R620 common list │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Common Magpie │ 320 │ — │ Magpie not on R620 common list │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Eurasian Skylark │ 5 │ — │ Not on R620 list │
├──────────────────────┼────────┼─────────┼───────────────────────────────────────┤
│ Total │ ~9,779 │ │ │
└──────────────────────┴────────┴─────────┴───────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────
Category B - Real Mislabels (need classification)
These are BirdNET labels that don't match any NZ species name, and the segments are actually NZ
birds:
┌─────────────────────────────────────┬─────────┬─────────────────────────────────┬──────────┐
│ BirdNET │ Count │ Suspected Code(s) │ Priority │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Common Crossbill │ 3,247 │ comred? comcha? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Javan Shortwing │ 2,824 │ tomtit1? nezrob3? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Yellow-bellied Flycatcher │ 1,018 │ nezfan1? tomtit1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Winter Wren │ 932 │ pipipi1? riflem1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Blue-backed Manakin │ 784 │ riflem1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Hermit Thrush │ 762 │ eurbla? sonthr1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Blue Whistling-Thrush │ 728 │ eurbla? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Eastern Wood-Pewee │ 712 │ tomtit1? nezfan1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Common Nightingale │ 678 │ nezrob3? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Red-breasted Flycatcher │ 678 │ tomtit1? nezfan1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Common Firecrest │ 608 │ silver3? riflem1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Eurasian Golden Oriole │ 548 │ tui1? nezbel1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Musician Wren │ 526 │ pipipi1? │ 🔴 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ White-browed Warbler │ 497 │ gryger1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Cedar Waxwing │ 487 │ eursta? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Iberian Chiffchaff │ 473 │ gryger1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Common Redstart │ 461 │ nezrob3? tomtit1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ European Greenfinch │ 454 │ comcha? comred? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Wood Thrush │ 432 │ eurbla? sonthr1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Pheasant Cuckoo │ 427 │ lotkoe1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Western Wood-Pewee │ 399 │ tomtit1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Greater Racket-tailed Drongo │ 376 │ ? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ White-eared Honeyeater │ 358 │ nezbel1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Broad-winged Hawk │ 351 │ Harrier? (not on list) │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Northern Pygmy-Owl │ 347 │ morepo2? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Black-capped Chickadee │ 345 │ ? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Bartlett's Tinamou │ 344 │ ? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Northern Saw-whet Owl │ 344 │ morepo2? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Varied Thrush │ 332 │ eurbla? sonthr1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Black-faced Antthrush │ 330 │ ? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Lesser Redpoll │ 324 │ comred │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Goldcrest │ 298 │ silver3? riflem1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Eurasian Pygmy-Owl │ 286 │ morepo2? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Common Chiffchaff │ 280 │ gryger1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Eurasian Siskin │ 270 │ comred? comcha? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ White-throated Gerygone │ 263 │ gryger1? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Two-barred Crossbill │ 262 │ comred? comcha? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Grey Shrikethrush │ 260 │ ? │ 🟡 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Little Friarbird │ 166 │ nezbel1? │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Great Tit │ 165 │ tomtit1? │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Golden-bellied Gerygone │ 161 │ gryger1? │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Red Wattlebird │ 151 │ nezbel1? │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Common Kingfisher │ 133 │ — (Kingfisher not on R620 list) │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Rufous Whistler │ 11 │ ? │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Rock Wren │ 15 │ — (Rock Wren not on R620 list) │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Nightingale Wren │ 159 │ ? │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Little Spiderhunter │ 117 │ ? │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ ... and ~1,400 more with count < 10 │ │ │ 🟢 │
├─────────────────────────────────────┼─────────┼─────────────────────────────────┼──────────┤
│ Total │ ~38,000 │ │ │
└─────────────────────────────────────┴─────────┴─────────────────────────────────┴──────────┘
segments with no calltype
┌─────────┬───────┬─────────────────────┬──────────────┐
│ Species │ Total │ Classified │ Unclassified │
├─────────┼───────┼─────────────────────┼──────────────┤
│ comcha │ 2022 │ 1950 (311+551+1088) │ 72 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ eurbla │ 467 │ 406 (81+182+89+54) │ 61 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ gryger1 │ 5879 │ 5382 (5372+8+2) │ 497 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ lotkoe1 │ 27 │ 12 │ 15 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ morepo2 │ 137 │ 135 (1+105+8+21) │ 2 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ nezbel1 │ 6919 │ 150 (11+26+7+106) │ 6769 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ nezfan1 │ 844 │ 839 (546+240+53) │ 5 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ nezpig2 │ 65 │ 65 │ 0 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ pipipi1 │ 31 │ 31 │ 0 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ silver3 │ 2876 │ 11 (5+6) │ 2865 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ tomtit1 │ 2167 │ 461 (200+261) │ 1706 │
├─────────┼───────┼─────────────────────┼──────────────┤
│ tui1 │ 1258 │ 2 │ 1256 │
└─────────┴───────┴─────────────────────┴──────────────┘
XNCKPSQERHNFZ6R4HHLDYJJUZEAPCKI4NXZITXVOMJOQXSUD6QQAC
npm install -g @earendil-works/gondolin bash
```
- import { RealFSProvider, VM } from "@earendil-works/gondolin";
+ import _gondolin from "@earendil-works/gondolin";
+ const { RealFSProvider, VM } = _gondolin;
```
for microvm:
pi -e pi-gondolin.ts
guru callers ./pkg/path/funcname
### Callgraph Commands
1. Full project-internal callgraph (best for understanding code flow):
```bash
callgraph -algo rta skraak 2>/dev/null | awk -F'\t' '$1 ~ /^skraak/ && $3 ~ /^skraak/' | grep -v '\$' | sed 's/--static-[0-9]*:[0-9]*-->/ →/; s/--dynamic-[0-9]*:[0-9]*-->/ →/' | sort -u
```
2. Cross-package calls only (who calls across package boundaries):
```bash
callgraph -algo rta skraak 2>/dev/null | awk -F'\t' '$1 ~ /^skraak/ && $3 ~ /^skraak/' | grep -v '\$' | awk -F'\t' '{split($1,a,"."); split($3,b,"."); if(a[1]!=b[1]) print}' | sed 's/--static-[0-9]*:[0-9]*-->/ →/; s/--dynamic-[0-9]*:[0-9]*-->/ →/' | sort -u
```
3. Trace a specific function — "what does CallsClip call?":
```bash
callgraph -algo rta skraak 2>/dev/null | grep 'tools.CallsClip\t' | grep -v '\$'
```
4. Who calls a specific function — "who uses GenerateSpectrogram?":
```bash
callgraph -algo rta skraak 2>/dev/null | grep 'GenerateSpectrogram' | grep -v '\$' | grep -v 'GenerateSpectrogram\t.*→.*GenerateSpectrogram'
```
5. Dynamic (interface/virtual) calls only — find where interfaces dispatch:
```bash
callgraph -algo rta skraak 2>/dev/null | grep 'dynamic' | grep '^skraak'
```
● Per-folder summary — summary_2026-05-08.jsonl (77 records)
Counts are summed across both filter versions (opensoundscape-kiwi-1.2 + 1.5), so a segment present in
both is counted twice for species/calltype.
Legend: files = data files read · segs = total segments · unrev/conf/dk = unreviewed/confirmed/dont_know ·
rev = reviewers
---
Empty folders (no data)
- AC34/2024-01-03, AC43/2024-01-03, R778/2025-02-25, R343/2024-05-06, B40/2025-12-12, GR8/2025-01-01 — 0
files, 0 segments.
Reviewed by David (2026-04-06 batch — fully triaged)
- C03/2026-04-06 — files=95 segs=149 · conf=119 dk=30 bookmarked=4 · Kiwi=102 (M=101, F=1); also
gryger1=13, morepo2=2, nezbel1=2.
- F09/2026-04-06 — files=376 segs=606 · conf=415 dk=191 bookmarked=48 · Kiwi=362 (M=320, F=23, Duet=19);
gryger1=34, morepo2=10, nezrob3=5, nezbel1=3, riflem1=1. Largest reviewed batch.
- G05/2026-04-06 — files=258 segs=444 · conf=356 dk=88 bookmarked=21 · Kiwi=338 (M=323, F=8, Duet=7);
gryger1=8, nezbel1=5, morepo2=4, nezkak1=1.
- H01/2026-04-06 — files=284 segs=357 · conf=93 dk=264 bookmarked=21 · Kiwi=67 (M=52, F=13, Duet=2);
gryger1=16, tui1=3, others.
- P09/2026-04-06 — files=71 segs=91 · conf=43 dk=48 bookmarked=3 · Kiwi=32 (M=30, F=2); tui1=5, gryger1=2,
riflem1=2.
- J11/2026-04-06 — files=213 segs=259 · conf=43 dk=216 bookmarked=11 · Kiwi=9 (M=7, Duet=2); gryger1=26,
riflem1=4, comcha=3.
Reviewed by David (partial — David listed but no confirmed)
- R778/2026-03-23 — files=1039 segs=1350 · unrev=1269 conf=6 dk=75 with_ct=39 · Kiwi=39 (M=37, F=2);
Gecko=6, gryger1=2.
- R743/2026-03-23 — files=1179 segs=1519 · unrev=1370 dk=149 with_ct=101 · Kiwi=101 (M=80, F=21),
Gecko=26.
- AC83/2023-01-03 — files=9 segs=9 · unrev=5 dk=4 with_ct=5 · Kiwi=5 (M=3, F=2).
Auto-classified, unreviewed — large backlog (≥1000 segments)
- R620/2026-03-23 — files=1684 segs=2324 · unrev=2126 dk=198 with_ct=173 · Kiwi=173 (M=172, F=1), Gecko=9.
Biggest folder overall.
- R306/2026-03-23 — files=1214 segs=1654 · unrev=1526 dk=128 with_ct=186 · Kiwi=186 (M=156, F=20,
Duet=10), Gecko=2.
- R343/2026-03-23 — files=1214 segs=1567 · unrev=1493 dk=74 with_ct=122 · Kiwi=122 (M=120, F=2).
- R620/2024-05-06 — files=1151 segs=1504 · unrev=173 dk=1331 with_ct=163 · Kiwi=163 (M=158, F=5),
Gecko=10.
- R287/2026-03-23 — files=903 segs=1153 · unrev=1083 dk=70 with_ct=43 · Kiwi=43 (M=38, F=5), Gecko=3.
- R18/2026-03-23 — files=1026 segs=1236 · unrev=1189 dk=47 with_ct=42 · Kiwi=42 (M=31, F=11), Gecko=7.
Auto-classified, unreviewed — medium (200–1000 segments)
- R635/2026-03-23 — files=1019 segs=1304 · unrev=1249 dk=55 with_ct=66 · Kiwi=66 (M=59, F=7), Gecko=9.
- R635/2025-02-25 — files=452 segs=548 · unrev=33 dk=515 with_ct=30 · Kiwi=30 (M=22, F=8), Gecko=3.
- R620/2025-02-25 — files=792 segs=979 · unrev=83 dk=896 with_ct=78 · Kiwi=78 (M=77, Duet=1), Gecko=5.
- R343/2025-02-25 — files=591 segs=696 · unrev=35 dk=661 with_ct=34 · Kiwi=34 (M=32, F=2).
- R287/2025-02-25 — files=279 segs=326 · unrev=28 dk=298 with_ct=28 · Kiwi=28 (M=22, F=6).
- R287/2024-05-06 — files=276 segs=335 · unrev=69 dk=266 with_ct=65 · Kiwi=65 (M=60, Duet=3, F=2),
Gecko=4.
- R219/2026-03-23 — files=736 segs=938 · unrev=865 dk=73 with_ct=29 · Kiwi=29 (M=23, F=4, Duet=2),
Gecko=3.
- R306/2025-02-25 — files=92 segs=102 · unrev=33 dk=69 with_ct=33 · Kiwi=33 (M=23, Duet=10).
- B29/2024-01-03 — files=407 segs=524 · unrev=112 dk=412 with_ct=112 · Kiwi=112 (M=102, Duet=9, F=1).
- B29/2025-04-16 — files=90 segs=134 · unrev=84 dk=50 with_ct=84 · Kiwi=84 (M=78, Duet=6).
- B29/2025-12-12 — files=95 segs=143 · unrev=64 dk=79 with_ct=64 · Kiwi=64 (M=54, Duet=7, F=3).
- C03/2023-09-11 — files=60 segs=87 · unrev=47 dk=40 with_ct=47 · Kiwi=47 (M=43, Duet=4).
- C03/2024-05-05 — files=110 segs=152 · unrev=48 dk=104 with_ct=48 · Kiwi=48 (M=41, Duet=4, F=3).
- C03/2025-02-25 — files=159 segs=235 · unrev=93 dk=142 with_ct=93 · Kiwi=93 (M=89, F=4).
- G05/2023-09-11 — files=27 segs=44 · unrev=28 dk=16 with_ct=28 · Kiwi=28 (M=26, Duet=2).
- G05/2024-05-05 — files=203 segs=322 · unrev=97 dk=225 with_ct=97 · Kiwi=97 (M=93, Duet=4).
- G05/2025-02-25 — files=356 segs=535 · unrev=278 dk=257 with_ct=275 · Kiwi=275 (M=253, Duet=12, F=10),
Gecko=3.
- H01/2023-09-11 — files=62 segs=80 · unrev=15 dk=65 with_ct=15 · Kiwi=15 (M=13, F=2).
- H01/2024-05-05 — files=410 segs=573 · unrev=72 dk=501 with_ct=72 · Kiwi=72 (M=63, Duet=6, F=3).
- H01/2025-02-25 — files=312 segs=407 · unrev=92 dk=315 with_ct=90 · Kiwi=90 (M=82, Duet=4, F=4), Gecko=2.
- P09/2023-09-11 — files=38 segs=51 · unrev=21 dk=30 with_ct=21 · Kiwi=21 (M=19, Duet=2).
- P09/2023-12-25 — files=74 segs=89 · unrev=6 dk=83 with_ct=6 · Kiwi=6 (M=6).
- P09/2024-05-05 — files=349 segs=440 · unrev=62 dk=378 with_ct=62 · Kiwi=62 (M=49, F=12, Duet=1).
- P09/2025-02-25 — files=398 segs=504 · unrev=74 dk=430 with_ct=74 · Kiwi=74 (M=61, F=8, Duet=5).
- J11/2023-09-11 — files=37 segs=49 · unrev=19 dk=30 with_ct=19 · Kiwi=19 (M=16, Duet=2, F=1).
- J11/2024-05-05 — files=203 segs=257 · unrev=29 dk=228 with_ct=28 · Kiwi=28 (M=23, Duet=4, F=1), Gecko=1.
- J11/2025-02-25 — files=419 segs=537 · unrev=30 dk=507 with_ct=25 · Kiwi=25 (M=17, F=5, Duet=3), Gecko=5.
- GR2/2025-01-01 — files=195 segs=222 · unrev=21 dk=201 with_ct=21 · Kiwi=21 (M=21).
- GR3/2025-01-01 — files=426 segs=472 · unrev=67 dk=405 with_ct=67 · Kiwi=67 (M=66, Duet=1).
- GR4/2025-01-01 — files=362 segs=426 · unrev=76 dk=350 with_ct=76 · Kiwi=76 (M=76).
- GR5/2025-01-01 — files=192 segs=222 · unrev=72 dk=150 with_ct=72 · Kiwi=72 (M=68, F=4).
- GR6/2025-01-01 — files=269 segs=303 · unrev=14 dk=289 with_ct=14 · Kiwi=14 (M=14).
- GR7/2025-01-01 — files=275 segs=305 · unrev=110 dk=195 with_ct=110 · Kiwi=110 (M=98, F=9, Duet=3).
- HCT_18103_H5/2025-06-16 — files=400 segs=452 · unrev=36 dk=416 with_ct=36 · Kiwi=36 (M=34, F=2).
- HCT_K04-14_H3/2025-06-16 — files=358 segs=403 · unrev=87 dk=316 with_ct=87 · Kiwi=87 (M=87).
- HCT_18807_H4/2026-05-04 — files=259 segs=293 · unrev=72 dk=221 with_ct=69 · Kiwi=69 (M=64, F=4, Duet=1),
Gecko=3.
- R778/2024-05-06 — files=286 segs=378 · unrev=36 dk=342 with_ct=35 · Kiwi=35 (M=31, F=4), Gecko=1.
Auto-classified, unreviewed — small (<200 segments)
- AC21/2024-01-03 — 7/9 · all dk · Don't Know=9 only.
- R778/2025-05-23 — 6/8 · all dk · Don't Know=8.
- B02/2024-01-03 — 27/39 · unrev=14 dk=25 · Kiwi=14 (M=14).
- B22/2023-10-29 — 33/39 · all dk · Don't Know=39.
- R635/2024-05-06 — 3/4 · unrev=2 dk=2 · Kiwi=2 (M=1, F=1).
- D03/2023-12-25 — 47/61 · unrev=27 dk=34 · Kiwi=27 (M=26, F=1).
- GR1/2025-01-01 — 17/20 · unrev=4 dk=16 · Kiwi=4 (M=4).
- GR9/2025-01-01 — 41/50 · unrev=37 dk=13 · Kiwi=37 (M=37).
- R16/2023-12-25 — 41/44 · unrev=6 dk=38 · Kiwi=6 (M=6).
- H09/2023-12-25 — 38/48 · unrev=5 dk=43 · Kiwi=5 (M=5).
- HCT10_Kaipo/2024-05-04 — 11/15 · unrev=4 dk=11 · Kiwi=4 (M=4).
- HCT6_Kaipo/2026-05-04 — 1/1 · all dk · Don't Know=1.
- HCT9_Kaipo/2026-05-04 — 118/147 · unrev=52 dk=95 · Kiwi=50 (M=50), Gecko=2.
- HCT_18807_H4/2025-06-16 — 92/105 · unrev=50 dk=55 · Kiwi=49 (M=49), Gecko=1.
- HCT_H13-74/2025-11-22 — 40/48 · unrev=1 dk=47 · Kiwi=1 (M=1).
- HCT_JOG_H7/2025-05-02 — 11/14 · unrev=8 dk=6 · Kiwi=7 (M=7), Gecko=1.
- HCT_KWR075/2025-05-04 — 4/4 · unrev=4 · Kiwi=4 (Duet=2, M=2). All-kiwi folder.
- HCT_K06-10_H1/2025-11-22 — 164/197 · unrev=25 dk=172 · Kiwi=22 (M=18, F=3, Duet=1), Gecko=3.
- HCT_K08-24_H2/2025-06-16 — 144/180 · unrev=26 dk=154 · Kiwi=25 (M=23, F=2), Gecko=1.
┌────────┬────────────┬────────────────────────────┬──────────────┐
│ Pass │ Promotions │ Breakdown │ Skipped @ 70 │
├────────┼────────────┼────────────────────────────┼──────────────┤
│ Male │ 508 │ 505 Male, 2 Female, 1 Duet │ 1374 │
├────────┼────────────┼────────────────────────────┼──────────────┤
│ Female │ 31 │ all → Male │ 65 │
├────────┼────────────┼────────────────────────────┼──────────────┤
│ Duet │ 31 │ all → Male │ 45 │
└────────┴────────────┴────────────────────────────┴──────────────┘
Good progress — the leaf files (sql, time, metadata, isnight, prepend, xxhash, export, replay, create,
update, cluster, dataset, location, pattern) are all os.Exit-free in their Run* functions. The remaining 103
are in:
- calls.go (internal functions like runCallsFromPreds, runCallsShowImages etc.)
- calls_*.go (subcommand implementations)
- import.go (internal runImport* functions)
- common.go (the requireFlags function)
This is a good stopping point for Stream 3c. The key architectural change is done: main.go has a single exit
point, and the core CRUD + utility commands return errors. The calls_* and import internal functions can be
converted in a follow-up.
Skraak Codebase Review: Recommendations
Based on thorough exploration with gosymdb, gocyclo, and direct file reading, here are my findings organized
by theme.
────────────────────────────────────────────────────────────────────────────────
1. Global Mutable State: dbPath Package Variable
File: tools/sql.go:16 — var dbPath string
Impact: Every DB-touching tool function implicitly reads this package-level variable. The cmd layer must call
tools.SetDBPath() before invoking any tool function — a hidden temporal coupling. 25 references across 11
files. This is the single biggest architectural smell.
Recommendation: Make dbPath an explicit parameter on every tool function that needs it. This makes data flow
visible, eliminates the SetDBPath function entirely, and makes the tools package side-effect-free and
testable in parallel without mutexes. The Input structs already exist for every tool — add DBPath string to
them (or a dedicated DBConfig struct embedded in each input).
────────────────────────────────────────────────────────────────────────────────
2. Duplicated Validation/Verification Functions
There are 8 local verify/validate functions in tools/ that duplicate or partially overlap with
db/validation.go:
┌─────────────────────────────────────────────────┬────────────────────────────────────┬────────────────────┐
│ Local (tools/) │ db/ equivalent │ Used by │
├─────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┤
│ verifyDatasetActive(*sql.DB) │ db.DatasetExistsAndActive(Querier) │ 1 caller │
│ │ │ (dataset.go) │
├─────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┤
│ verifyDatasetExistsAndActive(Querier) │ wraps db.DatasetExistsAndActive │ location.go │
├─────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┤
│ verifyLocationExistsAndActive(interface{QueryRo │ db.LocationBelongsToDataset(Querie │ location.go │
│ w...}) │ r) │ │
├─────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┤
│ verifyPatternExistsAndActive(*sql.DB) │ none in db/ │ pattern.go │
├─────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┤
│ verifyPatternExists(*LoggedTx) │ none │ cluster.go │
├─────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┤
│ validateClusterActive(*sql.DB) │ none │ cluster.go │
├─────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┤
│ validateClusterCyclicPattern(*sql.DB) │ none │ cluster.go │
├─────────────────────────────────────────────────┼────────────────────────────────────┼────────────────────┤
│ validateCyclicPattern(*sql.DB) │ none │ cluster.go │
└─────────────────────────────────────────────────┴────────────────────────────────────┴────────────────────┘
Problems:
- Inconsistent parameter types: some take *sql.DB, some take db.Querier, some take inline
interface{QueryRow...} — this is confusing
- verifyDatasetActive in dataset.go duplicates db.DatasetExistsAndActive but with a different signature and
error message
- verifyLocationExistsAndActive defines an anonymous interface instead of using db.Querier
Recommendation: Consolidate all entity-exists-and-active checks into db/validation.go using the db.Querier
interface consistently. Delete the local duplicates in tools/. The existing db.Querier interface is good —
extend it and use it everywhere.
────────────────────────────────────────────────────────────────────────────────
3. High Cyclomatic Complexity (40+ Functions ≥ 11)
Worst offenders (complexity 14):
- RunCalls, ExportDataset, CallsPropagateFolder, bulkReadCSV, loadSpeciesCalltypeIDs, normalizeFlat,
Model.View, Model.handleSwitchKey
Average per-package: tools and cmd both have many functions in the 11–14 range. The lint_test threshold is
14, so these are just barely passing.
Recommendation:
- Extract flag-parsing helpers in cmd/ — nearly every Run* function follows the same pattern: fs :=
flag.NewFlagSet(...), parse, validate required flags, call tools.SetDBPath, call tool function, printJSON.
The "validate required flags" block alone is duplicated ~20 times.
- Extract early-return guard clauses in the tools functions — many if err != nil { return ... } chains can be
collapsed.
- Break up the TUI's View() (complexity 14, 700+ lines) into sub-render methods (which is already partially
done, but the switch dispatch is the bottleneck).
────────────────────────────────────────────────────────────────────────────────
4. tools/ Package is a God Package (265 non-test functions, 28 source files)
556 symbols — 52% of the entire codebase's symbols live in tools/. It mixes:
- DB CRUD operations (cluster, dataset, location, pattern, import, export)
- File-processing pipelines (calls_from_preds, calls_classify, calls_clip, calls_clip_labels)
- AviaNZ format I/O (data file reading/writing)
- Audio processing (spectrogram generation, playback)
- Validation logic
- SQL execution
Recommendation: Split tools/ into 3–4 focused sub-packages:
- tools/calls — all calls_* files (from_preds, from_birda, from_raven, classify, clip, clip_labels, modify,
propagate, push_certainty, detect_anomalies, show_images, summarise)
- tools/crud — cluster, dataset, location, pattern create/update
- tools/import — import_file, import_files, import_segments, import_unstructured, bulk_file_import
- tools/sql — sql, export (or merge into db/)
This would also reduce merge conflicts, as these are largely independent domains.
────────────────────────────────────────────────────────────────────────────────
5. Repetitive cmd/ Boilerplate
Every Run* function in cmd/ repeats this pattern:
1. Create flag.NewFlagSet
2. Define flags (always --db)
3. Parse args
4. missing := []string{} + check required flags
5. tools.SetDBPath(*dbPath)
6. defer initEventLog(*dbPath)()
7. Build tools.XxxInput{...}
8. Call tools.Xxx(input)
9. printJSON(output) or error to stderr + os.Exit(1)
Recommendation: Create a cmd-internal helper:
```go
type CommandConfig struct {
DBPath string
EventLog bool
}
func parseCommand(args []string, requiredFlags ...string) (CommandConfig, *flag.FlagSet) { ... }
```
This would eliminate ~60% of the boilerplate in cmd/ files.
────────────────────────────────────────────────────────────────────────────────
6. MarshalJSON Boilerplate in db/types.go
Four types (Dataset, Location, Cluster, CyclicRecordingPattern) each have a nearly identical MarshalJSON
method that just reformats time.Time fields as RFC3339 strings. The pattern is: create anonymous struct
identical to the receiver but with string instead of time.Time, copy all fields, convert timestamps, marshal.
Recommendation: Use a helper or custom type:
- Option A: Define type JSONTime time.Time with a MarshalJSON that outputs RFC3339, and use it in the struct
tags.
- Option B: Use json.Marshal with a single helper that reflects over the struct and converts time.Time
fields.
- This would remove ~80 lines of boilerplate.
────────────────────────────────────────────────────────────────────────────────
7. AviaNZ Types Defined in calls_from_preds.go but Used Across Files
AviaNZMeta, AviaNZLabel, AviaNZSegment are defined in calls_from_preds.go but used by calls_from_birda.go,
calls_from_raven.go, and the shared writeDotDataFileSafe function. They're AviaNZ format concerns, not
prediction-specific.
Recommendation: Move these types to a dedicated file (e.g., tools/avianz_types.go or utils/data_file.go since
the rest of the AviaNZ parsing already lives there). This makes the dependency graph honest and reduces the
cognitive load of calls_from_preds.go (733 lines, one of the largest files).
────────────────────────────────────────────────────────────────────────────────
8. Mixed DB Access Patterns in tools/
The tools layer uses 4 different DB access patterns, sometimes within the same file:
┌───────────────────────────────────────┬───────────────────────────────────────────┬───────────────────────┐
│ Pattern │ Example │ Used in │
├───────────────────────────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ db.WithWriteTx(ctx, dbPath, toolName, │ cluster.go, dataset.go, location.go │ CRUD tools │
│ fn) │ │ │
├───────────────────────────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ db.OpenReadOnlyDB(dbPath) + manual │ bulk_file_import.go, export.go, │ Read-heavy tools │
│ db.Close() │ import_files.go │ │
├───────────────────────────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ db.OpenWriteableDB(dbPath) + manual │ bulk_file_import.go, import_file.go, │ Write-heavy without │
│ │ import_segments.go │ tx logging │
├───────────────────────────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ db.WithReadDB(dbPath, fn) │ import_files.go, import_unstructured.go │ Simple read queries │
└───────────────────────────────────────┴───────────────────────────────────────────┴───────────────────────┘
Some tools open a writeable DB but don't use WithWriteTx, meaning their mutations aren't logged for replay.
import_segments.go and import_file.go are notable — they open *sql.DB directly and do inserts without the
transaction logger.
Recommendation: Standardize on db.WithWriteTx for all mutations (it's already there for exactly this purpose)
and db.WithReadDB for reads. The manual Open*DB + defer Close() pattern should be the exception, not the
norm. This also fixes the audit-gap where some writes aren't replayable.
────────────────────────────────────────────────────────────────────────────────
9. tui/ Directly Imports tools for ClassifyState
The TUI package imports tools specifically for ClassifyState, ClassifyConfig, KeyBinding, BindingResult, and
FormatLabels. This creates a dependency from the presentation layer to the business-logic layer. The TUI
calls tools.FormatLabels() and directly manipulates tools.ClassifyState.
Recommendation: If tools/ is split per recommendation #4, the classify-related types should move to a
tools/calls sub-package, which the TUI can import. Alternatively, extract the types (ClassifyConfig,
KeyBinding, BindingResult) into utils/ since they're pure data with no DB dependency — ClassifyState is the
only one that has real logic (and it should stay in tools/calls).
────────────────────────────────────────────────────────────────────────────────
10. Error Handling: Inconsistent os.Exit in cmd/
There are 174 os.Exit calls across cmd/ files, with no structured error types. Every function exits directly
on any error. This makes testing cmd/ functions impossible and prevents composable error handling.
Recommendation: Return errors from Run* functions instead of calling os.Exit. Have a single top-level
dispatcher in main.go that handles os.Exit(1) based on the returned error. This would make the cmd/ layer
testable and allow future programmatic use of the CLI.
────────────────────────────────────────────────────────────────────────────────
Summary: Priority-Ordered Changes
┌───────────┬────────────────────────────────────────────────┬─────────────────────────────────────┬────────┐
│ Priority │ Change │ Impact │ Effort │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🔴 High │ Eliminate var dbPath — make it explicit │ Testability, parallel-safety, │ Medium │
│ │ parameter │ clarity │ │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🔴 High │ Consolidate duplicated verify/validate into │ Consistency, DRY │ Low │
│ │ db/validation.go │ │ │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🟡 Medium │ Split tools/ into sub-packages │ Maintainability, merge conflicts, │ High │
│ │ │ cognitive load │ │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🟡 Medium │ Standardize DB access patterns (use │ Auditability, consistency │ Medium │
│ │ WithWriteTx/WithReadDB) │ │ │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🟡 Medium │ Reduce cmd/ boilerplate with helper functions │ DRY, readability │ Low │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🟢 Low │ Deduplicate MarshalJSON in db/types.go │ DRY, ~80 lines removed │ Low │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🟢 Low │ Move AviaNZ types out of calls_from_preds.go │ Dependency clarity │ Low │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🟢 Low │ Return errors from cmd/ instead of os.Exit │ Testability │ Medium │
├───────────┼────────────────────────────────────────────────┼─────────────────────────────────────┼────────┤
│ 🟢 Low │ Reduce cyclomatic complexity in worst │ Maintainability │ Medium │
│ │ functions │ │ │
└───────────┴────────────────────────────────────────────────┴─────────────────────────────────────┴────────┘
#: 3
Item: Reduce cyclomatic complexity
Status: ⚠️ Partial
Notes: Still ~15 functions at 13–14 (e.g. ExportDataset, CallsPropagateFolder, bulkReadCSV,
loadSpeciesCalltypeIDs, RunCalls, handleSwitchKey). Same offenders as before — not yet broken up.
#: 8
Item: Standardize DB access patterns
Status: ⚠️ Partial
Notes: WithWriteTx / WithReadDB are used in CRUD tools, but tools/import/*.go (bulk_file_import, import_file,
import_files, import_segments) still call db.OpenWriteableDB directly without transaction logging. The
audit-gap recommendation #8 flagged is still present.
Evaluation: Recommendations Implementation Status
### 1. Global Mutable State: var dbPath — ✅ RESOLVED
Was: var dbPath package-level variable in tools/sql.go with SetDBPath() function, 25 references across 11
files.
Now: SetDBPath and var dbPath are completely eliminated. Every Input struct now has an explicit DBPath string
field (e.g., ClusterInput.DBPath, DatasetInput.DBPath, ExecuteSQLInput.DBPath). A small resolveDBPath()
helper delegates to db.ResolveDBPath(). The cmd layer passes *dbPath into each input struct directly. This is
exactly what was recommended — data flow is now visible, no temporal coupling, parallel-safe.
### 2. Duplicated Validation/Verification Functions — ✅ MOSTLY RESOLVED
Was: 8 local verify/validate functions in tools/ with inconsistent parameter types.
Now: db/validation.go has been significantly expanded with consistent db.Querier-based functions:
- DatasetExistsAndActive, LocationBelongsToDataset, LocationExistsAndActive, ClusterExistsAndActive,
ClusterBelongsToLocation, PatternExistsAndActive, ValidateLocationBelongsToDataset, plus type-specific
validators.
The old local duplicates in dataset.go, location.go, pattern.go are gone — those files now call db.*
directly.
Remaining issue: tools/cluster.go still has two local wrappers:
- validateClusterActive(database *sql.DB, ...) — just delegates to db.ClusterExistsAndActive
- validateClusterCyclicPattern(database *sql.DB, ...) — adds nil/empty check before calling
db.PatternExistsAndActive
These are thin wrappers (3–10 lines each) with a bit of business logic in the cyclic pattern one. Not a major
smell, but the first one is a pure pass-through that could be inlined.
### 3. High Cyclomatic Complexity — 🟡 PARTIALLY IMPROVED
Was: Worst offenders at complexity 14, many functions in the 11–14 range. Lint threshold at 14.
Now: No functions exceed 14 (the lint threshold), but 6 functions still sit at complexity 14 (the max):
- RunCalls (cmd), ExportDataset (tools), CallsPropagateFolder (calls), loadSpeciesCalltypeIDs (import),
bulkReadCSV (import), Model.handleSwitchKey (tui)
Plus ~28 functions in the 11–13 range. The TUI's View() was previously at complexity 14 and 700+ lines — now
the TUI is 874 lines but View no longer appears in the top offenders (it was likely broken into sub-methods
like handleSwitchKey, handleClipKey, etc.). So some progress on the TUI front.
Verdict: The ceiling hasn't changed (still 14), but there's been some redistribution. The lint threshold is
acting as a ceiling rather than a target — functions cluster just below/at it.
### 4. tools/ God Package — ✅ MOSTLY RESOLVED
Was: 265 non-test functions, 28 source files, 556 symbols — 52% of codebase.
Now: tools/ has been split into three packages:
┌───────────────┬───────┬───────────┬───────┐
│ Package │ Files │ Functions │ Lines │
├───────────────┼───────┼───────────┼───────┤
│ tools/ │ 12 │ 82 │ 2,746 │
├───────────────┼───────┼───────────┼───────┤
│ tools/calls/ │ 28 │ 271 │ 8,242 │
├───────────────┼───────┼───────────┼───────┤
│ tools/import/ │ 6 │ 37 │ 2,181 │
└───────────────┴───────┴───────────┴───────┘
This matches the recommended split almost exactly (calls → tools/calls/, import → tools/import/). The
remaining tools/ package is the CRUD core (cluster, dataset, location, pattern, sql, export, prepend, time).
The tools/sql sub-package recommendation wasn't adopted — export and sql remain in tools/, which is
reasonable given their smaller size.
### 5. Repetitive cmd/ Boilerplate — 🟡 PARTIALLY IMPROVED
Was: Every Run* function repeats 9 steps: flag.NewFlagSet, define flags, parse, validate required, SetDBPath,
initEventLog, build input, call tool, printJSON.
Now:
- SetDBPath step is eliminated (DBPath goes into input structs) ✅
- cmd/common.go was created with checkFlags() and checkNonZeroFlags() helpers that return errors instead of
calling os.Exit ✅
- Most Run* functions now return err instead of calling os.Exit directly ✅
- main.go has a clean dispatch map with a single os.Exit(1) on error ✅
Still present: 29 flag.NewFlagSet calls across 24 cmd files. Each function still creates its own FlagSet,
defines flags, and calls checkFlags. The "parseCommand" helper from the recommendation was not implemented.
The boilerplate is reduced (no SetDBPath, error returns instead of exit), but the flag-parsing pattern is
still repeated.
### 6. MarshalJSON Boilerplate in db/types.go — 🟡 PARTIALLY IMPROVED
Was: 4 nearly identical MarshalJSON methods, ~80 lines of boilerplate.
Now: A JSONTime type and jt() helper were introduced — this is exactly "Option A" from the recommendation.
However, the 4 MarshalJSON methods still exist. Each one creates an anonymous struct with JSONTime fields and
manually copies all fields. The JSONTime type eliminates the explicit time.Time → string conversion, but the
struct duplication remains. There are still 11 MarshalJSON references in db/types.go (4 methods + helper + jt
references).
Lines saved: Maybe ~20–30 lines reduced from the original ~80, but the fundamental boilerplate of duplicating
the struct definition persists.
### 7. AviaNZ Types — ✅ RESOLVED
Was: AviaNZMeta, AviaNZLabel, AviaNZSegment defined in calls_from_preds.go but used across files.
Now: Moved to tools/calls/avianz_types.go — a dedicated file in the calls sub-package. Clean separation,
honest dependency graph. The calls_from_preds.go file is down to 716 lines (from 733).
### 8. Mixed DB Access Patterns — 🟡 PARTIALLY IMPROVED
Was: 4 different DB access patterns across tools/.
Now: The CRUD operations in tools/ consistently use db.WithWriteTx for writes and db.WithReadDB for reads ✅.
tools/import/import_unstructured.go also uses WithWriteTx ✅.
Still an issue: 4 files in tools/import/ still use OpenWriteableDB + manual Close() without transaction
logging:
- import_file.go, import_files.go, import_segments.go, bulk_file_import.go
These writes bypass the audit/replay logger. The original concern about the "audit gap" remains for imports.
### 9. TUI Importing tools for ClassifyState — 🟡 PARTIALLY IMPROVED
Was: TUI imported tools directly for ClassifyState, ClassifyConfig, KeyBinding, BindingResult, FormatLabels.
Now: TUI imports tools/calls instead of tools. Since the calls sub-package was created (rec #4), this is the
"alternative" recommendation — the classify-related types live in tools/calls/ and TUI imports that. However,
the deeper coupling remains: TUI directly manipulates calls.ClassifyState, calls calls.FormatLabels(), and
calls.BindingResult. The types weren't extracted to utils/.
Verdict: The dependency is now on a more focused sub-package rather than the god package, which is an
improvement. But the TUI↔business-logic coupling remains.
### 10. Error Handling: os.Exit in cmd/ — ✅ MOSTLY RESOLVED
Was: 174 os.Exit calls across cmd/ files.
Now: Only 14 os.Exit calls remain in cmd/ (down from 174!). Of those:
- 5 are in cmd/common.go's mustValue/mustIntValue (unrecoverable CLI parse errors)
- 9 are in calls_classify.go, calls_modify.go, calls_push_certainty.go (TUI-related exit paths)
Most Run* functions now return error. main.go has a clean dispatch with if err := handler(...); err != nil →
single os.Exit(1). The ErrHelpRequested sentinel enables clean help exits.
Remaining: 9 os.Exit calls in the classify/modify/push-certainty commands — these are the TUI-adjacent
commands where the pattern is slightly different (TUI launch, signal handling). Still improvable but
dramatically better.
────────────────────────────────────────────────────────────────────────────────
Summary Table
┌────┬─────────────────────────┬─────────────┬──────────────────────────────────────────────────────────────┐
│ # │ Recommendation │ Status │ Notes │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 1 │ Eliminate var dbPath │ ✅ Done │ DBPath in every Input struct, SetDBPath gone │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 2 │ Consolidate │ ✅ Mostly │ db/validation.go expanded; 2 thin wrappers remain in │
│ │ verify/validate │ done │ cluster.go │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 3 │ Reduce cyclomatic │ 🟡 Marginal │ No functions >14, but 6 at exactly 14; still 28+ in 11–13 │
│ │ complexity │ │ range │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 4 │ Split tools/ god │ ✅ Done │ Split into tools/, tools/calls/, tools/import/ │
│ │ package │ │ │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 5 │ Reduce cmd/ boilerplate │ 🟡 Partial │ SetDBPath gone, checkFlags helper added, errors returned; │
│ │ │ │ but 29 FlagSets still repeated │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 6 │ Deduplicate MarshalJSON │ 🟡 Partial │ JSONTime helper added, but 4 MarshalJSON methods still │
│ │ │ │ duplicate structs │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 7 │ Move AviaNZ types │ ✅ Done │ Moved to tools/calls/avianz_types.go │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 8 │ Standardize DB access │ 🟡 Partial │ CRUD uses WithWriteTx consistently; 4 import files still use │
│ │ │ │ OpenWriteableDB │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 9 │ TUI→tools coupling │ 🟡 Partial │ Now imports tools/calls (more focused), but still deeply │
│ │ │ │ coupled to ClassifyState │
├────┼─────────────────────────┼─────────────┼──────────────────────────────────────────────────────────────┤
│ 10 │ Eliminate os.Exit in │ ✅ Mostly │ Down from 174→14; most functions return errors; main.go │
│ │ cmd/ │ done │ dispatches │
└────┴─────────────────────────┴─────────────┴──────────────────────────────────────────────────────────────┘
Overall: 3 fully resolved, 4 mostly/partially done with remaining work, 3 with marginal improvement. The
biggest wins were eliminating the global dbPath (rec #1), splitting the god package (rec #4), and the os.Exit
cleanup (rec #10). The remaining work is mostly medium-priority: the import audit gap (rec #8), the cmd/
FlagSet boilerplate (rec #5), and the MarshalJSON struct duplication (rec #6).