segment unstructured import into batches of 10000 files to keep within buffer limits, structured imports should be fine as we are talking 1 sd card (24/7 its 16000 max)
restrict db exports to structured, or handle unstructured : Andrew Digby
(base) skraak ➤ ./skraak export dataset --db ./db/skraak.duckdb --id 4Sh8_7p1ocks --output db/export.duckdb
Error: failed to copy file_dataset: Constraint Error: Violates foreign key constraint because key "id: CwN4rQCeOs2J4lOvijhxa" does not exist in the referenced table
ingest my training datasets
buy a drive to backup mac ~
include skraak metadata in written .data files at the end of relevant section BUT ONLY on insert to db at end, final review.
file level metadata: skraak_dataset_id, skraak_location_id, skraak_cluster_id, skraak_file_id, skraak_file_hash
label level metadata: skraak_segment_id, skraak_label_id
operator and reviewer in file metadata
make a psuedo schema for file and label metadata
Update tools could allow setting active to false??
look at nulls in schema, problem with updatde tool
Just the simplest clip viewer and sort in a tui, from db data not avianz data
Make segments from pred and save as json??? / insert into db
Make clip dynamicly in Julia and sort using model, get data from db, modify db weth changes
Img Viewer renders from db and files
User confirms
Update db, inc certainty
Make export clips tool
Edit selection tool to change start end. Frameing
Make import avianz, add skraak metadata
Where do I add skraak metadata
Make import freebird tool
Make import raven
##### NEED TO MERGE LABELS ###################
SELECT id, label, ebird_code, description, active
FROM species
WHERE ebird_code IN ('lotkoe1', 'liskiw1', 'grskiw1', 'okbkiw1', 'sobkiw1', 'nibkiw1');
┌──────────────┬──────────────────────┬────────────┬───────────────────────────────────────────────┬─────────┐
│ id │ label │ ebird_code │ description │ active │
│ varchar │ varchar │ varchar │ varchar │ boolean │
├──────────────┼──────────────────────┼────────────┼───────────────────────────────────────────────┼─────────┤
│ CNi9X-Hkz4-A │ Kiwi_Brown │ sobkiw1 │ FreebirdCode: 109, BiowebCode: 10857, Com… │ false │
│ D_pepf6f08kn │ Kiwi_Great_Spotted │ grskiw1 │ FreebirdCode: 110, BiowebCode: 10860, Com… │ false │
│ di-jxVg8tcVO │ Kiwi_Little_Spotted │ liskiw1 │ FreebirdCode: 111, BiowebCode: 10859, Com… │ false │
│ dZLV3Ue916E- │ Kiwi_Nth_Is_Brown │ nibkiw1 │ FreebirdCode: 112, BiowebCode: 10856, Com… │ false │
│ 74yg7hquUKT9 │ Kiwi_Okarito_Brown │ okbkiw1 │ FreebirdCode: 113, BiowebCode: 13151, Com… │ false │
│ ac0AgauTMvMT │ Kiwi_spp │ sobkiw1 │ FreebirdCode: 114, BiowebCode: 29709, Com… │ false │
│ 4JOC8g7OsLGW │ Cuckoo_long-tailed │ lotkoe1 │ FreebirdCode: 35, BiowebCode: 11248, Long-… │ false │
│ rqIcB4EkSy2G │ Long-tailed Koel │ lotkoe1 │ NULL │ true │
│ eYZ6Ns7XNbwR │ Kiwi pukupuku │ liskiw1 │ Kiwi pukupuku / Little Spotted Kiwi │ true │
│ TSwVuQdE0cEA │ Roroa │ grskiw1 │ Roroa / Great Spotted Kiwi │ true │
│ fDGmXv9c6xEn │ Rowi │ okbkiw1 │ Rowi / Okarito Brown Kiwi │ true │
│ jqEQrViqRFFX │ S. Fiordland Tokoeka │ sobkiw1 │ Southern Fiordland Tokoeka │ true │
│ ANhV8iZPfIh8 │ Haast Tokoeka │ sobkiw1 │ NULL │ true │
│ 7LNNjRcptMev │ Kiwi │ nibkiw1 │ Kiwi-nui / North Island Brown Kiwi │ true │
├──────────────┴──────────────────────┴────────────┴───────────────────────────────────────────────┴─────────┤
│ 14 rows 5 columns │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
on the duckdb cli How can i append this description
│ dZLV3Ue916E- │ Kiwi_Nth_Is_Brown │ nibkiw1 │ FreebirdCode: 112, BiowebCode: 10856, Com… │ false │
to the following, and also overwrite the Kiwi label with Kiwi_Nth_Is_Brown
7LNNjRcptMev │ Kiwi │ nibkiw1 │ Kiwi-nui / North Island Brown Kiwi │ true
then delete dZLV3Ue916E-
# Does not work.
UPDATE species
SET
label = 'Kiwi_Nth_Is_Brown',
description = description || ', ' || (
SELECT description
FROM species
WHERE id = 'dZLV3Ue916E-'
)
WHERE id = '7LNNjRcptMev';
DELETE FROM my_table
WHERE id = 'dZLV3Ue916E-';
time ./skraak calls from-preds --csv /media/david/SSD4/Twenty_Four_Seven/R620/2024-05-06/preds9_opensoundscape-multi-1.0_2025-07-22.csv > /media/david/SSD4/Twenty_Four_Seven/R620/2024-05-06/preds9_opensoundscape-multi-1.0_2025-07-22.json
time ./skraak calls from-preds --csv /media/david/SSD4/Twenty_Four_Seven/R620/2025-02-25/preds9_opensoundscape-multi-1.0_2025-07-22.csv > /media/david/SSD4/Twenty_Four_Seven/R620/2025-02-25/preds9_opensoundscape-multi-1.0_2025-07-22.json
./skraak calls classify \
--folder /media/david/SSD4/Pomona/C05/2025-11-08/ \
--reviewer David \
--bind k=Kiwi \
--bind d="Kiwi+Duet" \
--bind f="Kiwi+Female" \
--bind m="Kiwi+Male" \
--bind n="Don't Know" \
--bind p=Morepork \
--color \
--img-dims 224
./skraak calls classify \
--folder /media/david/SSD4/Twenty_Four_Seven/R620/2024-05-06/ \
--reviewer David \
--bind k=Kiwi \
--bind d="Kiwi+Duet" \
--bind f="Kiwi+Female" \
--bind m="Kiwi+Male" \
--bind n="Don't Know" \
--bind p=Morepork \
--color
file,start_time,end_time,ausbit1,bluduc1,comcha,comred,dunnoc1,eurbla,eursta,fernbi1,grskiw1,gryger1,kea1,liskiw1,lotkoe1,morepo2,nezbel1,nezfan1,nezkak1,nezpig2,nezrob3,nibkiw1,okbkiw1,parake,pipipi1,riflem1,shbcuc1,silver3,sobkiw2,soioys1,soiwre1,sonthr1,spocra2,tomtit1,tui1,varoys1,weka1,yellow2,weta,saddle3
ausbit1 Australasian Bittern
bluduc1 Blue Duck
comcha Common Chaffinch
comred Redpoll (Common)
dunnoc1 Dunnock
eurbla Eurasian Blackbird
eursta European Starling
fernbi1 New Zealand Fernbird
grskiw1 Great Spotted Kiwi
gryger1 Gray Gerygone
kea1 Kea
liskiw1 Little Spotted Kiwi
lotkoe1 Long-tailed Koel
morepo2 Morepork
nezbel1 New Zealand Bellbird
nezfan1 New Zealand Fantail
nezkak1 New Zealand Kaka
nezpig2 New Zealand Pigeon
nezrob3 South Island Robin
nibkiw1 North Island Brown Kiwi
okbkiw1 Okarito Brown Kiwi
parake parakeet sp.
pipipi1 Pipipi
riflem1 Rifleman
saddle3 South Island Saddleback
shbcuc1 Shining Bronze-Cuckoo
silver3 Silvereye
sobkiw2 Southern Brown Kiwi (South I.)
soioys1 South Island Oystercatcher
soiwre1 South Island Wren
sonthr1 Song Thrush
spocra2 Spotless Crake
tomtit1 Tomtit
tui1 Tui
varoys1 Variable Oystercatcher
weka1 Weka
yellow2 Yellowhammer
weta
bluduc1 Duck_Blue_Whio Blue Duck
comcha Chaffinch Common Chaffinch
dunnoc1 Dunnock_Hedge_Sparrow Dunnock
fernbi1 Fernbird New Zealand Fernbird
fernbi1 Fernbird_Nth Is New Zealand Fernbird
fernbi1 Fernbird_Sth Is New Zealand Fernbird
grskiw1 Kiwi_Great_Spotted Great Spotted Kiwi
grskiw1 Roroa Great Spotted Kiwi
gryger1 Warbler_Grey Gray Gerygone
kea1 Kea Kea
liskiw1 Kiwi pukupuku Little Spotted Kiwi
liskiw1 Kiwi_Little_Spotted Little Spotted Kiwi
lotkoe1 Cuckoo_long-tailed Long-tailed Koel
lotkoe1 Long-tailed Koel Long-tailed Koel
morepo2 Morepork Morepork
nezbel1 Bellbird New Zealand Bellbird
nezbel1 Bellbird_Poor_Knights New Zealand Bellbird
nezbel1 Bellbird_Three_Kings New Zealand Bellbird
nezfan1 Fantail New Zealand Fantail
nezfan1 Fantail_Chatham_Is New Zealand Fantail
nezfan1 Fantail_Nth_Is New Zealand Fantail
nezfan1 Fantail_Sth_Is New Zealand Fantail
nezkak1 Kaka New Zealand Kaka
nezrob3 Robin_Stewart_Is South Island Robin
nezrob3 Robin_Sth_Is South Island Robin
nibkiw1 Kiwi North Island Brown Kiwi
nibkiw1 Kiwi_Nth_Is_Brown North Island Brown Kiwi
okbkiw1 Kiwi_Okarito_Brown Okarito Brown Kiwi
okbkiw1 Rowi Okarito Brown Kiwi
riflem1 Rifleman Rifleman
riflem1 Rifleman_Nth_Is Rifleman
riflem1 Rifleman_Sth_Is Rifleman
saddle3 Saddleback_Sth_Is South Island Saddleback
shbcuc1 Cuckoo_Shining Shining Bronze-Cuckoo
silver3 Silvereye Silvereye
sobkiw2 Kiwi_Tokoeka_Haast Southern Brown Kiwi (South I.)
sonthr1 Thrush_Song Song Thrush
spocra2 Crake_Spotless Spotless Crake
tomtit1 Tomtit Tomtit
tomtit1 Tomtit_Auckland_Is Tomtit
tomtit1 Tomtit_Chatham_Is Tomtit
tomtit1 Tomtit_Nth_Is Tomtit
tomtit1 Tomtit_Snares_Is_Black Tomtit
tomtit1 Tomtit_Sth_Is Tomtit
tui1 Tui Tui
tui1 Tui_Chatham_Is Tui
varoys1 Oystercatcher_Variable Variable Oystercatcher
weka1 Weka_Buff Weka
weka1 Weka_Nth_Is Weka
weka1 Weka_Stewart_Is Weka
weka1 Weka_Western Weka
weka1 Weka_spp Weka
yellow2 Yellowhammer Yellowhammer
Not in species table: ausbit1, comred, eurbla, eursta, nezpig2, parake, pipipi1, soioys1, soiwre1, weta
c=comcha Chaffinch Common Chaffinch
d=dunnoc1 Dunnock_Hedge_Sparrow Dunnock
b=eurbla Eurasian Blackbird
g=gryger1 Warbler_Grey Gray Gerygone
k=kea1 Kea Kea
l=lotkoe1 Long-tailed Koel Long-tailed Koel
p=malpar2 Kakariki Malherbe's Parakeet (Orange-fronted Parakeet)
m=morepo2 Morepork Morepork
nezbel1 Bellbird New Zealand Bellbird
nezfan1 Fantail New Zealand Fantail
nezkak1 Kaka New Zealand Kaka
nezpig2 New Zealand Pigeon
nezrob3 Robin_Sth_Is South Island Robin
pipipi1 Pipipi Brown Creeper
riflem1 Rifleman Rifleman
saddle3 Saddleback_Sth_Is South Island Saddleback
shbcuc1 Cuckoo_Shining Shining Bronze-Cuckoo
silver3 Silvereye Silvereye
sobkiw2 Kiwi_Tokoeka_Haast Southern Brown Kiwi (South I.)
soioys1 South Island Oystercatcher
sonthr1 Thrush_Song Song Thrush
tomtit1 Tomtit Tomtit
tui1 Tui Tui
varoys1 Oystercatcher_Variable Variable Oystercatcher
yellow2 Yellowhammer Yellowhammer