As part of the July, 2019, Microsoft hackathon, two teams that focused on projects involving orcas and machine learning – Pod.Cast and OrcaHello — combined efforts in pursuit of “AI for Orcas”. Subsequently, and with increasing data access from the Orcasound hydrophone network and support from Microsoft’s AI for Earth program, this real-time inference system has been refined through collaborative efforts.
Cooperating during 2019-2021, the combined team launched an MVP of the live inference system in July, 2020, through the Microsoft hackathon. This ~3 year effort involved heroic dedication from ~25 Microsoft employees — all acting as volunteers within the 2020 July and 2021 October hackathon teams led by Prakruti Gogia, Akash Mahajan, Chris Hanke, Michelle Yang, Ayush Agrawal, Mike Cowan, Claire Goetschel, and Adele Bai. See the full list below in the LEAD CONTRIBUTORS AND COLLABORATORS section.
In September, 2020, beta-testing began with live data from a single Orcasound location with moderating bioacoustic experts Scott Veirs, David Bain, and Val Veirs. In November, the two other Orcasound hydrophone network locations were added to the beta-test. Development continued mainly through AI4Earth & OneWeek hackathons, but also many weekend hacks and asynchronous work via Teams.
By April, 2021, preliminary results include ~1300 candidate detections (~60 detections per month for each of the 3 Orcasound hydrophone locations). Confirmed candidates include 211 true positives, 15 “unknown” SRKW-like signals, and 1075 false positives.
Links
- OrcaHello moderator portal
- API documentation (REST API for interacting with detections in the OrcaHello DB)
In it’s current state, the OrcaHello system does the following:
- Ingest live audio data from Orcasound hydrophones via cloud-based storage
- Detect orca calls in short (2.45-second) data segments using a machine learning model
- Notify moderators and present 1-minute candidates via a moderation UI
- Validate labels based on moderator feedback
- For true positive candidates issues notifications to subscribers
For more details refer to the Github repository below.
Open-source code and open data
- aifororcas-livesystem : OrcaHello prod repository (Github)
- Pod.Cast dataset & annotation system
Presentations & features
Invited Talks
ML “in the wild”
Talk by Prakruti Gogia, Akash Mahajan, and Aayush Agrawal on Wednesday, December 16, 2020.
MERIDIAN webinar on “Sound detection and classification with deep learning”
Press Features
- Root & STEM Podcast: October 2021
- Ampere, Diving in data: How AI could help save the whales: Nov 2021
Lead contributors and collaborators
- Prakruti Gogia — Project lead (2019-2023)
- Akash Mahajan — Project lead (2019-2023)
- Dave Thaler — Project Github lead & maintainer (2023+)
- Chris Hanke — Project management (2019+)
- Patrick Pastore — Machine learning development (2023+)
- Michelle Yang — Project co-maintainer, full-stack development (2019+)
- Aayush Agrawal — Machine learning development (2019-2020)
- Mike Cowan — Moderation system, front end (2019-2023)
- Claire Goetschel — Moderation system, design (2019-2023)
Bioacoustic experts (partners / annotators / beta-testers)
- David Bain (2019+, Orca Conservancy)
- Scott Veirs (2019+, Beam Reach, Orcasound)
- Val Veirs (2020+, Beam Reach, Orcasound)
- Monika Wieland Shields (2021+, Orca Behavior Institute, Orcasound)
Hackathon participants (at least one hackathon in 2019-2021+)
- Nithya Govindarajan
- Herman Wu
- Dmitrii Vasilev
- Kenneth Rawlings
- Kunal Mehta
- Diego Rodríguez
- Adele Bai
- Kadrina Queyquep
- Trisha Hoy
- Athapan Arayasantiparb
- Ming Zhong
- Morgane Austern
- Joyce Cahoun
- Anurag Peshne
- Rob Boucher
- Shahrzad Gholami
- For more, see Orcasound Hacker Hall of Fame
Please refer to Github for the latest open-source contributors.
Note: this is volunteer-driven & is not an official product of Microsoft.
Support & credits
- Microsoft Garage on Twitter (organizers of MS Hack 2019)
- Microsoft AI for Earth
- $15k Azure credits and $15k labeling funds to Orca Conservancy (led by Dave Bain)
- $15k Azure credits to University of Washington and Orcasound (Valentina Staneva and Shima Abadi with Scott Veirs, Val Veirs, other Orcasound partners)
- Watkins Marine Mammal Sound Database, Woods Hole Oceanographic Institution (non-commercial academic or personal use of killer whale un/labeled recordings for initial Pod.Cast model training)
- Software: Pytorch | FastAI