Annotated images from the FathomNet Database were used to train and evaluate the new Segment Anything Model 3 (SAM 3), adding complex underwater scenes to the broad training dataset and expanding the variety of environments the model can recognize. 

Scenes in the ocean present some of the biggest challenges for computer vision models to analyze. Visibility varies by depth, some ocean environments bustle with overlapping animals, drifting sediment obscures a clear view, and many animals move together in large groups. As a result, it is rare to capture a single, clear image of one individual animal. Visual data about these rich underwater landscapes contain vital information about ocean health, but can be challenging for AI to analyze.

To address this, the FathomNet team created a database of expertly annotated ocean images that can be used to train computer vision models to analyze underwater imagery. The latest release from Meta, Segment Anything Model 3 (SAM 3), utilized labeled data from the FathomNet Database to enhance the model's ability to recognize ocean animals. Now, SAM 3 can be used to automatically detect, segment, and track objects in videos of the ocean. 

Adding Segmentation Annotations 

Segmentation annotations offer a more detailed and accurate understanding of underwater scenes compared to bounding boxes alone. This can help improve the performance of computer vision models and enable more accurate analysis of complex scenarios featuring multiple objects interacting with each other.  

AI researchers at Meta first processed a large set of images from the FathomNet Database with SAM 2, using existing bounding boxes as prompts. Then, they trained the new SAM 3 with these images, using SAM 2's results as labels. For 9,222 images, human reviewers refined the segmentation masks and used them to evaluate SAM 3’s performance. Two outputs from this process—the set of 280,118 segmentation masks used for training and the set of 14,247 segmentation masks used for testing—are now available to download from the FathomNet Database. 

Creation Process

1. Bounding boxes→masks

Existing FathomNet Database bounding box annotations were used as prompts for Meta's Segment Anything Model 2 to generate initial segmentation masks.

2. Train/test split

The dataset was divided into train and test subsets for proper evaluation.

3. Human verification

Human review was conducted on the test set to refine, correct, and validate the segmentation masks for accuracy.

4. Model training

The dataset was used to train and evaluate Meta's Segment Anything Model 3, enhancing its ability to generate segmentation masks for underwater imagery.

Expanding the Database and Streamlining Access

FathomNet is committed to streamlining access to AI solutions and ocean data. To unlock the full potential of big ocean data, we need equally big advances in AI. The addition of Meta’s segmentation masks to the FathomNet Database represents a significant step forward in bridging the gap between AI research and ocean science. We look forward to continuing to expand the FathomNet Database to help scientists everywhere study, understand, and protect the ocean.

We encourage machine learning practitioners to download and use the new segmentation annotations to refine and assess their models. Subscribe to our newsletter for updates on additional support and resources related to segmentation masks in the FathomNet Database.

Explore the FathomNet Database

Learn more about the FathomNet program

Join the FathomNet network