How to download images and bounding boxes from FathomNet using Python

FathomNet provides fathomnet-py, a Python client-side API to help scientists, researchers, and developers interact with FathomNet Database data. In this article, we’ll step through code demonstrating a common use case: downloading images and bounding boxes.

Setup

If you want to follow along, you’ll first need to install the FathomNet Python package via pip. You’ll need Python ≥3.8.

pip install fathomnet

Complete documentation of the fathomnet-py project is available at fathomnet-py.readthedocs.io, and the code is available at github.com/fathomnet/fathomnet-py.

1. Import the modules

First things first, we need to import the modules we want to use from fathomnet. The API modules live in the fathomnet.api subpackage, so we’ll import from there:

>>> from fathomnet.api import images, boundingboxes

Now, for example, if we want to get a count of all bounding boxes:

>>> boundingboxes.count_all()
Count(objectType='BoundingBox', count=134906)

2. Pick a concept

A concept is simply a term used to describe something seen in an image. It could be a species name, a description of a substrate, a piece of equipment, etc.

For the purpose of this demonstration, we’ll just pick a random concept from FathomNet to focus on. We can grab a list of all concepts used in FathomNet:

>>> all_concepts = boundingboxes.find_concepts()

and pick a random one:

>>> import random
>>> random_concept = random.choice(all_concepts)
>>> random_concept
'Paragorgia arborea'

Great! We’ve now randomly picked a concept (in this case, a coral: Paragorgia arborea) that has some existing images and bounding boxes in FathomNet.

3. Get a list of images for that concept

Now that we know what we want, let’s ask FathomNet for a list of images that have a bounding box of Paragorgia arborea in them. We’ll call the images.find_by_concept function and check how many images we’re working with:

>>> random_concept_images = images.find_by_concept(random_concept)
>>> len(random_concept_images)
1174

So FathomNet has records for 1174 images of Paragorgia arborea, which are now stored in our random_concept_images list. Let’s check one out.

>>> random_concept_images[746]
AImageDTO(
    id=2430897, 
    uuid='adee5c17-cf2b-426b-a2fe-7f40021c854c', 
    url='https://database.fathomnet.org/static/m3/framegrabs/Doc%20Ricketts/images/0850/04_03_26_00.png', 
    valid=True, 
    imagingType='ROV', 
    depthMeters=902.0, 
    height=1080, 
    lastValidation='2021-10-01T07:50:25.267686Z', 
    latitude=36.32948, 
    longitude=-122.312989, 
    salinity=34.4370002746582, 
    temperatureCelsius=4.065000057220459, 
    oxygenMlL=0.31200000643730164, 
    pressureDbar=911.4000244140625, 
    sha256='270bf3f6d7a1d855ad0fb5b03e966477e24cfce90097cc4f4460f84733e45faa', 
    contributorsEmail='brian@mbari.org', 
    tags=[
        ATagDTO(
            uuid='32290ba4-3f01-4ca7-aa99-9afd73bad1b8', 
            key='source', 
            mediaType='text/plain', 
            value='MBARI/VARS'
        )
    ], 
    timestamp='2016-06-02T18:10:47Z', 
    width=1920, 
    boundingBoxes=[
        ABoundingBoxDTO(
            uuid='d425d035-4892-460b-bb2b-38b71813ca45', 
            userDefinedKey='d39d288f-357d-438e-d76c-e5e0a9c4aa1e', 
            concept='Paragorgia arborea', 
            height=671, 
            occluded=None, 
            observer='lonny', 
            width=654, 
            x=834, 
            y=133, 
            rejected=False, 
            verified=False
        ), 
        ABoundingBoxDTO(
            uuid='a110bdbb-a5cf-4242-b1ff-e0465d9de23c', 
            userDefinedKey='71657a55-7f48-4299-7b63-fee0a9c4aa1e', 
            concept='Paragorgia arborea', 
            height=831, 
            observer='lonny', 
            width=439, 
            x=392, 
            y=40, 
            rejected=False, 
            verified=False
        ), 
        ABoundingBoxDTO(
            uuid='0adac0ed-ad1f-4734-8f0a-859db27d60bd', 
            userDefinedKey='803d87c7-bd30-484e-146f-68dfa9c4aa1e', 
            concept='Paragorgia arborea', 
            height=341, 
            observer='linda', 
            width=274, 
            x=939, 
            y=637, 
            rejected=False, 
            verified=False
        )
    ], 
    createdTimestamp='2021-09-29T21:23:44.533Z',
    lastUpdatedTimestamp='2021-10-01T07:50:25.277Z'
)

Here we can see all the data associated with that image, including the list of bounding boxes. At this point, we can download the images and munge the bounding boxes into our favorite format. But first, let’s follow the image URL and take a look!

Image of Paragorgia arborea, courtesy of MBARI

Aside: Advanced constraining

In the example above, we constrained our search by the concept alone by using the images.find_by_concept function. There are a few other ways to search for images (see the fathomnet-py documentation) as well as a generalized way to specify search constraints.

Let’s say we want to find images of Vampyroteuthis infernalis at depths between 600 and 800 meters. To do this, we set up a constraint object:

>>> from fathomnet.models import GeoImageConstraints
>>> constraints = GeoImageConstraints(
...     concept='Vampyroteuthis infernalis',
...     minDepth=600.0,
...     maxDepth=800.0
... )

and then call the general images.find function:

>>> vampire_squid_constrained_images = images.find(constraints)

Aside: Using the taxonomic tree

Oftentimes, we want to gather images and bounding boxes for multiple concepts by their taxonomic association. The taxa module provides functions to perform taxonomic look-ups to inform this process.

To use the functions in this module, we’ll need to specify a taxonomy provider. We can get a list of FathomNet’s available providers with taxa.list_taxa_providers:

>>> from fathomnet.api import taxa
>>> taxa.list_taxa_providers()
['mbari', 'worms']

A note on taxa providers: MBARI is mostly specific to California, but it’s fast and robust. worms is global but, due to the large number of available names, should only be used to find taxa at or below the genus level.

For example, let’s say we want images of Bathochordaeus and all its descendant concepts (according to the MBARI taxonomy provider). We’ll call the taxa.find_taxa function:

>>> all_batho_taxa = taxa.find_taxa('mbari', 'Bathochordaeus')

This gives us a list of Taxa objects:

[
    Taxa(name='Bathochordaeus', rank='genus'),
    Taxa(name='Bathochordaeus charon', rank='species'),
    Taxa(name='Bathochordaeus mcnutti', rank='species'),
    Taxa(name='Bathochordaeus stygius', rank='species')
]

We can then loop over those taxa and query all their images:

>>> all_batho_images = []
>>> for batho_taxa in all_batho_taxa:
...     batho_images = images.find_by_concept(batho_taxa.name)
...     all_batho_images.extend(batho_images)

Alternatively, we can do the same search by setting a taxaProviderName in the technique described in “Aside: Advanced constraining”. The search will use that taxa provider to look up the descendant taxa and automatically include them in the search results.

>>> from fathomnet.models import GeoImageConstraints
>>> constraints = GeoImageConstraints(
...     concept='Bathochordaeus',
...     taxaProviderName='mbari'
... )
>>> all_batho_images = images.find(constraints)

For more granular control of the concepts to use, we can walk up or down the taxonomic tree with the taxa.find_children and taxa.find_parent functions:

>>> taxa.find_children('mbari', 'Bathochordaeus')
[
Taxa(name='Bathochordaeus stygius', rank='species'),
Taxa(name='Bathochordaeus charon', rank='species'),
Taxa(name='Bathochordaeus mcnutti', rank='species')
]
>>> taxa.find_parent('mbari', 'Bathochordaeus')
Taxa(name='Bathochordaeinae', rank='subfamily')

4. Download the images and generate annotations

This section will walk through downloading the images and generating annotation files for the bounding boxes. There are a multitude of image annotation formats out there, but for the purpose of demonstration we’ll pick a common, human-readable one: Pascal VOC.

Install pascal-voc-writer

We will make use of the pascal-voc-writer package to help format the bounding boxes, so let’s install it:

pip install pascal-voc-writer

Create a target directory

To get the images themselves, we’ll extract the URL from each image record and download the image via HTTP. We need a directory to store the downloaded images and generated annotation files. Let’s make one named after the selected concept (in this case, Paragorgia arborea/):

>>> import os
>>> data_directory = random_concept
>>> os.makedirs(data_directory, exist_ok=True)

The plan is to loop through each image record, download its corresponding image, and write out a Pascal VOC annotation XML file. We’ll end up with a file structure like:

Paragorgia arborea/
    1d763849-5f8a-4cef-bbbc-c12968497abe.png
    1d763849-5f8a-4cef-bbbc-c12968497abe.xml
    eba151e3-35bf-4bb9-8bf4-9482dd2178e5.png
    eba151e3-35bf-4bb9-8bf4-9482dd2178e5.xml
    94b5660e-74d9-4f05-8bb3-1a6d782ff1e5.png
    94b5660e-74d9-4f05-8bb3-1a6d782ff1e5.xml
    5dc67d66-5fa0-425f-893c-f3543b1847e3.png
    5dc67d66-5fa0-425f-893c-f3543b1847e3.xml
    3a65854c-59b7-4d73-af49-d6a59bc08c24.png
    3a65854c-59b7-4d73-af49-d6a59bc08c24.xml
    ...

Run it

Now, let’s make it happen. We’ll define a function to download an image:

>>> from urllib.request import urlretrieve
>>> def download_image(image_record):
...     url = image_record.url                     # Extract the URL
...     extension = os.path.splitext(url)[-1]
...     uuid = image_record.uuid
...     image_filename = os.path.join(data_directory, 
...                                   image_record.uuid + extension)
...     urlretrieve(url, image_filename)        # Download the image
...     return image_filename

and a function to write an annotation:

>>> from pascal_voc_writer import Writer
>>> def write_annotation(image_record, image_filename):
...     writer = Writer(image_filename,
...                     image_record.width,
...                     image_record.height,
...                     database='FathomNet')
...     for box in image_record.boundingBoxes:
...         concept = box.concept
...         if box.altConcept is not None:
...             concept += ' ' + box.altConcept
...         writer.addObject(box.concept,
...                          box.x,
...                          box.y,
...                          box.x + box.width,
...                          box.y + box.height)
...     xml_filename = os.path.join(data_directory, 
...                                 image_record.uuid + '.xml')
...     writer.save(xml_filename)            # Write the annotation

Note: In this example, we’re renaming the file to be its image UUID (universally unique identifier) so that we won’t have filename conflicts.

Tying those two functions together, we can run the process:

>>> for image_record in random_concept_images:
...     image_filename = download_image(image_record)
...     write_annotation(image_record, image_filename)

Bada-bing bada-boom.

We have now downloaded images and bounding boxes from FathomNet. Awesome! To verify, we can check out a generated annotation file:

<annotation>
    <folder>Paragorgia arborea</folder>
    <filename>1d763849-5f8a-4cef-bbbc-c12968497abe.png</filename>
    <path>/data/Paragorgia arborea/1d763849-5f8a-4cef-bbbc-c12968497abe.png</path>
    <source>
        <database>FathomNet</database>
    </source>
    <size>
        <width>1920</width>
        <height>1080</height>
        <depth>3</depth>
    </size>
    <segmented>0</segmented>
    <object>
        <name>Paragorgia arborea</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <difficult>0</difficult>
        <bndbox>
            <xmin>259</xmin>
            <ymin>242</ymin>
            <xmax>191</xmax>
            <ymax>125</ymax>
        </bndbox>
    </object>
</annotation>

Thanks for reading!

If you want to use this code, here’s a gist with the snippets above in script form.

import os
import random

from urllib.request import urlretrieve

from fathomnet.api import images, boundingboxes
from pascal_voc_writer import Writer

# Get a count of all bounding boxes in FathomNet
print(boundingboxes.count_all())

# Grab a list of all concepts from FathomNet
all_concepts = boundingboxes.find_concepts()

# Pick a random one
random_concept = random.choice(all_concepts)
print(random_concept)

# Get a list of images for that concept from FathomNet
random_concept_images = images.find_by_concept(random_concept)
print(len(random_concept_images))

# Create a target directory
data_directory = random_concept
os.makedirs(data_directory, exist_ok=True)

# Function to download an image
def download_image(image_record):
    url = image_record.url                     # Extract the URL
    extension = os.path.splitext(url)[-1]
    image_filename = os.path.join(data_directory, 
                                  image_record.uuid + extension)
    urlretrieve(url, image_filename)        # Download the image
    return image_filename

# Function to write an annotation
def write_annotation(image_record, image_filename):
    writer = Writer(image_filename,
                    image_record.width,
                    image_record.height,
                    database='FathomNet')
    for box in image_record.boundingBoxes:
        concept = box.concept
        if box.altConcept is not None:
            box.concept += box.altConcept
        writer.addObject(box.concept,
                         box.x,
                         box.y,
                         box.x + box.width,
                         box.y + box.height)
    xml_filename = os.path.join(data_directory,
                                image_record.uuid + '.xml')
    writer.save(xml_filename)            # Write the annotation

# Download the images and generate annotations
for image_record in random_concept_images:
    image_filename = download_image(image_record)
    write_annotation(image_record, image_filename)