Uploading Large Files

This tutorial focuses on leveraging Python and a custom utility to streamline the process of handling large files for Ocean Exploration and Video Work.

Introduction

In ocean exploration and video work, handling large files is commonplace. Whether it's high-resolution videos from underwater expeditions or extensive datasets from oceanographic research, efficient uploading methods are essential. This tutorial focuses on leveraging Python and a custom utility to streamline the process using the Portal project.

Prerequisites

Before starting, ensure you have Python installed on your system and basic familiarity with the command line interface.

Setting Up the Environment

First, let's install the required dependencies for our utility. You can install the necessary packages using pip:

pip install tator-py

Understanding and Executing the Upload Process

To kickstart the upload process, we need to interface with the Portal project using our custom Python utility. Below is the code snippet to initiate the upload:

python3 tator.util.resumable_upload --host https://ovai.tator.io --token <API_TOKEN> --section <DESIRED_SECTION> --project <PROJECT_ID> --type <MEDIA_TYPE_ID> <PATH_TO_VIDEO_FILE>

The PROJECT_ID , MEDIA_TYPE_ID can be supplied by your project administrator. The API_TOKEN can be retrieved via the Tator UI exposed as part of the portal deployment . DESIRED_SECTION is the name of the section to upload the media to.

This code initializes the resumable uploader and starts uploading the specified file.

Uploading Large Video Files

Uploading large video files can be challenging due to their size. Our utility handles this efficiently by splitting files into manageable chunks and uploading them sequentially. This ensures reliability and minimizes the risk of upload failures.

Monitoring the Upload Progress

Monitoring the upload progress is crucial, especially for large files that may take some time to upload. While the utility handles most of the process automatically, it's helpful to have visibility into the progress to ensure everything is proceeding smoothly.

Handling Upload Failures

Despite our best efforts, upload failures can occur due to various reasons such as network issues or server errors. Our utility includes mechanisms to detect and handle such failures gracefully. It can resume uploads from the point of failure, minimizing data loss and ensuring the integrity of the uploaded files.

The upload tool outputs the command to use to resume the upload, generally the recipe is to replace --section and --project with the partially uploaded media controlled by --media-id as the example below:

python3 resumable_upload.py --host https://ovai.tator.io --token <TOKEN> --media-id <MEDIA_ID> <PATH_TO_VIDEO_FILE>

Finalizing Ingestion

Once all parts of the video are uploaded, we need to finalize the ingestion process. This involves joining the video parts and performing any necessary post-processing steps. This portion of the ingestion process would run on a cloud resource with a fast and stable connection to the uploaded material.

import_resumable.py --token <TOKEN> <MEDIA_ID>

Conclusion

Efficiently uploading large files is essential for ocean exploration and video work. With the right tools and strategies, such as our custom Python utility for the "Portal" project, you can streamline the process and focus more on exploration and discovery.

In this tutorial, we've explored how to initiate the upload process, handle large video files, monitor progress, handle failures, and finalize ingestion. Armed with this knowledge, you're well-equipped to tackle the challenges of uploading large files in your ocean exploration and video projects.