MarineLife-16K Read the paper

MarineLife-16K: An Attribute-Rich Video-Text Dataset for Motion-Aware Marine Video Understanding

Quang-Trung Truong1,2Thanh-Nghia Nguyen3Phu-Phong Vo3Jaeyeon Kim2Cheng Chen2Jie Zhang2Binh-Son Hua4Duc Thanh Nguyen5Ivor Tsang2Sai-Kit Yeung1

1Hong Kong University of Science and Technology, Hong Kong SAR 2Institute of Advanced Intelligence and Computing (IAIC), Agency for Science Technology and Research (A*STAR), Singapore, Singapore 3University of Information Technology, Vietnam National University, Ho Chi Minh city, Vietnam 4Trinity College Dublin, Ireland 5Deakin University, Australia

IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2027

Paper GitHub Hugging Face
View video examples

Overview

MarineLife-16K is an attribute-rich marine video–text dataset with 2,000 video–text pairs and 16K question–answer pairs. Its annotations cover subjects, appearance, behavior, environmental conditions and camera viewpoint.

The benchmark evaluates video question answering and captioning, with a focus on distinguishing camera movement from object motion in underwater scenes.

2,000video–text pairs
16Kquestion–answer pairs
Benchmark tasksVideo QA CaptioningObject, scene and motion understanding

Annotation Highlights

MarineLife-16K overview: underwater video sequences with object, scene and motion annotations connected to captioning and question-answering tasks.

Video, attribute and motion annotations support video QA and captioning.

Illustration of Motion Hallucination

Reef video viewpoints and camera trajectory beside Gemini and dataset descriptions, with highlighted differences about camera and object movement.

Gemini description and dataset reference for the same video, with differences in camera and object motion highlighted.

Data Examples

Dataset sample01 / 05

Reef fish

Big Island · January 2023

↓ Video

Five illustrative clips, not the full benchmark.

A Three-Stage, Human-in-the-Loop Caption Engine

  1. 01

    Annotate

    Human annotators record objects, appearance, activity, scene conditions and camera viewpoint.

  2. 02

    Caption & review

    A captioner composes descriptions; a reviewer checks them against the annotated attributes.

  3. 03

    Refine

    Descriptions that need revision pass through refinement before becoming dataset captions.

Three-stage annotation pipeline: human attribute annotation, caption generation and review, followed by refinement when required.

Attribute-to-Caption Example

Parrotfish video example showing human-extracted attributes, a color-coded generated caption and the human-refined caption.
Annotation Statistics
Bar charts of annotated object categories, size and shape, color, texture, lighting, video sharpness, camera position, camera angle and motion.

Evaluation Dimensions and Examples

Detailed examples of MarineLife-16K evaluation dimensions, including object recognition, morphology, camera viewpoint, behavior, environment, lighting and caption hallucinations.

Benchmark Results

Open- and closed-source VLMs compared across eight evaluation dimensions under static- and moving-camera settings.

Two radar plots comparing open-source VLMs with 7–8B and 30–80B parameters and closed-source VLMs under static-camera and moving-camera conditions across MBC-so, PMT, MBC-mo, PDO, OBK, SM-e, SM-l and OEMU.

See the paper for metric definitions, scoring scales and evaluation subsets.

Cite MarineLife-16K

BibTeX · WACV 2027
@inproceedings{truong2027marinelife16k,
  title={MarineLife-16K: An Attribute-Rich Video-Text Dataset for Motion-Aware Marine Video Understanding},
  author={Truong, Quang-Trung and Nguyen, Thanh-Nghia andVo, Phu-Phong and Kim, Jaeyeon and Chen, Cheng and Zhang, Jie and 
    Hua, Binh-Son and Nguyen, Duc Thanh and Tsang, Ivor and Yeung, Sai-Kit},
  booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2027}}