Marine ecosystems are under pressure from unsustainable fishing, with some populations declining faster than they can recover. Illegal, unreported, and unregulated (IUU) fishing further contributes to the problem, threatening biodiversity, economies, and global seafood supply chains.

While many fisheries have adopted Electronic Monitoring (EM) systems to support sustainable practices, these tools still depend heavily on manual video review, a process that is slow, expensive, and limited in scale.

In collaboration with The Nature Conservancy (TNC), we set out to change that. The challenge? Build a robust, near real-time monitoring system that could operate directly onboard vessels, processing hours of raw video footage into structured, actionable data without human intervention or cloud access.

It’s a problem few have tackled, and even fewer have solved in production. By combining edge computing, computer vision, and deep domain expertise, we built an AI-driven solution that detects species, tracks catch and discards, and generates daily reports, all in near real-time, right on the vessel.

This is AI applied in one of the ocean’s most demanding environments, and a clear example of how custom-built solutions can drive impact where off-the-shelf tools fall short.

While the solution is still evolving, early results are already promising. Continued testing and scaling will determine its full potential.

A closer look at onboard monitoring

Some industrial fishing vessels already carry video monitoring systems that record everything happening on deck. After each trip, trained observers review hours of footage to assess the catch, identify species, and evaluate whether discards were handled properly.

But this process is slow, expensive, and limited in scope. Reviewing all the footage isn’t feasible, so only a fraction is typically analyzed, leaving blind spots in the data. And because these reviews often happen weeks or months after a trip, any insights come too late to take corrective action.

Monitoring, however, is not just about ticking a compliance box, it’s a cornerstone of sustainable fisheries management. With reliable, timely data, it becomes possible to:

While Electronic Monitoring systems already capture valuable data through cameras, sensors, and GPS, they still depend on human-led review. This creates a major bottleneck.

That’s why innovation is essential. By combining edge computing with AI-powered video analysis, we can close that gap, turning raw footage into near real-time insights and helping fisheries managers act faster and more effectively.

The solution: AI-powered near real-time monitoring

Our approach focused on three key milestones:

1. Building a high-quality dataset
2. Developing an AI system capable of detecting, tracking, and counting fish species in near real-time
3. Generating daily reports that summarize onboard activity and flag potential compliance risks

The vessels we’re working with are equipped with EM systems, including two cameras, GPS sensors, and an NVIDIA Jetson edge device. The AI system runs directly on the edge, processing video footage in near real-time to identify species and count fish throughout the day. At the end of the day, it generates a report detailing the catch and assigns a risk score to flag potentially non-compliant or suspicious activity. Running AI models on the edge required carefully balancing model size, speed, and accuracy to perform reliably within limited onboard resources. While still under development, the system has already shown strong early results and promising potential to transform fisheries monitoring.

Building the dataset

This step was particularly challenging due to the complexities of labeling data. We initially began with Fishnet AI, a public dataset of fish images, which helped pretrain the model on general features. But differences in vessel types, species distribution, and camera angles made it unsuitable—so we shifted to data captured directly from the target vessels.

The EM systems onboard recorded daily fishing activity, with video stored locally and later transmitted to the cloud via Starlink. A team of marine biology experts reviewed the recordings to identify which species were caught and when, logging their observations in an Excel file. However, they did not annotate bounding boxes around the fish—doing so would have been prohibitively time-consuming and expensive.

To convert these expert annotations into usable training data, we combined them with the video and programmatically generated rough bounding boxes in frames where fish were expected to appear. Our labelers then manually refined those boxes as needed. This hybrid approach allowed us to incorporate expert insights while keeping labeling costs manageable.

Tracking fish across frames was one of the most labor-intensive steps. At 12 frames per second, a single hour of video produced more than 43,000 frames—far beyond what we could annotate internally.

We used CVAT, an open-source tool that also provides labeling services, to streamline the annotation process. To ensure quality and consistency, we manually labeled the first 10 haulings—each representing a full fishing session—and created detailed guidelines for the labeling team.

In total, we worked with data from 187 fishing days, covering over 5,000 individual fish across 63 species, resulting in more than 16,000 hours of annotated video to train and evaluate our system.

The AI system components

At the core of the solution is the AI system, designed to perform three key tasks: detecting, tracking, and counting fish. It’s built using the Ultralytics Python library, an open-source framework that enabled us to customize the pipeline and deploy efficiently on edge devices.

Fish detector

We use a medium-sized YOLOv11 model to detect and classify fish in each video frame. The model draws bounding boxes around each fish and assigns a species label.

Fish tracker

The BoT-SORT tracker assigns a unique ID to each individual fish and follows its trajectory across frames. To optimize performance, we manually tuned its parameters by testing different configurations on the validation set.

Fish counter

This step of the AI system handles two key scenarios. On one hand, it counts fish that cross a virtual line along the edge of the vessel’s deck, distinguishing between those retained onboard, and those returned to the sea (vessel discards). On the other, it tracks water discards events, where fish are released directly from the water without ever crossing the line.

How onboard counting works:

We implemented a custom rule-based counter to determine when fish are retained or released. This logic is built into a CustomObjectCounter class, which extends Ultralytics’ BaseObjectCounter and incorporates domain-specific rules tailored to fishing operations. Let’s explore both processes.

1. A virtual line is drawn along the vessel’s deck edge.
2. Each fish is tracked across frames, and when it crosses the line, the system logs the event—but waits to confirm it.
3. It waits a few more frames before confirming the event. This delay allows the model to capture clearer images once the fish is out of the water, since early detections of submerged fish are often too blurry or obscured for accurate identification.
4. To reduce false positives, the system only counts fish with sufficiently long tracks, ignoring quick, erratic detections that may actually be reflections, hoses, or other objects.
5. The direction of movement determines the type of event:

6. After each track is complete, the system reviews all detections for that fish and assigns the most frequently predicted species label—ensuring a reliable match between species ID and count.

Discard detection

Certain species—especially those that are threatened or protected—must be released immediately, without being brought onboard. These discard events are different from standard IN/OUT cases: the fish never cross the counting line and often appear only briefly in the water, sometimes obscured by foam or partially submerged.

Because discards are difficult to interpret in real time, this detection process runs retrospectively at the end of each fishing day. The system stores all detection and tracking data, and then a dedicated pipeline analyzes the full dataset to identify likely discard events.

Species-specific heuristics are applied, flagging consistent water-level detections without any corresponding IN or OUT event nearby in time. This helps distinguish actual discards from false positives or interrupted tracks.

By handling discards as a separate process, we ensure they are properly accounted for. This is an essential step in monitoring compliance and protecting vulnerable species.

How we evaluate the AI system

To ensure the AI system performs reliably in real-world conditions, we evaluate it on a carefully curated set of unseen videos. This is footage the model has never encountered during training. These test videos are selected to reflect the full diversity of operating conditions the system may face once deployed. This includes variation in species composition, catch volume, discard behavior, lighting conditions, geographic location, and seasonal patterns. By evaluating under a broad range of scenarios, we can better understand how the system will generalize beyond the training set.

Performance metrics

Once the test set is selected, we compare the AI's predictions to ground truth annotations. For a prediction to be counted as correct, it must match three things: the species, the event type (IN or OUT), and the frame at which the fish crosses the reference line—within a defined tolerance window. This strict evaluation ensures we’re not just matching totals by chance (with over- and under-counts canceling out), but actually detecting and classifying each individual event correctly.

We measure performance at three levels of granularity: overall catch-level accuracy (KPI 1), accuracy by broad taxonomic groups like fish, rays, sharks, and turtles (KPI 2), and per-species accuracy (KPI 3).

Across these levels, we compute standard metrics:

• Precision: The proportion of predicted events that were actually correct
• Recall: The proportion of real events that the system successfully detected
• F1-score: The harmonic mean of precision and recall, offering a balanced measure of accuracy
• Count error: The percentage difference between the predicted and ground truth counts, offering a direct view of how far off the system was from the true counts

Our primary benchmark is keeping the retained count error below 10% per species—a threshold defined by TNC as the minimum standard for the system to be considered reliable. After each model retraining, we evaluate performance on a dedicated test set and only deploy the new version if it outperforms the current one across key metrics.

This table presents an example of system performance on a representative set of evaluated catches. While it illustrates the solution’s potential, the system is still under active development and ongoing evaluation.

Seamless Deployment with CI/CD

To close the loop between development and deployment, we use a CI/CD pipeline that automates the delivery of updated components—fish detector, fish tracker, and fish counter—to Jetson devices on vessels. This ensures that every improvement to the model or supporting code is reliably and reproducibly shipped to the edge with minimal manual effort.

Whenever a new code release is created, the most recent trained fish detector model is uploaded to an AWS S3 bucket, and a Docker image containing the latest fish tracker and fish counter code is built and pushed to a Docker registry. From there, the pipeline remotely connects to each Jetson device, pulls the latest assets, and deploys them into a fresh Docker container, replacing the existing one.

Faster insights with the daily report

The daily report transforms model outputs into actionable information, enabling faster, more informed decision-making. Generated at the end of each fishing day by the AI system running on the vessel’s Jetson device, it provides a near real-time view of onboard activity without requiring manual review or port return.

It begins with a summary of the day’s fishing activity, including the number of retained, (kept onboard), vessel discards (caught and released), and water discards (fish released directly from the water without ever boarding).

The report also includes a daily risk score, a single metric designed to highlight potential irregularities in fishing behavior. This score is based on five key dimensions:

1. Elog alignment: Gaps between AI-predicted counts and the captain’s electronic logbook
2. Illegal species: Detection of species that are not permitted to be caught
3. Geolocation issues: Fishing activity in protected or restricted areas
4. Inference gaps: Incomplete or missing video processing
5. Operational anomalies: Issues such as missing footage, GPS signal loss, or camera movement

The risk score helps prioritize vessels for further inspection, guides targeted manual review, and contributes to future sustainability certification programs.

In addition to metrics, the report includes a detailed catch log. For each event, it records the species, event type (IN or OUT), model confidence, timestamp, and a representative frame showing the fish as it crosses the virtual counting line. These visuals provide essential context for validation and review.

To add geographic insight, the report maps the vessel’s route and marks the location of each catch in relation to boundaries such as marine protected areas and bathymetric zones.

Finally, all catch data is aggregated across multiple levels: group (e.g., fish, shark, ray), subgroup (e.g., tuna, marlin), and species. This structure provides stakeholders with both high-level overviews and detailed, species-specific insights.

Monitoring and alert system: a new layer of visibility

As a recent addition to the solution, we introduced a real-time monitoring and alert system powered by AWS Managed Grafana. This new component enhances situational awareness by giving teams live visibility into system performance and fishing activity—without waiting for end-of-day reports.

Integrated with our existing infrastructure, Grafana pulls data directly from AWS S3 via Athena and displays it through tailored dashboards. Each dashboard serves a specific audience:

• Operational dashboard: Built for developers and admins, it tracks device status, GPU usage, and logs, helping maintain system stability.
• Historical dashboard: Designed for business users, it visualizes long-term trends like species composition and recurring risk patterns.
• Real-time catch dashboard: Updates hourly to show what’s being caught and where—bringing elements of the daily report into live view.

To complement these insights, Grafana’s alerting system automatically notifies teams of critical events, such as:

1. Device disconnections or system failures
2. Missing catch events or incomplete inference
3. Detection of illegal species or suspicious fishing patterns
4. Activity inside protected areas or gaps in elog data

By surfacing both technical and behavioral anomalies as they happen, this real-time layer enables faster responses and smarter oversight.

What’s next: unlocking new capabilities

While the current solution marks a major step forward, we’re actively exploring new ways to expand its impact and intelligence.

Edge vs. cloud: finding the right balance

Edge deployment was a requirement for this project, given the remote nature of fishing operations and limited connectivity at sea. All processing needed to happen locally on the vessel to ensure near real-time performance without relying on internet access.

Edge computing delivered low-latency detection, offline operation, and timely reporting. However, its limited computing power restricts the size and complexity of models we can run, potentially affecting accuracy or advanced feature support.

By contrast, a cloud-based system offers greater processing capacity and scalability but depends on stable connectivity and introduces latency—making it unsuitable for many offshore scenarios.

Looking ahead, we’re exploring a hybrid model: using the edge for fast onboard analysis and offloading complex or uncertain cases to the cloud when possible. This could unlock both responsiveness and deeper insights across a range of conditions.

Size matters: estimating fish length in real time

For many regulated species, size determines whether a catch is legal. We're developing a fish size estimation feature to provide real-time length measurements directly from the video feed.

To ensure precision, we’ve already implemented:

• Lens calibration using chessboard patterns to correct image distortion
• Perspective correction via homography and known deck dimensions
• Camera monitoring to detect angle shifts that could affect measurement accuracy

This capability will give crews immediate feedback on catch legality, supporting compliance and sustainable fishing practices.

Continuous improvement: human-in-the-loop learning

Expert feedback from daily reports will fuel ongoing model retraining. By integrating human corrections into the loop, we’re building a system that not only performs well today but gets smarter over time.

Final thoughts

This project is a real-world example of AI solving a high-stakes, high-complexity problem, on board, in production, and in partnership with a leading conservation organization.

Every component was designed with the constraints of the marine environment in mind. And while the system continues to evolve, it’s already delivering measurable value: faster feedback, more consistent oversight, and the foundation for smarter, more sustainable fisheries management.

At Tryolabs, we specialize in bringing AI to challenging operational settings, whether on land, online, or offshore. This work with The Nature Conservancy is part of our broader AI for Good initiative, focused on using technology to drive meaningful impact at scale. Explore more of our AI for Good work.

If you’re working on a complex problem where AI could make a difference, we’d love to talk. Get in touch