ICV

Real-time computer vision for drilling rigs

ICV is a real-time computer vision system dvhb designed and built for an oilfield services company in 2020–2021. A server on the rig watches the drilling floor through the CCTV cameras, reads the rig’s sensors and warns the crew while an operation is going wrong, before a human error turns into an accident thousands of metres down.

Client
Oilfield services company, Europe
Years
2020–2021
Team
8 people at dvhb
Stage
Proof of concept on a working drilling rig
Stack
SSD object detection, optical flow, CVAT, RTSP, Modbus, MQTT and Protobuf, PostgreSQL, Grafana

The problem

Drilling crews follow strict procedures, and small slips are expensive. The client chose two to start with.

Drill string without movement

Left standing in the slips too long, the string can stick in the well, and freeing stuck pipe can stop drilling for days.

Casing make-up

Every casing connection needs the right number of turns. A bad one fails thousands of metres down, where it is very costly to fix.

An alert helps only while the crew can still act, so the system had to see and decide in real time, on the rig. Before writing code, the team went through the client’s incident database and interviewed drilling supervisors, drillers and assistant drillers about how these incidents happen.

Marker 1

Drill string without movement

Each rule ICV checks is a marker: a business rule over what the cameras and sensors see. For the first one the team trained an SSD object detector on four classes of 700 frames each: slips open, slips closed, slips closed and covered, and the spider. A region of interest and scene segmentation keep the model on the rotary table. A log analyzer turns frame-by-frame detections into time intervals, filters the noise and raises an alert when the string stands still longer than the rig’s limit, such as 3 or 5 minutes.

Rotary table seen from a ceiling camera on the rig, the detector's box around the slips labelled slips closed with confidence 0.98
What the rig camera sees: slips closed, confidence 0.98.
Drilling rig floor from a CCTV camera, with the pipe, the rotary table, the tongs and a worker each coloured by the segmentation model, labelled slips closed 0.998
Scene segmentation keeps the detector on the rotary table.
Three views of the same pipe on the rig floor: the optical flow field, the tracked feature points and the pipe highlighted with a detection box
Optical flow, tracked points and the pipe during make-up.

Marker 2

Counting casing turns

Casing pipes carry no mark that shows a full turn, so the team used the dirt: every pipe has its own pattern on its surface. ICV learns that pattern and counts a turn each time it comes round again, with tracking, image stabilisation, feature detection and optical flow following the pipe through the make-up.

Architecture

Every rig gets its own well-site server, so detection runs next to the cameras and keeps working when the internet doesn’t.

Architecture diagram: rig cameras and sensors feed a well-site server with video grabber, CV module, log analyzer and well manager, which sends local alarms and a dashboard and syncs over MQTT to a cloud server with notifications and mobile apps
System architecture. Tap the diagram to open it at full size.
Inputs
PoE cameras record to a video recorder that streams over RTSP; rig sensors connect over Modbus.
Well-site server
A video grabber feeds the CV module: detection, plus post-processing that drops impossible combinations. A sensor grabber publishes sensor data over MQTT with Protobuf. The log analyzer combines both, and the well manager applies the rig’s settings and sends alerts.
Rig floor
Sound and light alarms for the crew, and a dashboard for the supervisor with every event and its video clip.
Cloud
Events from all rigs over MQTT into PostgreSQL, the managers’ dashboard, mobile apps, and notifications by SMS, email, push and webhooks.
Supervisor dashboard design: a live rig floor view marked No risks with the drill string hold delay, and a list of incidents per shift with video clips, times, durations and events such as String hold > 180 s
Supervisor dashboard (design): the live view and every incident of the shift with its clip.

New markers had to be possible without new code, so the team also designed a visual marker composer, where a rule is a graph of detections, sensor values and conditions.

Data pipeline

The rig cameras produced terabytes of raw CCTV footage. The team automated the path from raw video to labelling in CVAT, GPU training and testing; more GPU capacity cut a training run from 18 hours to 2. A video archive with search, filters, automatic tagging, clip cutting and thumbnails let the team find and label new situations fast.

Field lessons

Image quality
Steam, snow, mud on the lens and night shifts, all of which the training data had to cover.
Connectivity
Mobile internet with a monthly data cap and low bandwidth, so video stays on the rig and only events and short clips go to the cloud.
Hardware
Overheating, GPU driver failures and power cuts; remote monitoring and debugging tools mattered as much as the models.
Night CCTV frame of the rig floor half hidden by steam, with snow piled in the foreground
A night shift on the rig floor: steam and snow in front of the camera.

Team

Eight people at dvhb: a product designer, a data scientist and ML engineer with a background in oil and gas, a principal backend architect for high-load computer vision and ML systems, backend and frontend engineers, a DevOps engineer and a data manager, led by Alex Samson as computer vision solution architect.