Practical case: Jetson Orin Nano OpenCV Motion Detection

Objective and use case

What you’ll build: A CSI camera motion‑detection pipeline on an NVIDIA Jetson Orin Nano Developer Kit using OpenCV and GStreamer. It captures from a Raspberry Pi Camera Module v2 (Sony IMX219), detects motion via background subtraction, overlays bounding boxes, logs events, and optionally records MP4s with the hardware H.264 encoder.

Why it matters / Use cases

• Smart security: Monitor a home entrance; when a person/vehicle moves, push an alert and save a 10‑second clip at 1080p30. Typical detection latency 50–80 ms; NVENC/GPU <10%, CPU 25–35% during motion.
• Industrial monitoring: Watch a conveyor; trigger GPIO or MQTT if motion ceases >2 s during production. Response <100 ms, stable 24/7 with ROI masking to reduce false alarms.
• Wildlife observation: Record only when animals enter a feeding station at night. 720p30 at 5 Mb/s yields ~4–6 MB per 10 s clip and 90%+ storage savings vs continuous recording.
• Occupancy analytics: Count entries/exits in a hallway via an ROI; log timestamps and estimated dwell time (±0.5 s). Handles 1,000+ events/day with CSV logs <1 MB/day.
• Safety zones: Detect motion in no‑go zones near machinery off‑hours; raise alarms and export a daily CSV of events. End‑to‑end detection 50–80 ms; false alarms reduced via min‑area and min‑duration filters.

Expected outcome

• Live 1080p30 preview with real‑time bounding boxes; stable 30 FPS (±1 FPS) and 50–80 ms end‑to‑end detection latency.
• Event logger writing CSV rows (start/stop time, ROI, bbox count/area) and optional MQTT messages; clock‑synced timestamps.
• On‑demand or auto MP4 recording using nvv4l2h264enc at 8 Mb/s (10 s ≈ ~10 MB), I‑frame interval 30; encoder overhead NVENC <10% and CPU +<5%.
• Resource profile at 1080p30 MOG2: CPU 25–35%, GPU/NVENC 5–10%, RAM <1 GB; system power ~5–7 W on Orin Nano Dev Kit for 24/7 operation.
• Configurable ROI, thresholds, and clip pre/post‑roll (e.g., 3 s pre + 7 s post) with log rotation and daily CSV export.

Audience: Embedded/vision developers, makers, facilities and operations engineers; Level: Intermediate.

Architecture/flow: nvarguscamerasrc (IMX219) → video/x-raw(memory:NVMM),1920×1080@30 → nvvidconv → appsink (OpenCV) → BackgroundSubtractorMOG2 → morphology/contours → bbox overlay + event state machine (min area/duration, hysteresis) → CSV/MQTT logging; frames optionally tee to nvv4l2h264enc bitrate=8M control-rate=CBR iframeinterval=30 → h264parse → qtmux → filesink for 10 s clips with pre/post‑roll.

Prerequisites

• NVIDIA Jetson Orin Nano Developer Kit running Ubuntu (JetPack/L4T preinstalled).
• Internet access via Ethernet/Wi‑Fi and a terminal (local or SSH).
• Python 3, OpenCV with GStreamer support, GStreamer core/plugins, and system tools included in JetPack.
• Confirm JetPack, kernel, and relevant NVIDIA packages:

cat /etc/nv_tegra_release
jetson_release -v

uname -a
dpkg -l | grep -E 'nvidia|tensorrt'

• Ensure OpenCV has GStreamer enabled:

python3 - << 'PY'
import cv2
info = cv2.getBuildInformation()
print('GStreamer: YES' if 'GStreamer:                   YES' in info else 'GStreamer: NO')
PY

• Optional but recommended: set MAXN performance (watch thermals; use a fan/heat sink):

sudo nvpmodel -q
sudo nvpmodel -m 0
sudo jetson_clocks

• You can revert performance settings later (see Cleanup at the end).

Materials

• EXACT device model: NVIDIA Jetson Orin Nano Developer Kit + Raspberry Pi Camera Module v2 (Sony IMX219).
• Included 15‑pin CSI ribbon cable for the camera.
• 5 V power supply for the Jetson Orin Nano Developer Kit (as recommended by NVIDIA).
• microSD card or NVMe storage with JetPack installed.
• Optional: active cooling (fan), HDMI display, keyboard, and mouse for local debugging (not required for CLI operation).

Setup/Connection

1) Power down the Jetson and unplug power.
2) Locate the two CSI connectors (CAM0/CAM1) on the Orin Nano Dev Kit. We will use CAM0.
3) Open the small black locking bar on CAM0. Insert the 15‑pin FFC ribbon from the IMX219 camera with the blue stiffener facing away from the connector pins (contacts toward the connector). Press the locking bar down to secure.

Table: CSI connection orientation summary

4) Place the IMX219 camera on a fixed stand directed at a static scene with some potential movement (e.g., your hands entering the frame).
5) Power the Jetson back on. Restart the Argus camera service just in case:

sudo systemctl restart nvargus-daemon

6) Quick GStreamer camera sanity check (headless, discard frames):

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! \
'video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1,format=NV12' ! \
fakesink -e

• If you see the pipeline run without errors for a few seconds, the CSI link is OK. Ctrl+C to stop.

Full Code

Below is a single Python script that:
– Opens the IMX219 via nvarguscamerasrc into OpenCV using GStreamer.
– Runs background subtraction (MOG2) with morphological filtering.
– Detects motion regions above a minimum area, draws bounding boxes, and logs events to CSV.
– Optionally writes hardware‑encoded H.264 MP4 video segments around motion events.

Save as motion_csi_opencv.py in your home directory.

#!/usr/bin/env python3
import cv2
import time
import csv
import os
from datetime import datetime

# Camera/GStreamer parameters
WIDTH = 1280
HEIGHT = 720
FPS = 30
SENSOR_ID = 0  # 0 for CAM0
FLIP_METHOD = 0  # 0..7 (sensor orientation / mounting)
# Optional exposure/gain tuning (uncomment/adjust if needed):
# EXPOSURE_RANGE = "exposuretimerange=100000 1000000"  # 100us..1,000,000us
# GAIN_RANGE = "gainrange=1 8"
# WBMODE = "wbmode=0"  # Auto

# Motion detection parameters
MIN_AREA_PX = 1200        # minimum contour area to consider as motion
DILATE_ITER = 2
ERODE_ITER = 1
HISTORY = 500             # background model history
VAR_THRESH = 16           # sensitivity of MOG2
DETECT_SHADOWS = True
EVENT_COOLDOWN_S = 1.0    # min seconds between events
DRAW = True               # overlay bounding boxes and text
SHOW_WINDOW = False       # set True if you have a local display

# MP4 recording parameters (set RECORD=True to save clips on motion)
RECORD = True
CLIP_SECONDS = 5          # per-event clip length
BITRATE = 4_000_000       # ~4 Mbps
OUT_DIR = "captures"
os.makedirs(OUT_DIR, exist_ok=True)

def gstreamer_csi_pipeline(width, height, fps, sensor_id=0, flip_method=0):
    # Build nvarguscamerasrc → nvvidconv → BGR → appsink pipeline
    return (
        f"nvarguscamerasrc sensor-id={sensor_id} ! "
        f"video/x-raw(memory:NVMM), width={width}, height={height}, framerate={fps}/1, format=NV12 ! "
        f"nvvidconv flip-method={flip_method} ! "
        f"video/x-raw, format=BGRx ! videoconvert ! "
        f"video/x-raw, format=BGR ! appsink drop=true sync=false max-buffers=2"
    )

def gstreamer_h264_writer_pipeline(outfile, width, height, fps, bitrate):
    # OpenCV → appsrc → hardware H.264 (nvv4l2h264enc) → MP4 (qtmux)
    # Note: qtmux requires EOS to finalize MP4; writer.release() will send EOS.
    return (
        f"appsrc ! videoconvert ! video/x-raw,format=BGR,width={width},height={height},framerate={fps}/1 ! "
        f"nvvidconv ! nvv4l2h264enc bitrate={bitrate} insert-sps-pps=true preset-level=1 "
        f"! h264parse ! qtmux ! filesink location={outfile} "
    )

def make_writer(path, width, height, fps, bitrate):
    gst = gstreamer_h264_writer_pipeline(path, width, height, fps, bitrate)
    writer = cv2.VideoWriter(gst, cv2.CAP_GSTREAMER, 0, float(fps), (width, height))
    if not writer.isOpened():
        raise RuntimeError("Failed to open GStreamer MP4 writer pipeline.")
    return writer

def main():
    cap = cv2.VideoCapture(gstreamer_csi_pipeline(WIDTH, HEIGHT, FPS, SENSOR_ID, FLIP_METHOD), cv2.CAP_GSTREAMER)
    if not cap.isOpened():
        raise RuntimeError("Could not open CSI camera via GStreamer. Check nvargus and CSI cable.")

    # Background subtractor
    backsub = cv2.createBackgroundSubtractorMOG2(history=HISTORY, varThreshold=VAR_THRESH, detectShadows=DETECT_SHADOWS)

    # Stats and logging
    last_event_ts = 0.0
    frame_count = 0
    fps_accum = 0.0
    fps_window_start = time.time()
    writer = None
    clip_end_ts = 0.0

    csv_path = os.path.join(OUT_DIR, f"events_{datetime.now().strftime('%Y%m%d_%H%M%S')}.csv")
    with open(csv_path, "w", newline="") as fcsv:
        csv_writer = csv.writer(fcsv)
        csv_writer.writerow(["timestamp", "frame_idx", "fps", "bbox_x", "bbox_y", "bbox_w", "bbox_h", "area"])

        t_prev = time.time()
        print("Press Ctrl+C to stop.")
        try:
            while True:
                ret, frame = cap.read()
                if not ret:
                    print("Frame grab failed; attempting to continue...")
                    time.sleep(0.01)
                    continue

                t_now = time.time()
                dt = t_now - t_prev
                t_prev = t_now
                fps_inst = 1.0 / dt if dt > 0 else 0.0
                fps_accum += fps_inst
                frame_count += 1

                # Background subtraction and morphology
                fgmask = backsub.apply(frame)
                # Shadows (value ~127) can be removed by thresholding if needed:
                _, thresh = cv2.threshold(fgmask, 200, 255, cv2.THRESH_BINARY)
                if ERODE_ITER > 0:
                    thresh = cv2.erode(thresh, None, iterations=ERODE_ITER)
                if DILATE_ITER > 0:
                    thresh = cv2.dilate(thresh, None, iterations=DILATE_ITER)

                contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

                # Assume the largest contour corresponds to main motion
                best_area = 0
                best_bbox = None
                for cnt in contours:
                    area = cv2.contourArea(cnt)
                    if area > best_area:
                        best_area = area
                        best_bbox = cv2.boundingRect(cnt)

                # Event detection
                event = False
                if best_bbox and best_area >= MIN_AREA_PX:
                    if (t_now - last_event_ts) >= EVENT_COOLDOWN_S:
                        event = True
                        last_event_ts = t_now

                # Start or continue recording when motion is active
                if RECORD:
                    if event and writer is None:
                        out_name = datetime.now().strftime("%Y%m%d_%H%M%S") + ".mp4"
                        out_path = os.path.join(OUT_DIR, out_name)
                        writer = make_writer(out_path, WIDTH, HEIGHT, FPS, BITRATE)
                        clip_end_ts = t_now + CLIP_SECONDS
                        print(f"[REC] Started: {out_path}")

                    if writer is not None:
                        writer.write(frame)  # write regardless; stop by time
                        if t_now >= clip_end_ts:
                            writer.release()
                            writer = None
                            print("[REC] Stopped (clip complete).")

                # Draw overlay
                if DRAW and best_bbox and best_area >= MIN_AREA_PX:
                    x, y, w, h = best_bbox
                    cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)
                    cv2.putText(frame, f"Motion area: {int(best_area)}", (x, max(0, y - 10)),
                                cv2.FONT_HERSHEY_SIMPLEX, 0.6, (0, 255, 0), 2)
                    # Log event line
                    avg_fps = (fps_accum / frame_count) if frame_count > 0 else 0.0
                    csv_writer.writerow([datetime.now().isoformat(), frame_count, f"{avg_fps:.2f}", x, y, w, h, int(best_area)])
                    fcsv.flush()

                # Periodic FPS print
                if time.time() - fps_window_start >= 2.0:
                    avg_fps = (fps_accum / frame_count) if frame_count > 0 else 0.0
                    print(f"[INFO] Frames: {frame_count}, Avg FPS: {avg_fps:.2f}")
                    fps_accum = 0.0
                    frame_count = 0
                    fps_window_start = time.time()

                # Optional window for local testing
                if SHOW_WINDOW:
                    cv2.imshow("CSI Motion", frame)
                    if cv2.waitKey(1) & 0xFF == ord('q'):
                        break

        except KeyboardInterrupt:
            print("Interrupted by user.")

        finally:
            if writer is not None:
                writer.release()
            cap.release()
            if SHOW_WINDOW:
                cv2.destroyAllWindows()
            print(f"Events CSV saved: {csv_path}")

if __name__ == "__main__":
    main()

Notes:
– If you mount the camera upside down, adjust FLIP_METHOD (0..7). For a 180° rotate, use 2.
– Recording is optional; set RECORD=False to disable MP4 writing.
– To adjust for low light, tweak the GStreamer source properties (exposure, gain); these can be appended to the nvarguscamerasrc element if needed.

Build/Flash/Run commands

1) Install dependencies (JetPack provides most of them, but ensure they’re present):

sudo apt update
sudo apt install -y python3-opencv python3-numpy gstreamer1.0-tools \
  gstreamer1.0-plugins-base gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \
  v4l-utils

2) (Optional) Install tegrastats helper and confirm its presence:

which tegrastats || echo "tegrastats should be preinstalled on Jetson."

3) Confirm camera with a short run:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! \
'video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1,format=NV12' ! \
fakesink -e

4) Run the motion detection script:

python3 motion_csi_opencv.py

• Output artifacts:
• CSV at captures/events_YYYYMMDD_HHMMSS.csv.
• MP4 files captures/YYYYMMDD_HHMMSS.mp4 (if RECORD=True).

5) Monitor performance while running:

In a separate terminal:

sudo tegrastats

Expect periodic lines with GR3D, EMC, RAM usage, etc. Use Ctrl+C to stop.

6) Optional: Performance mode and clocks (if you didn’t already):

sudo nvpmodel -m 0
sudo jetson_clocks

Warning: Higher thermals—ensure adequate cooling.

TensorRT path (A) for GPU sanity and FPS measurement

We will use TensorRT to build and run an ONNX model (ResNet50) once to confirm GPU acceleration and to capture FPS and power mode metrics. This is separate from the OpenCV motion detection, but it validates the Jetson’s AI stack and gives a baseline.

1) Download a small ONNX model:

mkdir -p ~/models && cd ~/models
wget -O resnet50-v1-12.onnx https://github.com/onnx/models/raw/main/vision/classification/resnet/model/resnet50-v1-12.onnx

2) Build and run a TensorRT engine with FP16:

/usr/src/tensorrt/bin/trtexec --onnx=resnet50-v1-12.onnx \
  --explicitBatch --fp16 --shapes=input:1x3x224x224 \
  --separateProfileRun --workspace=2048 --warmUp=10 --duration=30 --streams=2

• Capture the printed “mean latency” and “FPS” at the end (e.g., >100 FPS typical on Orin Nano for small batch). Run tegrastats simultaneously to see GR3D load and memory usage.

3) Optional INT8 build (requires calibration; not covered at Basic level). If you have an INT8 calibration cache, add –int8 –calib=/path/to/cache.

We stick to path A (TensorRT) for acceleration benchmarking; all inference commands above are reproducible CLI-only and provide quantitative FPS.

Step‑by‑step Validation

1) Validate JetPack and NVIDIA packages:

cat /etc/nv_tegra_release
uname -a
dpkg -l | grep -E 'nvidia|tensorrt'

• Expect an L4T release string and installed nvidia-l4t, tensorrt packages with versions matching your JetPack.

2) Validate camera pipeline:

sudo systemctl restart nvargus-daemon
gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1,format=NV12' ! fakesink -e

• Expected: No ERRs; clean shutdown on Ctrl+C.

3) Run the OpenCV motion detector:

python3 motion_csi_opencv.py

• Expected console messages:
• “Press Ctrl+C to stop.”
• Periodic “[INFO] Frames: N, Avg FPS: ~28–30” at 720p.
• When motion occurs: “[REC] Started: captures/2024…mp4” then “[REC] Stopped (clip complete).”
• Final line: “Events CSV saved: captures/events_…csv”

4) Quantitative metrics with tegrastats:

Open another terminal and run:

sudo tegrastats

• Sample output snippet (your numbers will vary):
• RAM 1600/3964MB (lfb 108x4MB) SWAP 0/1982MB
• CPU@1450MHz (5%@39C) GPU@918MHz GR3D 18% EMC 24%
• Criteria:
• FPS (script): ~30 FPS at 720p.
• GR3D during OpenCV MOG2: often <20% because most work is CPU + ISP/NVMM; spikes may occur due to conversion/encode.
• During trtexec (TensorRT), GR3D rises significantly (e.g., >70%), confirming GPU acceleration is functioning.

5) Validate outputs:
– CSV file contains rows only when motion crosses MIN_AREA_PX, with columns: timestamp, frame_idx, fps, bbox, area.
– MP4 file plays in VLC/ffplay; duration ~CLIP_SECONDS. No visible frame drops.

6) Adjust thresholds:
– For noisy scenes, set VAR_THRESH higher (20–24) and/or increase ERODE/DILATE to suppress small motion.
– Increase MIN_AREA_PX to ignore tiny movements (e.g., reflections, leaves).
– Tune exposure/gain via nvarguscamerasrc if night noise causes false positives.

Troubleshooting

• Error: “Could not open CSI camera via GStreamer.”
• Check cable orientation/seat on CAM0; power off and reseat.
• Restart the camera daemon: sudo systemctl restart nvargus-daemon
• Test: gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! fakesink -e

• dmesg | grep -i imx219 to verify the sensor is detected by the kernel.

• GStreamer element not found (e.g., nvv4l2h264enc):

• Ensure plugins installed: sudo apt install -y gstreamer1.0-plugins-bad

• On Jetson, nvv4l2h264enc is the V4L2 HW encoder; if missing, update/repair JetPack.

• OpenCV lacks GStreamer:

• Check build info (see Prerequisites). If NO, install the JetPack OpenCV: sudo apt install -y python3-opencv

• Avoid building OpenCV from source at Basic level unless necessary.

• VIDIOC_STREAMON / Argus timeout:

• Reseat the ribbon cable and ensure the IMX219 lens module is intact.
• Only one process should use the camera; close other apps or scripts.

• sudo systemctl restart nvargus-daemon

• MP4 not created or zero bytes:

• Ensure you let the script stop recording cleanly (writer.release() sends EOS for qtmux).
• Confirm sufficient disk space: df -h

• Validate your writer pipeline by testing a quick gst-launch-1.0 file encode separately.

• Low FPS:

• Confirm MAXN mode: sudo nvpmodel -m 0; sudo jetson_clocks
• Avoid SHOW_WINDOW over SSH; it can throttle.

• Reduce resolution to 960×540 or 640×480 for higher FPS if needed (update WIDTH/HEIGHT/FPS).

• False positives:

• Increase MIN_AREA_PX, VAR_THRESH; add ROI masks (crop or mask areas with known movement like fans/trees).
• Consider mean/median blur on the frame before backsub for denoising.

Improvements

• ROI and zones:

• Define multiple ROIs with different thresholds; log which zone triggered the event.

• Pre-/post-record buffers:

• Maintain a small ring buffer of frames to prepend a few seconds before a detected event.

• GPU-accelerated background subtraction:

• Use cv2.cuda if your OpenCV build supports CUDA on Jetson to offload MOG2 and morphology to the GPU.

• Smarter detection:

• Integrate a TensorRT-powered person/vehicle ONNX model for semantic filtering (trigger only if a person is detected).

• Streaming and alerts:

• Publish motion events to MQTT/HTTP; add timestamps and snapshots.

• Serve an RTSP stream with overlays using GStreamer and nvcompositor.

• Service deployment:

• Create a systemd unit to run at boot and restart on failure. Log to journald plus CSV.

• Power optimization:

• After validation, select a lower nvpmodel (e.g., sudo nvpmodel -m 1) for power savings; benchmark FPS vs. power.

Cleanup (reverting performance settings)

• Revert to default power mode and clocks:

sudo nvpmodel -q
# Choose a lower power profile; 1 is common on Orin Nano
sudo nvpmodel -m 1
sudo systemctl restart nvargus-daemon
# jetson_clocks has no "off"; reboot to return dynamic clocks
sudo reboot

Command reference and expected outcomes

Additional example: one-line CSI→OpenCV pipeline test

If you want a minimal capture-only test with FPS:

import cv2, time
gst = ("nvarguscamerasrc sensor-id=0 ! video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1,format=NV12 ! "
       "nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=true sync=false")
cap = cv2.VideoCapture(gst, cv2.CAP_GSTREAMER)
assert cap.isOpened()
t0 = time.time(); frames = 0
while frames < 300:
    ok, _ = cap.read()
    if not ok: break
    frames += 1
t1 = time.time()
print(f"Avg FPS over {frames} frames: {frames/(t1-t0):.2f}")
cap.release()

Run:

python3 csi_fps_test.py

Expect ~28–30 FPS at 720p.

Improvements to motion robustness

• Deghosting: Reset or “cool” the background model periodically (e.g., every few minutes) if the scene lighting drifts.
• Lighting changes: Use adaptive thresholds; monitor mean brightness; automatically increase MIN_AREA_PX when noise rises.
• Sensor parameters: For low light, you can expose the sensor longer by passing exposuretimerange/gainrange to nvarguscamerasrc. Example (conceptual) for gst-launch (adapt the same parameters in your pipeline if necessary):
• nvarguscamerasrc exposuretimerange=»100000 8000000″ gainrange=»1 10″ ! …

Final Checklist

• Hardware:
• [ ] Raspberry Pi Camera Module v2 (IMX219) connected to CAM0 with correct ribbon orientation.

• [ ] Jetson Orin Nano Developer Kit powered with adequate cooling.

• Software:

• [ ] JetPack/L4T verified: cat /etc/nv_tegra_release shows expected release.
• [ ] OpenCV with GStreamer support: python3 check prints “GStreamer: YES”.

• [ ] GStreamer plugins installed (good/bad/ugly), nvarguscamerasrc pipeline runs to fakesink.

• Performance and GPU validation:

• [ ] Optional: MAXN set and jetson_clocks applied for peak performance during testing.
• [ ] trtexec FP16 run completed with printed FPS and low mean latency.

• [ ] tegrastats shows expected GR3D/EMC utilization numbers during both OpenCV motion detection and TensorRT run.

• Application:

• [ ] motion_csi_opencv.py runs at ~30 FPS (720p) without frame drops.
• [ ] Motion events logged to CSV with bounding boxes and areas.

• [ ] MP4 clips saved using hardware encoder; files playable.

• Cleanup:

• [ ] nvpmodel reverted to a lower-power profile if desired; system rebooted to exit jetson_clocks state.

With this setup, you have a reproducible, command-line-only motion detection pipeline centered on OpenCV with a CSI camera, validated end-to-end on the NVIDIA Jetson Orin Nano Developer Kit. You also confirmed GPU acceleration via TensorRT (Path A), which provides a baseline for future upgrades such as DNN-powered object-aware motion filtering.