Practical case: RTSP face blur on Jetson Orin NX+IMX477

Objective and use case

What you’ll build: A GPU-accelerated RTSP service on NVIDIA Jetson that detects faces from a USB camera and applies real-time blur, streaming an H.264 feed consumable by VLC/ffplay/NVRs.

Why it matters / Use cases

• Privacy-aware CCTV in public venues: lobby/foyer feeds with auto-blurred faces to meet privacy regulations while preserving situational context.
• Retail analytics without PII: store cameras stream to an NVR with faces blurred, enabling heatmaps and occupancy analytics without biometric data.
• Smart home sharing: share a garage camera to relatives or a cloud dashboard with visitors’ faces anonymized.
• Developer testbed: prototype CV privacy features on edge hardware with an RTSP endpoint compatible with VLC, ffplay, and NVRs.
• Field deployments: mobile robots/kiosks send anonymized footage over constrained links while remaining compliant.

Expected outcome

• Real-time 720p face detection + blur at ≥15 FPS (on-device). Example: Jetson Xavier NX 25–30 FPS (~45–60% GPU, ~70–120 ms glass-to-glass); Jetson Nano 15–18 FPS (~70–85% GPU, ~120–180 ms).
• RTSP endpoint: rtsp://<jetson-ip>:8554/faceblur (H.264, UDP; typical 4–6 Mb/s at 720p30).
• Handles multiple faces per frame; blur confined to detected ROIs to retain scene detail; stable stream to standard clients (VLC, ffplay, NVRs).

Audience: Edge AI/CV engineers, systems integrators, robotics teams; Level: Intermediate–Advanced

Architecture/flow: USB camera (V4L2) → GStreamer capture → TensorRT face detector (CUDA) → ROI Gaussian blur (CUDA) → hardware H.264 encode (nvv4l2h264enc) → RTSP server (gst-rtsp-server) → client (VLC/ffplay/NVR); zero-copy where possible for low latency.

Prerequisites

• Platform: NVIDIA Jetson running JetPack (L4T) on Ubuntu (assumed).
• Exact device model used in this guide (as requested): NVIDIA Jetson Orin NX Developer Kit + Arducam USB3.0 Camera (Sony IMX477)
• Network access (Ethernet or Wi‑Fi), and shell access via HDMI+keyboard or SSH.
• Basic familiarity with terminal commands and Python.

Verify JetPack release, kernel, and NVIDIA packages:

cat /etc/nv_tegra_release
# Alternative if installed:
# jetson_release -v

# Kernel and NVIDIA/TensorRT packages present
uname -a
dpkg -l | grep -E 'nvidia|tensorrt'

Example expected snippets:
– /etc/nv_tegra_release prints “R36.x (release), REVISION: x.x” for JetPack 6 or “R35.x” for JetPack 5.
– dpkg shows nvidia-l4t-* packages and tensorrt runtime libraries.

Prefer GPU-accelerated AI. In this tutorial we take path C) PyTorch GPU:
– Confirm torch.cuda.is_available() in the Python code.
– We run a small face detector (MTCNN) on CUDA and time it (FPS reported).

Power/performance tools you’ll use:
– nvpmodel to set power mode; jetson_clocks to max clocks (test responsibly).
– tegrastats to measure CPU/GPU/EMC utilization and memory.

---

Materials

---

Setup/Connection

1. Physical connections
2. Plug the Arducam USB3.0 Camera (Sony IMX477) into a USB 3.0 port on the NVIDIA Jetson Orin NX Developer Kit.

3. Ensure the blue USB 3.0 port is used for maximum bandwidth.

4. Detect camera and enumerate formats

5. List the UVC device and its node:

lsusb | grep -i arducam
v4l2-ctl --list-devices

• Identify the video node (e.g., /dev/video0). List supported formats and frame sizes:

v4l2-ctl -d /dev/video0 --list-formats-ext

Typical UVC capabilities include MJPEG and YUYV at resolutions such as 1920×1080 and 1280×720 at 30 FPS or higher. We’ll use 1280×720@30 for this project to keep latency and compute modest.

1. Quick camera sanity test (no GUI)
2. Record a short 5-second H.264 clip (hardware-encoded) to verify HW pipeline and camera:

gst-launch-1.0 -e v4l2src device=/dev/video0 ! \
  image/jpeg,framerate=30/1,width=1280,height=720 ! jpegdec ! nvvidconv ! \
  nvv4l2h264enc insert-sps-pps=true bitrate=4000000 maxperf-enable=1 ! \
  h264parse ! qtmux ! filesink location=cam_test.mp4

• Play the file on another machine or with ffplay to confirm the camera and encoders work.

---

Full Code

This Python program:
– Captures frames from the Arducam USB3.0 (UVC) via GStreamer into OpenCV.
– Runs MTCNN face detection on the GPU (PyTorch).
– Blurs detected face regions using OpenCV (Gaussian blur).
– Serves the processed frames as an RTSP stream using GStreamer gst-rtsp-server with Jetson’s hardware H.264 encoder (nvv4l2h264enc).
– Prints FPS and detection counts for validation.

Save as rtsp_face_blur.py

#!/usr/bin/env python3
import os
import sys
import time
import threading
import socket

import numpy as np
import cv2

import torch
from facenet_pytorch import MTCNN

import gi
gi.require_version('Gst', '1.0')
gi.require_version('GstRtspServer', '1.0')
from gi.repository import Gst, GstRtspServer, GLib, GObject

# Configuration
DEVICE_PATH = "/dev/video0"
WIDTH = 1280
HEIGHT = 720
FPS = 30
BITRATE = 4000000  # bps for H.264 encoder
BLUR_KERNEL = (31, 31)
RTSP_PORT = 8554
RTSP_PATH = "/faceblur"

# Build GStreamer source pipeline for UVC camera -> BGR frames to appsink
# We request MJPEG @ 1280x720@30, decode, convert to BGR
GST_CAM_PIPE = (
    f"v4l2src device={DEVICE_PATH} ! "
    f"image/jpeg,framerate={FPS}/1,width={WIDTH},height={HEIGHT} ! "
    "jpegdec ! videoconvert ! video/x-raw,format=BGR ! "
    "appsink drop=true sync=false max-buffers=1"
)

class FacePrivacyBlur:
    def __init__(self):
        # Prefer CUDA if available
        self.use_cuda = torch.cuda.is_available()
        self.device = torch.device("cuda:0" if self.use_cuda else "cpu")

        # MTCNN face detector
        # Parameters are tuned for speed while keeping reasonable detection accuracy
        self.detector = MTCNN(
            keep_all=True,
            device=self.device,
            thresholds=[0.6, 0.7, 0.7],
            min_face_size=40
        )

        self.cap = cv2.VideoCapture(GST_CAM_PIPE, cv2.CAP_GSTREAMER)
        if not self.cap.isOpened():
            raise RuntimeError("Failed to open camera with GStreamer pipeline.")

        # Shared state for RTSP appsrc
        self.frame_lock = threading.Lock()
        self.latest_frame = None  # np.ndarray (BGR)
        self.running = True

        # Metrics
        self.fps_hist = []
        self.last_ts = time.time()

    def start(self):
        self.thread = threading.Thread(target=self._loop, daemon=True)
        self.thread.start()

    def stop(self):
        self.running = False
        if hasattr(self, 'thread'):
            self.thread.join(timeout=2.0)
        self.cap.release()

    def _blur_faces(self, frame):
        """Run GPU face detection + blur ROIs on BGR frame."""
        rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        boxes, probs = self.detector.detect(rgb)
        count = 0
        if boxes is not None:
            for box in boxes:
                x1, y1, x2, y2 = [int(max(0, v)) for v in box]
                x1 = max(0, min(x1, frame.shape[1]-1))
                y1 = max(0, min(y1, frame.shape[0]-1))
                x2 = max(0, min(x2, frame.shape[1]-1))
                y2 = max(0, min(y2, frame.shape[0]-1))
                if x2 > x1 and y2 > y1:
                    roi = frame[y1:y2, x1:x2]
                    if roi.size > 0:
                        # Gaussian blur ROI in-place
                        roi_blur = cv2.GaussianBlur(roi, BLUR_KERNEL, 0)
                        frame[y1:y2, x1:x2] = roi_blur
                        count += 1
        return frame, count

    def _loop(self):
        """Capture, detect, blur, and publish latest_frame."""
        print(f"[INFO] CUDA available: {self.use_cuda}; device: {self.device}")
        print(f"[INFO] Opening UVC camera at {DEVICE_PATH} -> {WIDTH}x{HEIGHT}@{FPS}")

        while self.running:
            ok, frame = self.cap.read()
            if not ok or frame is None:
                # short backoff to avoid tight loop on camera error
                time.sleep(0.01)
                continue

            t0 = time.time()
            frame, faces = self._blur_faces(frame)
            t1 = time.time()

            # Update shared latest_frame
            with self.frame_lock:
                self.latest_frame = frame

            dt = t1 - t0
            fps = 1.0 / max(dt, 1e-6)
            self.fps_hist.append(fps)
            if len(self.fps_hist) > 60:
                self.fps_hist.pop(0)

            # Periodic log
            now = time.time()
            if now - self.last_ts >= 2.0:
                avg_fps = sum(self.fps_hist) / max(len(self.fps_hist), 1)
                print(f"[METRICS] faces={faces} inst_fps={fps:.1f} avg_fps(2s)={avg_fps:.1f}")
                self.last_ts = now

    def get_latest_frame_bgr(self):
        with self.frame_lock:
            if self.latest_frame is None:
                return None
            return self.latest_frame.copy()


class AppSrcFactory(GstRtspServer.RTSPMediaFactory):
    def __init__(self, frame_src, width, height, fps, bitrate):
        super(AppSrcFactory, self).__init__()
        self.frame_src = frame_src
        self.number_frames = 0
        self.duration = Gst.SECOND // fps
        self.fps = fps
        self.width = width
        self.height = height
        self.bitrate = bitrate

        # Launch pipeline: appsrc (BGR) -> videoconvert -> I420 -> nvvidconv -> nvv4l2h264enc -> rtph264pay
        self.launch_str = (
            f"( appsrc name=src is-live=true block=true format=time "
            f"caps=video/x-raw,format=BGR,width={width},height={height},framerate={fps}/1 ! "
            f"videoconvert ! video/x-raw,format=I420 ! "
            f"nvvidconv ! "
            f"nvv4l2h264enc insert-sps-pps=true idrinterval={fps} bitrate={bitrate} "
            f"preset-level=1 maxperf-enable=1 ! "
            f"h264parse config-interval=-1 ! rtph264pay name=pay0 pt=96 )"
        )
        self.set_shared(True)

    def do_create_element(self, url):
        return Gst.parse_launch(self.launch_str)

    def do_configure(self, rtsp_media):
        self.number_frames = 0
        appsrc = rtsp_media.get_element().get_child_by_name("src")
        appsrc.set_property("emit-signals", True)
        appsrc.connect("need-data", self.on_need_data)

    def on_need_data(self, src, length):
        frame = self.frame_src.get_latest_frame_bgr()
        if frame is None:
            time.sleep(0.005)
            return

        data = frame.tobytes()
        buf = Gst.Buffer.new_allocate(None, len(data), None)
        buf.fill(0, data)

        # Timestamp
        pts = self.number_frames * self.duration
        buf.pts = pts
        buf.dts = pts
        buf.duration = self.duration
        self.number_frames += 1

        retval = src.emit("push-buffer", buf)
        if retval != Gst.FlowReturn.OK:
            print(f"[WARN] push-buffer returned {retval}")

def get_ip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        s.connect(("8.8.8.8", 80))
        ip = s.getsockname()[0]
    except Exception:
        ip = "127.0.0.1"
    finally:
        s.close()
    return ip

def main():
    GObject.threads_init()
    Gst.init(None)

    # Start capture + face blur worker
    blur = FacePrivacyBlur()
    blur.start()

    # RTSP server
    server = GstRtspServer.RTSPServer()
    server.props.service = str(RTSP_PORT)
    factory = AppSrcFactory(blur, WIDTH, HEIGHT, FPS, BITRATE)
    mount_points = server.get_mount_points()
    mount_points.add_factory(RTSP_PATH, factory)
    server.attach(None)

    ip = get_ip()
    print(f"[READY] RTSP stream at rtsp://{ip}:{RTSP_PORT}{RTSP_PATH}")
    print(f"[INFO] Press Ctrl+C to stop.")

    loop = GLib.MainLoop()
    try:
        loop.run()
    except KeyboardInterrupt:
        pass
    finally:
        blur.stop()
        loop.quit()

if __name__ == "__main__":
    main()

Notes:
– The pipeline uses hardware H.264 encoding on Jetson via nvv4l2h264enc.
– Frames are BGR in system memory; nvvidconv handles the conversion and feeding to NVENC.
– The app prints periodic metrics including faces per frame and FPS.

---

Build/Flash/Run commands

1. Optional: set performance mode (MAXN) and max clocks (monitor thermals)

# Query current mode
sudo nvpmodel -q

# Set to MAXN (mode 0) for peak performance and lock clocks (until reboot)
sudo nvpmodel -m 0
sudo jetson_clocks

Warning: Higher power and thermals; ensure adequate cooling. To revert later, see Cleanup below.

1. Install system packages (GStreamer, RTSP server GIR, V4L2 utils, Python dev)

sudo apt-get update
sudo apt-get install -y \
  python3 python3-pip python3-gi python3-gst-1.0 \
  gir1.2-gst-rtsp-server-1.0 \
  gstreamer1.0-tools gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good gstreamer1.0-plugins-bad \
  gstreamer1.0-plugins-ugly gstreamer1.0-libav \
  v4l-utils

1. Install Python dependencies
2. On Jetson, use NVIDIA’s L4T PyTorch wheels. The following works on recent JetPack releases (adjust version if needed):

# NVIDIA PyTorch wheels index for Jetson
export PIP_EXTRA_INDEX_URL="https://developer.download.nvidia.com/compute/redist"

# Install torch/torchvision matching your JetPack (example: JP6.x provides torch 2.1.x+nv)
python3 -m pip install --upgrade pip
python3 -m pip install 'torch==2.1.0+nv23.10' torchvision --extra-index-url $PIP_EXTRA_INDEX_URL

# Face detector and OpenCV
python3 -m pip install facenet-pytorch opencv-python

Verify CUDA is detected by PyTorch:

python3 -c "import torch; print('torch', torch.__version__, 'cuda?', torch.cuda.is_available())"

1. Save the program

nano rtsp_face_blur.py
# (Paste the full code above, save and exit)

1. Run the service and observe logs

python3 rtsp_face_blur.py

You should see logs similar to:
– CUDA available: True; device: cuda:0
– [READY] RTSP stream at rtsp://:8554/faceblur
– [METRICS] faces=1 inst_fps=24.3 avg_fps(2s)=22.8

1. Connect with an RTSP client from another machine on the same network

# Using ffplay
ffplay -fflags nobuffer -flags low_delay -rtsp_transport tcp rtsp://<jetson-ip>:8554/faceblur

# Or with VLC: open network stream with the same RTSP URL

---

Step-by-step Validation

1. Confirm Jetson/JetPack environment

2. Use the commands in Prerequisites. Ensure nvidia-l4t-* packages appear and that TensorRT/Torch are present (even though this project uses PyTorch primarily).

3. Confirm camera streamability and pixel formats

v4l2-ctl -d /dev/video0 --list-formats-ext

• Expect MJPEG at 1280×720 or higher. This pipeline uses MJPEG for efficient USB transport.

• Confirm base GStreamer capture and encode

gst-launch-1.0 -e v4l2src device=/dev/video0 ! \
  image/jpeg,framerate=30/1,width=1280,height=720 ! jpegdec ! nvvidconv ! \
  nvv4l2h264enc insert-sps-pps=true bitrate=4000000 maxperf-enable=1 ! \
  fakesink

• Should run without errors. Stop with Ctrl+C.

• Run the Python service and verify CUDA

python3 -c "import torch; print(torch.cuda.is_available())"
# Expect: True

1. Start the face-privacy RTSP service

python3 rtsp_face_blur.py

• Logs print “[READY] RTSP stream at rtsp://…” and periodic “[METRICS] …” lines.

• Typical FPS on the NVIDIA Jetson Orin NX Developer Kit at 1280×720 might be 15–30 FPS depending on lighting and number of faces.

• Monitor device utilization

• In another terminal, run:

sudo tegrastats

• Observe GR3D (GPU) utilization (e.g., 15–40% under load), EMC/CPU usage, and memory footprint. Record this along with the printed FPS to validate the “Expected outcome.”

• Connect from a client and check latency

ffplay -fflags nobuffer -flags low_delay -rtsp_transport tcp rtsp://<jetson-ip>:8554/faceblur

• The stream should display blurred faces. Latency on a LAN typically under ~400 ms with the given settings (zerolatency preset via speed of encoder and appsrc being live).

• Metrics to record (sample success criteria)

• avg_fps(2s) from the log ≥ 15 FPS at 1280×720 with 1–3 faces visible.
• tegrastats shows GR3D >10% when faces are present.
• No buffer underflows reported by appsrc (no continuous “[WARN] push-buffer” errors).
• RTSP playback stable for ≥5 minutes without drift or stutter.

---

Troubleshooting

• Camera node not found (/dev/video0 missing)
• Check cable/port. Try other USB 3.0 port. Replug camera and run:
  • lsusb, dmesg | tail -n 100
  • Ensure UVC driver is present (UVC is built-in to JetPack).

• If your camera enumerates as /dev/video1, update DEVICE_PATH accordingly.

• Poor performance (low FPS)

• Ensure GPU is used: the program prints “CUDA available: True”. If False:
  • Verify PyTorch wheel: use NVIDIA L4T wheels via PIP_EXTRA_INDEX_URL. Reinstall torch/torchvision.
  • Check that the JetPack CUDA/TensorRT stack is installed (dpkg -l | grep nvidia).
• Enable MAXN and clocks:
  • sudo nvpmodel -m 0; sudo jetson_clocks
• Reduce input resolution:
  • Change WIDTH/HEIGHT to 960×540 or 640×480 and restart.

• Adjust MTCNN thresholds or min_face_size upward for speed.

• High latency in client playback

• Try TCP transport explicitly (already shown with ffplay).
• Reduce encoder bitrate or increase iframeinterval for more consistent buffering.

• Set VLC/ffplay to low-latency options (already shown for ffplay).

• “Failed to open camera with GStreamer pipeline”

• Verify that v4l2-ctl shows MJPEG at the requested WIDTH/HEIGHT/FPS. If not, try YUYV:
  • Replace the source with:
  • «v4l2src device=/dev/video0 ! video/x-raw,format=YUY2,framerate=30/1,width=1280,height=720 ! videoconvert ! video/x-raw,format=BGR ! appsink …»

• Ensure gstreamer1.0-plugins-good and gstreamer1.0-libav are installed.

• RTSP not reachable

• Confirm IP address in the log and that both devices are on the same subnet.
• Check firewall: sudo ufw status (disable or allow 8554/tcp).

• Verify server is bound to the port: ss -lntp | grep 8554

• Crashes on start with GLib/Gst errors

• Ensure GIR packages are installed (gir1.2-gst-rtsp-server-1.0).

• Run with Python 3.8+ (default on JetPack 5/6).

• GPU very low utilization

• Some frames might skip detection if processing lags. Confirm the metrics print FPS in a healthy range. If GPU remains idle:
  • Add a torch.cuda.synchronize() around timing if measuring GPU compute precisely (optional).
  • Confirm the MTCNN device is set to CUDA in the code.

---

Improvements

• Use a TensorRT-optimized face detector (e.g., SCRFD converted to ONNX/INT8) to increase FPS. You can deploy via TensorRT or DeepStream for even lower latency and higher throughput on Orin NX.
• Add elliptical or mosaic blurring for improved aesthetics; vary kernel size based on bounding box size to keep blur strength proportional.
• Implement automatic exposure control or pre-processing (contrast/stretch) to improve detections in low light.
• Add an HTTP/metrics endpoint (Prometheus) exposing FPS, face count, and queue depth for monitoring.
• Add a motion gate: only run face detection when motion is detected (reduce compute).
• Switch to DeepStream (nvinfer + custom plugin for blur) for a fully GStreamer-native pipeline with GPU operators and lower CPU overhead.

---

Cleanup (revert performance settings)

If you enabled MAXN and max clocks:

# To relax clocks (return to dynamic governor)
sudo systemctl restart nvpmodel
# or simply reboot
sudo reboot

---

Exact commands recap

• Verify JetPack and NVIDIA stack:

cat /etc/nv_tegra_release
uname -a
dpkg -l | grep -E 'nvidia|tensorrt'

• Power mode and clocks:

sudo nvpmodel -q
sudo nvpmodel -m 0
sudo jetson_clocks

• Install dependencies:

sudo apt-get update
sudo apt-get install -y python3 python3-pip python3-gi python3-gst-1.0 \
  gir1.2-gst-rtsp-server-1.0 gstreamer1.0-tools gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly \
  gstreamer1.0-libav v4l-utils
export PIP_EXTRA_INDEX_URL="https://developer.download.nvidia.com/compute/redist"
python3 -m pip install --upgrade pip
python3 -m pip install 'torch==2.1.0+nv23.10' torchvision --extra-index-url $PIP_EXTRA_INDEX_URL
python3 -m pip install facenet-pytorch opencv-python

• Test camera:

v4l2-ctl -d /dev/video0 --list-formats-ext
gst-launch-1.0 -e v4l2src device=/dev/video0 ! \
  image/jpeg,framerate=30/1,width=1280,height=720 ! jpegdec ! nvvidconv ! \
  nvv4l2h264enc insert-sps-pps=true bitrate=4000000 maxperf-enable=1 ! \
  fakesink

• Run application:

python3 rtsp_face_blur.py

• Client playback:

ffplay -fflags nobuffer -flags low_delay -rtsp_transport tcp rtsp://<jetson-ip>:8554/faceblur

• Monitor utilization:

sudo tegrastats

---

Why we chose PyTorch GPU (Path C)

• Quick to prototype: MTCNN via facenet-pytorch provides straightforward, CUDA-accelerated face detection.
• Minimal glue code: We stay in Python while still leveraging Jetson’s hardware encoder and GStreamer RTSP server.
• Meets the objective: A working “rtsp-face-privacy-blur” solution with measurable FPS and GPU utilization, and easy to improve later with TensorRT/DeepStream.

---

Expected logs and metrics

During operation, you should see console output such as:
– [READY] RTSP stream at rtsp://192.168.1.42:8554/faceblur
– [INFO] CUDA available: True; device: cuda:0
– [METRICS] faces=2 inst_fps=21.7 avg_fps(2s)=20.9

With tegrastats in another terminal:
– GR3D: 20–40% during face activity
– CPU under ~150% total (varies by cores and capture resolution)
– EMC usage moderate depending on memory throughput

If the FPS drops below 10 or GR3D is ~0% while faces are present, revisit the Troubleshooting section.

---

Final Checklist

• Hardware
• [ ] Using the NVIDIA Jetson Orin NX Developer Kit + Arducam USB3.0 Camera (Sony IMX477).

• [ ] Camera enumerates (v4l2-ctl shows formats).

• Software

• [ ] JetPack shown via /etc/nv_tegra_release; NVIDIA packages present.
• [ ] GStreamer, RTSP GIR, and v4l-utils installed.

• [ ] PyTorch (L4T wheel) and facenet-pytorch installed; torch.cuda.is_available() returns True.

• Performance

• [ ] nvpmodel set to 0 (MAXN) and jetson_clocks enabled for tests (optional).

• [ ] tegrastats monitored; GPU utilization >10% during detection.

• Functionality

• [ ] Python app runs without errors.
• [ ] RTSP available at rtsp://:8554/faceblur.
• [ ] Faces are blurred on the client view.

• [ ] Logged FPS ≥ 15 at 1280×720, stable playback.

• Cleanup

• [ ] Revert power settings if needed (restart or reset nvpmodel).

This completes a practical, end-to-end “rtsp-face-privacy-blur” build on the NVIDIA Jetson Orin NX Developer Kit with an Arducam USB3.0 Camera (Sony IMX477), using GPU-accelerated face detection and hardware-encoded RTSP output suitable for LAN or NVR integration.