Sample raw frame from encoder pipeline

General Discussion
1

I am working on a video pipeline where I need to access a raw frame once in a while, ideally via http.

I need it when extracting frames for camera calibration. (intrinsics and distortion)

I guess I will just setup a regular encoder pipeline with a tee early in the pipeline.

But I am unsure how to extract a raw frame ocasionally. Do I use an appsink and code everything myself or is there a better way?

kind regards

Jesper

2

tee ! queue ! appsink max-buffers=1 or so would be one way of doing it.

You could also use an identity element’s "handoff" signal which is basically called for every buffer/frame that passes through, and usually you do nothing in there but if some kind of want_frame variable is set you send the frame to wherever it needs to go or something. You can achieve something similar with pad probes too. In either case you would have to wait for the next frame to pass through though and can’t just grab the current/latest one.

(Edit: of course you could also just keep around the last buffer yourself and store it somewhere)

1 Like
3

I ended up having something working by using this pipeline

gst-launch-1.0 videotestsrc num-buffers=-1 ! videoconvert ! video/x-raw,width=1280,height=720,format=RGB,framerate=30/1 ! tee name=t t. ! videoconvert ! queue ! autovideosink t. ! queue ! tcpserversink buffers-max=1 host=0.0.0.0 port=8008

I could then connect via a socket in python and read a frame

#!/usr/bin/python3
import cv2
import socket
import time
import numpy as np
SERVER_IP = '127.0.0.1'
SERVER_PORT = 8008
window_name = 'RGB Frame'
height = 720 
width = 1280
SZ = height * width * 3
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect((SERVER_IP, SERVER_PORT))
frame_data = b''
while len(frame_data) < SZ:
    data = client_socket.recv(SZ - len(frame_data))
    if not data:
        break
    frame_data += data
client_socket.close()
frame = np.frombuffer(frame_data, dtype=np.uint8)
frame = frame.reshape((height, width, 3))
cv2.imshow(window_name, frame)
cv2.waitKey(0)
cv2.destroyAllWindows()```