I am trying to do some object detection with the tfliteinference element, but I’m having trouble getting it to negotiate with the ssdtensordec element. I am using the GStreamer 1.28 packages from Debian Testing (version 1.28.1-2+b2) to get a recent enough version without having to compile the whole thing from source.
I’m using this ssdlite_mobiledet TFLite model for testing. Reading the docs I saw I need to create a modelinfo file, which I did with the helper script in gst devtools - I based it on the ssd_mobilenet example which AFAIK is for a similar ONNX model. The modelinfo file I came up with is this:
modelinfo file
[modelinfo]
version=1.0
group-id=ssd-mobilenet-v1-variant-1-out
[normalized_input_image_tensor]
id=image_tensor_0
type=uint8
dims=1,320,320,3
dir=input
ranges=0.0,255.0;0.0,255.0;0.0,255.0
[TFLite_Detection_PostProcess]
id=ssd-mobilenet-v1-variant-1-out-boxes
type=float32
dims=1,100,4
dir=output
[TFLite_Detection_PostProcess:1]
id=ssd-mobilenet-v1-variant-1-out-classes
type=float32
dims=1,100
dir=output
[TFLite_Detection_PostProcess:2]
id=ssd-mobilenet-v1-variant-1-out-scores
type=float32
dims=1,100
dir=output
[TFLite_Detection_PostProcess:3]
id=generic-variant-1-out-count
type=float32
dims=1
dir=output
Anyway, when trying to test it with this pipeline (I initially wanted to use the overlay to display bounding boxes on the detected objects, but replaced that with fakesink to isolate the problem)…
GST_DEBUG=basetransform:5,ssdtensordec:5,tfliteinference:5 gst-launch-1.0 filesrc location=bus.jpg ! jpegparse ! jpegdec ! videoconvertscale ! tfliteinference model-file=ssdlite_mobiledet_coco_qat_postprocess.tflite ! ssdtensordec label-file=coco_labels.txt ! objectdetectionoverlay ! fakesink
…I’m getting a negotiation error between the tfliteinference and ssdtensordec elements. The full debug log is very long so I won’t paste it here but this seems to be the relevant part:
0:00:00.052508356 1960 1963 DEBUG basetransform gstbasetransform.c:1280:gst_base_transform_acceptcaps_default:<ssdtensordec0> subset with pad template: video/x-raw, tensors=(structure)[tensorgroups, ssd-mobilenet-v1-variant-1-out=(/uniquelist){ (caps)"tensor/strided\,\ tensor-id\=\(string\)generic-variant-1-out-count\,\ dims\=\(int\)\<\ \[\ 0\,\ 2147483647\ \]\ \>\,\ dims-order\=\(string\)row-major\,\ type\=\(string\)float32", (caps)"tensor/strided\,\ tensor-id\=\(string\)ssd-mobilenet-v1-variant-1-out-scores\,\ dims\=\(int\)\<\ \[\ 0\,\ 2147483647\ \]\,\ 0\ \>\,\ dims-order\=\(string\)row-major\,\ type\=\(string\)float32", (caps)"tensor/strided\,\ tensor-id\=\(string\)ssd-mobilenet-v1-variant-1-out-boxes\,\ dims\=\(int\)\<\ \[\ 0\,\ 2147483647\ \]\,\ 0\,\ 4\ \>\,\ dims-order\=\(string\)row-major\,\ type\=\(string\)float32", (caps)"tensor/strided\,\ tensor-id\=\(string\)ssd-mobilenet-v1-variant-1-out-classes\,\ dims\=\(int\)\<\ \[\ 0\,\ 2147483647\ \]\,\ 0\ \>\,\ dims-order\=\(string\)row-major\,\ type\=\(string\)float32" };]
0:00:00.052518023 1960 1963 DEBUG basetransform gstbasetransform.c:1303:gst_base_transform_acceptcaps_default:<ssdtensordec0> caps can't intersect with the template
0:00:00.052529565 1960 1963 DEBUG basetransform gstbasetransform.c:1293:gst_base_transform_acceptcaps_default:<ssdtensordec0> accept-caps result: 0
0:00:00.052543231 1960 1963 DEBUG basetransform gstbasetransform.c:1237:gst_base_transform_find_transform:<tfliteinference0> FAILED to get peer of <tfliteinference0:src> to accept video/x-raw, width=(int)320, height=(int)320, interlace-mode=(string)progressive, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, pixel-aspect-ratio=(fraction)3/4, framerate=(fraction)0/1, format=(string)RGB, colorimetry=(string)1:1:0:0, tensors=(structure)[tensorgroups, ssd-mobilenet-v1-variant-1-out=(/uniquelist){ (caps)"tensor/strided\,\ dims-order\=\(string\)row-major\,\ tensor-id\=\(string\)ssd-mobilenet-v1-variant-1-out-boxes\,\ dims\=\(int\)\<\ 1\,\ 100\,\ 4\ \>\,\ type\=\(string\)float32", (caps)"tensor/strided\,\ dims-order\=\(string\)row-major\,\ tensor-id\=\(string\)ssd-mobilenet-v1-variant-1-out-classes\,\ dims\=\(int\)\<\ 1\,\ 100\ \>\,\ type\=\(string\)float32", (caps)"tensor/strided\,\ dims-order\=\(string\)row-major\,\ tensor-id\=\(string\)ssd-mobilenet-v1-variant-1-out-scores\,\ dims\=\(int\)\<\ 1\,\ 100\ \>\,\ type\=\(string\)float32", (caps)"tensor/strided\,\ dims-order\=\(string\)row-major\,\ tensor-id\=\(string\)generic-variant-1-out-count\,\ dims\=\(int\)\<\ 1\ \>\,\ type\=\(string\)float32" };]
From what I can understand, the dimensions of the output tensors don’t match, i.e. the ssd-mobilenet-v1-variant-1-out-boxes tensor has dimensions [1, 100, 4] while the ssdtensordec expects the second dimension (I guess number of detected objects?) there to be 0. I’m not sure how to get ssdtensordec to accept tensors with a dimension of 100 in that position, or if that’s even the right way to go about it. I also don’t know if the problem is with the modelinfo file I created, or if it’s something else. Can anyone give any pointers?