(V4L2codecs) feasibility of adding support for Raspberry Pi HEVC "SAND" formats NC12, NC30?

Plugin and library development
1

I’m investigating getting support for the Raspberry Pi (models 5,4B) v4l2-compliant HEVC decoding block driver “rpivid” into GStreamer, perhaps initially as a working patch to libgstv4l2codecs our users can apply, and then later as a PR, if it could be accepted.

The issue is the “SAND” (NC12, NC30) pixel formats the R Pi driver “rpivid” (in their kernel tree, not yet mainline) uses:

[Edit] A comment I just got from @6by9 at R Pi is:

AIUI Mainline GStreamer will only support formats that are defined in the mainline kernel, and upstreaming the HEVC decoder is still in progress

Once the format and driver are in the mainline kernel, I wouldn’t expect there to be any great obstacles to adding the format to GStreamer, but would ask Nicolas Dufresne as the person who knows that area best.

Sand was an internal Broadcom name for the format as it originally came from a company called Sand Video that was bought by them in 2004 - Broadcom acquires Sand Video • The Register

The rpivid driver uses /dev/video19:

$v4l2-ctl --list-formats-ext -d 19
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture Multiplanar

        [0]: 'NC12' (Y/CbCr 4:2:0 (128b cols))
        [1]: 'NC30' (10-bit Y/CbCr 4:2:0 (128b cols))

R Pi have provided a modified ffmpeg that supports this format, which seems to be a column format called “SAND”. These modifications appear to be compatible with LGPL 2.1.

The modifications appear to be

A comment (from R Pi engineer @6by9 ) about NC12 /NC30 describes vf_unsand:

There is code in FFmpeg to convert from this column format (known as SAND) to planar YUV420, but it looks like that only supports writing out to the 3 plane YUV420 (aka I420) rather than NV12/21 or other flavour.

Advice from GStreamer V4L2 maintainers (@ndufresne ) about whether adding support for these two formats in GStreamer might be feasible (or whether it is obviously a no-go) would be greatly appreciated!

  • as an alternative, since support would be available in the ffmpeg supplied on R Pi, would it be possible to leverage that into gstreamer libgstv4l2codecs by testing if ffmpeg support is present, and using it if it is, (maybe via gstreamer-libav?)
2

Originally, the only way to support a raw format was to add software conversion support. We don’t have any column tiling formats yet, but this is certainly feasible. The benefit is that you can validate the codec using automated tests like fluster.

Since 1.24, we have mechanism to allow opaque pixel formats to be passed through. The missing bit is to map the V4L2 definitions to their opaque DRM format + modifier form. This is not completely implemented yet, a draft is available here:

With that, mapping any opaque formats will become trivial (aka one liner for vendors that have not upstreamed their drivers and uses custom pixel formats).

Note that for the first approach, we can clearly land these as GstVideoFormat without any RPi media drivers upstream.

p.s. I haven’t check if these are format available in the ISP, so perhaps there is some progress, since the ISP is being upstreamed.

3

I’m slowly acquiring some knowledge of GStreamer internals, and the RPi ffmpeg code and rpivid driver code is available in their forks of the kernel and ffmpeg.

I’ll try to push them to get the format upstreamed to the kernel if it can be done earlier than the driver.

It could be possible to get working support for the driver in a fork of GStreamer
that would be ready for inclusion when that was possible.

The issue that no column formats are supported yet sounds daunting, though.
Ideally RPi would do all this, but they have focused on supporting ffmpeg.

1 Like
4

In practice, software support for column format is mostly for testing. This week Robert also made P030 work in his branch, so HDR work he’s doing can also be tested on the Pi. At this point, assuming we tolerate the downstream formats, we could get some support included for 1.26. One down side is that I heard that all this is very flexible at hardware level, and formats layout can still be changed.