[RZ/G1E] Display image is wrong with VSPM filtering

Hi all,

I am trying a scaling, displaying, and file saving on RZ/G1E (BSP for starter kit + multimedia package).
But, display image is wrong. I attached the image.

 

This is the flow of concrete processing.

  H.264 dec -> scaling & NV12 format convert with VSPM (DATA-A)
    DATA-A -> format convert with VSPM -> display(wayland)
    DATA-A -> NV12 file saving

[Command]
# gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! omxh264dec ! vspmfilter outbuf-alloc=true ! video/x-raw,format=NV12,width=800,height=480 ! tee tee0. ! vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA ! waylandsink sync=false async=false tee0. ! filesink location=output.dat sync=false async=false

BTW, if set I420 format, display image is good.

# gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! omxh264dec ! vspmfilter outbuf-alloc=true ! video/x-raw,format=I420,width=800,height=480 ! tee tee0. ! vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA ! waylandsink sync=false async=false tee0. ! filesink location=output.dat sync=false async=false

Do you know how to resolve?

Best Regards,
Kenji

  • Hi Kenji,

    Any update with your issue? Were you able to resolve it already?

    JB
    RenesasRulz Forum Moderator
  • Hi Kenji,

    Sorry for the late reply, we were clarifying the solution internally.

    FYI, the output of omxh264dec is in NV12 format so there is no need to use vspmfilter to convert it if you just need that format. Also, then running Weston in 'VSP compositor' mode (which is the default) it can display NV12 format directly so there is no need to convert it to BGRA.
    So, in simple scenarios you only need vspmfilter if you are doing scaling. Here is an example pipeline -

    gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! omxh264dec ! vspmfilter outbuf-alloc=false dmabuf-use=false ! video/x-raw,format=NV12,width=800,height=480 ! waylandsink

    Change 'NV12' to 'BGRA' when running Weston in 'GPU compositor' mode.

    In your case you are playing the video and at the same time recording it to a file, which makes things more complicated. Here is a pipeline that performs those tasks -

    gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! omxh264dec ! tee name=t \
    t. ! vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA,width=800,height=480 ! waylandsink sync=false async=false \
    t. ! vspmfilter outbuf-alloc=true ! video/x-raw,format=NV12,width=800,height=480 ! filesink location=output.dat sync=false async=false

    With this pipeline you can have independent scaling of the two streams.

    Regards,
    Georgi
  • In reply to Georgi:

    Hi Georgi,
    Thank you so much for your comment.

    Your independent scaling method is good.
    However, the method is two scaling, so it is not good for power consumption.
    I want to make it more efficient.

    With setting I420 format, I was able to solve on one scaling. But, I would like to save NV12 format file...
    Do you know why NV12 format cannot be worked?

    Best Regards,
    Kenji
  • In reply to Kenji:

    Hi Kenji,

    In both pipelines (your original pipeline and the one that I sent) vspmfilter is used twice so there isn't much difference there. The CPU load for the two pipelines is also very similar, so in terms of power consumption I think the two pipelines would be the same.

    I think the biggest contributor to power consumption in both cases would be writing the decoded video to a file in the SD card. The SD interface itself consumes quite a bit of power and writing all that data means it will be used a lot .
    Why do you need to store un-encoded video to a file?

    Regards,
    Georgi
  • In reply to Georgi:

    Hi Georgi,
    Thank you for your comment.

    My pipeline above (saving YUV) is for testing.
    What I really want to know is why it works in I420, but it does not work in NV12.
    I guess there is a bug or something.

    Best Regards,
    Kenji
  • In reply to Kenji:

    Hi Kenji,

    You meet problem with NV12 because your pipeline has 2 vspmfilter instances.

    vspmfilter is not designed to run 2 instances next to each other like that, so problem may happen. In this case the problem is in alignment.

    The hardware requires the buffer in page-alignt (or 4096-align) form. Normally omxh264dec, vspmfilter, and waylandsink will try to solve this alignment by themselves. But in your pipeline, combination of 2 vspmfilter instances make it impossible to run with correct alignment, which cause worng color.

    The problem will not happen if you use other video sizes which are already 4096-aligned (dividable to 4096), such as 640x480.
    In I420 format, system does not have this alignment requirement, so the alignment trouble does not occur in your pipeline with I420.

    Regards,
    Hung.

  • In reply to HungT:

    Hi Hung,
    Thanks for the useful information.

    I have two questions.

    1. Why the proble is not occurred with the above Georgi's proposal pipeline?
    This pipeline has two vspmfilter and don't use 4096-aligned size.

    gst-launch-1.0 filesrc location=input.mp4 ! qtdemux ! omxh264dec ! tee name=t \
    t. ! vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA,width=800,height=480 ! waylandsink sync=false async=false \
    t. ! vspmfilter outbuf-alloc=true ! video/x-raw,format=NV12,width=800,height=480 ! filesink location=output.dat sync=false async=false

    2. Is it Okay for the two processes to use one vspmfilter at the same time?

    gst-launch-1.0 filesrc location=videos/h264-hd-30.mp4 ! qtdemux ! omxh264dec ! \
    vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA,width=800,height=480 ! waylandsink & \
    gst-launch-1.0 filesrc location=videos/h264-hd-30.mp4 ! qtdemux ! omxh264dec ! \
    vspmfilter outbuf-alloc=true ! video/x-raw,format=NV12,width=800,height=480 ! filesink location=output.dat

    Best Regards,
    Kenji
  • In reply to Kenji:

    Hi Kenji,

    1. Georgi's pipelines use two vspmfilter instances, but in two different branches (after tee).
    Two vspmfilter instance do not connect to each other directly like your pipeline, thus there is no problem here.

    2. Yes, using vspmfilter in two process at the same time is OK.

    Regards,
    Hung.
  • In reply to HungT:

    Hi Hung,
    Thank you so much.

    I understand.
    I appreciate your help.

    Best Regards,
    Kenji
  • In reply to Kenji:

    Hi Hung,

    I tried the following command.

    # gst-launch-1.0 filesrc location=videos/h264-hd-30.mp4 ! qtdemux ! omxh264dec ! tee name=t \
    t. ! vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA,width=800,height=480 ! waylandsink sync=false async=false \
    t. ! queue ! vspmfilter outbuf-alloc=false ! video/x-raw,format=NV12,width=800,height=480 ! omxh264enc ! \
    rtph264pay ! udpsink host=192.168.0.1 port=10300 sync=false async=false

    But, the display image is wrong.
    When I change "outbuf-alloc" to "true" of vspmfilter for omxh264enc, the image is good.

    # gst-launch-1.0 filesrc location=videos/h264-hd-30.mp4 ! qtdemux ! omxh264dec ! tee name=t \
    t. ! vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA,width=800,height=480 ! waylandsink sync=false async=false \
    t. ! queue ! vspmfilter outbuf-alloc=true ! video/x-raw,format=NV12,width=800,height=480 ! omxh264enc ! \
    rtph264pay ! udpsink host=192.168.0.1 port=10300 sync=false async=false

    Are there also restrictions on "outbuf"?

    Best Regards,
    Kenji
  • In reply to Kenji:

    Hi Kenji,

    (sorry for so late reply, too busy lately).

    The problem you meet with first command is due to buffer mechanism of vspmfilter and omxh264enc.

    In different modes, vspmfilter has different buffer mechanism. With all options are false "outbuf-alloc=false", vspmfilter expects downstream element provides bufferpool for it to put output images.
    Unfortunately, omxh264enc does not support bufferpool (until now), so it cannot provide buffers to vspmfilter. Furthermore, omxh264enc expects upstream element provides input buffer for it.
    Due to this conflict in buffer mechanism, vspmfilter does not have correct buffers to run, which leads to the issue you saw.

    In "outbuf-alloc=true" mode, vspmfilter will allocate output buffers by itself. Thus it works correctly with omxh264enc.


    Regards,
    Hung.
  • In reply to HungT:

    Hi Hung,
    Thank you so much for your help.

    I understood about omxh264enc does not support bufferpool.

    BTW, when I use avidemux, vspmfilter and waylandsink, the display image is wrong.

    # gst-launch-1.0 filesrc do-timestamp=true location=input.avi typefind=true ! \
    avidemux ! h264parse ! omxh264dec ! \
    vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA,width=800,height=480 ! \
    waylandsink

    When I use mp4 file and qtdemux, the image is good.

    # gst-launch-1.0 filesrc do-timestamp=true location=input.mp4 typefind=true ! \
    qtdemux ! h264parse ! omxh264dec ! \
    vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA,width=800,height=480 ! \
    waylandsink

    Is this problem also related to outbuf-alloc ?

    Best Regards,
    Kenji
  • In reply to Kenji:

    Hi Kenji,

    No, this problem should not relate to vspmfilter (the demuxer has no relation with vspmfilter).
    There is chance that the video input.avi has a problem. Did you tried to play it on a PC?

    BTW, I notice you use "do-timestamp=true" for filesrc.
    FYI, option "do-timestamp" has no effect in this case. It is only used for a live source (like v4l2src)

    Regards,
    Hung.
  • In reply to HungT:

    Hi Hung,
    Sorry for late replay.

    There are no problem with input.avi. I attached this file.

    Bad case :

    # gst-launch-1.0 filesrc location=input.avi typefind=true ! \
     avidemux ! h264parse ! omxh264dec ! \
     vspmfilter outbuf-alloc=false ! video/x-raw,format=NV12,width=640,height=480 ! \
     waylandsink

    Good case :

    # gst-launch-1.0 filesrc location=input.avi typefind=true ! \
     avidemux ! h264parse ! omxh264dec ! \
     vspmfilter outbuf-alloc=false ! video/x-raw,format=BGRA,width=640,height=480 ! \
     waylandsink

    When I use mp4 file and qtdemux, both OK.

    Best Regards,
    Kenji

  • In reply to Kenji:

    I forgot to attach.Click here to play this video