Example using camera with XIP kernel 4.9

Hello,

Is there any example available of a working device-tree using a camera (any OVxxxx) for the 4.9 kernel in XIP with single QSPI ?

Best regards

Parents
  • Here's the status on camera interfacing (for RZ/A Linux that is).
    * There is a old V4L2 CEU driver in the kernel, but it's got some issues and more work that it's worth
    * There is some work going on in the upstream team to make a new V4L2 CEU driver, but it's not clear when that will get done
    * I created a very simple example at one point that basically sets up all the CEU and OV7670 registers from a user app (not a kernel driver). Basically, no V4L2 (just read camera frames to a buffer so you can pass them to OpenCV, libJPEG, the LCD, or whatever you want).

    Since #1 and #2 are not great options for right now, I was going to clean up #3 and post it. It was done with Linxu-3.14, but as I mentioned, it's all contained in a user app, so there isn't really and "porting" except add the pin mux setup in the Device Tree.

    So the question is, do you want the current ugly version now? Or wait till it's cleaned up and posted? (probably December time frame)

    NOTE: On the RZ/A1 RSK board, you can't use the LCD and CEU camera at the same time (pin conflict). So you have to save JPEGs to a file on a SD card/USB stick, or stream them via Ethernet/HTTP.

    Chris
  • Thanks for your reply Chris.
    I'll receive my hardware in early december (board based on RZ/A1L, single QSPI, OV6211) and my aim is to ensure our hardware can get frame from OV6211. I'm afraid I'll run out of time if I wait for the new driver.
    Actually I wanted to use 4.9 kernel to play around but if there is a better solution with the 3.14 I don't mind using it again :)
    As long as it works every solution suit me well !
  • >Actually I wanted to use 4.9 kernel to play around but if there is a better solution with the 3.14 I don't mind using it again :)

    The 4.9 BSP is going to be easier than 3.14, especially for a custom board. There are scripts now that will help you generate and customize the files (and device tree) for u-boot and kernel with the 4.9 BSP for your board. Basically you fill out a GUI with what's on your board, and 1/2 work will be done for you automatically. We have no plan to go back and add that for the 3.14 BSP.

    As for that #3 simple CEU camera driver, when i go back and clean it up, it will go into the 4.9 BSP, not the 3.14 BSP. I'll make a note to have a look at it and see if it's in a state that I can share it next week (it's been a while since i've built and ran it).


    NOTE: I just had a look at the OV6211 data sheet. It looks like it only outputs MIPI. The RZ/A1L doesn't support MIPI, only parallel.
  • Well then I will finally play with the 4.9 kernel :)
    > It looks like it only outputs MIPI. The RZ/A1L doesn't support MIPI, only parallel.
    Yes, we're aware of this issue, a FPGA will convert MIPI to parallel and I'll check if it's work.

    Thanks for the incoming release of your work !
  • Ahh, FPGA to the rescue!
  • When it comes to video, FPGA is sometimes the only solution :)
  • , did you find some time to cleanup your app ? I would like to give it a try.

  • Not yet.
    The reason was that with the RZ/A1H RSK board, you can't use the LCD and camera at the same time. Also, with the RZ/A1L Stream-it board, you can't use the SDRAM and camera at the same time.
    However, a new version of the Stream it board is coming out that with a resistor change, you can use SDRAM, LCD and camera at the same time. So, that was what I was going to base my example code on.
  • Just FYI, I cleaned up that camera code and added it to the 4.9 BSP.
    github.com/.../6cfd6f6758596b06c6adcb9af4583daf825e2d76

    I used a new Stream it V2.3 board that can be easily modified to support the CEU interface and Linux at the same time.

    The code supports both OV7670 and OV7740 cameras which came with the Stream It kit.

    The app code is pretty generic, and should work with any board (but you have to set up the CEU pins yourself)
  • Thanks a lot Chris ! I'll give it a try. Currently I'm using the ongoing upstream CEU driver and a 4.15 kernel it works so far with an OV7670 (I'm still working on OV6211)
Reply Children
  • I've been meaning to try that new CEU driver and backport it to 4.14 (our next BSP kernel version). I wasn't sure if it would back port easily to 4.9, so I just put up the sample code that sets the registers from user space.

    Out of curiosity, is everything you need (RZ/A1 drivers) in the 4.15 upstream release?
  • TBH I tried to backport it to the 4.9 kernel but I gave up a few hours after that, lot of things changed and it make the backport too heavy (for me). But I guess it will be much more easier to backport it to the 4.14 kernel.
    Regarding the 4.15 kernel I only had to add and patch AXFS to the kernel. I don't think I had to modify something else (except devicetree) Maybe I reused some ASM files (startup code) from the 4.9 to solve some compilations errors ? I didn't wrote down all the changes I've made.

  • > Regarding the 4.15 kernel I only had to add and patch AXFS to the kernel.

    Can I get your AXFS patch??? It's the only thing I need left before we can release the 4.14 BSP. I've been meaning to get back to it, but haven't.

    If it looks good, maybe you can submit a pull request to the official repo ( github.com/.../ ).
  • > Can I get your AXFS patch???
    Sure, I'll open a PR before the end of the day.
  • I just saw the PR. Thanks!

    At first glance, I have some questions on it, but I'll ask them via github.

    One more think I forgot to mention was that now as of 4.15, cramfs has an XIP option now. However unlike AXFS, it's all or nothing when you choose to XIP a file (you can't select just the pages you want XIP-ed). However, it is an 'upstream' option. But, you do need a new mkcramfs tool to use it. I plan on updating Buildroot (upstream) at some point (hopefully sooner than later)
  • Now you're talking about XIP option in upstream kernel, I remember I had to do some dirty hacks to be able to build my kernel as XIP.
  • Yes, no matter how many times I've tried, and as many different patches I have submitted, the upstream ARM maintainer will not let me enable XIP for MMU enabled systems. (although he is OK with keeping the underlying code working).

    So, you always have to apply this simple patch to be able to enable XIP_KERNEL:

    diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
    index 7f61372a6462..d33e20ffcbe6 100644
    --- a/arch/arm/Kconfig
    +++ b/arch/arm/Kconfig
    @@ -331,7 +331,7 @@ config ARCH_MULTIPLATFORM
    bool "Allow multiple platforms to be selected"
    depends on MMU
    select ARM_HAS_SG_CHAIN
    - select ARM_PATCH_PHYS_VIRT
    + select ARM_PATCH_PHYS_VIRT if !XIP_KERNEL
    select AUTO_ZRELADDR
    select TIMER_OF
    select COMMON_CLK
    @@ -1966,7 +1966,7 @@ endchoice

    config XIP_KERNEL
    bool "Kernel Execute-In-Place from ROM"
    - depends on !ARM_LPAE && !ARCH_MULTIPLATFORM
    + depends on !ARM_LPAE
    help
    Execute-In-Place allows the kernel to run from non-volatile storage
    directly addressable by the CPU, such as NOR flash. This saves RAM


    Since 4.16.-rc1 was released yesterday, I'll have to confirm sometime this week that nothing broke for XIP_KERNEL.
  • Hello!

    I'm also trying to use an OV7670 with Linux-4.9, but in a RSK+ (RZA1H). With "ceu_example", the code stucked in the while around "ceu_is_capturing" function (CAPSR register always 0x01). You said we must change CEU pin setup in u-boot or kernel, but I can't figure out where... The ideia is "streaming" the images to a webpage, just like 3.14 example.

    Thanks a lot!
  • Remember, on the RSK board, you can not use camera and LCD at the same time. They share pins.
    So, you should remove LCD display section (vdc5) from the Device Tree so the pins do not get configured as LCD.

    Or, you can set the status to disabled.
    &vdc50 {
    pinctrl-names = "default";
    pinctrl-0 = <&vdc50_pins>;

    display = <&display0>;
    status = "disabled";


    Then, to set the pins to CEU, you can do this in u-boot, add this code to function board_early_init_f( )
    [file] u-boot-2017.05/board/renesas/rskrza1/rskrza1.c

    /* Enable CEU Camera interface */
    pfc_set_pin_function(10, 1, ALT1, 0, 0); /* VIO_VD */
    pfc_set_pin_function(10, 2, ALT1, 0, 0); /* VIO_HD */
    pfc_set_pin_function(10, 0, ALT1, 0, 0); /* VIO_CLK */

    pfc_set_pin_function(10, 4, ALT1, 0, 0); /* VIO_D0 */
    pfc_set_pin_function(10, 5, ALT1, 0, 0); /* VIO_D1 */
    pfc_set_pin_function(10, 6, ALT1, 0, 0); /* VIO_D2 */
    pfc_set_pin_function(10, 7, ALT1, 0, 0); /* VIO_D3 */

    pfc_set_pin_function(10, 8, ALT1, 0, 0); /* VIO_D4 */
    pfc_set_pin_function(10, 9, ALT1, 0, 0); /* VIO_D5 */
    pfc_set_pin_function(10, 10,ALT1, 0, 0); /* VIO_D6 */
    pfc_set_pin_function(10, 11,ALT1, 0, 0); /* VIO_D7 */


    Or, instead you can add the pin setup to the Device Tree in the kernel.
    The easiest is to just add the CEU pins to a driver that you already know will be used.
    For example, the serial console. Then, when the serial console is loaded, the CEU pins will also get set up.
    Just remember that you have to remove the ";" in the list (after RxD2) and and replace it with a ","

    Something like this:
    [FILE] linux-4.9/arch/arm/boot/dts/r7s72100-rskrza1.dts

    &pinctrl {

    /* Serial Console */
    scif2_pins: serial2 {
    pinmux = <RZA1_PINMUX(3, 0, 6)>, /* TxD2 */
    - <RZA1_PINMUX(3, 2, 4)>; /* RxD2 */
    + <RZA1_PINMUX(3, 2, 4)>, /* RxD2 */
    +
    + /* Enable CEU Camera interface */
    + <RZA1_PINMUX(10, 1, 1>, /* VIO_VD */
    + <RZA1_PINMUX(10, 2, 1>, /* VIO_HD */
    + <RZA1_PINMUX(10, 0, 1>, /* VIO_CLK */
    +
    + <RZA1_PINMUX(10, 4, 1>, /* VIO_D0 */
    + <RZA1_PINMUX(10, 5, 1>, /* VIO_D1 */
    + <RZA1_PINMUX(10, 6, 1>, /* VIO_D2 */
    + <RZA1_PINMUX(10, 7, 1>, /* VIO_D3 */
    +
    + <RZA1_PINMUX(10, 8, 1>, /* VIO_D4 */
    + <RZA1_PINMUX(10, 9, 1>, /* VIO_D5 */
    + <RZA1_PINMUX(10, 10,1>, /* VIO_D6 */
    + <RZA1_PINMUX(10, 11,1>; /* VIO_D7 */
    +
    }


    That should do it.
  • Hi Chris, thanks for your help!

    I've disabled "vdc50" and added "PINMUX" on Device Tree, but no success... App still stucking on "ceu_is_capturing"... I've also tried "pfc_set_pin_function", with no luck... Double checked the connections, they're fine too!

    Thanks again
  • Don't you have some soldering to do (moving some resistors) to be able to use the camera on this board ? I never used this board.
  • First make sure something is coming out of the camera pins. That will confirm that the I2C connection is working.

    Even if you set up the CEU pins correctly, if the CEU never sees a VSYNC, HSYNC, or clock, it will never trigger.
  • Actually, now that you mention it, the pin multiplexer (IC38) need to be changed to redirect the pins/signals from LCD to CEU. But, you don't have to change any resistors that I can see or remember (no little yellow stickers on my boards that say 'modified for camera' like I normally do).

    So in u-boot, before you boot the board, change the pin multiplexer.
    There's a special u-boot command ("io_mux") just for the RSK board.

    => io_mux
    io_mux - Change io mux

    Usage:
    io_mux Change the I/O Multiplexers by changing the output of I2C Port Expander 2.
    Usage: io_mux [a|b|c|d]
    a: PX1_EN0=L: P10/P11 = LCD(CN44)
    b: PX1_EN0=H: P10/P11 = CEU(CN41) and RSPI-1(CN25,CN26)

    c: PX1_EN1=L: P2/P3 = IO(JA1), RSPI-4(CN15), SIM(CN4)
    d: PX1_EN1=H: P2/P3 = Ethernet

    Current PX1_EN0 = L
    PX1_EN1 = H


    So you will want to run the command:
    => io_mux b

    then, boot your board.
  • Hello guys

    I only had to solder a connector for the camera, even some capacitor around was already soldered... And about i2c, looks like it's working fine: if I remove the camera VCC, "ceu_omni" shows me "camera not found"; putting VCC again, I've got "ov7670 found" (I edited the "I2C_BUS_SCAN_" routine)...
  • Chris is right, since the number of pins is limited, some of them are shared. So you need to manually tell your u-boot or whatever you want to physically connect VS, HS, CLK to your camera.
    I2C is working (this is normal) but you're stuck here because there is no end of frame interrupt. You're not receiving this interrupt because you're missing at least one signal from your camera and CEU is still waiting for it.