Example using camera with XIP kernel 4.9


Is there any example available of a working device-tree using a camera (any OVxxxx) for the 4.9 kernel in XIP with single QSPI ?

Best regards

  • Here's the status on camera interfacing (for RZ/A Linux that is).
    * There is a old V4L2 CEU driver in the kernel, but it's got some issues and more work that it's worth
    * There is some work going on in the upstream team to make a new V4L2 CEU driver, but it's not clear when that will get done
    * I created a very simple example at one point that basically sets up all the CEU and OV7670 registers from a user app (not a kernel driver). Basically, no V4L2 (just read camera frames to a buffer so you can pass them to OpenCV, libJPEG, the LCD, or whatever you want).

    Since #1 and #2 are not great options for right now, I was going to clean up #3 and post it. It was done with Linxu-3.14, but as I mentioned, it's all contained in a user app, so there isn't really and "porting" except add the pin mux setup in the Device Tree.

    So the question is, do you want the current ugly version now? Or wait till it's cleaned up and posted? (probably December time frame)

    NOTE: On the RZ/A1 RSK board, you can't use the LCD and CEU camera at the same time (pin conflict). So you have to save JPEGs to a file on a SD card/USB stick, or stream them via Ethernet/HTTP.

  • Thanks for your reply Chris.
    I'll receive my hardware in early december (board based on RZ/A1L, single QSPI, OV6211) and my aim is to ensure our hardware can get frame from OV6211. I'm afraid I'll run out of time if I wait for the new driver.
    Actually I wanted to use 4.9 kernel to play around but if there is a better solution with the 3.14 I don't mind using it again :)
    As long as it works every solution suit me well !
  • TBH I tried to backport it to the 4.9 kernel but I gave up a few hours after that, lot of things changed and it make the backport too heavy (for me). But I guess it will be much more easier to backport it to the 4.14 kernel.
    Regarding the 4.15 kernel I only had to add and patch AXFS to the kernel. I don't think I had to modify something else (except devicetree) Maybe I reused some ASM files (startup code) from the 4.9 to solve some compilations errors ? I didn't wrote down all the changes I've made.

  • > Regarding the 4.15 kernel I only had to add and patch AXFS to the kernel.

    Can I get your AXFS patch??? It's the only thing I need left before we can release the 4.14 BSP. I've been meaning to get back to it, but haven't.

    If it looks good, maybe you can submit a pull request to the official repo ( github.com/.../ ).
  • > Can I get your AXFS patch???
    Sure, I'll open a PR before the end of the day.
  • I just saw the PR. Thanks!

    At first glance, I have some questions on it, but I'll ask them via github.

    One more think I forgot to mention was that now as of 4.15, cramfs has an XIP option now. However unlike AXFS, it's all or nothing when you choose to XIP a file (you can't select just the pages you want XIP-ed). However, it is an 'upstream' option. But, you do need a new mkcramfs tool to use it. I plan on updating Buildroot (upstream) at some point (hopefully sooner than later)
  • Now you're talking about XIP option in upstream kernel, I remember I had to do some dirty hacks to be able to build my kernel as XIP.
  • Yes, no matter how many times I've tried, and as many different patches I have submitted, the upstream ARM maintainer will not let me enable XIP for MMU enabled systems. (although he is OK with keeping the underlying code working).

    So, you always have to apply this simple patch to be able to enable XIP_KERNEL:

    diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
    index 7f61372a6462..d33e20ffcbe6 100644
    --- a/arch/arm/Kconfig
    +++ b/arch/arm/Kconfig
    @@ -331,7 +331,7 @@ config ARCH_MULTIPLATFORM
    bool "Allow multiple platforms to be selected"
    depends on MMU
    select ARM_HAS_SG_CHAIN
    - select ARM_PATCH_PHYS_VIRT
    select AUTO_ZRELADDR
    select TIMER_OF
    select COMMON_CLK
    @@ -1966,7 +1966,7 @@ endchoice

    config XIP_KERNEL
    bool "Kernel Execute-In-Place from ROM"
    - depends on !ARM_LPAE && !ARCH_MULTIPLATFORM
    + depends on !ARM_LPAE
    Execute-In-Place allows the kernel to run from non-volatile storage
    directly addressable by the CPU, such as NOR flash. This saves RAM

    Since 4.16.-rc1 was released yesterday, I'll have to confirm sometime this week that nothing broke for XIP_KERNEL.
  • Hello!

    I'm also trying to use an OV7670 with Linux-4.9, but in a RSK+ (RZA1H). With "ceu_example", the code stucked in the while around "ceu_is_capturing" function (CAPSR register always 0x01). You said we must change CEU pin setup in u-boot or kernel, but I can't figure out where... The ideia is "streaming" the images to a webpage, just like 3.14 example.

    Thanks a lot!
  • Remember, on the RSK board, you can not use camera and LCD at the same time. They share pins.
    So, you should remove LCD display section (vdc5) from the Device Tree so the pins do not get configured as LCD.

    Or, you can set the status to disabled.
    &vdc50 {
    pinctrl-names = "default";
    pinctrl-0 = <&vdc50_pins>;

    display = <&display0>;
    status = "disabled";

    Then, to set the pins to CEU, you can do this in u-boot, add this code to function board_early_init_f( )
    [file] u-boot-2017.05/board/renesas/rskrza1/rskrza1.c

    /* Enable CEU Camera interface */
    pfc_set_pin_function(10, 1, ALT1, 0, 0); /* VIO_VD */
    pfc_set_pin_function(10, 2, ALT1, 0, 0); /* VIO_HD */
    pfc_set_pin_function(10, 0, ALT1, 0, 0); /* VIO_CLK */

    pfc_set_pin_function(10, 4, ALT1, 0, 0); /* VIO_D0 */
    pfc_set_pin_function(10, 5, ALT1, 0, 0); /* VIO_D1 */
    pfc_set_pin_function(10, 6, ALT1, 0, 0); /* VIO_D2 */
    pfc_set_pin_function(10, 7, ALT1, 0, 0); /* VIO_D3 */

    pfc_set_pin_function(10, 8, ALT1, 0, 0); /* VIO_D4 */
    pfc_set_pin_function(10, 9, ALT1, 0, 0); /* VIO_D5 */
    pfc_set_pin_function(10, 10,ALT1, 0, 0); /* VIO_D6 */
    pfc_set_pin_function(10, 11,ALT1, 0, 0); /* VIO_D7 */

    Or, instead you can add the pin setup to the Device Tree in the kernel.
    The easiest is to just add the CEU pins to a driver that you already know will be used.
    For example, the serial console. Then, when the serial console is loaded, the CEU pins will also get set up.
    Just remember that you have to remove the ";" in the list (after RxD2) and and replace it with a ","

    Something like this:
    [FILE] linux-4.9/arch/arm/boot/dts/r7s72100-rskrza1.dts

    &pinctrl {

    /* Serial Console */
    scif2_pins: serial2 {
    pinmux = <RZA1_PINMUX(3, 0, 6)>, /* TxD2 */
    - <RZA1_PINMUX(3, 2, 4)>; /* RxD2 */
    + <RZA1_PINMUX(3, 2, 4)>, /* RxD2 */
    + /* Enable CEU Camera interface */
    + <RZA1_PINMUX(10, 1, 1>, /* VIO_VD */
    + <RZA1_PINMUX(10, 2, 1>, /* VIO_HD */
    + <RZA1_PINMUX(10, 0, 1>, /* VIO_CLK */
    + <RZA1_PINMUX(10, 4, 1>, /* VIO_D0 */
    + <RZA1_PINMUX(10, 5, 1>, /* VIO_D1 */
    + <RZA1_PINMUX(10, 6, 1>, /* VIO_D2 */
    + <RZA1_PINMUX(10, 7, 1>, /* VIO_D3 */
    + <RZA1_PINMUX(10, 8, 1>, /* VIO_D4 */
    + <RZA1_PINMUX(10, 9, 1>, /* VIO_D5 */
    + <RZA1_PINMUX(10, 10,1>, /* VIO_D6 */
    + <RZA1_PINMUX(10, 11,1>; /* VIO_D7 */

    That should do it.
  • Hi Chris, thanks for your help!

    I've disabled "vdc50" and added "PINMUX" on Device Tree, but no success... App still stucking on "ceu_is_capturing"... I've also tried "pfc_set_pin_function", with no luck... Double checked the connections, they're fine too!

    Thanks again
  • Don't you have some soldering to do (moving some resistors) to be able to use the camera on this board ? I never used this board.
  • First make sure something is coming out of the camera pins. That will confirm that the I2C connection is working.

    Even if you set up the CEU pins correctly, if the CEU never sees a VSYNC, HSYNC, or clock, it will never trigger.
  • Actually, now that you mention it, the pin multiplexer (IC38) need to be changed to redirect the pins/signals from LCD to CEU. But, you don't have to change any resistors that I can see or remember (no little yellow stickers on my boards that say 'modified for camera' like I normally do).

    So in u-boot, before you boot the board, change the pin multiplexer.
    There's a special u-boot command ("io_mux") just for the RSK board.

    => io_mux
    io_mux - Change io mux

    io_mux Change the I/O Multiplexers by changing the output of I2C Port Expander 2.
    Usage: io_mux [a|b|c|d]
    a: PX1_EN0=L: P10/P11 = LCD(CN44)
    b: PX1_EN0=H: P10/P11 = CEU(CN41) and RSPI-1(CN25,CN26)

    c: PX1_EN1=L: P2/P3 = IO(JA1), RSPI-4(CN15), SIM(CN4)
    d: PX1_EN1=H: P2/P3 = Ethernet

    Current PX1_EN0 = L
    PX1_EN1 = H

    So you will want to run the command:
    => io_mux b

    then, boot your board.
  • Hello guys

    I only had to solder a connector for the camera, even some capacitor around was already soldered... And about i2c, looks like it's working fine: if I remove the camera VCC, "ceu_omni" shows me "camera not found"; putting VCC again, I've got "ov7670 found" (I edited the "I2C_BUS_SCAN_" routine)...
  • Chris is right, since the number of pins is limited, some of them are shared. So you need to manually tell your u-boot or whatever you want to physically connect VS, HS, CLK to your camera.
    I2C is working (this is normal) but you're stuck here because there is no end of frame interrupt. You're not receiving this interrupt because you're missing at least one signal from your camera and CEU is still waiting for it.
  • Hi guys

    Even changing "io_mux" (both pin "High"), no luck... I'll try to find a Scope so I can check the 3 pins signal...

    Thanks again!
Reply Children
  • Hello guys

    Now the camera is working! The problem was "RZA1_PINMUX" for CEU interface. Alternative function "f" is 6, according to datasheet, instead of 1:

    <RZA1_PINMUX(10, 0, 6)>

    Each capture is displaying on a simple webpage (slowly, but it's ok!).

    @Chris, any chance to have this camera driver in kernel again?
  • To be honest, I think it will be better for you to switch to a 4.17 kernel.
    The reworked driver has been released, see git.kernel.org/.../renesas-ceu.c
    As I said previously in this post, backporting this driver to the 4.9 kernel will be complicated. Maybe it will be easier to backport it to the 4.14 kernel maintain by Renesas but It may take some time.

  • The newer mainlined CEU driver is much better than the old one. But as Dylan pointed out, it didn't get in until 4.17.

    We're in the process of backporting it to our 4.14 BSP. You can't just copy the driver back because some files in the video subsystem changed between 4.14 and 4.17. After we get it stable and tested, we'll push it to the 4.14 repo on github.

    As for backporting it all the way to 4.9, we will probably not do that.
  • Thanks guys for the information!

    So I'll wait the 4.14 update!

    Thanks again!