TDA4VM: TDA4VM 线性三帧HDR VISS的merge

peng wei

Prodigy 120 points

Other Parts Discussed in Thread: TDA4VM

Hardware and Software Environment

Hardware Platform

TDA4VM-J721EXCPXEVM Evaluation Board
SoM-J721EXSOMXEVM

Software Environment

PROCESSOR-SDK-RTOS-J721E — RTOS SDK for DRA829 & TDA4VM Jacinto Processors
Version: 09.01.00.06
SC2210

Configuration Note
The camera registers are configured for three Virtual Channels (VC), which independently output long, medium, and short exposure data.

Problem Description & Attempts

Sensor Driver Configuration

Configuration 1

static IssSensor_CreateParams SC2210CreatePrms = {
SENSOR_SC2210, /* sensor name */
0x6, /* i2cInstId */
{
0x30, 0x30, 0x30, 0x30, /* #define SENSOR_I2C_ADDR 0x60 */
0, 0, 0, 0
}, /* i2cAddrSensor */
{
0, 0, 0, 0,
0, 0, 0, 0
}, /* i2cAddrSer */

/* IssSensor_Info */
{
{

1080, /* height - SC2210 actual resolution */
1,

{
{TIVX_RAW_IMAGE_16_BIT, 9},
},
0, /* meta_height_before */
0, /* meta_height_after */
},
ISS_SENSOR_SC2210_FEATURES, /* features */
ALGORITHMS_ISS_AEWB_MODE_AEWB, /* aewbMode */
30, /* fps - outputs 30fps (actual sensor samples at 90fps) */
4, /* numDataLanes - 4lane MIPI */
{1, 2, 3, 4}, /* dataLanesMap */
{0, 0, 0, 0}, /* dataLanesPolarity */
CSIRX_LANE_BAND_SPEED_720_TO_800_MBPS, /* csi_laneBandSpeed */
},
2, /* numChan - associated with i2cAddrSensor */
2210, /* dccId */
};

Method for Acquiring Three-Frame HDR

Configuration 2

Apply the following modifications in the capture module:
tda4vm/vision_apps/modules/src/app_capture_module.c

captureObj->params.numCh = sensorObj->num_cameras_enabled * 3;
captureObj->raw_image_arr[q] = vxCreateObjectArray(context, (vx_reference)raw_image, sensorObj->num_cameras_enabled * 3);

To enable three-frame HDR data acquisition, three independent channels are created for a single sensor. The system simultaneously receives three frames of data from the three independent CSI channels (CH0, CH1, and CH2).

exp0_ch0
exp0_ch1
exp0_ch2

Observed Phenomenon

The only changes made were to these two parameters in Configuration 1.

3, /* num_exposures - 3:1 WDR mode */
vx_false_e, /* line_interleaved - WDR requires line interleaving */
{
{TIVX_RAW_IMAGE_16_BIT, 9}, /* dataFormat and MSB [0] 10-bit RAW, MSB=9 Unpacked 16-bit: 2304*1296*2 = 5,971,968 bytes */
{TIVX_RAW_IMAGE_16_BIT, 9},
{TIVX_RAW_IMAGE_16_BIT, 9},
},

Saved data:

Only these two locations in Configuration 2 were modified.

captureObj->params.numCh = sensorObj->num_cameras_enabled * 3;

captureObj->raw_image_arr[q] = vxCreateObjectArray(context, (vx_reference)raw_image, sensorObj->num_cameras_enabled * 3);

Saved data:

Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp0_ch_0_00000473.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp0_ch_1_00000473.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp0_ch_2_00000473.raw ..

Modify both Configuration 1 and Configuration 2 simultaneously.

captureObj->params.numCh = sensorObj->num_cameras_enabled * 3;

captureObj->raw_image_arr[q] = vxCreateObjectArray(context, (vx_reference)raw_image, sensorObj->num_cameras_enabled * 3);

Saved data:

Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp0_ch_0_00000245.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp1_ch_0_00000245.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp2_ch_0_00000245.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp0_ch_1_00000245.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp1_ch_1_00000245.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp2_ch_1_00000245.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp0_ch_2_00000245.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp1_ch_2_00000245.raw ..
Done!
Writing /opt/vision_apps/app_cam_out/csix_raw_output_1920x1080_16bpp_exp2_ch_2_00000245.raw ..
Done!

Current Status

Currently, by modifying Configuration 2, I can successfully capture the Long, Medium, and Short exposure data from the SC2210 sensor. These are output via three independent channels (CH0, CH1, CH2) corresponding to the first exposure frame (exp0).

Questions

1. Regarding RTOS SDK support for SC2210 3-frame HDR:
Since I can already acquire the Long, Medium, and Short exposures from the three independent channels (CH0, CH1, CH2) on exp0, how should I configure the system to route these three independent channels to the VISS (Vision ISP) for HDR merging?

2. Implementation details:
If this is supported, how can it be implemented? Which specific modules or APIs should I focus on?

3. Workaround for unsupported scenarios:
If this specific routing is not currently supported, what would be the recommended approach to enable it?

1 个月前

0 Alice 1 个月前

TI__Mastermind 48670 points

Hello,

How are you capturing 3 frames of exposures? Are they treated as separate frames/channels and then you are opening the OpenVX node in multiple channels mode?

I dont think we have any examples of multi-exposure merge in the SDK, even though driver supports it in VISS. Capture and VISS node framework do not support capturing three exposures and then merging them..

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

I implemented a workaround in the file tda4vm/vision_apps/modules/src/app_capture_module.c by adding two channels

code:

captureObj->params.numCh = sensorObj->num_cameras_enabled * 3;

captureObj->raw_image_arr[q] = vxCreateObjectArray(context, (vx_reference)raw_image, sensorObj->num_cameras_enabled * 3);

Saved data:

exp0_ch_0, exp0_ch_1, and exp0_ch_2 store the long, medium, and short exposure data from the sensor, respectively.

If the SDK doesn't support it natively, could we implement a node to rearrange the pointers—mapping exp0_ch_0/1/2 to exp0_ch_0, exp1_ch_0, and exp2_ch_0—so that the VISS merge can handle it?

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

Here are the details on how to acquire exposure frames. Please review Configuration 1 and Configuration 2.

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

Is the block diagram in 1.1 supported by your native architecture? The diagram in 1.2 represents the current issue that needs to be resolved. Can we handle this by adding a new node to reorganize the data before feeding it into the VISS?

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

Reorganize the first frame data from the three channels (originally exp0_ch_0, exp0_ch_1, exp0_ch_2) to target the same channel index across different exposures (i.e., exp0_ch_0, exp1_ch_0, exp2_ch_0).

Input (capture output):

[exp0_ch_0_cam0, exp1_ch_0_cam0, exp2_ch_1_cam0, exp0_ch_1_cam1, exp1_ch_1_cam1, exp2_ch_1_cam1, exp0_ch_2_cam2, exp1_ch_2_cam2, exp2_ch_2_cam2]

Map the sequence [exp0_ch_0_cam0, exp0_ch_1_cam1, exp0_ch_2_cam2] to the output [exp0_ch_0_cam0, exp1_ch_0_cam1, exp2_ch_0_cam0] (HDR fusion expectation).

0 Alice 1 个月前回复 peng wei

TI__Mastermind 48670 points

Hello,

Are these exposures output of same bit depths? Are they all outputting in RAW12 format?

I want to first understand how CSIRX will output these three frames..

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

Are these exposure images the same bit depth?
Yes. The configuration explicitly specifies 10bit. This means all three exposure frames are output with the same bit depth.
Are they output in RAW12 format?
No. They are not output in RAW12. As indicated by the 10bit parameter in the configuration string, the sensor is configured to output data in RAW10 format.

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

Here is a detailed breakdown of the relevant parameters in your configuration string: cleaned_special_0x13_SC2210_MIPI_24Minput_4lane_10bit_594Mbps_237.6Mpclk_1920x1080_VC_SHDR_3-exp_30fps_continue

0 Alice 1 个月前回复 peng wei

TI__Mastermind 48670 points

Hello,

So CSIRX will store all three channels in object array? But when we connect this to VISS, VISS node will process only first channel. If we replicate VISS node, each channel will be separately processed.

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

When you mention "three channels," do you mean exp0, exp1, exp2 or ch0, ch1, ch2?

My understanding is that ch0 contains exp0, exp1, and exp2. If by "channel" you refer to ch0, can VISS process all three exposures (exp0, exp1, exp2) within it, or is it limited to processing only exp0?

0 Alice 1 个月前回复 peng wei

TI__Mastermind 48670 points

Hello,

But that's what i am asking. How are these exposures captured by CSIRX? Are you creating an object array to store these three exposures and treating them as channels? Because in OpenVX, object array contains different channel's data.

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

Yes, we use a vx_object_array to encapsulate the three exposures as separate raw_image objects, forming a multi-channel-like data structure.

0 Alice 1 个月前回复 peng wei

TI__Mastermind 48670 points

Hello，

That means you treating exposures as channels, isn't it? VISS node currently do not support extracting exposures from object array..

0 peng wei 1 个月前回复 Alice

Prodigy 120 points

So, how does VISS currently support triple exposure? Or how can triple exposure be fed into VISS?

0 Alice 1 个月前回复 peng wei

TI__Mastermind 48670 points

Hello，

Unfortunately we have not validated/supported merge in the VISS node. VISS node supports raw_image data structure as input and this data structure does support multiple exposure pointers..

处理器

处理器论坛

TDA4VM: TDA4VM 线性三帧HDR VISS的merge

Current Status

Questions