TMS320C6657: 作为PCIE EP设备与ARM SOC进行PCIE通信数据读写及测试读写速率例程需求

panjiang weng

Prodigy 40 points

Part Number: TMS320C6657

1.开发环境

CCS版本号：V5.5.0.0007

PDK版本号：1.1.2.6

2.例程需求描述

2.1 设计框图如下：

（1）ARM端作为PCIE-RC设备，运行的是Ubuntu18.04系统；
（2）TMS320C6657作为PCIE-EP设备，运行的固件是“D:\ti\pdk_C6657_1_1_2_6\packages\ti\drv\exampleProjects\PCIE_exampleProject”的PCIE_exampleProject

3.问题及需求描述

问题：在PCIE_exampleProject例程中，PCIE BAR的配置只是映射了L2 SRAM的地址，如何将DDR3的地址映射到PCIE域进行数据读写测试，

需求：

（1）如何配置PCIE的参数及地址，完成DDR3的地址映射？是否有参考代码？

（2）完成参数及地址配置后，如何向DDR3内存地址中写入数据，便于RC端读取？如何读取DDR3内存地址中的数据，来确认RC端已经向PCIE EP写入数据?是否有参考代码？

（3）如不能在PCIE_exampleProject例程进行修改完成需求，那是否有其他PCIE数据通信和读写速率的例程可以参考？

1 年多前

0 Gary Lu 1 年多前

TI__Mastermind 27060 points

您好，

对于TMS320C6657的PCIE BAR配置以及DDR3地址映射的需求，这是一个相当复杂的问题，需要深入的硬件和软件知识。在这里，我可以给你一些个人的指导，请参考：

1. 配置PCIE的参数及地址，完成DDR3的地址映射：
- 首先，你需要了解TMS320C6657的PCIE控制器和DDR3控制器的寄存器配置，以及PCIE BAR的设置。你需要配置PCIE BAR，将DDR3的地址映射到PCIE域。这通常需要对PCIE控制器和DDR3控制器进行初始化和配置。

2. 向DDR3内存地址中写入数据，以及读取DDR3内存地址中的数据：
- 一旦PCIE BAR和DDR3地址映射完成，你可以使用PCIE接口进行对DDR3的读写操作。这涉及到PCIE的DMA控制和数据传输。
- 你需要编写PCIE EP端和RC端的代码，以实现对DDR3的数据读写操作。这可能需要使用PCIE的驱动程序和API。

0 panjiang weng 1 年多前回复 Gary Lu

Prodigy 40 points

1.我已经了解了并完成了PCIE BAR的设置，但是不清楚如何将DDR3的地址映射到PCIE域。以下分别是我的PCIE 配置的代码和PCIE BAR配置代码

/**
 * @brief configure PCIe in End Point Mode
 *
 * @param handle pcie register handle
 *
 * @return successful execution of the program
 *     @retval 0 successful
 *     @retval 1 Invalid handle
 */
int pcie_endpoint_cfg(Pcie_Handle handle)
{
    int                    ret;
    pcieCmdStatusReg_t     cmdStatus;
    pcieObSizeReg_t        obSize;
    pcieGen2Reg_t          gen2;
    pcieType0Bar32bitIdx_t type0Bar32bitIdx;
    pcieStatusCmdReg_t     statusCmd;
    pcieVndDevIdReg_t      vndDevId;
    pcieRevIdReg_t         revId;
    pcieDevStatCtrlReg_t   devStatCtrl;
    pcieAccrReg_t          accr;
    pcieRegisters_t        setRegs;
    pcieRegisters_t        getRegs;
    uint32_t               i;
    /* BAR mask */
    //PCIE_BAR0_MASK       0x00000FFF /* 4KB */
    //PCIE_BAR1_MASK       0x0007FFFF /* 512KB */
    //PCIE_BAR2_MASK       0x000FFFFF /* 1M */
    //PCIE_BAR3_MASK       0x00FFFFFF /* 16M */
    uint32_t               bar_mask[4] = { PCIE_BAR0_MASK, PCIE_BAR1_MASK,
                                            PCIE_BAR2_MASK, PCIE_BAR3_MASK };

    memset(&cmdStatus,        0, sizeof(cmdStatus));
    memset(&cmdStatus,        0, sizeof(vndDevId));
    memset(&cmdStatus,        0, sizeof(revId));
    memset(&obSize,           0, sizeof(obSize));
    memset(&gen2,             0, sizeof(gen2));
    memset(&type0Bar32bitIdx, 0, sizeof(type0Bar32bitIdx));
    memset(&statusCmd,        0, sizeof(statusCmd));
    memset(&devStatCtrl,      0, sizeof(devStatCtrl));
    memset(&accr,             0, sizeof(accr));

    /*Disable link training*/
    ret = pcie_ltssm_ctrl(handle, FALSE);
    if(ret) {
        uart_printf("Failed to disable Link Training!\n");
        return ret;
    }

    /* Configure the vendor and device ID */
    memset(&setRegs, 0, sizeof(setRegs));
    vndDevId.devId = 0x104c;
    vndDevId.vndId = 0xb005;
    setRegs.vndDevId = &vndDevId;
    ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
    if (ret)
    {
        printf("Configure the vendor and device ID failed !\n");
        return ret;
    }
    
    /* Configure the class code and revision ID */
    memset(&setRegs, 0, sizeof(setRegs));
    revId.classCode = 0x048000;
    revId.revId = 0x01;
    setRegs.revId = &revId;
    ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
    if (ret)
    {
        printf("Configure the class code and revision ID failed !\n");
        return ret;
    }
    obSize.size = pcie_OB_SIZE_8MB;
    setRegs.obSize = &obSize;

    if ((ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs)))
    {
        printf("Configure the vendor and device ID failed !\n");
        return ret;
    }

    /* Setting PL_GEN2 */
    memset(&setRegs, 0, sizeof(setRegs));
    gen2.numFts = 0xF;
    gen2.dirSpd = 0x0;
    gen2.lnEn   = 1;
    setRegs.gen2 = &gen2;
    ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
    if(ret) {
        uart_printf("SET GEN2 register failed!\n");
        return ret;
    }

    /* First need to enable writing on BAR mask registers */
    memset(&setRegs, 0, sizeof(setRegs));
    memset(&getRegs, 0, sizeof(getRegs));
    memset(&cmdStatus,  0, sizeof(cmdStatus));

    getRegs.cmdStatus = &cmdStatus;
    ret = Pcie_readRegs(handle, pcie_LOCATION_LOCAL, &getRegs);
    if(ret) {
        uart_printf("Read CMD STATUS register failed!\n");
        return ret;
    }

    cmdStatus.dbi = 1;
    setRegs.cmdStatus = &cmdStatus;
    ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
    if(ret) {
        uart_printf("SET CMD STATUS register failed!\n");
        return ret;
    }

    /* Configure BAR0-3 Masks */
    for(i = 0; i < 4; i++) {
        memset(&setRegs, 0, sizeof(setRegs));
        type0Bar32bitIdx.idx = i;
        type0Bar32bitIdx.reg.reg32 = bar_mask[i];
        setRegs.type0Bar32bitIdx = &type0Bar32bitIdx;
        ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
        if(ret) {
            uart_printf("SET BAR MASK register failed!\n");
            return ret;
        }
    }

    /* Disable writing on BAR Masks */
    memset(&setRegs, 0, sizeof(setRegs));
    memset(&getRegs, 0, sizeof(getRegs));
    memset(&cmdStatus,  0, sizeof(cmdStatus));
    getRegs.cmdStatus = &cmdStatus;
    ret = Pcie_readRegs(handle, pcie_LOCATION_LOCAL, &getRegs);
    if(ret) {
        uart_printf ("Read CMD STATUS register failed!\n");
        return ret;
    }

    cmdStatus.dbi = 0;
    setRegs.cmdStatus = &cmdStatus;
    ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
    if(ret) {
        uart_printf ("SET CMD STATUS register failed!\n");
        return ret;
    }

    /* Enable memory access and mastership of the bus */
    memset(&setRegs, 0, sizeof(setRegs));
    memset(&getRegs, 0, sizeof(getRegs));
    getRegs.statusCmd = &statusCmd;
    ret = Pcie_readRegs(handle, pcie_LOCATION_LOCAL, &getRegs);
    if(ret) {
        uart_printf("Read Status Comand register failed!\n");
        return ret;
    }

    statusCmd.memSp  = 1;
    statusCmd.busMs  = 1;
    statusCmd.resp   = 1;
    statusCmd.serrEn = 1;
    setRegs.statusCmd = &statusCmd;
    ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
    if(ret) {
        uart_printf("SET Status Command register failed!\n");
        return ret;
    }

    /* Enable Error Reporting */
    memset(&setRegs, 0, sizeof(setRegs));
    memset(&getRegs, 0, sizeof(getRegs));
    getRegs.devStatCtrl = &devStatCtrl;
    ret = Pcie_readRegs(handle, pcie_LOCATION_LOCAL, &getRegs);
    if(ret) {
        uart_printf("Regad Device Status Control register failed!\n");
        return ret;
    }

    devStatCtrl.reqRp = 1;
    devStatCtrl.fatalErRp = 1;
    devStatCtrl.nFatalErRp = 1;
    devStatCtrl.corErRp = 1;
    setRegs.devStatCtrl = &devStatCtrl;
    ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
    if(ret) {
        uart_printf("SET Device Status Control register failed!\n");
        return ret;
    }

    /* Enable ECRC */
    memset(&setRegs, 0, sizeof(setRegs));
    accr.chkEn=1;
    accr.chkCap=1;
    accr.genEn=1;
    accr.genCap=1;
    setRegs.accr = &accr;
    ret = Pcie_writeRegs(handle, pcie_LOCATION_LOCAL, &setRegs);
    if(ret) {
        uart_printf("SET ACCR register failed!\n");
        return ret;
    }

    return 0;
}

/**
 * @brief configure PCIe BAR mapping
 *
 * @param void
 *
 * @return successful execution of the program
 *     @retval 0 successful
 *     @retval 1 Invalid handle
 */
int pcie_bar_cfg(void)
{
    int              ret;
    pcieIbTransCfg_t ibCfg;
    pcieBarCfg_t     barCfg;
    void             *pcieBase;

    /* Configure Address Translation */
    barCfg.location = pcie_LOCATION_LOCAL;
    barCfg.mode     = pcie_EP_MODE;
    barCfg.base     = PCIE_BAR0_LO_ADDR;        //PCIE_BAR0_LO_ADDR = CSL_PCIE_REGS(0x60000000)
    barCfg.prefetch = pcie_BAR_NON_PREF;
    barCfg.type     = pcie_BAR_TYPE32;
    barCfg.memSpace = pcie_BAR_MEM_MEM;
    barCfg.idx      = PCIE_BAR0_REGION;         //PCIE_BAR0_REGION = 0

    ret = Pcie_cfgBar(handle, &barCfg);
    if(ret) {
        uart_printf("Failed to configure BAR!\n");
        return ret;
    }

    ibCfg.ibBar         = 0; /* BAR0 */
    ibCfg.ibStartAddrLo = PCIE_BAR0_LO_ADDR;    //PCIE_BAR0_LO_ADDR = CSL_PCIE_REGS(0x60000000)
    ibCfg.ibStartAddrHi = PCIE_BAR0_HI_ADDR;    //PCIE_BAR0_HI_ADDR = 0
    ibCfg.ibOffsetAddr  = PCIE_BAR0_MAP_ADDR;   //PCIE_BAR0_MAP_ADDR = CSL_PCIE_CONFIG_REGS(0x21800000)
    ibCfg.region        = PCIE_BAR0_REGION;     //PCIE_BAR0_REGION = 0

    ret = pcie_inbound_cfg(handle, &ibCfg);
    if(ret) {
        uart_printf("Failed to configure BAR0 Inbound Translation!\n", (int)ret);
        return ret;
    }

    /* Configure Address Translation */
    barCfg.location = pcie_LOCATION_LOCAL;
    barCfg.mode     = pcie_EP_MODE;
    barCfg.base     = PCIE_BAR1_LO_ADDR;        //PCIE_BAR1_LO_ADDR = CSL_PCIE_REGS(0x60000000) + 0x4000000
    barCfg.prefetch = pcie_BAR_NON_PREF;
    barCfg.type     = pcie_BAR_TYPE32;
    barCfg.memSpace = pcie_BAR_MEM_MEM;
    barCfg.idx      = PCIE_BAR1_REGION;         //PCIE_BAR1_REGION = 1

    ret = Pcie_cfgBar(handle, &barCfg);
    if(ret) {
        uart_printf("Failed to configure BAR!\n");
        return ret;
    }

    ibCfg.ibBar         = 1; /* BAR1 */
    ibCfg.ibStartAddrLo = PCIE_BAR1_LO_ADDR;    //PCIE_BAR1_LO_ADDR = CSL_PCIE_REGS(0x60000000) + 0x4000000
    ibCfg.ibStartAddrHi = PCIE_BAR1_HI_ADDR;    //PCIE_BAR1_HI_ADDR = 0
    ibCfg.ibOffsetAddr  = PCIE_BAR1_MAP_ADDR;	//PCIE_BAR1_MAP_ADDR = CSL_DSP0_L2_SRAM_REGS(0x10800000)
    ibCfg.region        = PCIE_BAR1_REGION;		//PCIE_BAR1_REGION = 1

    ret = pcie_inbound_cfg(handle, &ibCfg);
    if(ret) {
        uart_printf("Failed to configure BAR1 Inbound Translation!\n", (int)ret);
        return ret;
    }
    
    Pcie_getMemSpaceRange (handle, &pcieBase, NULL);
    uart_printf("pcieBase=0x%p\n",pcieBase);

    /* Configure Address Translation */
    barCfg.location = pcie_LOCATION_LOCAL;
    barCfg.mode     = pcie_EP_MODE;
    barCfg.base     = PCIE_BAR2_LO_ADDR;		//PCIE_BAR2_LO_ADDR = CSL_PCIE_REGSCSL_PCIE_REGS(0x60000000) + 0x8000000
    barCfg.prefetch = pcie_BAR_NON_PREF;
    barCfg.type     = pcie_BAR_TYPE32;
    barCfg.memSpace = pcie_BAR_MEM_MEM;
    barCfg.idx      = PCIE_BAR2_REGION;			//PCIE_BAR2_REGION = 2

    ret = Pcie_cfgBar(handle, &barCfg);
    if(ret) {
        uart_printf("Failed to configure BAR!\n");
        return ret;
    }

    ibCfg.ibBar         = 2; /* BAR2 */
    ibCfg.ibStartAddrLo = PCIE_BAR2_LO_ADDR;	//PCIE_BAR2_LO_ADDR = CSL_PCIE_REGSCSL_PCIE_REGS(0x60000000) + 0x8000000
    ibCfg.ibStartAddrHi = PCIE_BAR2_HI_ADDR;	//PCIE_BAR2_HI_ADDR = 0
    ibCfg.ibOffsetAddr  = PCIE_BAR2_MAP_ADDR;	//PCIE_BAR2_MAP_ADDR = CSL_MSMC_SRAM_REGS(0x0c000000)
    ibCfg.region        = PCIE_BAR2_REGION;		//PCIE_BAR2_REGION = 2

    ret = pcie_inbound_cfg(handle, &ibCfg);
    if(ret) {
        uart_printf("Failed to configure BAR1 Inbound Translation!\n", (int)ret);
        return ret;
    }

    /* Configure Address Translation */
    barCfg.location = pcie_LOCATION_LOCAL;
    barCfg.mode     = pcie_EP_MODE;
    barCfg.base     = PCIE_BAR3_LO_ADDR;		//PCIE_BAR3_LO_ADDR = CSL_PCIE_REGSCSL_PCIE_REGS(0x60000000) + 0xc000000
    barCfg.prefetch = pcie_BAR_NON_PREF;
    barCfg.type     = pcie_BAR_TYPE32;
    barCfg.memSpace = pcie_BAR_MEM_MEM;
    barCfg.idx      = PCIE_BAR3_REGION;			//PCIE_BAR3_REGION = 3

    ret = Pcie_cfgBar(handle, &barCfg);
    if(ret) {
        uart_printf("Failed to configure BAR!\n");
        return ret;
    }

    ibCfg.ibBar         = 3; /* BAR3 */
    ibCfg.ibStartAddrLo = PCIE_BAR3_LO_ADDR;	//PCIE_BAR3_LO_ADDR = CSL_PCIE_REGSCSL_PCIE_REGS(0x60000000) + 0xc000000
    ibCfg.ibStartAddrHi = PCIE_BAR3_HI_ADDR;	//PCIE_BAR3_HI_ADDR = 0
    ibCfg.ibOffsetAddr  = PCIE_BAR3_MAP_ADDR;	//PCIE_BAR3_MAP_ADDR = CSL_DDR3_DATA_0_REGS(0x80000000)
    //dstBuf.buf地址为0x817138
    // ibCfg.ibOffsetAddr  = (uint32_t)Convert_CoreLocal2GlobalAddr ((uint32_t)dstBuf.buf);  //经过转换后的地址后=0x10817138
    ibCfg.region        = PCIE_BAR3_REGION;		//PCIE_BAR3_REGION = 3
    uart_printf("ibCfg.ibOffsetAddr=0x%08x\n", &(ibCfg.ibOffsetAddr));

    ret = pcie_inbound_cfg(handle, &ibCfg);
    if(ret) {
        uart_printf("Failed to configure BAR1 Inbound Translation!\n", (int)ret);
        return ret;
    }

    return 0;
}

地址映射关系如下图所示：

我们申请了一个dstBuf.buf用来接收和处理数据，并且通过函数Convert_CoreLocal2GlobalAddr（如下代码）将地址转换成全局地址。

/**
 * @brief This function convert L2SARM inter address to global address
 *
 * @param addr L2SRAM inter address
 *
 * @return 32bit-addr
 */
uint32_t Convert_CoreLocal2GlobalAddr(uint32_t addr)
{
    uint32_t corenum;
    uint32_t	ret;
    /* Get the core number. */
    corenum = CSL_chipReadReg(CSL_CHIP_DNUM);
    printf("addr=0x%p\n",addr);
    ret = ((1 << 28) | (corenum << 24) | (addr & 0x00ffffff));
    printf("ret=0x%p\n",ret);
    return ret; 
}

并将Convert_CoreLocal2GlobalAddr函数的返回值作为BAR3的ibCfg.ibOffsetAddr的值。如下是测试的buf地址数据（在BAR配置中也有写，但是测试时无法访问被注释掉了）

//dstBuf.buf地址为0x817138
// ibCfg.ibOffsetAddr = (uint32_t)Convert_CoreLocal2GlobalAddr ((uint32_t)dstBuf.buf); //经过转换后的地址后=0x10817138。

此时我们在ARM端可以看到C6657有4个Region(0-3)，如下图所示。想在ARM端访问Region 3,从而访问到C6657 的dstBuf.buf的内容，但是ARM端访问报错。

疑问点：

（1）Convert_CoreLocal2GlobalAddr 函数能否将一个buf地址转换成DDR3内存上的地址？

（2）PCIE地址映射能否像PCIE BAR配置中BAR3的那样进行映射？即将Convert_CoreLocal2GlobalAddr函数的返回值作为BAR3的ibCfg.ibOffsetAddr的值

（3）您所说的“PCIE接口进行对DDR3的读写操作”是指哪个接口？是否有详细的例程或文档描述？

（4）在文档KeyStone Architecture Peripheral Component Interconnect Express (PCIe)（Literature Number: SPRUGS6D）的第2.7.1和2.7.2章节中有对inbound和outbound的地址转换进行举例，但是没有明白Example2-1中的“0x9D9A1234”和Example2-2中的“0x12345678_ABC50000” 与BAR配置代码中的哪个配置参数的含义是一样的？

0 Gary Lu 1 年多前回复 panjiang weng

TI__Mastermind 27060 points

1. Convert_CoreLocal2GlobalAddr函数只能将一个buf地址转换成Core Local地址，无法将其转换为DDR3内存上的地址。

2. PCIE地址映射可以像PCIE BAR配置中BAR3那样进行映射。将Convert_CoreLocal2GlobalAddr函数的返回值作为BAR3的ibCfg.ibOffsetAddr的值是可行的。

3. “PCIE接口进行对DDR3的读写操作”是指在PCIE设备和主机之间进行数据传输时，可以使用DMA引擎或直接访问DDR3内存进行读写操作。

4. 在Example2-1中，“0x9D9A1234”与BAR配置代码中的ibCfg.ibBaseAddr参数的含义是一样的，表示PCIE设备的Inbound空间的起始地址。在Example2-2中，“0x12345678_ABC50000”与BAR配置代码中的obCfg.obBaseAddr参数的含义是一样的，表示PCIE设备的Outbound空间的起始地址。

0 panjiang weng 1 年多前回复 Gary Lu

Prodigy 40 points

1.如果将Convert_CoreLocal2GlobalAddr函数中的第15行“ret = ((1 << 28) | (corenum << 24) | (addr & 0x00ffffff));”改成“ret = ((8 << 28) | (corenum << 24) | (addr & 0x00ffffff));”，如下代码所示，是不是就可以将buf地址转换成DDR3内存上的地址?根据datasheet描述，DDR3的地址范围为：0x80000000~FFFFFFFF

/**
 * @brief This function convert L2SARM inter address to global address
 *
 * @param addr L2SRAM inter address
 *
 * @return 32bit-addr
 */
uint32_t Convert_CoreLocal2GlobalAddr(uint32_t addr)
{
    uint32_t corenum;
    uint32_t	ret;
    /* Get the core number. */
    corenum = CSL_chipReadReg(CSL_CHIP_DNUM);
    printf("addr=0x%p\n",addr);
    ret = ((8 << 28) | (corenum << 24) | (addr & 0x00ffffff));
    printf("ret=0x%p\n",ret);
    return ret; 
}

2.无疑问了

3.可以使用DMA引擎或直接访问DDR3内存进行读写操作，DMA的操作和直接访问DDR3内存是否有文档或SDK中有参考例程？

4.我提供的代码中并没有ibCfg.ibBaseAddr和obCfg.obBaseAddr，您是指ibCfg.ibStartAddrL还是barCfg.base？