This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

[FAQ] [参考译文] [常见问题解答] TDA4VM:如何使用 edgea-benchmark repo 对具有预编译模型工件的 TDA4VM SoC 进行推理/基准测试?

Guru**** 1091470 points
Other Parts Discussed in Thread: TDA4VM
请注意,本文内容源自机器翻译,可能存在语法或其它翻译错误,仅供参考。如需获取准确内容,请参阅链接中的英语原文或自行翻译。

https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1235203/faq-tda4vm-how-to-inference-benchmark-on-tda4vm-soc-with-precompiled-model-artifacts-using-edgeai-benchmark-repo

器件型号:TDA4VM

您好!

如何对作为目标 SoC 上的示例分析的一部分的物体检测、分类和分段模型进行基准测试?

目标 SoC 上的基准模型支持哪些 KPI?

感谢您的帮助。  

  • 请注意,本文内容源自机器翻译,可能存在语法或其它翻译错误,仅供参考。如需获取准确内容,请参阅链接中的英语原文或自行翻译。

    您好!

    用户可以使用以下步骤对 edgeai-Benchmark 存储库上的所有受支持模型进行基准测试:

    在您的 PC 上克隆 edgeai-Benchmark 存储库,并按照此处列出的设置说明进行操作: 链接

    对于本常见问题解答、我们使用了 Linux SDK 版本8.6、因此我们目前正在分支中 r8.6.

    我们在 edgeai-benchmark repo 中有许多受支持的模型,您可以检查文件 setting_base.yaml 以供参考。

    在本常见问题解答中、我们考虑了分级模型 V1. 该模型存在于 Model Zoo 中、具有参考标签  cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite

    将 SETTING_BASE.YAML 文件  MODEL_SELECTION :空更改为 MODE_SELECTION : cl-0000这将只消耗所选的模型用于编译部分。

    如果 MODEL_SELECTION 参数设置为空、则默认情况下、所有模型都将被考虑进行编译。

    确保内部没有任何链接文件 edgeai-benchmark/work_dirs/modelimports 否则将跳过编译。

    运行基准测试脚本、

    run_benchmarks_pc.sh TDA4VM

    成功执行 run_becnhmark_pc.sh 脚本后、控制台日志将如下所示。

    (benchmark) user@udtensorlab1:~/model-training/edgeai-benchmark$ ./run_benchmarks_pc.sh TDA4VM
    
    target_device/SOC: TDA4VM
    Pass the appropriate commandline argument to use another target_device
    find: ‘./work_dirs/modelartifacts/TDA4VM/8bits/’: No such file or directory
    TIDL_TOOLS_PATH=/home/user/model-training/edgeai-benchmark/tools/TDA4VM/tidl_tools
    LD_LIBRARY_PATH=/home/user/model-training/edgeai-benchmark/tools/TDA4VM/tidl_tools
    PYTHONPATH=:
    -------------------------------------------------------------------
    argv: ['./scripts/benchmark_modelzoo.py', 'settings_import_on_pc.yaml', '--target_device', 'TDA4VM']
    settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/TDA4VM_package', 'datasets_path': './dependencies/datasets', 'target_device': 'TDA4VM', 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
    work_dir: ./work_dirs/modelartifacts/TDA4VM_package/8bits
    
    INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/imagenetv2c/val
    
    INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=0.30s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.36s)
    creating index...
    index created!
    
    INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/ycbv
    loading annotations into memory...
    Done (t=0.04s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.03s)
    creating index...
    index created!
    
    INFO:20230607-173717: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=0.58s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.76s)
    creating index...
    index created!
    
    INFO:20230607-173720: dataset exists - will reuse - ./dependencies/datasets/widerface
    loading annotations into memory...
    Done (t=0.49s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.12s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.12s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.12s)
    creating index...
    index created!
    
    INFO:20230607-173722: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=0.46s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.46s)
    creating index...
    index created!
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/VOCdevkit/VOC2012
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/nyudepthv2
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
    
    INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
    download_ok: True
    configs to run: ['cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite']
    number of configs: 1
    TASKS                                                       |          |     0% 0/1| [< ]
    INFO:20230607-173733: starting process on parallel_device - 0   0%|          || 0/1 [00:00<?, ?it/s]
    
    INFO:20230607-173734: starting - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
    INFO:20230607-173734: model_path - /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
    INFO:20230607-173734: model_file - /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
    Downloading 1/1: /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
    Downloading software-dl.ti.com/.../mobilenet_v1_1.0_224.tflite to /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
    16908288it [00:08, 1992659.04it/s]                                                                                                                                                                                 
    Download done for /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
    /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
    
    INFO:20230607-173742: running - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
    INFO:20230607-173742: pipeline_config - {'task_type': 'classification', 'dataset_category': 'imagenet', 'calibration_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0x7f84f3177250>, 'input_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0x7f84f3177190>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7f84f3177110>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7f84f33c9c50>, 'session': <edgeai_benchmark.sessions.tflitert_session.TFLiteRTSession object at 0x7f845a1b5cd0>, 'metric': {'label_offset_pred': -1}, 'model_info': {'metric_reference': {'accuracy_top1%': 71.676}, 'model_shortlist': 10}}
    INFO:20230607-173742: import  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time...
     Preliminary number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
     
    Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal
    
     ************** Frame index 1 : Running float import ************* 
    ****************************************************
    **                ALL MODEL CHECK PASSED          **
    ****************************************************
    
    The soft limit is 2048
    The hard limit is 2048
    MEM: Init ... !!!
    MEM: Init ... Done !!!
     0.0s:  VX_ZONE_INIT:Enabled
     0.21s:  VX_ZONE_ERROR:Enabled
     0.48s:  VX_ZONE_WARNING:Enabled
     0.2456s:  VX_ZONE_INIT:[tivxInit:184] Initialization Done !!!
    
     ************ Frame index 1 : Running float inference **************** 
    
     ************ Frame index 2 : Running float inference **************** 
    
     ************ Frame index 3 : Running float inference **************** 
    
     ************ Frame index 4 : Running float inference **************** 
    
     ************ Frame index 5 : Running fixed point mode for calibration **************** 
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1822.48  .... ..... ... .... .....
    #    1 . .. T    1824.50  .... ..... ... .... .....
    #    2 . .. T    1841.23  .... ..... ... .... .....
    #    3 . .. T    1804.06  .... ..... ... .... .....
    #    4 . .. T    1822.88  .... ..... ... .... .....
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1622.77  .... ..... ... .... .....
    #    1 . .. T    1624.90  .... ..... ... .... .....
    #    2 . .. T    1569.75  .... ..... ... .... .....
    #    3 . .. T    1561.55  .... ..... ... .... .....
    #    4 . .. T    1617.24  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 0 completed ************************ 
     
     
     
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1572.60  .... ..... ... .... .....
    #    1 . .. T    1578.75  .... ..... ... .... .....
    #    2 . .. T    1552.07  .... ..... ... .... .....
    #    3 . .. T    1563.12  .... ..... ... .... .....
    #    4 . .. T    1559.22  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 1 completed ************************ 
     
     
     
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1623.64  .... ..... ... .... .....
    #    1 . .. T    1603.79  .... ..... ... .... .....
    #    2 . .. T    1608.56  .... ..... ... .... .....
    #    3 . .. T    1628.29  .... ..... ... .... .....
    #    4 . .. T    1609.37  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 2 completed ************************ 
     
     
     
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1562.18  .... ..... ... .... .....
    #    1 . .. T    1567.63  .... ..... ... .... .....
    #    2 . .. T    1552.51  .... ..... ... .... .....
    #    3 . .. T    1611.97  .... ..... ... .... .....
    #    4 . .. T    1550.12  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 3 completed ************************ 
     
     
     
    
    ~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
    
    Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt 
     Freeing memory for user provided Net
     ----------------------- TIDL Process with REF_ONLY FLOW ------------------------
    
    #    0 . .. T    1565.97  .... ..... ... .... .....
    #    1 . .. T    1599.54  .... ..... ... .... .....
    #    2 . .. T    1612.83  .... ..... ... .... .....
    #    3 . .. T    1561.91  .... ..... ... .... .....
    #    4 . .. T    1553.31  .... ..... ... .... .....
     
     
     *****************   Calibration iteration number 4 completed ************************ 
     
     
     
    
    ------------------ Network Compiler Traces -----------------------------
    Error : Error Code = <ERR_UNSUPPORTED_DATA_TYPE>
    Could not open /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_net/perfSimInfo.bin
    Rerunning network compiler for reshape
    
    ------------------ Network Compiler Traces -----------------------------
    successful Memory allocation
    ****************************************************
    **                ALL MODEL CHECK PASSED          **
    ****************************************************
    
    
     Final number of subgraphs:1 , 34 nodes delegated to accelerator 
     
    
    INFO:20230607-173854: import completed  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 72 sec
    INFO:20230607-173854: infer  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time...
     Number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
     
    infer : cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_| 100%|##########|| 5/5 [00:01<00:00,  4.76it/s]
    
    INFO:20230607-173856: infer completed  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 1 sec
    
    SUCCESS:20230607-173856: benchmark results - {'infer_path': 'cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite', 'accuracy_top1%': 100.0, 'num_subgraphs': 1, 'infer_time_core_ms': 198.201599, 'infer_time_subgraph_ms': 198.192176, 'ddr_transfer_mb': 18446744073709.55, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0}
    MEM: Deinit ... !!!
    MEM: Alloc's: 52 alloc's of 147541958 bytes 
    MEM: Free's : 52 free's  of 147541958 bytes 
    MEM: Open's : 0 allocs  of 0 bytes 
    MEM: Deinit ... Done !!!
    TASKS                                                       | 100%|██████████|| 1/1 [01:27<00:00, 87.60s/it]
    TASKS                                                       | 100%|██████████|| 1/1 [01:27<00:00, 87.60s/it]
    
    -------------------------------------------------------------------
    -------------------------------------------------------------------
    settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/_package', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
    no results found - no report to generate.
    Report generated at ./work_dirs/modelartifacts/_package
    -------------------------------------------------------------------
    

    请注意、该日志包含的迭代次数较少、因为根据 CALIBRATION_FRAMES 设置可能会有所不同。

    模型编译完成后、您可以在中检查生成的模型工件 edgea-benchmark/work_dirs/modelimits/TDA4VM 文件。

    下一步是封装此模型工件、以便它们可用于目标基准测试。

    运行打包脚本、

    run_package_artifacts_for_evm.sh TDA4VM

    成功执行上述脚本后、将在  edgeai-benchmark/work_dirs/modelimits/TDA4VM_packet 目录中创建一个软件包

    在目标器件上安装出现在 PC 中的 edgeai-Benchmark 存储库、使用已安装的目录运行进一步的命令。

    在目标端、我们需要安装  requirements_evm.txt 文件中提供的所有依赖项软件包。

    运行以下命令以将依赖项 列表安装为:

    pip3 install -r requirements_evm.txt

    然后再运行 run_benchmarks_evm.sh 确保这一点 modelimits_path 中的设置正确 setting_base.yam

    因为我们已将编译的模型工件存储在这里 TDA4VM_PACKAGE 我们可以将路径设置为

    modelartifacts_path : './work_dirs/modelartifacts/{target_device}_package'

    最后、要对目标运行基准测试脚本、请运行以下命令、如所示

    ./run_benchmarks_evm.sh TDA4VM

    执行上述脚本后、您将看到下面列出的内容 KPI。

    ' infer_path ":"CL-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite"、

    ' 精度_top1 %':80.0,

    ' num_subgraphs ':1、

    ' refer_time_core_ms ':1.965482、

    ' Infer_time_subgraph_ms ':1.950902、

    ' DDR_TRANSMIT_MB ':6.163737、

    ' perssim_time_ms ':0.0、

    ' perfsim_DDR_transfer_MB ':0.0、

    ' perssim_gmacs ':0.0

    请参阅以下日志以进行参考。

    root@tda4vm-sk:/opt/mounted-benchmark# ./run_benchmarks_evm.sh TDA4VM
    TARGET_SOC: TDA4VM
    Pass the appropriate commandline argument to use another one.
    TIDL_TOOLS_PATH=/opt/mounted-benchmark/tools/TDA4VM/tidl_tools
    LD_LIBRARY_PATH=/opt/mounted-benchmark/tools/TDA4VM/tidl_tools
    PYTHONPATH=:/usr/lib/python3.8/site-packages/
    ===================================================================
    Please install EdgeAI Linux SDK StarterKit for TDA4VM
    from: www.ti.com/.../PROCESSOR-SDK-LINUX-SK-TDA4VM
    All the required dependencies are available in the SDK
    For more information, please visit: www.ti.com/.../SK-TDA4VM
    -------------------------------------------------------------------
    ===================================================================
    argv: ['./scripts/benchmark_modelzoo.py', 'settings_infer_on_evm.yaml', '--target_device', 'TDA4VM']
    settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/TDA4VM_package', 'datasets_path': './dependencies/datasets', 'target_device': 'TDA4VM', 'target_machine': 'evm', 'run_suffix': None, 'parallel_devices': None, 'tensor_bits': 8, 'runtime_options': None, 'run_import': False, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_infer_on_evm.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
    work_dir: ./work_dirs/modelartifacts/TDA4VM_package/8bits
    
    INFO:20230607-130458: dataset exists - will reuse - ./dependencies/datasets/imagenetv2c/val
    
    INFO:20230607-130459: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=0.76s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.76s)
    creating index...
    index created!
    
    INFO:20230607-130500: dataset exists - will reuse - ./dependencies/datasets/ycbv
    loading annotations into memory...
    Done (t=0.12s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.10s)
    creating index...
    index created!
    
    INFO:20230607-130523: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=2.36s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=2.39s)
    creating index...
    index created!
    
    INFO:20230607-130533: dataset exists - will reuse - ./dependencies/datasets/widerface
    loading annotations into memory...
    Done (t=1.55s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=1.66s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.40s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=0.38s)
    creating index...
    index created!
    
    INFO:20230607-130541: dataset exists - will reuse - ./dependencies/datasets/coco
    loading annotations into memory...
    Done (t=2.88s)
    creating index...
    index created!
    loading annotations into memory...
    Done (t=1.59s)
    creating index...
    index created!
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/VOCdevkit/VOC2012
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/nyudepthv2
    
    INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
    
    INFO:20230607-130552: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
    download_ok: True
    configs to run: ['cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite']
    number of configs: 1
    
    INFO:20230607-130552: starting - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
    INFO:20230607-130555: model_path - /opt/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
    INFO:20230607-130555: model_file - /opt/mounted-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
    
    INFO:20230607-130555: running - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
    INFO:20230607-130555: pipeline_config - {'task_type': 'classification', 'dataset_category': 'imagenet', 'calibration_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0xffff5eae7550>, 'input_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0xffff5eae7610>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0xffff5eae7640>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0xffff5eae76d0>, 'session': <edgeai_benchmark.sessions.tflitert_session.TFLiteRTSession object at 0xffff5eae7880>, 'metric': {'label_offset_pred': -1}, 'model_info': {'metric_reference': {'accuracy_top1%': 71.676}, 'model_shortlist': 10}}
    INFO:20230607-130555: infer  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time...
     Number of subgraphs:1 , 34 nodes delegated out of 34 nodes 
     
    APP: Init ... !!!
    MEM: Init ... !!!
    MEM: Initialized DMA HEAP (fd=6) !!!
    MEM: Init ... Done !!!
    IPC: Init ... !!!
    IPC: Init ... Done !!!
    REMOTE_SERVICE: Init ... !!!
    REMOTE_SERVICE: Init ... Done !!!
     21226.775752 s: GTC Frequency = 200 MHz
    APP: Init ... Done !!!
     21226.787708 s:  VX_ZONE_INIT:Enabled
     21226.787741 s:  VX_ZONE_ERROR:Enabled
     21226.788333 s:  VX_ZONE_WARNING:Enabled
     21226.789310 s:  VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!!
     21226.789824 s:  VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!!
    infer : cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_| 100%|##########|| 5/5 [00:00<00:00, 19.63it/s]
    
    INFO:20230607-130557: infer completed  - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 1 sec
    
    SUCCESS:20230607-130557: benchmark results - {'infer_path': 'cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite', 'accuracy_top1%': 80.0, 'num_subgraphs': 1, 'infer_time_core_ms': 1.965482, 'infer_time_subgraph_ms': 1.950902, 'ddr_transfer_mb': 6.163737, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0}
     21227.153310 s:  VX_ZONE_INIT:[tivxHostDeInitLocal:107] De-Initialization Done for HOST !!!
     21227.155829 s:  VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!!
    APP: Deinit ... !!!
    REMOTE_SERVICE: Deinit ... !!!
    REMOTE_SERVICE: Deinit ... Done !!!
    IPC: Deinit ... !!!
    IPC: DeInit ... Done !!!
    MEM: Deinit ... !!!
    DDR_SHARED_MEM: Alloc's: 7 alloc's of 5358322 bytes 
    DDR_SHARED_MEM: Free's : 7 free's  of 5358322 bytes 
    DDR_SHARED_MEM: Open's : 0 allocs  of 0 bytes 
    DDR_SHARED_MEM: Total size: 536870912 bytes 
    MEM: Deinit ... Done !!!
    APP: Deinit ... Done !!!
    -------------------------------------------------------------------
    ===================================================================
    settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/_package', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'evm', 'run_suffix': None, 'parallel_devices': None, 'tensor_bits': 8, 'runtime_options': None, 'run_import': False, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_infer_on_evm.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
    no results found - no report to generate.
    Report generated at ./work_dirs/modelartifacts/_package
    -------------------------------------------------------------------