This thread has been locked.
If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.
您好！
用户可以使用以下步骤对 edgeai-Benchmark 存储库上的所有受支持模型进行基准测试：
在您的 PC 上克隆 edgeai-Benchmark 存储库,并按照此处列出的设置说明进行操作: 链接
对于本常见问题解答、我们使用了 Linux SDK 版本8.6、因此我们目前正在分支中 r8.6.
我们在 edgeai-benchmark repo 中有许多受支持的模型,您可以检查文件 setting_base.yaml 以供参考。
在本常见问题解答中、我们考虑了分级模型 V1. 该模型存在于 Model Zoo 中、具有参考标签 cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
将 SETTING_BASE.YAML 文件 行 MODEL_SELECTION :空更改为 MODE_SELECTION : cl-0000这将只消耗所选的模型用于编译部分。
如果 MODEL_SELECTION 参数设置为空、则默认情况下、所有模型都将被考虑进行编译。
确保内部没有任何链接文件 edgeai-benchmark/work_dirs/modelimports 否则将跳过编译。
运行基准测试脚本、
run_benchmarks_pc.sh TDA4VM
成功执行 run_becnhmark_pc.sh 脚本后、控制台日志将如下所示。
(benchmark) user@udtensorlab1:~/model-training/edgeai-benchmark$ ./run_benchmarks_pc.sh TDA4VM
target_device/SOC: TDA4VM
Pass the appropriate commandline argument to use another target_device
find: ‘./work_dirs/modelartifacts/TDA4VM/8bits/’: No such file or directory
TIDL_TOOLS_PATH=/home/user/model-training/edgeai-benchmark/tools/TDA4VM/tidl_tools
LD_LIBRARY_PATH=/home/user/model-training/edgeai-benchmark/tools/TDA4VM/tidl_tools
PYTHONPATH=:
-------------------------------------------------------------------
argv: ['./scripts/benchmark_modelzoo.py', 'settings_import_on_pc.yaml', '--target_device', 'TDA4VM']
settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/TDA4VM_package', 'datasets_path': './dependencies/datasets', 'target_device': 'TDA4VM', 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
work_dir: ./work_dirs/modelartifacts/TDA4VM_package/8bits
INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/imagenetv2c/val
INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/coco
loading annotations into memory...
Done (t=0.30s)
creating index...
index created!
loading annotations into memory...
Done (t=0.36s)
creating index...
index created!
INFO:20230607-173711: dataset exists - will reuse - ./dependencies/datasets/ycbv
loading annotations into memory...
Done (t=0.04s)
creating index...
index created!
loading annotations into memory...
Done (t=0.03s)
creating index...
index created!
INFO:20230607-173717: dataset exists - will reuse - ./dependencies/datasets/coco
loading annotations into memory...
Done (t=0.58s)
creating index...
index created!
loading annotations into memory...
Done (t=0.76s)
creating index...
index created!
INFO:20230607-173720: dataset exists - will reuse - ./dependencies/datasets/widerface
loading annotations into memory...
Done (t=0.49s)
creating index...
index created!
loading annotations into memory...
Done (t=0.12s)
creating index...
index created!
loading annotations into memory...
Done (t=0.12s)
creating index...
index created!
loading annotations into memory...
Done (t=0.12s)
creating index...
index created!
INFO:20230607-173722: dataset exists - will reuse - ./dependencies/datasets/coco
loading annotations into memory...
Done (t=0.46s)
creating index...
index created!
loading annotations into memory...
Done (t=0.46s)
creating index...
index created!
INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016
INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/VOCdevkit/VOC2012
INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/nyudepthv2
INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
INFO:20230607-173726: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd
download_ok: True
configs to run: ['cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite']
number of configs: 1
TASKS | | 0% 0/1| [< ]
INFO:20230607-173733: starting process on parallel_device - 0 0%| || 0/1 [00:00<?, ?it/s]
INFO:20230607-173734: starting - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
INFO:20230607-173734: model_path - /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
INFO:20230607-173734: model_file - /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
Downloading 1/1: /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
Downloading software-dl.ti.com/.../mobilenet_v1_1.0_224.tflite to /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
16908288it [00:08, 1992659.04it/s]
Download done for /home/user/model-training/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite
/home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite
INFO:20230607-173742: running - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite
INFO:20230607-173742: pipeline_config - {'task_type': 'classification', 'dataset_category': 'imagenet', 'calibration_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0x7f84f3177250>, 'input_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0x7f84f3177190>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0x7f84f3177110>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0x7f84f33c9c50>, 'session': <edgeai_benchmark.sessions.tflitert_session.TFLiteRTSession object at 0x7f845a1b5cd0>, 'metric': {'label_offset_pred': -1}, 'model_info': {'metric_reference': {'accuracy_top1%': 71.676}, 'model_shortlist': 10}}
INFO:20230607-173742: import - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time...
Preliminary number of subgraphs:1 , 34 nodes delegated out of 34 nodes
Warning : Requested Output Data Convert Layer is not Added to the network, It is currently not Optimal
************** Frame index 1 : Running float import *************
****************************************************
** ALL MODEL CHECK PASSED **
****************************************************
The soft limit is 2048
The hard limit is 2048
MEM: Init ... !!!
MEM: Init ... Done !!!
0.0s: VX_ZONE_INIT:Enabled
0.21s: VX_ZONE_ERROR:Enabled
0.48s: VX_ZONE_WARNING:Enabled
0.2456s: VX_ZONE_INIT:[tivxInit:184] Initialization Done !!!
************ Frame index 1 : Running float inference ****************
************ Frame index 2 : Running float inference ****************
************ Frame index 3 : Running float inference ****************
************ Frame index 4 : Running float inference ****************
************ Frame index 5 : Running fixed point mode for calibration ****************
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt
Freeing memory for user provided Net
----------------------- TIDL Process with REF_ONLY FLOW ------------------------
# 0 . .. T 1822.48 .... ..... ... .... .....
# 1 . .. T 1824.50 .... ..... ... .... .....
# 2 . .. T 1841.23 .... ..... ... .... .....
# 3 . .. T 1804.06 .... ..... ... .... .....
# 4 . .. T 1822.88 .... ..... ... .... .....
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt
Freeing memory for user provided Net
----------------------- TIDL Process with REF_ONLY FLOW ------------------------
# 0 . .. T 1622.77 .... ..... ... .... .....
# 1 . .. T 1624.90 .... ..... ... .... .....
# 2 . .. T 1569.75 .... ..... ... .... .....
# 3 . .. T 1561.55 .... ..... ... .... .....
# 4 . .. T 1617.24 .... ..... ... .... .....
***************** Calibration iteration number 0 completed ************************
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt
Freeing memory for user provided Net
----------------------- TIDL Process with REF_ONLY FLOW ------------------------
# 0 . .. T 1572.60 .... ..... ... .... .....
# 1 . .. T 1578.75 .... ..... ... .... .....
# 2 . .. T 1552.07 .... ..... ... .... .....
# 3 . .. T 1563.12 .... ..... ... .... .....
# 4 . .. T 1559.22 .... ..... ... .... .....
***************** Calibration iteration number 1 completed ************************
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt
Freeing memory for user provided Net
----------------------- TIDL Process with REF_ONLY FLOW ------------------------
# 0 . .. T 1623.64 .... ..... ... .... .....
# 1 . .. T 1603.79 .... ..... ... .... .....
# 2 . .. T 1608.56 .... ..... ... .... .....
# 3 . .. T 1628.29 .... ..... ... .... .....
# 4 . .. T 1609.37 .... ..... ... .... .....
***************** Calibration iteration number 2 completed ************************
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt
Freeing memory for user provided Net
----------------------- TIDL Process with REF_ONLY FLOW ------------------------
# 0 . .. T 1562.18 .... ..... ... .... .....
# 1 . .. T 1567.63 .... ..... ... .... .....
# 2 . .. T 1552.51 .... ..... ... .... .....
# 3 . .. T 1611.97 .... ..... ... .... .....
# 4 . .. T 1550.12 .... ..... ... .... .....
***************** Calibration iteration number 3 completed ************************
~~~~~Running TIDL in PC emulation mode to collect Activations range for each layer~~~~~
Processing config file #0 : /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_io_.qunat_stats_config.txt
Freeing memory for user provided Net
----------------------- TIDL Process with REF_ONLY FLOW ------------------------
# 0 . .. T 1565.97 .... ..... ... .... .....
# 1 . .. T 1599.54 .... ..... ... .... .....
# 2 . .. T 1612.83 .... ..... ... .... .....
# 3 . .. T 1561.91 .... ..... ... .... .....
# 4 . .. T 1553.31 .... ..... ... .... .....
***************** Calibration iteration number 4 completed ************************
------------------ Network Compiler Traces -----------------------------
Error : Error Code = <ERR_UNSUPPORTED_DATA_TYPE>
Could not open /home/user/model-training/edgeai-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/artifacts/tempDir/87_tidl_net/perfSimInfo.bin
Rerunning network compiler for reshape
------------------ Network Compiler Traces -----------------------------
successful Memory allocation
****************************************************
** ALL MODEL CHECK PASSED **
****************************************************
Final number of subgraphs:1 , 34 nodes delegated to accelerator
INFO:20230607-173854: import completed - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 72 sec
INFO:20230607-173854: infer - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time...
Number of subgraphs:1 , 34 nodes delegated out of 34 nodes
infer : cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_| 100%|##########|| 5/5 [00:01<00:00, 4.76it/s]
INFO:20230607-173856: infer completed - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 1 sec
SUCCESS:20230607-173856: benchmark results - {'infer_path': 'cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite', 'accuracy_top1%': 100.0, 'num_subgraphs': 1, 'infer_time_core_ms': 198.201599, 'infer_time_subgraph_ms': 198.192176, 'ddr_transfer_mb': 18446744073709.55, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0}
MEM: Deinit ... !!!
MEM: Alloc's: 52 alloc's of 147541958 bytes
MEM: Free's : 52 free's of 147541958 bytes
MEM: Open's : 0 allocs of 0 bytes
MEM: Deinit ... Done !!!
TASKS | 100%|██████████|| 1/1 [01:27<00:00, 87.60s/it]
TASKS | 100%|██████████|| 1/1 [01:27<00:00, 87.60s/it]
-------------------------------------------------------------------
-------------------------------------------------------------------
settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/_package', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'pc', 'run_suffix': None, 'parallel_devices': 1, 'tensor_bits': 8, 'runtime_options': None, 'run_import': True, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_import_on_pc.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None}
no results found - no report to generate.
Report generated at ./work_dirs/modelartifacts/_package
-------------------------------------------------------------------
请注意、该日志包含的迭代次数较少、因为根据 CALIBRATION_FRAMES 设置可能会有所不同。
模型编译完成后、您可以在中检查生成的模型工件 edgea-benchmark/work_dirs/modelimits/TDA4VM 文件。
下一步是封装此模型工件、以便它们可用于目标基准测试。
运行打包脚本、
run_package_artifacts_for_evm.sh TDA4VM
成功执行上述脚本后、将在 edgeai-benchmark/work_dirs/modelimits/TDA4VM_packet 目录中创建一个软件包
在目标器件上安装出现在 PC 中的 edgeai-Benchmark 存储库、使用已安装的目录运行进一步的命令。
在目标端、我们需要安装 requirements_evm.txt 文件中提供的所有依赖项软件包。
运行以下命令以将依赖项 列表安装为：
pip3 install -r requirements_evm.txt
然后再运行 run_benchmarks_evm.sh 确保这一点 modelimits_path 中的设置正确 setting_base.yam 长
因为我们已将编译的模型工件存储在这里 TDA4VM_PACKAGE 我们可以将路径设置为
modelartifacts_path : './work_dirs/modelartifacts/{target_device}_package'
最后、要对目标运行基准测试脚本、请运行以下命令、如所示
./run_benchmarks_evm.sh TDA4VM
执行上述脚本后、您将看到下面列出的内容 KPI。
' infer_path "："CL-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite"、
' 精度_top1 %'：80.0，
' num_subgraphs '：1、
' refer_time_core_ms '：1.965482、
' Infer_time_subgraph_ms '：1.950902、
' DDR_TRANSMIT_MB '：6.163737、
' perssim_time_ms '：0.0、
' perfsim_DDR_transfer_MB '：0.0、
' perssim_gmacs '：0.0
请参阅以下日志以进行参考。
root@tda4vm-sk:/opt/mounted-benchmark# ./run_benchmarks_evm.sh TDA4VM TARGET_SOC: TDA4VM Pass the appropriate commandline argument to use another one. TIDL_TOOLS_PATH=/opt/mounted-benchmark/tools/TDA4VM/tidl_tools LD_LIBRARY_PATH=/opt/mounted-benchmark/tools/TDA4VM/tidl_tools PYTHONPATH=:/usr/lib/python3.8/site-packages/ =================================================================== Please install EdgeAI Linux SDK StarterKit for TDA4VM from: www.ti.com/.../PROCESSOR-SDK-LINUX-SK-TDA4VM All the required dependencies are available in the SDK For more information, please visit: www.ti.com/.../SK-TDA4VM ------------------------------------------------------------------- =================================================================== argv: ['./scripts/benchmark_modelzoo.py', 'settings_infer_on_evm.yaml', '--target_device', 'TDA4VM'] settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/TDA4VM_package', 'datasets_path': './dependencies/datasets', 'target_device': 'TDA4VM', 'target_machine': 'evm', 'run_suffix': None, 'parallel_devices': None, 'tensor_bits': 8, 'runtime_options': None, 'run_import': False, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_infer_on_evm.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None} work_dir: ./work_dirs/modelartifacts/TDA4VM_package/8bits INFO:20230607-130458: dataset exists - will reuse - ./dependencies/datasets/imagenetv2c/val INFO:20230607-130459: dataset exists - will reuse - ./dependencies/datasets/coco loading annotations into memory... Done (t=0.76s) creating index... index created! loading annotations into memory... Done (t=0.76s) creating index... index created! INFO:20230607-130500: dataset exists - will reuse - ./dependencies/datasets/ycbv loading annotations into memory... Done (t=0.12s) creating index... index created! loading annotations into memory... Done (t=0.10s) creating index... index created! INFO:20230607-130523: dataset exists - will reuse - ./dependencies/datasets/coco loading annotations into memory... Done (t=2.36s) creating index... index created! loading annotations into memory... Done (t=2.39s) creating index... index created! INFO:20230607-130533: dataset exists - will reuse - ./dependencies/datasets/widerface loading annotations into memory... Done (t=1.55s) creating index... index created! loading annotations into memory... Done (t=1.66s) creating index... index created! loading annotations into memory... Done (t=0.40s) creating index... index created! loading annotations into memory... Done (t=0.38s) creating index... index created! INFO:20230607-130541: dataset exists - will reuse - ./dependencies/datasets/coco loading annotations into memory... Done (t=2.88s) creating index... index created! loading annotations into memory... Done (t=1.59s) creating index... index created! INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016 INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ADEChallengeData2016 INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/VOCdevkit/VOC2012 INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/nyudepthv2 INFO:20230607-130551: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd INFO:20230607-130552: dataset exists - will reuse - ./dependencies/datasets/ti-robokit_semseg_zed1hd download_ok: True configs to run: ['cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite'] number of configs: 1 INFO:20230607-130552: starting - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite INFO:20230607-130555: model_path - /opt/edgeai-modelzoo/models/vision/classification/imagenet1k/mlperf/mobilenet_v1_1.0_224.tflite INFO:20230607-130555: model_file - /opt/mounted-benchmark/work_dirs/modelartifacts/TDA4VM_package/8bits/cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite/model/mobilenet_v1_1.0_224.tflite INFO:20230607-130555: running - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite INFO:20230607-130555: pipeline_config - {'task_type': 'classification', 'dataset_category': 'imagenet', 'calibration_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0xffff5eae7550>, 'input_dataset': <edgeai_benchmark.datasets.imagenetv2.ImageNetV2C object at 0xffff5eae7610>, 'postprocess': <edgeai_benchmark.postprocess.PostProcessTransforms object at 0xffff5eae7640>, 'preprocess': <edgeai_benchmark.preprocess.PreProcessTransforms object at 0xffff5eae76d0>, 'session': <edgeai_benchmark.sessions.tflitert_session.TFLiteRTSession object at 0xffff5eae7880>, 'metric': {'label_offset_pred': -1}, 'model_info': {'metric_reference': {'accuracy_top1%': 71.676}, 'model_shortlist': 10}} INFO:20230607-130555: infer - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - this may take some time... Number of subgraphs:1 , 34 nodes delegated out of 34 nodes APP: Init ... !!! MEM: Init ... !!! MEM: Initialized DMA HEAP (fd=6) !!! MEM: Init ... Done !!! IPC: Init ... !!! IPC: Init ... Done !!! REMOTE_SERVICE: Init ... !!! REMOTE_SERVICE: Init ... Done !!! 21226.775752 s: GTC Frequency = 200 MHz APP: Init ... Done !!! 21226.787708 s: VX_ZONE_INIT:Enabled 21226.787741 s: VX_ZONE_ERROR:Enabled 21226.788333 s: VX_ZONE_WARNING:Enabled 21226.789310 s: VX_ZONE_INIT:[tivxInitLocal:130] Initialization Done !!! 21226.789824 s: VX_ZONE_INIT:[tivxHostInitLocal:93] Initialization Done for HOST !!! infer : cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_| 100%|##########|| 5/5 [00:00<00:00, 19.63it/s] INFO:20230607-130557: infer completed - cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite - 1 sec SUCCESS:20230607-130557: benchmark results - {'infer_path': 'cl-0000_tflitert_imagenet1k_mlperf_mobilenet_v1_1.0_224_tflite', 'accuracy_top1%': 80.0, 'num_subgraphs': 1, 'infer_time_core_ms': 1.965482, 'infer_time_subgraph_ms': 1.950902, 'ddr_transfer_mb': 6.163737, 'perfsim_time_ms': 0.0, 'perfsim_ddr_transfer_mb': 0.0, 'perfsim_gmacs': 0.0} 21227.153310 s: VX_ZONE_INIT:[tivxHostDeInitLocal:107] De-Initialization Done for HOST !!! 21227.155829 s: VX_ZONE_INIT:[tivxDeInitLocal:193] De-Initialization Done !!! APP: Deinit ... !!! REMOTE_SERVICE: Deinit ... !!! REMOTE_SERVICE: Deinit ... Done !!! IPC: Deinit ... !!! IPC: DeInit ... Done !!! MEM: Deinit ... !!! DDR_SHARED_MEM: Alloc's: 7 alloc's of 5358322 bytes DDR_SHARED_MEM: Free's : 7 free's of 5358322 bytes DDR_SHARED_MEM: Open's : 0 allocs of 0 bytes DDR_SHARED_MEM: Total size: 536870912 bytes MEM: Deinit ... Done !!! APP: Deinit ... Done !!! ------------------------------------------------------------------- =================================================================== settings: {'include_files': None, 'pipeline_type': 'accuracy', 'num_frames': 5, 'calibration_frames': 5, 'calibration_iterations': 5, 'configs_path': './configs', 'models_path': '../edgeai-modelzoo/models', 'modelartifacts_path': './work_dirs/modelartifacts/_package', 'datasets_path': './dependencies/datasets', 'target_device': None, 'target_machine': 'evm', 'run_suffix': None, 'parallel_devices': None, 'tensor_bits': 8, 'runtime_options': None, 'run_import': False, 'run_inference': True, 'run_missing': True, 'detection_threshold': 0.3, 'detection_top_k': 200, 'detection_nms_threshold': None, 'detection_keep_top_k': None, 'save_output': False, 'num_output_frames': 50, 'model_selection': 'cl-0000', 'model_shortlist': None, 'model_exclusion': None, 'task_selection': None, 'runtime_selection': None, 'session_type_dict': {'onnx': 'onnxrt', 'tflite': 'tflitert', 'mxnet': 'tvmdlr'}, 'dataset_type_dict': {'imagenet': 'imagenetv2c'}, 'dataset_selection': None, 'dataset_loading': True, 'config_range': None, 'enable_logging': True, 'verbose': False, 'capture_log': False, 'experimental_models': False, 'rewrite_results': False, 'with_udp': True, 'flip_test': False, 'model_transformation_dict': None, 'report_perfsim': False, 'tidl_offload': True, 'input_optimization': None, 'run_dir_tree_depth': None, 'settings_file': 'settings_infer_on_evm.yaml', 'basic_keys': ['include_files', 'pipeline_type', 'num_frames', 'calibration_frames', 'calibration_iterations', 'configs_path', 'models_path', 'modelartifacts_path', 'datasets_path', 'target_device', 'target_machine', 'run_suffix', 'parallel_devices', 'tensor_bits', 'runtime_options', 'run_import', 'run_inference', 'run_missing', 'detection_threshold', 'detection_top_k', 'detection_nms_threshold', 'detection_keep_top_k', 'save_output', 'num_output_frames', 'model_selection', 'model_shortlist', 'model_exclusion', 'task_selection', 'runtime_selection', 'session_type_dict', 'dataset_type_dict', 'dataset_selection', 'dataset_loading', 'config_range', 'enable_logging', 'verbose', 'capture_log', 'experimental_models', 'rewrite_results', 'with_udp', 'flip_test', 'model_transformation_dict', 'report_perfsim', 'tidl_offload', 'input_optimization', 'run_dir_tree_depth', 'settings_file'], 'dataset_cache': None} no results found - no report to generate. Report generated at ./work_dirs/modelartifacts/_package -------------------------------------------------------------------