Installation and setup:
=======================
1. Unzip ut-tool-*.zip into a desired installation directory.
2. On Windows, open the Command Prompt with administrator privileges.
3. Go to the installation directory.
    cd <install-dir>
4. Set up the environment.
    Windows (Command Prompt):
        ut-vars.cmd
    Windows (PowerShell):
        . .\ut-vars.ps1
   	Linux:
        source ut-vars.sh
5. (Linux only) Run the following commands to install the EMON driver.
   On Windows, UT automatically manages the driver installation and uninstallation for each session.
        cd <install-dir>/bin/collectors/emon/sepdk/src
        ./build-driver -ni
        sudo ./insmod-sep -g <driver-access-group> (<driver-access-group> should be a group that the user has access to)
6. Make sure the relevant GPU and/or NPU drivers are installed on the system. Both device trace profiling and metric collection depend on driver availability - without the correct driver, collection will fail.

   Device Driver Downloads:
        Intel NPU: https://www.intel.com/content/www/us/en/download/794734/intel-npu-driver-windows.html
        Intel Arc GPU: https://www.intel.com/content/www/us/en/download/785597/intel-arc-graphics-windows.html

Help menu:
==========
ut -h | --help
    Usage: ut [options]
    Options:
    -o, --output <file>              Output file name for the collected data (incompatible with -c)
    -l, --list-features              Print a list of available features
    --config-level <level>           Overhead configuration level for all collectors (low|medium|high, default: low)
    -p, --pwr-config <file>          User-defined JSON configuration file path for power collector
                                     (default: bin/configs/pwr-low.json, changes with --config-level)
    -z, --ze-config <file>           User-defined JSON configuration file path for GPU/NPU collector
                                     (default: bin/configs/ze-low.json, changes with --config-level)
    -e, --emon-config <file>         User-defined JSON configuration file path for EMON collector
                                     (default: bin/configs/emon-low.json, changes with --config-level)
    -w, --etw-config <file>          User-defined JSON configuration file path for ETW collector
                                     (default: bin/configs/etw-low.json, changes with --config-level, Windows only)
    --xpu-os-config <file>           User-defined JSON configuration file path for XPU OS collector
                                     (default: bin/configs/xpu-os-low.json, changes with --config-level, Windows only)
    -k, --vk-config <file>           User-defined JSON configuration file path for Vulkan collector
                                     (default: bin/configs/vk.json, Windows only, if built with Vulkan support)
    -f, --fp-config <file>           Enable function profiler with config file (if built with function profiler support)
    --mem-trace                      Enable memory trace (capture malloc/free args, if built with function profiler support)
    -t, --time-duration <sec>        Collection duration in seconds (default: 1)
    --enable <list>                  Comma-separated list of components to enable.
                                     Overrides all other component flags.
                                     Collectors: socwatch,level-zero,emon,etw (Windows),xpu-os (Windows)
                                     Tracers: itt,xpu-trace,vulkan*,func-profiler*
                                     (* = available only when built with corresponding feature enabled)
    --enable-all                     Enable all collectors and tracers.
    -a, --app "<appcommand>"         Application to be launched
    -s, --sampling-interval <ms>     Sampling interval in milliseconds for all collectors
                                     (overrides config file values, continuous mode only)
    -c, --continuous                 Collect in continuous mode (incompatible with -o)
    -v, --version                    Print version information and exit

Selective Collection with --enable:
====================================
Use --enable to specify exactly which collectors and tracers to activate:

    Collect only EMON metrics:
        ut -o data/d1 --enable emon -a "<appcommand>"

    Collect only power and CPU metrics with ITT tracing:
        ut -o data/d1 --enable socwatch,emon,itt -a "<appcommand>"

    Collect with device (GPU/NPU) tracing only:
        ut -o data/d1 --enable level-zero,xpu-trace -a "<appcommand>"

    Enable ETW collection on Windows:
        ut.exe -o data\d1 --enable socwatch,emon,etw -a "<appcommand>"

    Enable all collectors and tracers:
        ut -o data/d1 --enable-all -a "<appcommand>"

When --enable is specified, only the listed components are active.
When --enable is not specified, all available components are enabled by default.
Use --enable-all as a shorthand to explicitly enable every available collector and tracer.

Run Collection:
===============
1. Make sure the environment is set up (see Step 4 in "Installation and setup").
2. Launch your application with UT to collect performance data until the application exits.

    Windows:
        ut.exe -o data\d1 -a "path\to\your\application.exe <args>"
    Linux:
        ut -o data/d1 -a "path/to/your/application <args>"

   Example with OpenVINO benchmark_app:

   Note: ITT instrumentation collection requires OpenVINO 2026.1 or later, which includes ITT hooks for NPU, GPU, and CPU.
         Download from: https://github.com/openvinotoolkit/openvino/releases/tag/2026.1.0

    Windows:
        ut.exe -o data\d1 -a "<openvino-install-dir>\tools\benchmark_app.exe -m <fullmodelpath> -d CPU"
    Linux:
        ut -o data/d1 -a "<openvino-install-dir>/tools/benchmark_app -m <fullmodelpath> -d CPU"
   This creates the following binary data files in the "data" folder after the collection completes (note the 'd1' filename prefix specified via '-o').

   Output Files:
    d1.ut.main.bin       Session metadata (application info, timestamps)
    d1.emon.bin          CPU hardware performance counters
    d1.itt.bin           Instrumentation regions (model execution, user-defined markers)
    d1.l0_gpu.bin        GPU metrics
    d1.l0_npu.bin        NPU metrics
    d1.socwatch.bin      Power and thermal metrics
    d1.xpu-os.bin        Per-process GPU/NPU utilization (Windows only)
    d1.l0_graph.bin      Graph profiling data
    d1.l0_trace.bin      Level Zero API tracing data
    d1.pti_host.bin      Host-side API call traces with timestamps (GPU/NPU)
    d1.gpu_device.bin    GPU device-side execution traces (kernel/command durations)
    d1.npu_host.bin      NPU host-side API call traces with timestamps
    d1.npu_device.bin    NPU device-side execution traces (layer/task durations)

   Note: If all required drivers and collectors are available, you will see a subset of these files depending on which collectors and tracers are enabled in your configuration. You can pass any subset of these files to bin2perfetto depending on which data you need in the timeline.

Process Data:
=============
1. Make sure the environment is set up (see Step 4 in "Installation and setup").
2. Stitch the data files into a single Perfetto JSON file.
    Windows:
        bin2perfetto.exe -i data\d1* -o data\d1
    Linux:
        bin2perfetto -i data/d1* -o data/d1
   This creates the d1.json file in the "data" folder. This process may take a few minutes depending on the data file sizes.
3. Open the d1.json file in Perfetto (https://ui.perfetto.dev). You should see various CPU, GPU, NPU, and model execution metrics in the timeline.

bin2perfetto Options:
    -i, --input <file>          Input .bin file(s) — comma-separated list or wildcard (required)
    -f, --format <type>         Output format: json (default), perfetto, console
    -o, --output <file>         Output file name (default: trace_output)

   Output formats:
    json     — Chrome Trace JSON, viewable in chrome://tracing or Perfetto UI. Automatically generates flow events linking host API calls to device executions and ITT regions to child tasks.
    perfetto — Perfetto trace format for https://ui.perfetto.dev. Supports track separation and annotations.
    console  — Human-readable text output for quick inspection.

Device Tracing (GPU/NPU):
==========================
To collect device-side profiling data, use the --enable flag to include xpu-trace:

    Windows:
        ut.exe -o data\d1 --enable socwatch,level-zero,emon,itt,xpu-trace -a "<appcommand>"
    Linux:
        ut -o data/d1 --enable socwatch,level-zero,emon,itt,xpu-trace -a "<appcommand>"

Note: xpu-trace captures both host API calls and device-side execution on GPU and NPU. This generates additional output files as shown in the output files table above.

NPU Device Tracing Requirements:
---------------------------------
CRITICAL: NPU device tracing requires the model to be compiled with profiling instrumentation enabled.
If you use a pre-compiled model blob without profiling instrumentation, you will not get performance data.

For OpenVINO applications:

Enable profiling instrumentation using one of the following methods:

    Option 1: ov::enable_profiling property - Set this property to true when compiling the model:
        ov::CompiledModel compiled_model = core.compile_model(model, "NPU", ov::enable_profiling(true));

    Option 2: -pc flag (benchmark_app only) - Pass -pc to enable per-layer performance counters:
        benchmark_app -m <model> -d NPU -pc

For ONNX Runtime / WinML applications:

To enable NPU device events, you must set the PERF_COUNT provider option to YES before compiling the model.

    Option 1: Set the option directly in code

    Embed the configuration as a JSON string and pass it to the execution provider via the load_config option:

        const char* config_json = R"({"NPU":{"PERF_COUNT":"YES"}})";
        Ort::KeyValuePairs ep_options;
        ep_options.Add("load_config", config_json);

    Use this approach when the configuration is stable and you don't need to change it without rebuilding.

    Option 2: Point to a JSON config file (recommended)

    Store the settings in a config file on disk and pass the file path to the load_config option. The execution provider will read the file at session creation and apply the settings.

    Application code:
        Ort::KeyValuePairs ep_options;
        ep_options.Add("load_config", "path/to/config.json");

    Example config.json:
        {
          "NPU": {
            "PERF_COUNT": "YES"
          }
        }

    Advantage: Option 2 allows you to ship the application once and adjust settings later - for example, toggling performance counters on or off across runs without recompiling.

For additional NPU execution provider options, see:
https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/intel_npu/README.md

Continuous Mode Collection:
===========================
Continuous mode (-c flag) streams collected data via callbacks rather than writing to a single output file.
This mode is useful for long-running or real-time monitoring scenarios.

Note: Only the SocWatch (power) and XPU OS (Windows) collectors produce data in continuous mode.
      Continuous mode (-c) cannot be used with the -o output flag.

    Windows:
        ut.exe -c -t 30 -a "myapp.exe"
    Linux:
        ut -c -t 30 -a "myapp.exe"

Sampling Interval Override:
---------------------------
Use -s, --sampling-interval <ms> to override the sampling interval for all collectors.
This flag is only available in continuous mode.

        ut.exe -c -s 500 -t 60 -a "myapp.exe"

SocWatch Continuous Configs:
----------------------------
In continuous mode, SocWatch automatically uses pwr-continuous-{level}.json config files
instead of the standard pwr-{level}.json files. The --config-level flag determines which
variant is selected (e.g., pwr-continuous-low.json for --config-level low).

The XPU OS collector uses the same xpu-os-{level}.json config files for both snapshot and continuous modes.

Metric Availability:
====================
The metric details associated with the EMON, SoCWatch, and XPU OS collectors are available in the following locations.
    EMON: docs\README-EMON.md
    SoCWatch: docs\README-SOCWATCH.md
    XPU OS: See sample_configs\package\README.md (Windows only)

FAQ:
====
1. Why am I seeing the following error?
   Failed to configure socwatch collector
   [ERROR]: Unable to launch application (error 740)
   [EMON][ERR] Failed to install EMON driver (exit code: 23)
   Failed to configure emon collector

   Solution: Make sure the Command Prompt is opened with administrator privileges.

2. Why am I seeing a Level Zero collector warning about metric timestamps?
   [l0][ERROR] Known driver issue: metric timestamps returned in incorrect units - automatically correcting - trace timestamps may be inaccurate

   or:

   [l0][ERROR] Known driver issue: unable to confirm metric timestamp units - trace timestamps may be inaccurate.

   Solution: This is a known GPU driver bug where the Level Zero API returns metric timestamps in incorrect units. UT attempts to detect and correct this automatically, but the correction is best-effort - trace timestamps may still be inaccurate, and L0 metric data may not align correctly with other collectors in the timeline. The collected metric values themselves are correct. If automatic correction is not happening, exclude l0_gpu.bin when converting to JSON format using bin2perfetto.

3. How do I manually manage the EMON driver installation and uninstallation on Windows?

   By default, UT automatically manages the EMON driver installation and uninstallation ('autoManageDriver' in bin\configs\emon-<low|medium|high>.json is set to true).
   To manage the driver manually, set 'autoManageDriver' to false and use the following commands:

   To install the EMON driver (required only for the first time on a system), run:
        <install-dir>\bin\collectors\emon\bin64\sepreg -i

   To uninstall the EMON driver, run:
        <install-dir>\bin\collectors\emon\bin64\sepreg -u pax

4. Which configuration files does UT use?

   The configuration file used depends on the collector, the mode (snapshot vs. continuous),
   and the --config-level setting (default: low).

   Snapshot mode (default):
       - SocWatch (Power): -p flag, uses pwr-{low|medium|high}.json (All platforms)
       - EMON (CPU): -e flag, uses emon-{low|medium|high}.json (All platforms)
       - Level Zero (GPU/NPU): -z flag, uses ze-{low|medium|high}.json (All platforms)
       - ETW: -w flag, uses etw-{low|medium|high}.json (Windows only)
       - XPU OS: --xpu-os-config flag, uses xpu-os-{low|medium|high}.json (Windows only)
       - Vulkan: -k flag, uses vk.json (Windows only)

   Continuous mode (-c flag): Only SocWatch and XPU OS produce data in this mode.
       - SocWatch (Power): -p flag, uses pwr-continuous-{low|medium|high}.json (All platforms)
       - XPU OS: --xpu-os-config flag, uses xpu-os-{low|medium|high}.json (Windows only)

   Key notes:
       - --config-level determines which {low|medium|high} variant is used.
       - SocWatch is the only collector with separate continuous config files.
       - -s overrides sampling intervals across all collector configs (continuous mode only).
       - XPU OS is enabled by default on Windows; disable by using --enable without xpu-os.

   XPU OS config level details:
       - low: 2000 ms sampling interval, 3 metrics (3D, Compute, Neural)
       - medium: 1000 ms sampling interval, 3 metrics (3D, Compute, Neural)
       - high: 500 ms sampling interval, 5 metrics (3D, Compute, Copy, Video, Neural)
