================================================================================
Copyright (C) 2021 Intel Corporation

This software and the related  documents are Intel  copyrighted  materials,  and
your use  of them  is governed  by the  express  license under  which  they were
provided to you (License).  Unless the  License provides otherwise,  you may not
use,  modify, copy,  publish, distribute,  disclose or transmit this software or
the related documents without Intel's prior written permission.

This software and the  related documents are provided as is,  with no express or
implied warranties, other than those that are expressly stated in the License.
================================================================================

================================================================================

 Intel(R) oneMKL Examples
================================================================================

---------------------
 CMake-related Notes
---------------------
* The minimum supported CMake version is 3.13.
* CMAKE_BUILD_TYPE option is locked in "Release" mode for CMake versions below 3.25.2 due to lack of compiler-cmake integration.
* Use CMake version 3.25.2 or newer for full native support of Intel(R) oneAPI DPC++/C++ Compiler.

-------------------
 Examples Contents
-------------------

* c            - examples for the oneMKL C API
* c_mpi        - examples for the oneMKL C Cluster API
* c_offload    - examples for the oneMKL C API with OpenMP Offload
* f            - examples for the oneMKL Fortran API
* f95          - examples for the oneMKL Fortran 95 API
* f_mpi        - examples for the oneMKL Fortran Cluster API
* f_offload    - examples for the oneMKL Fortran API with OpenMP Offload
* sycl         - examples for the oneMKL SYCL API
* sycl_device  - examples for the oneMKL SYCL Device API
* sycl_mpi     - examples for the oneMKL SYCL MPI API (Linux-only)
* cmake        - CMake scripts
* <MKLROOT>/lib/cmake/mkl/MKLConfig.cmake
               - CMake Config-Mode file for oneMKL
* <MKLROOT>/lib/cmake/mkl/MKLConfigVersion.cmake
               - CMake Config-Mode Version file for oneMKL(helps find the right package version)

NOTE: this list could be limited to what is supported on current OS

-------------------------------
 How to build and run examples
-------------------------------

NOTE: CMake Config-Mode file for oneMKL by default defines MKLROOT as '../../../' 
    and assumes the package structure is '<MKLROOT>/lib/cmake/mkl/MKLConfig.cmake', 
    in case of different package structure $MKLROOT environment variable should be 
    defined by user manually or via oneMKL environment script.

1. Choose the oneMKL API type and change directory to the selected oneMKL API type.

    Example: $> cd $MKLROOT/share/doc/mkl/examples/c

2. Create the build folder and change directory to it.

    Example: $> mkdir build && cd build

3. Call CMake to configure the build and customize build options if needed (see
"Examples build options" below).

    Example: $> cmake .. -G "Ninja" -DCMAKE_Fortran_COMPILER=ifx -DTARGET_DOMAINS=blas

4. Call CMake to build the configured project.
Optional: use the CMake `-j` option to specify the number of parallel jobs.
Optional: use the CMake `--verbose` option to see compilation/link lines for each example.

    Example: $> cmake --build . -j 24 --verbose

5. Run examples using the CTest tool.
Optional: use the CTest `--verbose` option to see examples output.

    Example: $> ctest --verbose

------------------------
 Examples build options
------------------------
CMake Generator option `-G "<value>"`
Supported CMake Generators on Linux   : Ninja, Unix Makefiles
Supported CMake Generators on Windows : Ninja, NMake Makefiles
NOTE: The default Windows generator is NOT supported. The Windows CMake Generator option must be provided.

All build options should be defined as `-D<option>=<value>` at the CMake configuration
step (see #3 in "How to build and run examples" above).

+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| Option                   | Description             | Supported                       | Default      | MKLConfig    |
|                          |                         | values                          | value        | option       |
+==========================+=========================+=================================+==============+==============+
| CMAKE_C_COMPILER         | Define C compiler       | lnx: icc gcc clang icx          | icx          | No           |
|                          |                         | win: icl cl icx                 |              |              |
|                          |                         | C MPI wrappers: mpicc or mpi*   |              |              |
|                          +-------------------------+---------------------------------+--------------+--------------+
|                          | OpenMP Offload API      | icx                             | icx          | No           |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| CMAKE_Fortran_COMPILER   | Define Fortran compiler | lnx: ifort ifx gfortran         | ifx          | No           |
|                          |                         | win: ifort ifx                  |              |              |
|                          |                         | Fortran MPI wrapper mpiifort    |              |              |
|                          +-------------------------+---------------------------------+--------------+              |
|                          | OpenMP Offload API      | ifx                             | ifx          |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| CMAKE_CXX_COMPILER       | Define C++ compiler     | lnx: icpx                       | lnx: icpx    | No           |
|                          | (SYCL examples only)    | win: icx                        | win: icx     |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_DIR                  | Specify path to         | Valid full path to              | Search       | No           |
|                          | MKLConfig.cmake         | MKLConfig.cmake                 | Paths**      |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_LINK                 | Define oneMKL link type | static dynamic sdl              | dynamic      | Yes          |
|                          |                         | Note: oneMKL SYCL static library|              |              |
|                          |                         |       is deprecated and will be |              |              |
|                          |                         |       removed in oneMKL 2026.0  |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_INTERFACE            | Define Integer size for | ilp64 lp64                      | ilp64        | Yes          |
|                          | C/Fortran API           |                                 |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_THREADING            | Define oneMKL threading | sequential tbb_thread           | intel_thread | Yes          |
|                          | for C/Fortran API       | intel_thread gnu_thread         |              |              |
|                          +-------------------------+---------------------------------+--------------+              |
|                          | OpenMP Offload API      | sequential tbb_thread           | intel_thread |              |
|                          |                         | intel_thread                    |              |              |
|                          +-------------------------+---------------------------------+--------------+              |
|                          | SYCL APIs               | sequential tbb_thread           | tbb_thread   |              |
|                          |                         | intel_thread                    |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| MKL_MPI                  | Define MPI vendor       | lnx: intelmpi openmpi mpich     | intelmpi     | Yes          |
|                          | (Cluster examples only) | win: intelmpi mshpc msmpi       |              |              |
|                          +-------------------------+---------------------------------+--------------+              |
|                          | Define MPI vendor       | lnx: intelmpi                   | intelmpi     |              |
|                          | (SYCL MPI examples only)| win: N/A                        |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| TARGET_DEVICES           | Define list of devices  | cpu gpu                         | cpu;gpu      | No           |
|                          | (SYCL examples only)    |                                 |              |              |
|                          +-------------------------+---------------------------------+--------------+              |
|                          | Define list of devices  | lnx: gpu                        | gpu          |              |
|                          | (SYCL MPI examples only)| win: N/A                        |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| TARGET_DOMAINS           | Define list of domains  | Any domains listed in           | All          | No           |
|                          | to build and run        | <oneMKL API type> folder        |              |              |
|                          |                         | E: TARGET_DOMAINS="blas vml"    |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| TARGET_FUNCTIONS         | Define list of functions| Any function listed in          | All          | No           |
|                          | to build and run as     | <domain>/<domain>.lst           |              |              |
|                          | <domain>/<function>     | E: TARGET_FUNCTIONS="vml/vdsin" |              |              |
|                          | TARGET_DOMAINS  is      |                                 |              |              |
|                          | ignored.                | The regex used is the CMake     |              |              |
|                          | Regex expressions* can  | standard.                       |              |              |
|                          | be used to identify     | E: TARGET_FUNCTIONS="vml/vd.*"  |              |              |
|                          | specific sets of        |   will run all vml examples     |              |              |
|                          | functions to run.       |    starting with "vd"           |              |              |
|                          |                         |    using wildcard ".*"          |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| TARGET_OFFLOAD_PRECISION | Define list of examples | sp (for single precision)       | All          | No           |
|                          | to build and run based  | dp (for double precision)       |              |              |
|                          | on precision. Applies   |                                 |              |              |
|                          | only to c/f_offload     |                                 |              |              |
|                          | examples and to domains |                                 |              |              |
|                          | that have separate lists|                                 |              |              |
|                          | <domain>_[sp|dp].lst    |                                 |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+
| FAIL_ON_MISSING_DEVICES  | Enable failing of test  | yes no                          | No           | No           |
|                          | if defined device in    |                                 |              |              |
|                          | TARGET_DEVICES variable |                                 |              |              |
|                          | cannot be found while   |                                 |              |              |
|                          | examples are running.   |                                 |              |              |
+--------------------------+-------------------------+---------------------------------+--------------+--------------+

* See https://cmake.org/cmake/help/latest/command/string.html#regex-specification for the regex 
specification
** Note: https://cmake.org/cmake/help/latest/command/find_package.html#id5
