Module development

Introduction

System/hardware modules are responsible for analysing a workflow SDFG in the context of a specific component or its part in a system graph. They describe how the component should be modelled/profiled and can provide optional information of interest to a user such as profiling results.

SDFG documentation

SDFGs are explained in this page of the DaCe documentation.

Currently, all outputs produced by modules should be saved to disk using the Adaptyst module API explained in the following section. These outputs can be loaded later in Adaptyst Analyser using its module API described in the Adaptyst Analyser implementation section. The same API allows for developing a custom UI for inspecting the outputs.

In the future Adaptyst versions, as indicated in the roadmap, the modules will also play a significant role in the software-hardware co-design goal of the tool, i.e. determining how well an SDFG or part of it matches a specific system component or its part (e.g. whether a specific code segment is more suitable for a CPU or a GPU). The documentation will be updated accordingly as the R&D work on this progresses.

Compatibility of changes to modules due to R&D work

The changes planned to be made to modules as part of the R&D work explained above will be backward compatible.

Adaptyst implementation

Diagram
Diagram explaining how modules work technically.

As shown in the figure above, all modules are implemented in form of a shared library linking against libadaptyst.so containing the Adaptyst module API. The C interface is used. In turn, libadaptyst.so loads modules dynamically through dlopen(), using the lib<MODULE NAME>.so file located inside the <MODULE NAME> folder in the /opt/adaptyst/modules module path (e.g. in case of linuxperf, the full shared library path is /opt/adaptyst/modules/linuxperf/liblinuxperf.so by default).

Defining installation of modules for Adaptyst

There is no pre-defined standard for installing modules for Adaptyst, so you are expected to provide users with installation instructions of your module.

Note that this is not the case for Adaptyst Analyser, see the Adaptyst Analyser implementation section.

Module path changes

The /opt/adaptyst/modules path can be changed by a user, either during Adaptyst compilation or through the ADAPTYST_MODULE_DIR runtime environment variable.

Therefore, it is crucial to:

  • allow the user to change the installation directory of your module or at least not hardcode that directory (you can e.g. use the ADAPTYST_MODULE_PATH variable in CMake when importing the Adaptyst CMake target, see below),
  • check the value of ADAPTYST_MODULE_DIR at all points where you refer to the module path or make all variables dependent on the module path changeable by the user through module options.

When Adaptyst is installed, libadaptyst.so and the header files are copied to the usual library and include directories unless specified otherwise. At the same time, the adaptyst::adaptyst CMake target for libadaptyst.so and the header files is created automatically. To use it, put e.g. these two lines in your CMakeLists.txt file:

find_package(adaptyst REQUIRED)
target_link_libraries(example PUBLIC adaptyst::adaptyst)

The CMake target also makes available the ADAPTYST_MODULE_PATH variable storing the module path (i.e. /opt/adaptyst/modules by default). One example of its usage is the following:

find_package(adaptyst REQUIRED)
set(INSTALL_DIR "${ADAPTYST_MODULE_PATH}/my_module")

The shared library must implement the following C functions:

// These three lines are REQUIRED.
#define ADAPTYST_MODULE_ENTRYPOINT
#include <adaptyst/hw.h>
amod_t module_id = 0;

// Called once when a module instance is initialised by
// Adaptyst (there may be several module instances within
// an entity up to the limit defined in max_count_per_entity).
//
// The return value indicates whether the initialisation
// has succeeded.
bool adaptyst_module_init() {
    return true;
}

// Called when a module instance is expected to process
// a workflow SDFG stored in a serialised format in the
// "sdfg" parameter. It is up to the module to decide
// what parts of the SDFG to process. Adaptyst module
// API methods should be used for saving module outputs.
//
// This function is guaranteed to be called after
// adaptyst_module_init().
//
// The return value indicates whether the module
// has successfully processed the SDFG.
bool adaptyst_module_process(const char *sdfg) {
    return true;
}

// Called once when a module instance is finalised/closed
// by Adaptyst.
//
// This function is guaranteed to be called last.
void adaptyst_module_close() {
    
}

Serialisation of SDFGs

Serialised SDFGs are in JSON and the serialisation is described in this page of the DaCe documentation. You are welcome to make your own experiments with DaCe to see what the JSON format of SDFGs is like.

Additionally, it must define the following constants/variables in the global scope:

// The human-friendly name of a module.
volatile const char *name = "MyModule";

// The human-friendly version of a module.
volatile const char *version = "1.0.0";

// The non-negative version numbers of a module, with
// the last element being negative (-1 is standard here).
//
// Note that the last element is always ignored, it is used only
// for indicating the end of the array.
//
// At least one non-negative number must be defined.
//
// Version comparisons are done lexicographically,
// e.g. { 3, <anything> } > { 2, <anything> },
// { 3, 2, <anything> } > { 3, 1, <anything> },
// { 11, 2, -1 } > { 11, -1 },
// { 12, 5, 3, -1 } > { 12, 4, 88, -1 } etc.
//
// Please bear in mind that {<xyz>, 0, ..., 0, -1} > {<xyz>, -1},
// where <xyz> is any combination of non-negative numbers and
// the number of zeroes on the left-hand side is anything greater
// than or equal to 1.
volatile const int version_nums[] = { 1, 0, 0, -1 };

// The maximum number of instances of a module that can be
// spawned in an entity. 0 means unlimited.
//
// This constant is optional, the default value is 0.
volatile const unsigned int max_count_per_entity = 0;

volatile const char *options[] = {
    // The names of all user-settable options provided by
    // a module, with the last element being NULL.
    NULL
};

volatile const char *tags[] = {
    // The module tags that will be attached
    // automatically to a node, with the last element
    // being NULL.
    //
    // Tags can be used for a variety of reasons,
    // e.g. to verify whether a GPU module is attached
    // to a node that is connected with a CPU node.
    //
    // All tags used by a module are public-facing
    // and thus must be visibly documented to the outside
    // world, especially if a module is closed-source.
    NULL
};

volatile const char *log_types[] = {
    // The names of all types of logs that a module
    // can produce, with the last element being NULL.
    NULL
};

// Each option XYZ can accept either a single or an array value, NOT both.

// *** For each option XYZ in the "options" array with a single value: ***

// Help message, required.
volatile const char *XYZ_help = "XYZ help message";

// Option value type, required.
volatile const option_type XYZ_type = UNSIGNED_INT;

// Default option value, optional (the option becomes
// required to be set by a user if this is not provided).
//
// The type of this constant is determined by the value
// of XYZ_type. See the Doxygen documentation of
// option_type for matchings between option_type values
// and C types.
volatile const unsigned int XYZ_default = 1;

// *** For each option XYZ in the "options" array with an array value: ***

// Help message, required.
volatile const char *XYZ_help = "XYZ help message";

// Value type of every option array element, required.
volatile const option_type XYZ_array_type = UNSIGNED_INT;

// Default option value, optional (the option becomes
// required to be set by a user if this is not provided).
//
// The type of this array constant is determined by the value
// of XYZ_array_type. See the Doxygen documentation of
// option_type for matchings between option_type values
// and C types.
volatile const unsigned int XYZ_array_default[] = {};

// Number of elements in the default option value, required
// if XYZ_array_default is set.
volatile const unsigned int XYZ_array_default_size = 0;

You are very likely to use at least some Adaptyst module API functions in your module: please consult the Doxygen documentation of the API here. All essential Adaptyst module API functions compatible with C can be found in the adaptyst/*.h files. Similarly, extra Adaptyst module API features compatible with C++20 (not C) can be found in the adaptyst/*.hpp files.

Technically speaking, modules are used in the Adaptyst flow in one of the two ways:

  • When workflow execution is Adaptyst-handled and a module indicates that it will profile a workflow:
    1. Adaptyst calls adaptyst_module_init() in the module. Then, in the module, a call to the adaptyst_set_will_profile() API method is made to indicate that the module will profile the workflow.
    2. Adaptyst compiles an SDFG to an executable.
    3. Adaptyst starts the executable and makes it wait for all profiling modules in all entities to send a “ready” notification.
    4. Adaptyst calls adaptyst_module_process() in the module in a separate thread.
    5. The module notifies Adaptyst through the adaptyst_profile_notify() API call that it is ready for profiling.
    6. The executable resumes its work as soon as it receives a “ready” notification from all profiling modules in all entities.
    7. The executable finishes its work and the module finishes its processing.
    8. Adaptyst calls adaptyst_module_close() in the module.
  • In all other cases:
    1. Adaptyst calls adaptyst_module_init() in the module.
    2. Adaptyst calls adaptyst_module_process() in the module in a separate thread.
    3. The module finishes its processing.
    4. Adaptyst calls adaptyst_module_close() in the module.

adaptyst_module_process() is always called in all modules within an entity in separate threads, each with its own copy of an SDFG.

Adaptyst Analyser implementation

In contrary to the Adaptyst part of a module, the Adaptyst Analyser part has a pre-defined directory structure and is installable by adaptyst-analyser:

| <root directory>
   | python
      | <module name>
         | __init__.py
         | ...
         | (Python files)
         | ...
   | web
      | <module name>
         | backend.js
         | backend.css
         | settings.html
         | settings.css
         | deps
            | ...
            | (extra .js/.cjs/.css files to be loaded by Adaptyst Analyser)
            | ...
   | metadata.yml

Because Adaptyst Analyser is a Flask app, the Adaptyst Analyser part of a module consists of two components: a server-side one in Python and a client-side one in HTML/CSS/JavaScript/jQuery. The server-side files are stored in python/<module name> and the client-side files are stored in web/<module name> (the module name must be the same in both cases). The general information about a module is stored in metadata.yml.

Adaptyst Analyser JavaScript documentation

The remainder of this section assumes that you have looked at the Adaptyst Analyser JavaScript documentation.

The client-side component must have these files:

  • backend.js: a JavaScript file implementing at least the following (you can use jQuery, elements from settings.html are available and can be used):
// Called when a user double-clicks a node in a system graph
// and asks to open a specific module. This function should
// create a new window representing the starting point
// for exploring outputs produced by the module.
//
// Parameters:
// * entity_id (String): the ID of an entity of a node double-clicked
//   by a user.
// * node_id (String): the ID of a node double-clicked by a user.
// * session (Object): a Session object corresponding to the
//   currently-selected performance analysis session.
function createRootWindow(entity_id, node_id, session) {
    
}

export { createRootWindow };
  • backend.css: a CSS stylesheet of HTML elements produced by backend.js. If you want to apply CSS to parts of a window created by your module, you may want to use one of the class names indicated by the diagram below, where <type> is the return value of getType() in your JavaScript Window-inheriting class.
.<type>_content for window contents, .<type>_window for a whole window
Class names for a window in Adaptyst Analyser.
  • settings.html: an HTML file implementing a settings window of a module. JavaScript functions from backend.js cannot be used. Settings widgets to be used by backend.js must have IDs. It is highly recommended that the ID of every element is prefixed by the name of a module, e.g. linuxperf_option123. Otherwise, there may be conflicts between the IDs of elements from two different modules, leading to an unexpected behaviour.
  • settings.css: a CSS stylesheet of the settings window. If you want to apply CSS to the content of the window, you may want to use #<module name>_settings_block.

Other files can be added if used by the above.

Important notes

  • When defining HTML in backend.js, do not use IDs: use classes instead to distinguish elements of your layout. This is because multiple windows with the same layout can be opened at the same time by a user.
  • When defining CSS in backend.css, nest your classes in .<type>_content or .<type>_window, do not put them in the global scope. Otherwise, there may be conflicts between the class names of elements from two different modules or from a module and the Adaptyst Analyser core, leading to an unexpected behaviour.

The client-side component may also have the “deps” folder containing any extra .js, .cjs, and .css files to be loaded by Adaptyst Analyser (this is meant for JavaScript dependencies along with their CSS stylesheets if any). If you use it, please define all filenames expected in the directory in metadata.yml: see the metadata.yml part of the documentation later. Also, if you don’t want to store the files locally in “deps”, you can specify their HTTP(S) URLs in metadata.yml so that Adaptyst Analyser downloads them automatically when installing a module: again, see the metadata.yml documentation for more details.

Dependency conflicts

Both files stored in “deps” and files downloaded from metadata.yml URLs are installed into the same directory in Adaptyst Analyser.

If there are any name conflicts, a user will be given a choice of file version to keep when installing a module. It is not possible to have two different versions of the same file stored at the same time because loading them simultaneously in Adaptyst Analyser may lead to an undefined behaviour.

When writing an HTML code, you may find Adaptyst-Analyser-defined classes useful for formatting purposes: check the main stylesheet in the Adaptyst Analyser code. Similarly, you may find the locally-stored selection of Google Material icons useful: you can embed them in your HTML in getContentCode() in backend.js in the following way:

<svg xmlns="http://www.w3.org/2000/svg" data-icon="<icon type>"
 other-attributes>
  Your extra SVG tags, e.g. <title>
</svg>

where <icon type> can be one of:

Icon typeIcon
general
warning
replace
download
delete
copy

If you want more Google Material icons to be embeddable, don’t hesitate to contact us!

For communication between the client-side and server-side components, POST requests sent by a client-side part through the sendRequest() JavaScript function in a Window-inheriting class or Session are used. These are transmitted to process() in the Python code of your module. Therefore, the server-side component must export the following Python functions through __init__.py:

def process(storage, identifier, entity, node, data):
    """
    Process a request sent by the client-side of a module.
    
    The return value is a (code, d) tuple, where "code" is
    an HTTP code to send back to the client and "d" is
    data to send back to the client (e.g. a JSON string).
    
    If your HTTP code for the client is 200 (OK),
    the return value can also be just data to send back to
    the client rather than the tuple above.
    
    :param str storage: The path to the *parent* directory of
                        results of a performance analysis
                        session the request corresponds to.
    :param str identifier: The ID of a performance analysis
                           session the request corresponds to.
    :param str entity: The name of an entity of a node in a
                       system graph the request corresponds to.
    :param str node: The name of a node in a system graph
                     the request corresponds to.
    :param dict data: JSON dictionary sent by the client.
    """

metadata.yml is a YAML file and its structure is as follows:

name: ModuleName
version: 1.0.0
short_desc: Short module description.

# The minimum supported version of the corresponding
# Adaptyst module. If a user tries to inspect data produced
# by an older version of the module, they will get an error.
# 
# Use the same versioning scheme as in version_nums in an
# Adaptyst module shared library (negative numbers shouldn't be
# included).
min_module_version: [1, 0, 0]

# Python dependencies. Use the pip requirements.txt format.
python_dependencies:
  - package1
  - package2
  - package3

# JavaScript dependencies expected to be found in "deps" folder
# along with CSS stylesheets if any, in form of full filenames including
# extensions. Only .js, .cjs, and .css files are accepted.
js_dependencies:
  - script1.js
  - script2.js
  - script2.css

# JavaScript dependencies to be downloaded along with CSS
# stylesheets if any, in form of HTTP(S) URLs (e.g. pointing to CDNs).
# Only .js, .cjs, and .css files are accepted.
js_url_dependencies:
  - https://example.com/example.js
  - https://example.com/example.css

Once you create the first version of the above files, you can use adaptyst-analyser to install the Adaptyst Analyser part of your module in development mode, i.e. with all changes to your code being immediately propagated to the Adaptyst Analyser installation directory so that they can be tested directly in a real environment (similarly to how pip install -e works in Python):

adaptyst-analyser -d <path to the directory with your module>