# Module development<no value>
{{< callout context="caution" title="Adaptyst intermediate representation change" icon="outline/alert-triangle" >}}
*Last update: 25 March 2026*

We are changing the intermediate representation used by Adaptyst from SDFGs to
[MLIR](https://mlir.llvm.org). See the announcement [here](https://adaptyst-blog.web.cern.ch/ir-used-by-adaptyst-is-changing-to-mlir).

The first stage of this switch is already live: DaCe is no longer a dependency
of Adaptyst and the internals of the tool have been rewritten to accommodate the
future MLIR integration while providing higher API stability for module developers.

The single-command-analysis mode currently uses
neither SDFGs nor MLIR: later, there will be also the MLIR version of it. Both
variants will stand on an equal footing and be maintained.

**You are free to continue working on ALL parts of your module once you implement
the IR-abstracting Adaptyst API changes made by us: no more breaking API
changes are foreseen for finalising the MLIR integration.**
{{< /callout >}}

## Introduction
System/hardware modules are responsible for analysing a workflow intermediate representation (IR)
in the context of a specific component or its part in a system graph. They
describe how the component should be modelled/profiled and can provide
optional information of interest to a user such as profiling results.

Currently, all outputs produced by modules should be saved to disk using the
Adaptyst module API explained in the following section. These outputs can be loaded
later in Adaptyst Analyser using its module API described in the
[Adaptyst Analyser implementation](#adaptyst-analyser-implementation)
section. The same API allows for developing a custom UI for inspecting
the outputs.

In the future Adaptyst versions,
the modules will also play a significant
role in the software-hardware co-design goal of the tool, i.e. determining how
well an IR or part of it matches a specific
system component or its part (e.g. whether a specific code segment
is more suitable for a CPU or a GPU). The documentation will be updated
accordingly as the R&D work on this progresses.

{{< callout context="note" title="Compatibility of changes to modules due to R&D work" icon="outline/info-circle" >}}
The changes planned to be made to modules as part of the R&D work explained above will be **backward compatible**.
{{< /callout >}}

{{< callout context="caution" title="API stability" icon="outline/alert-triangle" >}}
We try to keep our APIs as stable as possible already now, but due to the early development stage of Adaptyst, we cannot guarantee the stability yet.

~~However, all breaking changes are announced in advance.~~ If there are any breaking changes in a new update, they are announced at the release time or earlier (**it's unfortunately difficult to anticipate these changes in advance at the moment**). If you use Adaptyst / Adaptyst Analyser APIs, [let us know](/contact): by staying in touch with us, we can ensure that the impact of any API changes on your work is minimised.
{{< /callout >}}

## Adaptyst implementation
{{< figure src="/images/module_flow.png" class="ratio"
    alt="Diagram"
    caption="Diagram explaining how modules work technically." >}}

As shown in the figure above, all modules are implemented
in form of a shared library linking against libadaptyst.so containing
the Adaptyst module API. The C interface is used. In turn, libadaptyst.so
loads modules **dynamically** through ```dlopen()```, using the
```lib<MODULE NAME>.so``` file located inside the ```<MODULE NAME>``` folder in
the ```<user install prefix>/opt/adaptyst/modules``` module path (e.g. in case of linuxperf,
the full shared library path is ```/usr/opt/adaptyst/modules/linuxperf/liblinuxperf.so```
by default if a user has installed Adaptyst into ```/usr```).

{{< callout context="caution" title="Defining installation of modules for Adaptyst" icon="outline/alert-triangle" >}}
There is **no** pre-defined standard for installing modules for Adaptyst,
so you are expected to provide users with installation instructions of your module.

Note that this is **not** the case for Adaptyst Analyser, see the
[Adaptyst Analyser implementation](#adaptyst-analyser-implementation) section.
{{< /callout >}}

{{< callout context="caution" title="Module path changes" icon="outline/alert-triangle" >}}
The ```<user install prefix>/opt/adaptyst/modules``` path can be changed by a user,
either during Adaptyst compilation or through the ```ADAPTYST_MODULE_DIRS``` runtime environment variable.

Therefore, it is crucial to:
* allow the user to change the installation directory of your module **or** at least not
hardcode that directory (you can e.g. use the ```ADAPTYST_MODULE_PATH``` variable in CMake when
importing the Adaptyst CMake target, see below),
* use the ```adaptyst_get_library_dir()``` API method to get the module path in your code (alternatively:
check the value of ```ADAPTYST_MODULE_DIRS``` at all points where you refer to the module path **or**
make all module-path-dependent variables changeable by the user through module options).
{{< /callout >}}

When Adaptyst is installed, libadaptyst.so and the header files are copied to the usual
library and include directories unless specified otherwise. At the same time,
the ```adaptyst::adaptyst``` CMake target for libadaptyst.so and
the header files is created automatically. To use it, put e.g. these two lines in your
CMakeLists.txt file:
```cmake
find_package(adaptyst REQUIRED)
target_link_libraries(example PUBLIC adaptyst::adaptyst)
```

The CMake target also makes available the ```ADAPTYST_MODULE_PATH``` variable storing the
module path (i.e. ```<user install prefix>/opt/adaptyst/modules``` by default). One example of its usage is
the following:
```cmake
find_package(adaptyst REQUIRED)
set(INSTALL_DIR "${ADAPTYST_MODULE_PATH}/my_module")
```

The shared library must implement the following C functions:
```c
// These two lines are REQUIRED.
#define ADAPTYST_MODULE_ENTRYPOINT
#include <adaptyst/hw.h>

// Called once when a module instance is initialised by
// Adaptyst (there may be several module instances within
// an entity up to the limit defined in max_count_per_entity).
//
// Use the value of the "module_id" parameter in all Adaptyst
// API calls requiring an ID of type amod_t.
//
// The return value indicates whether the initialisation
// has succeeded.
bool adaptyst_module_init(amod_t module_id) {
    return true;
}

// Called when a module instance is expected to process
// a workflow IR stored in the "ir" parameter.
// It is up to the module to decide what parts of the IR to
// process. Adaptyst module API methods should be used for
// saving module outputs.
//
// The type of the "ir" parameter is a C struct compatible
// with many different IR types: see the
// Doxygen documentation of adaptyst/hw.h to see how it
// is structured.
//
// Use the value of the "module_id" parameter in all Adaptyst
// API calls requiring an ID of type amod_t.
//
// This function is guaranteed to be called after
// adaptyst_module_init().
//
// The return value indicates whether the module
// has successfully processed the IR.
bool adaptyst_module_process(amod_t module_id, ir workflow) {
    return true;
}

// Called once when a module instance is finalised/closed
// by Adaptyst.
//
// Use the value of the "module_id" parameter in all Adaptyst
// API calls requiring an ID of type amod_t.
//
// This function is guaranteed to be called last.
void adaptyst_module_close(amod_t module_id) {
    
}

// Called when a workflow calls adaptyst_region_start()
// (see "Running Adaptyst" -> "Code regionisation"),
// with the following arguments:
// * module_id: use its value in all Adaptyst API calls requiring
//   an ID of type amod_t.
// * name: the name of the region as passed to adaptyst_region_start().
// * part_id: a string indicating what part of the workflow
//   adaptyst_region_start() has been called in. At the moment,
//   it is always in form of "<PID>_<TID>" where <PID> and <TID>
//   are the process and thread ID respectively.
// * timestamp_str: a string representing the timestamp of the
//   adaptyst_region_start() call in nanoseconds, it's "-1" if
//   undefined/unknown.
//
// The return value indicates whether the module has processed
// the region start successfully.
bool adaptyst_region_start(amod_t module_id, const char *name,
                           const char *part_id, const char *timestamp_str) {
                           
}

// Called when a workflow calls adaptyst_region_end()
// (see "Running Adaptyst" -> "Code regionisation"),
// with the following arguments:
// * module_id: use its value in all Adaptyst API calls requiring
//   an ID of type amod_t.
// * name: the name of the region as passed to adaptyst_region_end().
// * part_id: a string indicating what part of the workflow
//   adaptyst_region_end() has been called in. At the moment,
//   it is always in form of "<PID>_<TID>" where <PID> and <TID>
//   are the process and thread ID respectively.
// * timestamp_str: a string representing the timestamp of the
//   adaptyst_region_end() call in nanoseconds, it's "-1" if
//   undefined/unknown.
//
// The return value indicates whether the module has processed
// the region end successfully.
bool adaptyst_region_end(amod_t module_id, const char *name,
                         const char *part_id, const char *timestamp_str) {
                         
}
```

Additionally, it must define the following constants/variables in the global scope:
```c
// The human-friendly name of a module.
volatile const char *name = "MyModule";

// The human-friendly version of a module.
volatile const char *version = "1.0.0";

// The non-negative version numbers of a module, with
// the last element being negative (-1 is standard here).
//
// Note that the last element is always ignored, it is used only
// for indicating the end of the array.
//
// At least one non-negative number must be defined.
//
// Version comparisons are done lexicographically,
// e.g. { 3, <anything> } > { 2, <anything> },
// { 3, 2, <anything> } > { 3, 1, <anything> },
// { 11, 2, -1 } > { 11, -1 },
// { 12, 5, 3, -1 } > { 12, 4, 88, -1 } etc.
//
// Please bear in mind that {<xyz>, 0, ..., 0, -1} > {<xyz>, -1},
// where <xyz> is any combination of non-negative numbers and
// the number of zeroes on the left-hand side is anything greater
// than or equal to 1.
volatile const int version_nums[] = { 1, 0, 0, -1 };

// The maximum number of instances of a module that can be
// spawned in an entity. 0 means unlimited.
//
// This constant is optional, the default value is 0.
volatile const unsigned int max_count_per_entity = 0;

volatile const char *options[] = {
    // The names of all user-settable options provided by
    // a module, with the last element being NULL.
    NULL
};

volatile const char *tags[] = {
    // The module tags that will be attached
    // automatically to a node, with the last element
    // being NULL.
    //
    // Tags can be used for a variety of reasons,
    // e.g. to verify whether a GPU module is attached
    // to a node that is connected with a CPU node.
    //
    // All tags used by a module are public-facing
    // and thus must be visibly documented to the outside
    // world, especially if a module is closed-source.
    NULL
};

volatile const char *log_types[] = {
    // The names of all types of logs that a module
    // can produce, with the last element being NULL.
    NULL
};

// Each option XYZ can accept either a single or an array value, NOT both.

// *** For each option XYZ in the "options" array with a single value: ***

// Help message, required.
volatile const char *XYZ_help = "XYZ help message";

// Option value type, required.
volatile const option_type XYZ_type = UNSIGNED_INT;

// Default option value, optional (the option becomes
// required to be set by a user if this is not provided).
//
// The type of this constant is determined by the value
// of XYZ_type. See the Doxygen documentation of
// option_type for matchings between option_type values
// and C types.
volatile const unsigned int XYZ_default = 1;

// *** For each option XYZ in the "options" array with an array value: ***

// Help message, required.
volatile const char *XYZ_help = "XYZ help message";

// Value type of every option array element, required.
volatile const option_type XYZ_array_type = UNSIGNED_INT;

// Default option value, optional (the option becomes
// required to be set by a user if this is not provided).
//
// The type of this array constant is determined by the value
// of XYZ_array_type. See the Doxygen documentation of
// option_type for matchings between option_type values
// and C types.
volatile const unsigned int XYZ_array_default[] = {};

// Number of elements in the default option value, required
// if XYZ_array_default is set.
volatile const unsigned int XYZ_array_default_size = 0;
```

You are very likely to use at least some Adaptyst module API functions in your
module: please consult the Doxygen documentation of the API [here](/docs/adaptyst/doxygen-docs).
All essential Adaptyst module API functions compatible with C can be found in the
```adaptyst/*.h``` files. Similarly, extra Adaptyst module API features
compatible with C++20 (not C) can be found in the ```adaptyst/*.hpp``` files.

Technically speaking, modules are used in the Adaptyst flow in one of the two ways:
* **When workflow execution is Adaptyst-handled and a module indicates that
  it will profile a workflow:**
  1. Adaptyst calls ```adaptyst_module_init()``` in the module. Then, in the
     module, a call to the ```adaptyst_set_will_profile()``` API method is made to
     indicate that the module will profile the workflow.
  2. Adaptyst compiles an IR to an executable if applicable.
  3. Adaptyst starts the workflow and makes it wait for all profiling modules
     in all entities to send a "ready" notification.
  4. Adaptyst calls ```adaptyst_module_process()``` in the module **in a separate
     thread**.
  5. The module notifies Adaptyst through the ```adaptyst_profile_notify()```
     API call that it is ready for profiling.
  6. The executable resumes its work as soon as it receives a "ready"
     notification from all profiling modules in all entities.
  7. The executable finishes its work and the module finishes its processing.
  8. Adaptyst calls ```adaptyst_module_close()``` in the module.
* **In all other cases:**
  1. Adaptyst calls ```adaptyst_module_init()``` in the module.
  2. Adaptyst calls ```adaptyst_module_process()``` in the module **in a separate
     thread**.
  3. The module finishes its processing.
  4. Adaptyst calls ```adaptyst_module_close()``` in the module.

```adaptyst_module_process()``` is always called in all modules within an entity in separate
threads, each with its own copy of an IR.

## Adaptyst Analyser implementation
In contrary to the Adaptyst part of a module, the Adaptyst Analyser part has a pre-defined
directory structure and is installable by ```adaptyst-analyser```:
```
| <root directory>
   | python
      | <module name>
         | __init__.py
         | ...
         | (Python files)
         | ...
   | web
      | <module name>
         | backend.js
         | backend.css
         | settings.html
         | settings.css
         | deps
            | ...
            | (extra .js/.cjs/.css files to be loaded by Adaptyst Analyser)
            | ...
   | metadata.yml
```

Because Adaptyst Analyser is a [Flask](https://flask.palletsprojects.com) app, the Adaptyst
Analyser part of a module consists of two components: a server-side one in Python and a 
client-side one in HTML/CSS/JavaScript/jQuery. The server-side files are stored in
```python/<module name>``` and the client-side files are stored in
```web/<module name>``` (the module name **must be the same** in both cases).
The general information about a module is stored in
```metadata.yml```.

{{< callout context="note" title="Adaptyst Analyser JavaScript documentation" icon="outline/info-circle" >}}
The remainder of this section assumes that you have looked at
[the Adaptyst Analyser JavaScript documentation](/docs/adaptyst/adaptyst-analyser-js-docs).
{{< /callout >}}

The client-side component must have these files:
* ```backend.js```: a JavaScript file implementing at least the following (you can
use jQuery, elements from ```settings.html``` are available and can be used):
```javascript
// Called when a user double-clicks a node in a system graph
// and asks to open a specific module. This function should
// create a new window representing the starting point
// for exploring outputs produced by the module.
//
// Parameters:
// * entity_id (String): the ID of an entity of a node double-clicked
//   by a user.
// * node_id (String): the ID of a node double-clicked by a user.
// * session (Object): a Session object corresponding to the
//   currently-selected performance analysis session.
function createRootWindow(entity_id, node_id, session) {
    
}

export { createRootWindow };
```
* ```backend.css```: a CSS stylesheet of HTML elements produced by ```backend.js```.
If you want to apply CSS to parts of a window created by your module, you may want
to use one of the class names indicated by the diagram below, where ```<type>```
is the return value of ```getType()``` in your JavaScript
```Window```-inheriting class.

{{< figure src="/images/window_classes.png" class="ratio"
    alt=".<type>_content for window contents, .<type>_window for a whole window"
    caption="Class names for a window in Adaptyst Analyser." >}}

* ```settings.html```: an HTML file implementing a settings window of a
  module. JavaScript functions from ```backend.js``` **cannot** be used. Settings widgets
  to be used by ```backend.js``` must have IDs. **It is highly recommended that the ID of
  every element is prefixed by the name of a module, e.g. ```linuxperf_option123```.**
  Otherwise, there may be conflicts between the IDs of elements from two different modules,
  leading to an unexpected behaviour.
* ```settings.css```: a CSS stylesheet of the settings window. If you want to
  apply CSS to the content of the window, you may want to use
  `#<module name>_settings_block`.
  
Other files can be added if used by the above.
  
{{< callout context="caution" title="Important notes" icon="outline/alert-triangle" >}}
* When defining HTML in ```backend.js```, **do not use IDs**: use classes instead to distinguish
  elements of your layout. This is because multiple windows with the same layout can be opened
  at the same time by a user.
* When defining CSS in ```backend.css```, **nest your classes in ```.<type>_content```
  or ```.<type>_window```, do not put them in the global scope**. Otherwise, there may be
  conflicts between the class names of elements from two different modules or from a module
  and the Adaptyst Analyser core, leading to an unexpected behaviour.
{{< /callout >}}

The client-side component may also have the "deps" folder containing any extra
.js, .cjs, and .css files to be loaded by Adaptyst Analyser (this is meant for
JavaScript dependencies along with their CSS stylesheets if any). If you use it,
please define all filenames expected in the directory in metadata.yml: see the
metadata.yml part of the documentation later. Also, if you don't want
to store the files locally in "deps", you can specify their HTTP(S) URLs in metadata.yml
so that Adaptyst Analyser downloads them automatically when installing a module: again,
see the metadata.yml documentation for more details.

{{< callout context="caution" title="Dependency conflicts" icon="outline/alert-triangle" >}}
Both files stored in "deps" and files downloaded from metadata.yml URLs are installed
into the same directory in Adaptyst Analyser.

If there are any name conflicts, a user will be given a choice of file version
to keep when installing a module. **It is not possible to have two different versions
of the same file stored at the same time** because loading them simultaneously in Adaptyst
Analyser may lead to an undefined behaviour.
{{< /callout >}}

When writing an HTML code, you may find Adaptyst-Analyser-defined classes
useful for formatting purposes: check [the main stylesheet](https://github.com/Adaptyst/adaptyst-analyser/blob/main/src/adaptystanalyser/static/viewer.css) in
the Adaptyst Analyser code. Similarly, you may find the locally-stored 
selection of [Google Material icons](https://fonts.google.com/icons) useful:
you can embed them in your HTML in ```getContentCode()``` in ```backend.js```
in the following way:
```html
<svg xmlns="http://www.w3.org/2000/svg" data-icon="<icon type>"
 other-attributes>
  Your extra SVG tags, e.g. <title>
</svg>
```
where ```<icon type>``` can be one of:
| Icon type      | Icon |
|----------------|------|
| ```general```  | <svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#000000" style="background:white"><path d="M280-280h80v-200h-80v200Zm320 0h80v-400h-80v400Zm-160 0h80v-120h-80v120Zm0-200h80v-80h-80v80ZM200-120q-33 0-56.5-23.5T120-200v-560q0-33 23.5-56.5T200-840h560q33 0 56.5 23.5T840-760v560q0 33-23.5 56.5T760-120H200Zm0-80h560v-560H200v560Zm0-560v560-560Z"/></svg>
| ```warning```  | <svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#000000" style="background:white"><path d="m40-120 440-760 440 760H40Zm138-80h604L480-720 178-200Zm302-40q17 0 28.5-11.5T520-280q0-17-11.5-28.5T480-320q-17 0-28.5 11.5T440-280q0 17 11.5 28.5T480-240Zm-40-120h80v-200h-80v200Zm40-100Z"/></svg>
| ```replace```  | <svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#000000" style="background:white"><path d="M164-560q14-103 91.5-171.5T440-800q59 0 110.5 22.5T640-716v-84h80v240H480v-80h120q-29-36-69.5-58T440-720q-72 0-127 45.5T244-560h-80Zm620 440L608-296q-36 27-78.5 41.5T440-240q-59 0-110.5-22.5T240-324v84h-80v-240h240v80H280q29 36 69.5 58t90.5 22q72 0 127-45.5T636-480h80q-5 36-18 67.5T664-352l176 176-56 56Z"/></svg>
| ```download``` | <svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#000000" style="background:white"><path d="M480-320 280-520l56-58 104 104v-326h80v326l104-104 56 58-200 200ZM240-160q-33 0-56.5-23.5T160-240v-120h80v120h480v-120h80v120q0 33-23.5 56.5T720-160H240Z"/></svg>
| ```delete```   | <svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#000000" style="background:white"><path d="M280-120q-33 0-56.5-23.5T200-200v-520h-40v-80h200v-40h240v40h200v80h-40v520q0 33-23.5 56.5T680-120H280Zm400-600H280v520h400v-520ZM360-280h80v-360h-80v360Zm160 0h80v-360h-80v360ZM280-720v520-520Z"/></svg>
| ```copy```     | <svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#000000" style="background:white"><path d="M120-220v-80h80v80h-80Zm0-140v-80h80v80h-80Zm0-140v-80h80v80h-80ZM260-80v-80h80v80h-80Zm100-160q-33 0-56.5-23.5T280-320v-480q0-33 23.5-56.5T360-880h360q33 0 56.5 23.5T800-800v480q0 33-23.5 56.5T720-240H360Zm0-80h360v-480H360v480Zm40 240v-80h80v80h-80Zm-200 0q-33 0-56.5-23.5T120-160h80v80Zm340 0v-80h80q0 33-23.5 56.5T540-80ZM120-640q0-33 23.5-56.5T200-720v80h-80Zm420 80Z"/></svg>

If you want more Google Material icons to be embeddable, don't hesitate to
[contact us](/contact)!

For communication between the client-side and server-side components,
POST requests sent by a client-side part through the ```sendRequest()``` JavaScript
function in a `Window`-inheriting class or `Session` are used. These are transmitted
to ```process()``` in the Python code of your module. Therefore, the server-side
component must export the following Python functions through ```__init__.py```:
```python3
def process(storage, identifier, entity, node, data):
    """
    Process a request sent by the client-side of a module.
    
    The return value is a (code, d) tuple, where "code" is
    an HTTP code to send back to the client and "d" is
    data to send back to the client (e.g. a JSON string).
    
    If your HTTP code for the client is 200 (OK),
    the return value can also be just data to send back to
    the client rather than the tuple above.
    
    :param str storage: The path to the *parent* directory of
                        results of a performance analysis
                        session the request corresponds to.
    :param str identifier: The ID of a performance analysis
                           session the request corresponds to.
    :param str entity: The name of an entity of a node in a
                       system graph the request corresponds to.
    :param str node: The name of a node in a system graph
                     the request corresponds to.
    :param dict data: JSON dictionary sent by the client.
    """
```

```metadata.yml``` is a YAML file and its structure is as follows:
```yaml
name: ModuleName
version: 1.0.0
short_desc: Short module description.

# The minimum supported version of the corresponding
# Adaptyst module. If a user tries to inspect data produced
# by an older version of the module, they will get an error.
# 
# Use the same versioning scheme as in version_nums in an
# Adaptyst module shared library (negative numbers shouldn't be
# included).
min_module_version: [1, 0, 0]

# Python dependencies. Use the pip requirements.txt format.
python_dependencies:
  - package1
  - package2
  - package3

# JavaScript dependencies expected to be found in "deps" folder
# along with CSS stylesheets if any, in form of full filenames including
# extensions. Only .js, .cjs, and .css files are accepted.
js_dependencies:
  - script1.js
  - script2.js
  - script2.css

# JavaScript dependencies to be downloaded along with CSS
# stylesheets if any, in form of HTTP(S) URLs (e.g. pointing to CDNs).
# Only .js, .cjs, and .css files are accepted.
js_url_dependencies:
  - https://example.com/example.js
  - https://example.com/example.css
```

Once you create the first version of the above files, you can use ```adaptyst-analyser```
to install the Adaptyst Analyser part of your module in development mode, i.e. with all
changes to your code being immediately propagated to the Adaptyst Analyser installation
directory so that they can be tested directly in a real environment (similarly to how
```pip install -e``` works in Python):
```bash
adaptyst-analyser -d <path to the directory with your module>
```