RU | EN

OnnxRunner

This example demonstrates how to integrate ONNX and load the TinyStories-33M model on a mobile device. It showcases a locally running chatbot and the process of local text generation based on the TinyStories-33M model.

Build status:

1. example -
2. dev -

Table of Contents

• Compatibility
• Build features
• Branch info
• Install and running
• Screenshots
• Use cases
• Project Structure
• Terms of Use and Participation in Development

Compatibility

The project is compatible with all current versions of the Aurora OS.

Build features

The project is built using CMake. Two large binary dependencies are fetched from the moshub Generic Package Registry by rpm/ru.auroraos.OnnxRunner.spec during the install phase, so they live outside git:

• ONNX Runtime 1.26.0 cross-compiled for Aurora OS, with libstdc++.so.6 and libgcc_s.so.1 from the GCC 15 toolchain bundled alongside (since Aurora's system libstdc++ from GCC 12 is missing GLIBCXX 3.4.31+ symbols required by the .so). Source and build flow: aurora-psdk-onnx-runtime-devel.
• TinyStories 33M INT8 model (~110 MB), dynamically quantized so XNNPACK + kleidiai micro-kernels can accelerate matmul on ARMv8.

Bundled .sos have $ORIGIN rpath baked in, so they resolve each other inside /usr/share/<app>/lib/ at runtime. The build link adds -Wl,--allow-shlib-undefined so the older PSDK linker doesn't reject newer libstdc++ symbols transitively referenced by libonnxruntime.so — they get resolved at runtime against the bundled libstdc++.

Branch info

Application versions conform to the branch naming convention

Install and running

Installation and build are performed according to the Build example instruction.

Screenshots

Use cases

The application demonstrates LLM functionality via text generation:

Text Generation

• Model: TinyStories-33M (≈33M parameters), a compact GPT-2 compatible LLM for short story generation
• Features: GPT-2 BPE tokenization (~50k vocabulary), token-by-token progressive generation, dynamic INT8 weight quantization
• Model source: Hugging Face — roneneldan/TinyStories-33M
• Convert to ONNX: use scripts/universal_onnx_converter.py

Benchmark

The About page has a Run benchmark (CPU vs XNNPACK) button that runs 20 forward passes of the loaded model on each backend in a background thread and reports avg/min/max latency and tokens/sec. Useful to verify XNNPACK + kleidiai acceleration on ARMv8 targets without leaving the app.

Project Structure

The project has a standard structure of an application based on C++ and QML for Aurora OS with LLM capabilities.

• CMakeLists.txt file describes the project structure for the CMake build system and ONNX Runtime dependencies.
• icons directory contains the application icons for different screen resolutions.
• qml directory contains the QML source code and the UI resources.
  • cover directory contains the application cover implementations.
  • icons directory contains the additional custom UI icons.
  • pages directory contains the application pages.
  • OnnxRunner.qml file provides the application window implementation.
• rpm directory contains the rpm-package build settings.
  • ru.auroraos.OnnxRunner.spec file is used by rpmbuild tool.
• src directory contains the C++ source code with ONNX Runtime integration.
  • main.cpp file is the application entry point.
  • llmrunner.h header file defines the LLM service interface.
  • llmrunner.cpp implementation of LLM inference using ONNX Runtime.
  • chatmodel.h and chatmodel.cpp chat message management.
  • chatmessage.h and chatmessage.cpp individual message handling.
• translations directory contains the UI translation files.
• models directory contains ONNX models and tokenizer files.
• scripts directory contains model conversion scripts and usage docs. See scripts/README.md.
• ru.auroraos.OnnxRunner.desktop file defines the display and parameters for launching the application.

Terms of Use and Participation in Development

The source code of the project is provided under license, which allows to use it in third-party applications.

To participate in the development of the project, please read the member agreement. If you plan to submit your own source code for inclusion in the project, you will need to accept the CLA terms and conditions.

Participant information is listed in the AUTHORS file.

The Code of Conduct is the current set of rules of the Open Mobile Platform Company, which informs about the expectations for interaction between community members when communicating and working on projects.