OnnxRunner
This example demonstrates how to integrate ONNX and load the TinyStories-33M model on a mobile device. It showcases a locally running chatbot and the process of local text generation based on the TinyStories-33M model.
Build status:
- example -
- dev -
Table of Contents
- Compatibility
- Build features
- Branch info
- Install and running
- Screenshots
- Use cases
- Project Structure
- Terms of Use and Participation in Development
Compatibility
The project is compatible with all current versions of the Aurora OS.
Build features
The project is built using CMake build system. The project uses Conan package manager to obtain dependencies including MLPack library and its dependencies.
Branch info
Application versions conform to the branch naming convention
Install and running
Installation and build are performed according to the Build example instruction.
Screenshots
Use cases
The application demonstrates LLM functionality via text generation:
Text Generation
- Model: TinyStories-33M (≈33M parameters), a compact GPT-2 compatible LLM for short story generation
- Features: GPT-2 BPE tokenization (~50k vocabulary), token-by-token progressive generation, optimized ONNX variant with INT8 quantization support to reduce model size
- Model source: Hugging Face — roneneldan/TinyStories-33M
- Convert to ONNX: use scripts/universal_onnx_converter.py
Project Structure
The project has a standard structure of an application based on C++ and QML for Aurora OS with LLM capabilities.
- CMakeLists.txt file describes the project structure for the CMake build system and ONNX Runtime dependencies.
- icons directory contains the application icons for different screen resolutions.
-
qml directory contains the QML source code and the UI resources.
- cover directory contains the application cover implementations.
- icons directory contains the additional custom UI icons.
- pages directory contains the application pages.
- OnnxRunner.qml file provides the application window implementation.
-
rpm directory contains the rpm-package build settings.
- ru.auroraos.OnnxRunner.spec file is used by rpmbuild tool.
-
src directory contains the C++ source code with ONNX Runtime integration.
- main.cpp file is the application entry point.
- llmrunner.h header file defines the LLM service interface.
- llmrunner.cpp implementation of LLM inference using ONNX Runtime.
- chatmodel.h and chatmodel.cpp chat message management.
- chatmessage.h and chatmessage.cpp individual message handling.
- translations directory contains the UI translation files.
- models directory contains ONNX models and tokenizer files.
- scripts directory contains model conversion scripts and usage docs. See scripts/README.md.
- ru.auroraos.OnnxRunner.desktop file defines the display and parameters for launching the application.
Terms of Use and Participation in Development
The source code of the project is provided under license, which allows to use it in third-party applications.
To participate in the development of the project, please read the member agreement. If you plan to submit your own source code for inclusion in the project, you will need to accept the CLA terms and conditions.
Participant information is listed in the AUTHORS file.
The Code of Conduct is the current set of rules of the Open Mobile Platform Company, which informs about the expectations for interaction between community members when communicating and working on projects.