IOWarp Core
A Comprehensive Platform for Context Management in Scientific Computing
Overview ·
Components ·
Getting Started ·
Documentation ·
Contributing
Overview
IOWarp Core is a unified framework that integrates multiple high-performance components for context management, data transfer, and scientific computing. Built with a modular architecture, IOWarp Core enables developers to create efficient data processing pipelines for HPC, storage systems, and near-data computing applications.
IOWarp Core provides:
- High-Performance Context Management: Efficient handling of computational contexts and data transformations
- Heterogeneous-Aware I/O: Multi-tiered, dynamic buffering for accelerated data access
- Modular Runtime System: Extensible architecture with dynamically loadable processing modules
- Advanced Data Structures: Shared memory compatible containers with GPU support (CUDA, ROCm)
- Distributed Computing: Seamless scaling from single node to cluster deployments
Architecture
IOWarp Core follows a layered architecture integrating five core components:
┌──────────────────────────────────────────────────────────────┐
│ Applications │
│ (Scientific Workflows, HPC, Storage Systems) │
└──────────────────────────────────────────────────────────────┘
│
┌─────────────────────┼─────────────────────┐
│ │ │
┌───────────────┐ ┌──────────────────┐ ┌────────────────┐
│ Context │ │ Context │ │ Context │
│ Exploration │ │ Assimilation │ │ Transfer │
│ Engine │ │ Engine │ │ Engine │
└───────────────┘ └──────────────────┘ └────────────────┘
│ │ │
└─────────────────────┼─────────────────────┘
│
┌─────────────────┐
│ Chimaera │
│ Runtime │
│ (ChiMod System)│
└─────────────────┘
│
┌─────────────────────────┐
│ Context Transport │
│ Primitives │
│ (Shared Memory & IPC) │
└─────────────────────────┘
Components
IOWarp Core consists of five integrated components, each with its own specialized functionality:
1. Context Transport Primitives
Location: context-transport-primitives/
High-performance shared memory library containing data structures and synchronization primitives compatible with shared memory, CUDA, and ROCm.
Key Features:
- Shared memory compatible data structures (vector, list, unordered_map, queues)
- GPU-aware allocators (CUDA, ROCm)
- Thread synchronization primitives
- Networking layer with ZMQ transport
- Compression and encryption utilities
2. Chimaera Runtime
Location: context-runtime/
High-performance modular runtime for scientific computing and storage systems with coroutine-based task execution.
Key Features:
- Ultra-high performance task execution (< 10μs latency)
- Modular ChiMod system for dynamic extensibility
- Coroutine-aware synchronization (CoMutex, CoRwLock)
- Distributed architecture with shared memory IPC
- Built-in storage backends (RAM, file-based, custom block devices)
3. Context Transfer Engine
Location: context-transfer-engine/
Heterogeneous-aware, multi-tiered, dynamic I/O buffering system designed to accelerate I/O for HPC and data-intensive workloads.
Key Features:
- Programmable buffering across memory/storage tiers
- Multiple I/O pathway adapters
- Integration with HPC runtimes and workflows
- Improved throughput, latency, and predictability
4. Context Assimilation Engine
Location: context-assimilation-engine/