AQuery++ Database

Introduction

AQuery++ Database is a cross-platform, In-Memory Column-Store Database that incorporates compiled query execution.

Architecture

AQuery Compiler

• The query is first processed by the AQuery Compiler which is composed of a frontend that parses the query into AST and a backend that generates target code that delivers the query.
• Front end of AQuery++ Compiler is built on top of mo-sql-parsing with modifications to handle AQuery dialect and extension.
• Backend of AQuery++ Compiler generates target code dependent on the Execution Engine. It can either be the C++ code for AQuery Execution Engine or sql and C++ post-processor for Hybrid Engine or k9 for the k9 Engine.

Execution Engines

• AQuery++ supports different execution engines thanks to the decoupled compiler structure.
• AQuery Execution Engine: executes queries by compiling the query plan to C++ code. Doesn't support joins and udf functions.
• Hybrid Execution Engine: decouples the query into two parts. The sql-compliant part is executed by an Embedded version of Monetdb and everything else is executed by a post-process module which is generated by AQuery++ Compiler in C++ and then compiled and executed.
• K9 Execution Engine: (discontinued).

Roadmap

• SQL Parser -> AQuery Parser (Front End)
• AQuery-C++ Compiler (Back End)
  • Schema and Data Model
  • Data acquisition/output from/to csv file
• Execution Engine
  • Projections and single-group Aggregations
  • Group by Aggregations
  • Filters
  • Order by
  • Assumption
  • Flatten
  • UDFs (Hybrid Engine only)
  • User Module
  • Triggers
  • Join (Hybrid Engine only)
  • Subqueries
• Query Optimization
  • Selection/Order by push-down
  • Join Optimization (Only in Hybrid Engine)

TODO:

• User Module load syntax parsing (fn definition/registration)
• User Module initialize location
• User Module test -> Interval based triggers
• Optimize Compilation Process, using static libraries, hot reloading server binary
• Bug fixes: type deduction misaligned in Hybrid Engine -> Investigation: Using postproc only for q1 in Hybrid Engine (make is_special always on)
• Limitation: putting ColRefs back to monetdb.
• C++ Meta-Programming: Eliminate template recursions as much as possible.
• Limitation: Date and Time, String operations, Funcs in groupby agg.

Installation

Requirements

1. Recent version of Linux, Windows or MacOS, with recent C++ compiler that has C++17 (1z) support. (however c++20 is recommended if available for heterogeneous lookup on unordered containers)

   • GCC: 9.0 or above (g++ 7.x, 8.x fail to handle fold-expressions due to a compiler bug)
   • Clang: 5.0 or above (Recommended)
   • MSVC: 2017 or later (2022 or above is recommended)

2. Monetdb for Hybrid Engine

   • On windows, the required libraries and headers are already included in the repo.
   • On Linux, see Monetdb Easy Setup for instructions.
   • On MacOS, Monetdb can be easily installed in homebrew brew install monetdb.

3. Python 3.6 or above and install required packages in requirements.txt by python3 -m pip install -r requirements.txt

Usage

python3 prompt.py will launch the interactive command prompt. The server binary will be autometically rebuilt and started.

Commands:

• <sql statement>: parse sql statement

• f <filename>: parse all sql statements in file

• dbg start debugging session

• print: printout parsed sql statements

• exec: execute last parsed statement(s) with AQuery Execution Engine. AQuery Execution Engine executes query by compiling it to C++ code and then executing it.

• xexec: execute last parsed statement(s) with Hybrid Execution Engine. Hybrid Execution Engine decouples the query into two parts. The sql-compliant part is executed by an Embedded version of Monetdb and everything else is executed by a post-process module which is generated by AQuery++ Compiler in C++ and then compiled and executed.

• r: run the last generated code snippet

• save <OPTIONAL: filename>: save current code snippet. will use random filename if not specified.

• exit: quit the prompt

Example:

f moving_avg.a
xexec