From 15f11f1eab25e950ccafc3d6b762a05aa7a02cc9 Mon Sep 17 00:00:00 2001 From: Bill Date: Tue, 27 Sep 2022 05:25:25 +0800 Subject: [PATCH] Updated instructions, bulid drivers, bug fixes --- AQuery2 | 2616 ------------------------------- MonetDB-release-epel.noarch.rpm | Bin 6588 -> 0 bytes 2 files changed, 2616 deletions(-) delete mode 100644 AQuery2 delete mode 100644 MonetDB-release-epel.noarch.rpm diff --git a/AQuery2 b/AQuery2 deleted file mode 100644 index 4478769..0000000 --- a/AQuery2 +++ /dev/null @@ -1,2616 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - GitHub - sunyinqi0508/AQuery2: An in-memory column-store time-series database that uses query compilation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- Skip to content - - - - - - - - - - - - -
- -
- - - - - - - -
- - - - - -
- - - - - - - - - - - -
-
-
- - - - - - - - - - - - -
- -
- -
-
- - - - / - - AQuery2 - - - Public -
- -
- - - -
- -
-
-

- An in-memory column-store time-series database that uses query compilation -

- -

License

- - - - -
-
- -
- -
-
- -
- - - - -
- - - - - -
- - - - - - -

sunyinqi0508/AQuery2

-
- - -
- - -
- -
- - - - - -
-
- -
- -
-
- - - master - - - - -
-
-
- Switch branches/tags - -
- - - -
- -
- -
- - -
- -
- - - - - - - - - - - - - - - -
- - -
-
-
-
- -
- -
- - - - -
- - - - - - - - -
- Code -
- -
-
- -
- - -
-
- - - - - - -
-
-

Latest commit

-
- -
-
 
-
-

Git stats

- -
-
-
-

Files

- - - - - Permalink - -
- - - Failed to load latest commit information. - - - -
-
-
-
Type
-
Name
-
Latest commit message
-
Commit time
-
- -
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- data -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- docs -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- engine -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- lib -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- monetdb -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- msvs-py -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- sdk -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- server -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- tests -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- LICENSE -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- Makefile -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- README.md -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- build.py -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- csv.h -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- dbconn.py -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- mmw.cpp -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- -
- prompt.py -
- -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
-
- -
- - - -
-
 
-
- -
-
 
-
- -
-
- -
- -
- - -
- - - - -
- - - -
-

AQuery++ Database

-

Introduction

-

AQuery++ Database is a cross-platform, In-Memory Column-Store Database that incorporates compiled query execution.

-

Requirements

-
    -
  1. -

    Recent version of Linux, Windows or MacOS, with recent C++ compiler that has C++17 (1z) support. (however c++20 is recommended if available for heterogeneous lookup on unordered containers)

    -
      -
    • GCC: 9.0 or above (g++ 7.x, 8.x fail to handle fold-expressions due to a compiler bug)
    • -
    • Clang: 5.0 or above (Recommended)
    • -
    • MSVC: 2019 or later (2022 or above is recommended)
    • -
    -
  2. -
  3. -

    Monetdb for Hybrid Engine

    -
      -
    • On windows, the required libraries and headers are already included in the repo.
    • -
    • On Linux, see Monetdb Easy Setup for instructions.
    • -
    • On MacOS, Monetdb can be easily installed in homebrew brew install monetdb.
    • -
    -
  4. -
  5. -

    Python 3.6 or above and install required packages in requirements.txt by python3 -m pip install -r requirements.txt

    -
  6. -
-

Installation

-

AQuery is tested on mainstream operating systems such as Windows, macOS and Linux

-

Windows

-

There're multiple options to run AQuery on Windows. You can use the native toolchain from Microsoft Visual Studio or gcc from Cygwin/MinGW or run it under Windows Subsystem for Linux.

-
    -
  • -

    For WSL, Docker or Linux virtual machines, see Linux, Docker sections below

    -
  • -
  • -

    For Visual Studio:

    -
      -
    1. Install python3.6 or above from official website or Microsoft Store.
    2. -
    3. Install Microsoft Visual Studio 2022 or later with Desktop development with C++ selected.
    4. -
    5. Clone AQuery repo from Github
    6. -
    7. Install python requirements with pip python3 -m pip install -r requirements.txt
    8. -
    9. Change the build driver from aquery_config.py to "MSBuild"
    10. -
    11. The libraries and headers for Monetdb are already included in msc-plugins, however you can also choose to download them from Monetdb Easy Setup and put them in the same place.
    12. -
    -
  • -
  • -

    For MinGW:

    -
      -
    1. Install gcc and python3 using the package manager. (For Msys2, pacman -S gcc python3)
    2. -
    3. Clone AQuery repo from Github
    4. -
    5. Install python requirements
    6. -
    7. The prebuilt binaries are included in ./lib directory. However, you could also rebuild them from source.
    8. -
    -
  • -
  • -

    Note that it might be possible to use python from python.org or Microsoft store with gcc from MinGW. However, it might not work because of ABI breakage. So the better way is to use gcc with MinGW python from pacman or use clang/MSVC instead.

    -
  • -
-

macOS

-
    -
  • Install a package manager such as homebrew
  • -
  • Install python3 and monetdb using homebrew brew install python3 monetdb
  • -
  • Install C++ compiler come with Xcode commandline tool by xcode-select --install or from homebrew
  • -
  • If you have multiple C++ compilers on the system. Specify C++ compiler by setting the CXX environment variable. e.g. export CXX=clang
  • -
  • Install python packages from requirements.txt
  • -
-

for arm64 macOS users

-
    -
  • In theory, AQuery++ could work on both native arm64 and x86_64 through Rosetta. But for maximum performance, running native is preferred.
  • -
  • However, they can't be mixed up, i.e. make sure every component, python binary, C++ compiler, monetdb library and system commandline utilities such as uname should have the same architecture.
  • -
  • Because I can't get access to an arm-based mac to fully test this setup, there might still be issues. Please open an issue if you encounter any problems.
  • -
-

Linux

-
    -
  • Install monetdb, see Monetdb Easy Setup for instructions.
  • -
  • Install python3, C++ compiler and git. (For Ubuntu, run apt update && apt install -y python3 python3-pip clang-14 libmonetdbe-dev git )
  • -
  • Install required python packages by python3 -m pip install -r requirements.txt
  • -
  • If you have multiple C++ compilers on the system. Specify C++ compiler by setting the CXX environment variable. e.g. export CXX=clang++-14
  • -
-

Docker:

-
    -
  • Alternatively, you can also use docker to run AQuery.
  • -
  • Type make docker to build the docker image from scratch.
  • -
  • For Arm-based Mac users, you would need to build and run the x86_64 docker image because MonetDB doesn't offer official binaries for arm64 Linux.
  • -
-

Usage

-

python3 prompt.py will launch the interactive command prompt. The server binary will be automatically rebuilt and started.

-

Commands:

-
    -
  • -

    <sql statement>: parse AQuery statement

    -
  • -
  • -

    f <filename>: parse all AQuery statements in file

    -
  • -
  • -

    dbg start debugging session

    -
  • -
  • -

    print: printout parsed AQuery statements

    -
  • -
  • -

    xexec: execute last parsed statement(s) with Hybrid Execution Engine. Hybrid Execution Engine decouples the query into two parts. The standard SQL (MonetDB dialect) part is executed by an Embedded version of Monetdb and everything else is executed by a post-process module which is generated by AQuery++ Compiler in C++ and then compiled and executed.

    -
  • -
  • -

    save <OPTIONAL: filename>: save current code snippet. will use random filename if not specified.

    -
  • -
  • -

    exit: quit the prompt

    -
  • -
  • -

    r: run the last generated code snippet

    -
  • -
-

Example:

-

f moving_avg.a
-xexec

-

See ./tests/ for more examples.

-

Architecture

-

Architecture

-

AQuery Compiler

-
    -
  • The query is first processed by the AQuery Compiler which is composed of a frontend that parses the query into AST and a backend that generates target code that delivers the query.
  • -
  • Front end of AQuery++ Compiler is built on top of mo-sql-parsing with modifications to handle AQuery dialect and extension.
  • -
  • Backend of AQuery++ Compiler generates target code dependent on the Execution Engine. It can either be the C++ code for AQuery Execution Engine or sql and C++ post-processor for Hybrid Engine or k9 for the k9 Engine.
  • -
-

Execution Engines

-
    -
  • AQuery++ supports different execution engines thanks to the decoupled compiler structure.
  • -
  • AQuery Execution Engine: executes queries by compiling the query plan to C++ code. Doesn't support joins and udf functions.
  • -
  • Hybrid Execution Engine: decouples the query into two parts. The sql-compliant part is executed by an Embedded version of Monetdb and everything else is executed by a post-process module which is generated by AQuery++ Compiler in C++ and then compiled and executed.
  • -
  • K9 Execution Engine: (discontinued).
  • -
-

Roadmap

-
    -
  • SQL Parser -> AQuery Parser (Front End)
  • -
  • AQuery-C++ Compiler (Back End) -
      -
    • Schema and Data Model
    • -
    • Data acquisition/output from/to csv file
    • -
    -
  • -
  • Execution Engine -
      -
    • Projections and single-group Aggregations
    • -
    • Group by Aggregations
    • -
    • Filters
    • -
    • Order by
    • -
    • Assumption
    • -
    • Flatten
    • -
    • UDFs (Hybrid Engine only)
    • -
    • User Module
    • -
    • Triggers
    • -
    • Join (Hybrid Engine only)
    • -
    • Subqueries
    • -
    -
  • -
  • Query Optimization -
      -
    • Selection/Order by push-down
    • -
    • Join Optimization (Only in Hybrid Engine)
    • -
    -
  • -
-

Known Issues:

-
    -
  • Interval based triggers
  • -
  • Hot reloading server binary
  • -
  • Bug fixes: type deduction misaligned in Hybrid Engine
  • -
  • Investigation: Using postproc only for q1 in Hybrid Engine (make is_special always on)
  • -
  • C++ Meta-Programming: Eliminate template recursions as much as possible.
  • -
  • Functionality: Basic helper functions in aquery
  • -
  • Improvement: More DDLs, e.g. drop table, update table, etc.
  • -
  • Bug: Join-Aware Column management
  • -
  • Bug: Order By after Group By
  • -
-
-
-
- -
- - -
-
- -
-
-
-

About

- -

- An in-memory column-store time-series database that uses query compilation -

- -

Topics

- - -

Resources

- - -

License

- - - - - - - -

Stars

- - -

Watchers

- - -

Forks

- - -
-
- - - - - - - - - - - -
-
-

Languages

-
- - - - - -
- - -
-
-
-
- -
- -
- - -
- -
- - -
-
- -
- - - - - - - - - - - - - - - - - - - - - diff --git a/MonetDB-release-epel.noarch.rpm b/MonetDB-release-epel.noarch.rpm deleted file mode 100644 index 3edafcff9245dc48aa9a7c196bb2052f04843bd7..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 6588 zcma)92UHZv)^1?PNrIxNI0lq7!@vLok|cxVBtg=I9vGSA1d;>=BpFdebU{#1LBfh6 z2q-ElA}YqkfJzim5kbU+%3lr5zTNlld1w2a>RVskP`B=_y4C%@apjiyj6P*c1BzD3BMe0#qbh zgouGeq@bk>mSa-0k53A(6??7tI*R@5NA2%X@S6}9-Ppic0?A0vn;TJ6mx6|u4ryj$ z^T@@YtnRtre?FADb+|x6z2;iATJrVe*KvN8g%?C^lofZmYD{^Qm^*2wTNYQr{;hi@ z*jk|`q!->%Ow4F-!eZ_E^!^^|{S2GTk=^bCTkOVn!Oh=u<@(0CmwCS5A_ukmDniZ9 ziBWI-vS=>cYu?BUO&?BqaA1f1zTC!6@w-=k6J(UpOEc4oxehPwzJ2I$%81*(_`Kbv zq25nvaU*G(rOIDi_E<{e$2YpA;!S&6%*t8Dkx4xlL)u+(75B`)>NdaT)KIXBvu)VK z8MzA1$bhYLb=ZrKn>8eU=mtwFGc2YDJ_u!XR*bxCyHNJf>{4{bA0#=NjZ!$#$bdv6 zP|QqWI*mvo(8*-jh;Cv^Gcz_ahK)!DW-x(hYDNYbfS?6S{#A@u{WZpSx+Q-|8Du0# zYwXk-SJCP!@9~_t+pQhL%}~uvS(|feL(^I11B@wwa>(hMCXWdjhobB(@?JRhE^}kT z20?mPny#%*{`x&`P^)Hq6~vQK=jL2C#Fp$x_pA8GR2Rm-efp*B_6EZ05goUEP^~cM zZ1J|My+>UX--Ju8(ck2;s(Q)JGaZ6&4J$|1cgZ)X3g!DMC9y(x@dd+TsUJ7K%~NQ@ zoxd_5yi~Wc+ppzd_mS&c-h1H*m%Yd6`*%bKSGB(w9JnwUF_eri?o_K1@39CI{k*c( zmJ}(Qh|^6u|Jdh<-&(b(lH7}1bXJVwlfV6ZcG~hs06SFY%bx4)B*&ZtD$O~s_K(JV zc2)M?W^q>!(#wl43XoVTm4MgG`Iw%b-V6Hk{&IfSf^A<1#;idD`(LsQ;e&=`K?*eF znL97w{6L~8egi0i)liI_9|SK(F>-#SLDNL>dlWmMcoa}@oXxQ{wEZFp2ecdDE@+CbrgRE6wEI=H{TDGAA<6KqBsS`zfin=7GuENGv|FLpkU1y zWNo0NiRPggSsQ|p2qSzMK!J{EA&O-I1@RLtL$TZ}7F&&Cq#glzF)b7;qI?31ksJVh zF_51b`>KEfAH?hd1^K|t2NcogqI@I|hN^*$GD*w@dJ_D z3@BpX6i{FvgV;l;8K8)K7K#zS02W95MfkR}SYj2Rh`!V8xP%3uNPL`8Y>#3WK!Lsl zGC#r(MEPDQ4nq0JIKmG``2s)@`x{X{;va&MyaGQYwgQUChXRV&&q8q+iYrh|0TirP z;trsQK5Z6DqWc8{qj){ar=!>%#Y_}?q8RZD%r6-QC^Ekg#dGihq#Fq#B*WbGaHzK$vkS544Fy= zpA$SXFwED(`g*(JX1IawpaXnm10ys(UPi_*3M_+5rbZ*vb9tN?1`QYzM8Q}#OojP; zG7=sJ6B*;oEC6BsC5$q>|AdjlM&rZBA}a$6pfJ&;`+!9{gP1vhiFsTe14P1&!4}40 zwHSC9uP5WncMF|94 z{#t!~knXvygiN7N6I|Kx_^YeBhfbsJWOo+sd|;A)W%?E0IQ|nVAv^ zBr=^~02&Na$t0R7!Hi;RWMoF96HJK)Bm%|M&r5KX8AW1^Xn0fj_1GBYuQDKHTxQp|{CBAI4FBGC!P1TqX8nwU`y=tKgI zNH8#^BL{l!;c++u=)YcQJpad+D4x%w;(1&a1c7^U=J_Wblew7x&C=}5ppeM?cjLh{ zaG>UdAz+`$plI2FJ;VcB5T-eT4d_8;!F=sG9x`Ijh=loq|BQK&94j{1WdBdug&&xp4M9Xa0dIsefQ;}ac)|=t-pDBUDu;~m zhBITPU~@r1Y4g@0&G#byvago$I^Rtkm`B69gCma2!oyndXGYeMsjQiOiN#@OKJKA^ zCy7k;?<5WWwIs3_94LS+P7FAg944q5j7TSrMF2~}%V4ou ze=AE=G8@b0#3FNn5=z4g+0-a9I}-Fj#i4LOWu4J9nK6eN(FUtn5S7c|Kye9>AzI16k(@c1Gt~_j$wqGMzds_7 zjjx@H3swN`C5Xo0&CDAAcbPea#B=5wdV2at1nAHENlFL=A1yL{Q*uW|q2(e9U=ZWw zY|^V#EE66IEtG$Op%{J>%5FV&QPEps%QaWEJbqvzhPpI}Hnyfzxn|o~rC4Yx>14zn zIhNM+kGs-e^*iwpsV6!PLF|qDX4WF~{6vn1dzX&#lpzANq*zzc;@m zA#uyPMYyT*+u041(i#Utnx`D3Qr@aAt;kKfry-Xgxnso@<){&_9}(rTip%VBu3IE; zHB7R9saEbkKhS%#$n)^vZB@?%1{>O4eP8q`_>Su4yR3y}_@~Ok~{e(1}_(XkUAK#OX$OCSIylB;}|^KgOhT@s7;IiG9_&1EN9m zMr}OA16$;qMH_bqSQC`)ea;(lY+bneeA06<`aX#_vW27XxhoqD63Zh`OX&Ipr0E4W zJm_taQA@X8JO7-y$pv-R6tr;7(s^xX<9|xZzgl1SJ%L!GR4S-klCnrvS=z3Xxm9OP zcgeYnvYETgP4AYxv@+h2c+fh$Uu-zMif*+~;?06vUsf8+W?lM@BPurc-nQ5^f1jVW zvZz#+#4&S=B`#-_y$^a4Yd*Eb=cuPT9^6#bB^4I`J&h)IRdz$svR{=wPX|RGxtxuv zXA+OQJ52QOcI_-Ul^^m$(!FX5wYws1z`1(rK!A9K%J)7J3&Vh&aFIfk( zZ`N-Q{4uWg=7z`T16p|@#~Nc6(MMbj{L@zLx#bt0MJkkqriy9O*(N6zy6o7gWrPV7 z8}~F#sjy59T@u8zjICRpPl`L$=CUQCYt-a7h4+^yKbQPGE;TXLN%wFvXbNAhn0#fp z_mqT+QKX>%BB#$k=>0&9_MM&B5T2BNc~Lv$ymLLvYR^ zY0cu*U4hBM1=_9HeY+Otw~nn%T^Ch+Li13(nQFm-7LwJjweQ1k(%g>Sj_bX4MQ5eo z-k9yeMOxf*-V0jPnkG&=wPf!-zPm!{Wrfx5Bg->>$Qq9rkErbKabbHH9fS<@K7yJ7b!gCd8FfZ z=GVyWij=~JtKsKOT`NU;{N0LaWQDh*J&oON^pKKQ8r`7}yQ!9s*N?JUw#RLs^R`VU z&2wnKQ8X++y!p__!^ZII^{Pjbe?R$?>9C+9$Xn=jLDNxqBDF0n0r#neb$^3A#z(3O$%l6)Wtc|&(P^~jmge#P>|uGqiudW+YL;D27c9No z-&y3AV?!_4gwt=^hz@pM5fAJeY|mO+Z+1ze&|GQM>)rX0?hHTE&uX>iis^C@$JKvH zbkoL;cfLKSD!<}ac89!f=%Tj5MSZ%v5}JmsuP?YCAuGP#;jX;NQSE}fi_Z@oSknHG z5dD;tTyW;>o=tZ?pEW0y<$Sf_+$6S=u6Z~H>ctS7Z@x`_OXRs0wA*A{9PfPA7I!#h zjaHcTyOjCY<7oJ0vFTWi$rS7gZCk_s{xLtofa&Pbt%Csr-#*o?9FS98sUxvszhPpA z{I%Y~H8!Cw6SM}$d98=!mOqtY?6-Spw@*S&BhSUPvhLZ_a|DCQR)NmuMpK!>ONFR%VkfxNL2<`{hXu3}gEIjL=NszDq%Dnms zE4=e~FZhpngSpr0s*Tq*P8?l-qG#99@vhvQn}X?+^pWF7T%Ht|jMo=^xF>VDQ~syH zRC+UeSrGoddO*03>7J3A{ak;aZt({v*E7z1TlN~WCb@k0h5o+xMHch*Dxyx@=_q@D zw#p~bC)Rsx1LfWoN?h7U^)=deht6ev({UJSaK`m}hZHu+1@Z^y>(IYRIPTNuHIkD~ z{yvnZPid_b5b@J0mySPm`cy-IS$v?`bsuHF*X6Tf%cnoDveuAnZQIfsa4EoAdP|G> z8$zkBro!LAzw$Ax5-BL07lj%gbcXf`KSB&S* zIwzG>j~=HnSzVnqAq#Wb6`Uy6>&ibcHSfrJPIp{95WQ87cR1VK_o`|lCp9cq@H5F? z&O**MKcBlr?pezInt-0cye2GWL6Os&qBO%*n{+E1x*8i}b4nFop2tP1?{Lic<2 zf#T&~OK-qc+j4L8M{eKs_H|a^)+dKAPsN;CCQpwR*0gp;`*)#&KN&H#Qz^9NZO1S9Ed^EO)oMx)?T;z!F|>>n$#Ymz z<+XgrnN^E5+^myh?gg(j=nAf)=RR}`Tv}(lH{Ekg^N#cDg6|Ou&*RQJ`;xBwdZuz} zEK?;<*DBFirBJyK-|@iL=z{iuonmst#87zg#5G6VSD(~L*rnN|$G=WLi`RdwA!YO| zysMRY@>?jaWwFZHq7`(&jtrt?&;Md_2@a+))s$>R$=v|inD zjd4!6-mta_pQzHDbAjeivgv7Q_Y+l#F6GnYbHPeaFKrvKmaM;16r0}jqNC!0r1Qn{ z$>1N)FKDN173xlg25o$9Y(ain7H3y;vZv_N&9s9LA8GeLu_W)KPpN{6z)|6r&gxrDr=9FR4+fhLZ^1==>;&UIsaJv#*Oqw)2oYzzG)H9*R`yDNLH`+ zHSc@7{@eDwUs+wo6QTv_eiok5@kzwtyPod@OsD l?Y+|4+>60kDjy+{{qsv!LQt~Hjm-KtJxgYO3}k|W{|}$