Updated instructions, bulid drivers, bug fixes

README.md

@@ -28,21 +28,23 @@ There're multiple options to run AQuery on Windows. You can use the native toolc
   2. Install Microsoft Visual Studio 2022 or later with **Desktop development with C++** selected.
   3. Clone AQuery repo from [Github](https://github.com/sunyinqi0508/AQuery2)
   4. Install python requirements with pip `python3 -m pip install -r requirements.txt`
-  5. Change the build driver from aquery_config.py to "MSBuild"
+  5. Change the build_driver variable in aquery_config.py to "MSBuild"
   6. The libraries and headers for Monetdb are already included in msc-plugins, however you can also choose to download them from [Monetdb Easy Setup](https://www.monetdb.org/easy-setup/) and put them in the same place.
 
 - For Winlibs (Recommended):
   - Download latest winlibs toolchain from the [official website](https://winlibs.com/)
   - Since winlibs is linked with native windows runtime libraries (UCRT or MSVCRT), it offers better interoperatibility with other libraries built with MSVC such as python and monetdb.
   - Other steps can be either the same as Visual Studio or Cygwin/Mingw (below) without ABI break.
+  - Copy or link `mingw64/libexec/gcc/<arch>/<version>/liblto-plugin.dll` to `mingw64/lib/bfd-plugins/` For Link time optimization support on gcc-ar and gcc-ranlib 
   
 - For CygWin/MinGW:
-   1. Install gcc and python3 using its **builtin package manager** instead of the one from python.org or windows store. (For Msys2, `pacman -S gcc python3`)
+   1. Install gcc and python3 using its **builtin package manager** instead of the one from python.org or windows store. (For Msys2, `pacman -S gcc python3`). Otherwise, ABI breakage may happen.
    2. Clone AQuery repo from Github
    3. Install python requirements
    4. The prebuilt binaries are included in ./lib directory. However, you could also rebuild them from [source](https://github.com/MonetDB/MonetDB).
    
 ### macOS
+- If you're using an arm-based mac (e.g. M1, M2 processors). Please go to the Application folder and right-click on the Terminal app, select 'Get Info' and ensure that the 'Open using Rosetta' option is unchecked. See the section below for more notes for arm-based macs.
 - Install a package manager such as [homebrew](https://brew.sh) 
 - Install python3 and monetdb using homebrew `brew install python3 monetdb`
 - Install C++ compiler come with Xcode commandline tool by `xcode-select --install` or from homebrew
@@ -50,8 +52,10 @@ There're multiple options to run AQuery on Windows. You can use the native toolc
 - Install python packages from **requirements.txt**
   
 **for arm64 macOS users**
-- In theory, AQuery++ could work on both native arm64 and x86_64 through Rosetta. But for maximum performance, running native is preferred. 
-- However, they can't be mixed up, i.e. make sure every component, `python` binary, `C++ compiler`, `monetdb` library and system commandline utilities such as `uname` should have the same architecture. 
+- In theory, AQuery++ can work on both native arm64 and x86_64 through Rosetta. But for maximum performance, running native is preferred. 
+- However, they can't be mixed up, i.e. make sure every component, `python` , `C++ compiler`, `monetdb` library and system commandline utilities such as `uname` should have the same architecture. 
+- Use the script `./arch-check.sh` to check if relevant binaries all have the same architecture.
+- In the case where binaries have different architectures, install the software with desired architecture and make an alias or link to ensure the newly installed binary is referred to. 
 - Because I can't get access to an arm-based mac to fully test this setup, there might still be issues. Please open an issue if you encounter any problems.
   
 ### Linux
@@ -59,19 +63,23 @@ There're multiple options to run AQuery on Windows. You can use the native toolc
 - Install python3, C++ compiler and git. (For Ubuntu, run `apt update && apt install -y python3 python3-pip clang-14 libmonetdbe-dev git `)
 - Install required python packages by `python3 -m pip install -r requirements.txt`
 - If you have multiple C++ compilers on the system. Specify C++ compiler by setting the **CXX** environment variable. e.g. `export CXX=clang++-14`
+- Note for anaconda users: the system libraries included in anaconda might differ from the ones your compiler is using. In this case, you might get errors similar to: 
+  >ImportError: libstdc++.so.6: version `GLIBCXX_3.4.26' not found 
+   
+   In this case, upgrade anaconda or your compiler or use the python from your OS or package manager instead. Or (**NOT recommended**) copy/link the library from your system (e.g. /usr/lib/x86_64-linux-gnu/libstdc++.so.6) to anaconda's library directory (e.g. ~/Anaconda3/lib/).
+
 ### Docker: 
    - Alternatively, you can also use docker to run AQuery.
    - Type `make docker` to build the docker image from scratch. 
-   - For Arm-based Mac users, you would need to build and run the x86_64 docker image because MonetDB doesn't offer official binaries for arm64 Linux. 
+   - For Arm-based Mac users, you would have to build and run the x86_64 docker image because MonetDB doesn't offer official binaries for arm64 Linux. 
 ## Usage
 `python3 prompt.py` will launch the interactive command prompt. The server binary will be automatically rebuilt and started.
 #### Commands:
 - `<sql statement>`: parse AQuery statement
 - `f <filename>`: parse all AQuery statements in file
+- `exec`: execute last parsed statement(s) with Hybrid Execution Engine. Hybrid Execution Engine decouples the query into two parts. The standard SQL (MonetDB dialect) part is executed by an Embedded version of Monetdb and everything else is executed by a post-process module which is generated by AQuery++ Compiler in C++ and then compiled and executed.
 - `dbg` start debugging session 
 - `print`: printout parsed AQuery statements
-
-- `xexec`: execute last parsed statement(s) with Hybrid Execution Engine. Hybrid Execution Engine decouples the query into two parts. The standard SQL (MonetDB dialect) part is executed by an Embedded version of Monetdb and everything else is executed by a post-process module which is generated by AQuery++ Compiler in C++ and then compiled and executed.
 - `save <OPTIONAL: filename>`: save current code snippet. will use random filename if not specified.
 - `exit`: quit the prompt
 - `r`: run the last generated code snippet

aquery_config.py

@@ -2,7 +2,7 @@
 
 ## GLOBAL CONFIGURATION FLAGS
 
-version_string = '0.4.6a'
+version_string = '0.4.7a'
 add_path_to_ldpath = True
 rebuild_backend = False
 run_backend = True

arch-check.sh

@@ -0,0 +1,21 @@
+ARCH=`uname -m`
+ARCH2=`arch`
+echo Current architechure: $ARCH $ARCH2
+echo Current shell: $SHELL
+PASSED=1
+for i in python3 c++ make ranlib libtool $SHELL
+do
+	FILEPATH=`which $i`
+	FILEINFO=`file $FILEPATH`
+	if [[ $FILEINFO =~ $ARCH ]]; then
+		echo $i@$FILEPATH: passed
+	else
+		echo "\033[1;31mERROR\033[0m: Architecture of $i is not $ARCH: $FILEINFO"
+		PASSED=0
+	fi
+done
+
+if [[ PASSED -eq 1 ]]; then
+	echo "\033[1;32mBinary archtechure check passed\033[0m"
+fi
+

build.py

@@ -94,8 +94,8 @@ class build_manager:
             ret = True
             for c in self.build_cmd:
                 if c:
-                    try:
-                        ret = subprocess.call(c, stdout = stdout, stderr = stderr) and ret
+                    try: # only last success matters
+                        ret = not subprocess.call(c, stdout = stdout, stderr = stderr) # and ret
                     except (FileNotFoundError):
                         ret = False
                         pass
@@ -205,21 +205,23 @@ class build_manager:
              self.cache_status.sources or 
              self.cache_status.env
         ):
-            self.driver.pch()
-            self.driver.libaquery_a()
-            self.driver.server()
+            success &= self.driver.pch()
+            success &= self.driver.libaquery_a()
+            success &= self.driver.server()
         else:
             if self.cache_status.libaquery_a:
-                success = self.driver.libaquery_a() and success
+                success &= self.driver.libaquery_a() 
             if self.cache_status.pch_hpp_gch:
-                success = self.driver.pch() and success
+                success &= self.driver.pch() 
             if self.cache_status.server:
-                success = self.driver.server() and success
+                success &= self.driver.server() 
         if success:
             current.calc(self.cxx, libaquery_a)
             with open('.cached', 'wb') as cache_sig:
                 cache_sig.write(pickle.dumps(current))
         else:
+            if aquery_config.os_platform == 'mac':
+                os.system('./arch-check.sh')
             try:
                 os.remove('./.cached')
             except:

engine/utils.py

@@ -141,3 +141,6 @@ nullstream = open(os.devnull, 'w')
 
 def clamp(val, minval, maxval):
     return min(max(val, minval), maxval)
+
+def escape_qoutes(string : str):
+    return re.sub(r'^\'', r'\'',re.sub(r'([^\\])\'', r'\1\'', string))

prompt.py

@@ -43,7 +43,7 @@ dbg:
 print: 
     printout parsed sql statements
 exec: 
-    execute last parsed statement(s) with AQuery Execution Engine
+    execute last parsed statement(s) with AQuery Execution Engine (disabled)
 xexec: 
     execute last parsed statement(s) with Hybrid Execution Engine
 r: 
@@ -336,7 +336,7 @@ def prompt(running = lambda:True, next = lambda:input('> '), state = None):
                 time.sleep(.00001)
             og_q : str = next()
             q = og_q.lower().strip()
-            if q == 'exec': # generate build and run (AQuery Engine)
+            if False and q == 'exec': # generate build and run (AQuery Engine)
                 state.cfg.backend_type = Backend_Type.BACKEND_AQuery.value
                 cxt = engine.exec(state.stmts, cxt, keep)
                 if state.buildmgr.build_dll() == 0:
@@ -352,7 +352,7 @@ def prompt(running = lambda:True, next = lambda:input('> '), state = None):
                 else:
                     print(prompt_help)
                 continue
-            elif q.startswith('xexec'): # generate build and run (MonetDB Engine)
+            elif q.startswith('xexec') or q.startswith('exec'): # generate build and run (MonetDB Engine)
                 state.cfg.backend_type = Backend_Type.BACKEND_MonetDB.value
                 cxt = xengine.exec(state.stmts, cxt, keep)
                 

reconstruct/ast.py

@@ -617,6 +617,7 @@ class join(ast_node):
         self.join_conditions = []
         # self.tmp_name = 'join_' + base62uuid(4)
         # self.datasource = TableInfo(self.tmp_name, [], self.context)
+        
     def append(self, tbls, __alias = ''):
         alias = lambda t : t + ' ' + __alias if len(__alias) else t
         if type(tbls) is join:
@@ -661,8 +662,11 @@ class join(ast_node):
                     self.have_sep = True
                     j = join(self, node[keys[0]])
                     tablename = f' {keys[0]} {j}'
-                    if len(keys) > 1 and keys[1].lower() == 'on':
-                        tablename += f' on {expr(self, node[keys[1]])}' 
+                    if len(keys) > 1 :
+                        if keys[1].lower() == 'on':
+                            tablename += f' ON {expr(self, node[keys[1]])}' 
+                        elif keys[1].lower() == 'using':
+                            tablename += f' USING {expr(self, node[keys[1]])}'
                     self.joins.append((tablename, self.have_sep))
                     self.tables += j.tables
                     self.tables_dir = {**self.tables_dir, **j.tables_dir}
@@ -731,8 +735,9 @@ class join(ast_node):
 class filter(ast_node):
     name = 'where'
     def produce(self, node):
-        self.add(expr(self, node).sql)
-    
+        filter_expr = expr(self, node)
+        self.add(filter_expr.sql)
+        self.datasource.join_conditions += filter_expr.join_conditions
         
 class create_table(ast_node):
     name = 'create_table'

reconstruct/expr.py

@@ -12,6 +12,11 @@ from engine.types import *
 
 class expr(ast_node):
     name='expr'
+    valid_joincond = {
+        0 : ('and', 'eq', 'not'),
+        1 : ('or', 'neq', 'not'),
+        2 : ('', '', '')
+    }
     @property
     def udf_decltypecall(self):
         return self._udf_decltypecall if self._udf_decltypecall else self.sql
@@ -46,6 +51,7 @@ class expr(ast_node):
         self.node = node
         self.supress_undefined = supress_undefined
         if(type(parent) is expr):
+            self.next_valid = parent.next_valid
             self.inside_agg = parent.inside_agg
             self.is_udfexpr = parent.is_udfexpr
             self.is_agg_func = parent.is_agg_func
@@ -53,6 +59,8 @@ class expr(ast_node):
             self.c_code = parent.c_code
             self.builtin_vars = parent.builtin_vars
         else:
+            self.join_conditions = []
+            self.next_valid = 0
             self.is_agg_func = False
             self.is_udfexpr = type(parent) is udf
             self.root : expr = self
@@ -93,8 +101,17 @@ class expr(ast_node):
                 if len(node) > 1:
                     print(f'Parser Error: {node} has more than 1 dict entry.')
 
+                is_joincond = False
                 for key, val in node.items():
                     key = key.lower()
+                    if key not in self.valid_joincond[self.next_valid]:
+                        self.next_valid = 2
+                    else:
+                        if key == self.valid_joincond[self.next_valid][2]:
+                            self.next_valid = not self.next_valid
+                        elif key == self.valid_joincond[self.next_valid][1]:
+                            self.next_valid = 2
+                            is_joincond = True
                     if key in self.operators:
                         if key in builtin_func:
                             if self.is_agg_func:
@@ -200,6 +217,9 @@ class expr(ast_node):
                     else:
                         print(f'Undefined expr: {key}{val}')
                 
+                if (is_joincond and len(self.children) == 2
+                    and all([c.is_ColExpr for c in self.children])) :
+                    self.root.join_conditions.append((c.raw_col for c in self.children))
                     
         if type(node) is str:
             if self.is_udfexpr:
@@ -342,9 +362,11 @@ class expr(ast_node):
                             exec(f'loc["{b}"] = lambda : "{b}"')
                     
             x = self.c_code if c_code is None else c_code
+            from engine.utils import escape_qoutes
             if decltypestr:
-                return eval('f\'' + self.udf_decltypecall + '\'')
-            return eval('f\'' + self.sql + '\'')
+                return eval('f\'' + escape_qoutes(self.udf_decltypecall) + '\'')
+            self.sql.replace("'", "\\'")
+            return eval('f\'' + escape_qoutes(self.sql) + '\'')
         if self.is_recursive_call_inudf or (self.need_decltypestr and self.is_udfexpr) or gettype:
             return call
         else:

test.aquery

@@ -1,5 +1,8 @@
 #!aquery
 
+select "hello world"
+xexec
+
 echo Testing Insert, Filters and Nested Aggregation
 f stock.a
 xexec