Single Table query part 1

dev
Bill Sun 3 years ago
parent 4dd571b3d2
commit c46ce578d4

@ -1,4 +1,13 @@
OS_SUPPORT =
ifeq ($(OS),Windows_NT)
OS_SUPPORT += server/winhelper.cpp
endif
$(info $(OS_SUPPORT))
server.bin:
g++ server/server.cpp $(OS_SUPPORT) --std=c++1z -O3 -march=native -o server.bin
snippet: snippet:
g++ -shared --std=c++1z out.cpp -O3 -march=native -o dll.so g++ -shared -fPIC --std=c++1z out.cpp -O3 -march=native -o dll.so
clean: clean:
rm *.shm -rf rm *.shm -rf

@ -1,239 +1,20 @@
# AQuery Compiler # AQuery Compiler
AQuery Compiler that compiles AQuery into [c](https://shakti.com/). AQuery Compiler that compiles AQuery into C++11.
Frontend built on top of [mo-sql-parsing](https://github.com/klahnakoski/mo-sql-parsing). Frontend built on top of [mo-sql-parsing](https://github.com/klahnakoski/mo-sql-parsing).
## Roadmap ## Roadmap
- [x] SQL Parser -> AQuery Parser - [x] SQL Parser -> AQuery Parser (Front End)
- [ ] Data acquisition/output from/to csv file (By Jan. 21) - [ ] AQuery-C++ Compiler (Back End)
- -> AQuery-c Compiler - [x] Schema and Data Model
- Simple query (By Jan. 21) - [x] Data acquisition/output from/to csv file
- [ ] Nested queries (Jan. 28) - [ ] Single table queries
- [x] Projections and Single Table Aggregations
- [x] Group by Aggregations
- [x] Filters
- [ ] Order by
- [ ] Assumption
- [ ] Multi-table
- [ ] Join
- [ ] Subqueries
- [ ] -> Optimizing Compiler - [ ] -> Optimizing Compiler
# Descriptions from mo-sql-parsing:
Parse SQL into JSON so we can translate it for other datastores!
[See changes](https://github.com/klahnakoski/mo-sql-parsing#version-changes)
## Problem Statement
SQL is a familiar language used to access databases. Although, each database vendor has its quirky implementation, there is enough standardization that the average developer does not need to know of those quirks. This familiar core SQL (lowest common denominator, if you will) is useful enough to explore data in primitive ways. It is hoped that, once programmers have reviewed a datastore with basic SQL queries, and they see the value of that data, and they will be motivated to use the datastore's native query format.
## Objectives
The objective is to convert SQL queries to JSON-izable parse trees. This originally targeted MySQL, but has grown to include other database engines. *Please [paste some SQL into a new issue](https://github.com/klahnakoski/mo-sql-parsing/issues) if it does not work for you*
## Project Status
November 2021 - There are [over 800 tests](https://app.travis-ci.com/github/klahnakoski/mo-sql-parsing). This parser is good enough for basic usage, including:
* inner queries,
* with clauses,
* window functions
* create/drop tables and views
* insert/update/delete statements
* lambda (`->`) functions
## Install
pip install mo-sql-parsing
## Parsing SQL
>>> from aquery_parser import parse
>>> parse("select count(1) from jobs")
{'select': {'value': {'count': 1}}, 'from': 'jobs'}
Each SQL query is parsed to an object: Each clause is assigned to an object property of the same name.
>>> parse("select a as hello, b as world from jobs")
{'select': [{'value': 'a', 'name': 'hello'}, {'value': 'b', 'name': 'world'}], 'from': 'jobs'}
The `SELECT` clause is an array of objects containing `name` and `value` properties.
### SQL Flavours
There are a few parsing modes you may be interested in:
#### SQLServer Identifiers (`[]`)
SQLServer uses square brackets to delimit identifiers. For example
SELECT [Timestamp] FROM [table]
which conflicts with BigQuery array constructor (eg `[1, 2, 3, 4]`). You may use the SqlServer flavour with
from aquery_parser import parse_sqlserver as parse
#### NULL is None
The default output for this parser is to emit a null function `{"null":{}}` wherever `NULL` is encountered in the SQL. If you would like something different, you can replace nulls with `None` (or anything else for that matter):
result = parse(sql, null=None)
this has been implemented with a post-parse rewriting of the parse tree.
#### Normalized function call form
The default behaviour of the parser is to output function calls in `simple_op` format: The operator being a key in the object; `{op: params}`. This form can be difficult to work with because the object must be scanned for known operators, or possible optional arguments, or at least distinguished from a query object.
You can have the parser emit function calls in `normal_op` format
>>> from aquery_parser import parse, normal_op
>>> parse("select trim(' ' from b+c)", calls=normal_op)
which produces calls in a normalized format
{"op": op, "args": args, "kwargs": kwargs}
here is the pretty-printed JSON from the example above:
```
{'select': {'value': {
'op': 'trim',
'args': [{'op': 'add', 'args': ['b', 'c']}],
'kwargs': {'characters': {'literal': ' '}}
}}}
```
#### MySQL literal strings
MySQL uses both double quotes and single quotes to declare literal strings. This is not ansi behaviour, but it is more forgiving for programmers coming from other languages. A specific parse function is provided:
result = parse_mysql(sql)
## Generating SQL
You may also generate SQL from the a given JSON document. This is done by the formatter, which is in Alpha state (Oct2021).
>>> from aquery_parser import format
>>> format({"from":"test", "select":["a.b", "c"]})
'SELECT a.b, c FROM test'
## Contributing
In the event that the parser is not working for you, you can help make this better but simply pasting your sql (or JSON) into a new issue. Extra points if you describe the problem. Even more points if you submit a PR with a test. If you also submit a fix, then you also have my gratitude.
### Run Tests
See [the tests directory](https://github.com/klahnakoski/mo-sql-parsing/tree/dev/tests) for instructions running tests, or writing new ones.
## More about implementation
SQL queries are translated to JSON objects: Each clause is assigned to an object property of the same name.
# SELECT * FROM dual WHERE a>b ORDER BY a+b
{
"select": "*",
"from": "dual",
"where": {"gt": ["a", "b"]},
"orderby": {"value": {"add": ["a", "b"]}}
}
Expressions are also objects, but with only one property: The name of the operation, and the value holding (an array of) parameters for that operation.
{op: parameters}
and you can see this pattern in the previous example:
{"gt": ["a","b"]}
## Array Programming
The `mo-sql-parsing.scrub()` method is used liberally throughout the code, and it "simplifies" the JSON. You may find this form a bit tedious to work with because the JSON property values can be values, lists of values, or missing. Please consider converting everything to arrays:
```
def listwrap(value):
if value is None:
return []
elif isinstance(value, list)
return value
else:
return [value]
```
then you may avoid all the is-it-a-list checks :
```
for select in listwrap(parsed_result.get('select')):
do_something(select)
```
## Version Changes
### Version 8
*November 2021*
* Prefer BigQuery `[]` (create array) over SQLServer `[]` (identity)
* Added basic DML (`INSERT`/`UPDATE`/`DELETE`)
* flatter `CREATE TABLE` structures. The `option` list in column definition has been flattened:<br>
**Old column format**
{"create table": {
"columns": {
"name": "name",
"type": {"decimal": [2, 3]},
"option": [
"not null",
"check": {"lt": [{"length": "name"}, 10]}
]
}
}}
**New column format**
{"create table": {
"columns": {
"name": "name",
"type": {"decimal": [2, 3]}
"nullable": False,
"check": {"lt": [{"length": "name"}, 10]}
}
}}
### Version 7
*October 2021*
* changed error reporting; still terrible
* upgraded mo-parsing library which forced version change
### Version 6
*October 2021*
* fixed `SELECT DISTINCT` parsing
* added `DISTINCT ON` parsing
### Version 5
*August 2021*
* remove inline module `mo-parsing`
* support `CREATE TABLE`, add SQL "flavours" emit `{null:{}}` for None
### Version 4
*November 2021*
* changed parse result of `SELECT DISTINCT`
* simpler `ORDER BY` clause in window functions

@ -1,10 +1,11 @@
from operator import index
from engine.utils import base62uuid from engine.utils import base62uuid
from copy import copy
# replace column info with this later. # replace column info with this later.
class ColRef: class ColRef:
def __init__(self, cname, _ty, cobj, cnt, table:'TableInfo', name, id, compound = False): def __init__(self, cname, _ty, cobj, cnt, table:'TableInfo', name, id, compound = False):
self.cname = cname self.cname = cname # column object location
self.cxt_name = None self.cxt_name = None # column object in context
self.type = _ty self.type = _ty
self.cobj = cobj self.cobj = cobj
self.cnt = cnt self.cnt = cnt
@ -28,6 +29,7 @@ class ColRef:
base_name = f'{base_name}_{counter}' base_name = f'{base_name}_{counter}'
self.cxt_name = base_name self.cxt_name = base_name
cxt.columns_in_context[self] = base_name cxt.columns_in_context[self] = base_name
# TODO: change this to cname;
cxt.emit(f'auto& {base_name} = *(ColRef<{self.type}> *)(&{self.table.cxt_name}->colrefs[{self.id}]);') cxt.emit(f'auto& {base_name} = *(ColRef<{self.type}> *)(&{self.table.cxt_name}->colrefs[{self.id}]);')
elif self.cxt_name is None: elif self.cxt_name is None:
self.cxt_name = cxt.columns_in_context[self] self.cxt_name = cxt.columns_in_context[self]
@ -44,7 +46,9 @@ class ColRef:
self.__arr__[key] = value self.__arr__[key] = value
def __str__(self): def __str__(self):
return self.cname return self.reference()
def __repr__(self):
return self.reference()
class TableInfo: class TableInfo:
@ -87,6 +91,7 @@ class TableInfo:
self.cxt.emit(f'auto& {base_name} = *(TableInfo{type_tags} *)(cxt->tables[{self.table_name}]);') self.cxt.emit(f'auto& {base_name} = *(TableInfo{type_tags} *)(cxt->tables[{self.table_name}]);')
def refer_all(self): def refer_all(self):
self.reference()
for c in self.columns: for c in self.columns:
c.reference() c.reference()
def add_cols(self, cols, new = True): def add_cols(self, cols, new = True):
@ -95,12 +100,13 @@ class TableInfo:
def add_col(self, c, new = True, i = 0): def add_col(self, c, new = True, i = 0):
_ty = c['type'] _ty = c['type']
if new: if new:
cname =f'{self.table_name}->colrefs[{i}].scast<int>()' cname =f'get<{i}>({self.table_name})'
_ty = _ty if type(c) is ColRef else list(_ty.keys())[0] _ty = _ty if type(c) is ColRef else list(_ty.keys())[0]
col_object = ColRef(cname, _ty, c, 1, self,c['name'], len(self.columns)) col_object = ColRef(cname, _ty, c, 1, self,c['name'], len(self.columns))
else: else:
col_object = c col_object = c
cname = c.cname cname = c.cname
c.table = self
self.cxt.ccols_byname[cname] = col_object self.cxt.ccols_byname[cname] = col_object
self.columns_byname[c['name']] = col_object self.columns_byname[c['name']] = col_object
self.columns.append(col_object) self.columns.append(col_object)
@ -156,7 +162,7 @@ class TableInfo:
self.cxt.tables_byname[alias] = self self.cxt.tables_byname[alias] = self
self.alias.add(alias) self.alias.add(alias)
def parse_tablenames(self, colExpr, materialize = True): def parse_tablenames(self, colExpr, materialize = True, raw = False):
self.get_col = self.get_col if materialize else self.get_col_d self.get_col = self.get_col if materialize else self.get_col_d
parsedColExpr = colExpr.split('.') parsedColExpr = colExpr.split('.')
@ -168,13 +174,12 @@ class TableInfo:
if datasource is None: if datasource is None:
raise ValueError(f'Table name/alias not defined{parsedColExpr[0]}') raise ValueError(f'Table name/alias not defined{parsedColExpr[0]}')
else: else:
ret = datasource.get_col(parsedColExpr[1]) ret = datasource.parse_tablenames(parsedColExpr[1], raw)
if self.groupinfo is not None and ret: from engine.expr import index_expr
ret = f"{ret.reference()}[{'start' if ret in self.groupinfo.referenced else 'range'}]" string = ret.reference() + index_expr
else: if self.groupinfo is not None and ret and ret in self.groupinfo.raw_groups:
ret = ret.reference() string = f'get<{self.groupinfo.raw_groups.index(ret)}>({{y}})'
return ret return string, ret if raw else string
class View: class View:
def __init__(self, context, table = None, tmp = True): def __init__(self, context, table = None, tmp = True):
self.table: TableInfo = table self.table: TableInfo = table
@ -187,7 +192,11 @@ class View:
self.context.emit(f'{self.name}:()') self.context.emit(f'{self.name}:()')
class Context: class Context:
function_head = 'extern \"C\" int dllmain(Context* cxt){ \n' function_head = '''
extern "C" int __DLLEXPORT__ dllmain(Context* cxt) {
using namespace std;
using namespace types;
'''
def __init__(self): def __init__(self):
self.tables:List[TableInfo] = [] self.tables:List[TableInfo] = []
self.tables_byname = dict() self.tables_byname = dict()
@ -208,7 +217,8 @@ class Context:
# and will be deactivated when the `from' is out of scope # and will be deactivated when the `from' is out of scope
self.datasource = None self.datasource = None
self.ds_stack = [] self.ds_stack = []
self.scans = []
self.removing_scan = False
def add_table(self, table_name, cols): def add_table(self, table_name, cols):
tbl = TableInfo(table_name, cols, self) tbl = TableInfo(table_name, cols, self)
self.tables.append(tbl) self.tables.append(tbl)
@ -257,6 +267,10 @@ class Context:
return ds return ds
else: else:
return None return None
def remove_scan(self, scan, str_scan):
self.emit(str_scan)
self.scans.remove(scan)
def finalize(self): def finalize(self):
if not self.finalized: if not self.finalized:
headers = '' headers = ''
@ -282,8 +296,6 @@ class ast_node:
self.context = parent.context if context is None else context self.context = parent.context if context is None else context
self.parent = parent self.parent = parent
self.datasource = None self.datasource = None
for h in self.header:
self.context.headers.add(h)
self.init(node) self.init(node)
self.produce(node) self.produce(node)
self.spawn(node) self.spawn(node)

@ -1,20 +1,34 @@
# code-gen for data decl languages # code-gen for data decl languages
from engine.ast import ColRef, TableInfo, ast_node, include from engine.ast import ColRef, TableInfo, ast_node, Context, include
from engine.utils import base62uuid from engine.utils import base62uuid
class create_table(ast_node): class create_table(ast_node):
name = 'create_table' name = 'create_table'
def __init__(self, parent: "ast_node", node, context: Context = None, cexpr = None):
self.cexpr = cexpr
super().__init__(parent, node, context)
def produce(self, node): def produce(self, node):
if type(node) is not TableInfo:
ct = node[self.name] ct = node[self.name]
tbl = self.context.add_table(ct['name'], ct['columns']) tbl = self.context.add_table(ct['name'], ct['columns'])
else:
tbl = node
col_type_str = ','.join([c.type for c in tbl.columns])
# create tables in c # create tables in c
self.emit(f"auto {tbl.table_name} = new TableInfo(\"{tbl.table_name}\", {tbl.n_cols});") self.emit(f"auto {tbl.table_name} = new TableInfo<{col_type_str}>(\"{tbl.table_name}\", {tbl.n_cols});")
self.emit("cxt->tables.insert({\"" + tbl.table_name + f"\", {tbl.table_name}"+"});") self.emit("cxt->tables.insert({\"" + tbl.table_name + f"\", {tbl.table_name}"+"});")
self.context.tables_in_context[tbl] = tbl.table_name self.context.tables_in_context[tbl] = tbl.table_name
tbl.cxt_name = tbl.table_name tbl.cxt_name = tbl.table_name
for i, c in enumerate(ct['columns']): tbl.refer_all()
# TODO: more self awareness if self.cexpr is None:
self.emit(f"{tbl.table_name}->colrefs[{i}].ty = types::AINT;") for c in tbl.columns:
self.emit(f"{c.cxt_name}.init();")
else:
for i, c in enumerate(tbl.columns):
self.emit(f"{c.cxt_name}.init();")
self.emit(f"{c.cxt_name} = {self.cexpr[i]()};")
class insert(ast_node): class insert(ast_node):
name = 'insert' name = 'insert'
@ -42,18 +56,25 @@ class c(ast_node):
class load(ast_node): class load(ast_node):
name="load" name="load"
def produce(self, node): def produce(self, node):
self.context.headers.add('"csv.h"')
node = node[self.name] node = node[self.name]
table:TableInfo = self.context.tables_byname[node['table']] table:TableInfo = self.context.tables_byname[node['table']]
n_keys = len(table.columns) table.refer_all()
keys = '' csv_reader_name = 'csv_reader_' + base62uuid(6)
for _ in range(n_keys): col_types = [c.type for c in table.columns]
keys+='`tk'+base62uuid(6) col_tmp_names = ['tmp_'+base62uuid(8) for _ in range(len(table.columns))]
tablename = 'l'+base62uuid(7) # col_type_str = ",".join(col_types)
col_names = ','.join([f'"{c.name}"' for c in table.columns])
self.emit(f"{tablename}:({keys}!(+(`csv ? 1:\"{node['file']['literal']}\")))[{keys}]")
self.emit(f'io::CSVReader<{len(col_types)}> {csv_reader_name}("{node["file"]["literal"]}");')
self.emit(f'{csv_reader_name}.read_header(io::ignore_extra_column, {col_names});')
for t, n in zip(col_types, col_tmp_names):
self.emit(f'{t} {n};')
self.emit(f'while({csv_reader_name}.read_row({",".join(col_tmp_names)})) {{ \n')
for i, c in enumerate(table.columns): for i, c in enumerate(table.columns):
self.emit(f'{c.cname}:{tablename}[{i}]') self.emit(f'{c.cxt_name}.emplace_back({col_tmp_names[i]});')
self.emit('}')
class outfile(ast_node): class outfile(ast_node):
name="_outfile" name="_outfile"

@ -1,5 +1,7 @@
from engine.ast import ast_node from engine.ast import ast_node, ColRef
start_expr = 'f"'
index_expr = '{\'\' if x is None and y is None else f\'[{x}]\'}'
end_expr = '"'
class expr(ast_node): class expr(ast_node):
name='expr' name='expr'
@ -8,7 +10,7 @@ class expr(ast_node):
'min': 'min', 'min': 'min',
'avg': 'avg', 'avg': 'avg',
'sum': 'sum', 'sum': 'sum',
'mod':'mod', 'count' : 'count',
'mins': ['mins', 'minsw'], 'mins': ['mins', 'minsw'],
'maxs': ['maxs', 'maxsw'], 'maxs': ['maxs', 'maxsw'],
'avgs': ['avgs', 'avgsw'], 'avgs': ['avgs', 'avgsw'],
@ -19,29 +21,32 @@ class expr(ast_node):
'sub':'-', 'sub':'-',
'add':'+', 'add':'+',
'mul':'*', 'mul':'*',
'div':'%', 'div':'/',
'mod':'%',
'and':'&', 'and':'&',
'or':'|', 'or':'|',
'xor' : '^',
'gt':'>', 'gt':'>',
'lt':'<', 'lt':'<',
'le':'<=',
'gt':'>='
} }
compound_ops = { compound_ops = {
'ge' : [2, lambda x: f'~({x[0]}<{x[1]})'],
'le' : [2, lambda x: f'~({x[0]}>{x[1]})'],
'count' : [1, lambda x: f'#({x[0]})']
} }
unary_ops = { unary_ops = {
'neg' : '-', 'neg' : '-',
'not' : '~' 'not' : '!'
} }
coumpound_generating_ops = ['mod', 'mins', 'maxs', 'sums'] + \ coumpound_generating_ops = ['mod', 'mins', 'maxs', 'sums'] + \
list( binary_ops.keys()) + list(compound_ops.keys()) + list(unary_ops.keys() ) list( binary_ops.keys()) + list(compound_ops.keys()) + list(unary_ops.keys() )
def __init__(self, parent, node, materialize_cols = True): def __init__(self, parent, node, materialize_cols = True, abs_col = False):
self.materialize_cols = materialize_cols self.materialize_cols = materialize_cols
self.raw_col = None
self.__abs = abs_col
ast_node.__init__(self, parent, node, None) ast_node.__init__(self, parent, node, None)
def init(self, _): def init(self, _):
@ -54,7 +59,8 @@ class expr(ast_node):
else: else:
self.datasource = self.context.datasource self.datasource = self.context.datasource
self.udf_map = parent.context.udf_map self.udf_map = parent.context.udf_map
self.cexpr = '' self._expr = ''
self.cexpr = None
self.func_maps = {**self.udf_map, **self.builtin_func_maps} self.func_maps = {**self.udf_map, **self.builtin_func_maps}
def produce(self, node): def produce(self, node):
@ -62,30 +68,31 @@ class expr(ast_node):
for key, val in node.items(): for key, val in node.items():
if key in self.func_maps: if key in self.func_maps:
# if type(val) in [dict, str]: # if type(val) in [dict, str]:
self.context.headers.add('"./server/aggregations.h"')
if type(val) is list and len(val) > 1: if type(val) is list and len(val) > 1:
cfunc = self.func_maps[key] cfunc = self.func_maps[key]
cfunc = cfunc[len(val) - 1] if type(cfunc) is list else cfunc cfunc = cfunc[len(val) - 1] if type(cfunc) is list else cfunc
self.cexpr += f"{cfunc}(" self._expr += f"{cfunc}("
for i, p in enumerate(val): for i, p in enumerate(val):
self.cexpr += expr(self, p).cexpr + (';'if i<len(val)-1 else '') self._expr += expr(self, p)._expr + (','if i<len(val)-1 else '')
else: else:
funcname = self.func_maps[key] funcname = self.func_maps[key]
funcname = funcname[0] if type(funcname) is list else funcname funcname = funcname[0] if type(funcname) is list else funcname
self.cexpr += f"{funcname}(" self._expr += f"{funcname}("
self.cexpr += expr(self, val).cexpr self._expr += expr(self, val)._expr
self.cexpr += ')' self._expr += ')'
elif key in self.binary_ops: elif key in self.binary_ops:
l = expr(self, val[0]).cexpr l = expr(self, val[0])._expr
r = expr(self, val[1]).cexpr r = expr(self, val[1])._expr
self.cexpr += f'({l}{self.binary_ops[key]}{r})' self._expr += f'({l}{self.binary_ops[key]}{r})'
elif key in self.compound_ops: elif key in self.compound_ops:
x = [] x = []
if type(val) is list: if type(val) is list:
for v in val: for v in val:
x.append(expr(self, v).cexpr) x.append(expr(self, v)._expr)
self.cexpr = self.compound_ops[key][1](x) self._expr = self.compound_ops[key][1](x)
elif key in self.unary_ops: elif key in self.unary_ops:
self.cexpr += f'({expr(self, val).cexpr}{self.unary_ops[key]})' self._expr += f'({expr(self, val)._expr}{self.unary_ops[key]})'
else: else:
print(f'Undefined expr: {key}{val}') print(f'Undefined expr: {key}{val}')
@ -101,10 +108,18 @@ class expr(ast_node):
while type(p) is expr and not p.isvector: while type(p) is expr and not p.isvector:
p.isvector = True p.isvector = True
p = p.parent p = p.parent
self.cexpr = self.datasource.parse_tablenames(node, self.materialize_cols)
self._expr, self.raw_col = self.datasource.parse_tablenames(node, self.materialize_cols, True)
self.raw_col = self.raw_col if type(self.raw_col) is ColRef else None
if self.__abs and self.raw_col:
self._expr = self.raw_col.reference() + index_expr
elif type(node) is bool: elif type(node) is bool:
self.cexpr = '1' if node else '0' self._expr = '1' if node else '0'
else: else:
self.cexpr = f'{node}' self._expr = f'{node}'
def toCExpr(_expr):
return lambda x = None, y = None : eval(start_expr + _expr + end_expr)
def consume(self, _):
self.cexpr = expr.toCExpr(self._expr)
def __str__(self): def __str__(self):
return self.cexpr return self.cexpr

@ -1,56 +1,61 @@
from engine.ast import ast_node from engine.ast import TableInfo, ast_node
from engine.scan import scan
from engine.utils import base62uuid from engine.utils import base62uuid
from engine.expr import expr from engine.expr import expr
import engine.types
class groupby(ast_node): class groupby(ast_node):
name = '_groupby' name = '_groupby'
def init(self, _): def init(self, _):
self.context.headers.add('"./server/hasher.h"')
self.context.headers.add('unordered_map')
self.group = 'g' + base62uuid(7) self.group = 'g' + base62uuid(7)
self.group_type = 'record_type' + base62uuid(7)
self.datasource = self.parent.datasource self.datasource = self.parent.datasource
self.scanner = None
self.datasource.rec = [] self.datasource.rec = []
self.raw_groups = []
def produce(self, node): def produce(self, node):
if type(node) is not list: if type(node) is not list:
node = [node] node = [node]
g_contents = '(' g_contents = ''
g_contents_list = []
first_col = '' first_col = ''
for i, g in enumerate(node): for i, g in enumerate(node):
v = g['value'] v = g['value']
e = expr(self, v).cexpr e = expr(self, v)
self.raw_groups.append(e.raw_col)
e = e._expr
# if v is compound expr, create tmp cols # if v is compound expr, create tmp cols
if type(v) is not str: if type(v) is not str:
tmpcol = 't' + base62uuid(7) tmpcol = 't' + base62uuid(7)
self.emit(f'{tmpcol}:{e}') self.emit(f'auto {tmpcol} = {e};')
e = tmpcol e = tmpcol
if i == 0: if i == 0:
first_col = e first_col = e
g_contents += e + (';'if i < len(node)-1 else '') g_contents_list.append(e)
g_contents_decltype = [f'decltype({c})' for c in g_contents_list]
self.emit(f'{self.group}:'+g_contents+')') g_contents = expr.toCExpr(','.join(g_contents_list))
self.emit(f'typedef record<{expr.toCExpr(",".join(g_contents_decltype))(0)}> {self.group_type};')
self.emit(f'unordered_map<{self.group_type}, vector_type<uint32_t>, '
f'transTypes<{self.group_type}, hasher>> {self.group};')
self.n_grps = len(node) self.n_grps = len(node)
if self.n_grps <= 1: self.scanner = scan(self, None, expr.toCExpr(first_col)()+'.size')
self.emit(f'{self.group}:={self.group}') self.scanner.add(f'{self.group}[forward_as_tuple({g_contents(self.scanner.it_ver)})].emplace_back({self.scanner.it_ver});')
else:
self.emit(f'{self.group}:groupby[({self.group},(,!(#({first_col}))))]')
def consume(self, _): def consume(self, _):
self.referenced = self.datasource.rec self.referenced = self.datasource.rec
self.datasource.rec = None self.datasource.rec = None
return super().consume(_) self.scanner.finalize()
def finalize(self, cexprs, out:TableInfo):
gscanner = scan(self, self.group)
key_var = 'key_'+base62uuid(7)
val_var = 'val_'+base62uuid(7)
gscanner.add(f'auto &{key_var} = {gscanner.it_ver}.first;')
gscanner.add(f'auto &{val_var} = {gscanner.it_ver}.second;')
gscanner.add(';\n'.join([f'{out.columns[i].reference()}.emplace_back({ce(x=val_var, y=key_var)})' for i, ce in enumerate(cexprs)])+';')
def finalize(self, ret, out): gscanner.finalize()
self.groupby_function = 'fgrp'+base62uuid(4)
grp = self.group
if self.n_grps <= 1:
cfn = "{[range] start:*range;"+ ret + "}"
self.emit(f'{out}:(({cfn}\'{grp})[!{grp}])')
self.parent.inv = False
else:
cfn = "{[ids;grps;ll;dim;x] " + \
"start:grps[x][dim];" + \
"end:$[x=0;ll;grps[x-1][dim]];" + \
"range:(end-start)#((start-ll)#ids);" + \
"start:ids[start];" + \
ret + '}'
self.emit(f'{self.groupby_function}:{cfn}')
self.emit(f'{out}:+({self.groupby_function}' + \
f'[{grp}[1];{grp}[0];(#{grp}[0])+1;(#({grp}[0][0]))-1]\'!(#({grp}[0])))')

@ -5,7 +5,7 @@ from engine.expr import expr
from engine.orderby import orderby from engine.orderby import orderby
from engine.scan import filter from engine.scan import filter
from engine.utils import base62uuid, enlist, base62alp from engine.utils import base62uuid, enlist, base62alp
from engine.ddl import outfile from engine.ddl import create_table, outfile
import copy import copy
class projection(ast_node): class projection(ast_node):
@ -75,7 +75,7 @@ class projection(ast_node):
if self.group_node is not None: if self.group_node is not None:
# There is group by; # There is group by;
has_groupby = True has_groupby = True
cexpr = f'(' cexprs = []
flatten = False flatten = False
cols = [] cols = []
self.out_table = TableInfo('out_'+base62uuid(4), [], self.context) self.out_table = TableInfo('out_'+base62uuid(4), [], self.context)
@ -89,27 +89,23 @@ class projection(ast_node):
if type(proj) is dict: if type(proj) is dict:
if 'value' in proj: if 'value' in proj:
e = proj['value'] e = proj['value']
if type(e) is str: sname = expr(self, e)._expr
cname = e # TODO: deal w/ alias fname = expr.toCExpr(sname)
cexpr += (f"{self.datasource.parse_tablenames(proj['value'])}") absname = expr(self, e, abs_col=True)._expr
elif type(e) is dict:
p_expr = expr(self, e)
cname = p_expr.cexpr
compound = True compound = True
cexpr += f"{cname}" cexprs.append(fname)
cname = ''.join([a if a in base62alp else '' for a in cname]) cname = ''.join([a if a in base62alp else '' for a in fname()])
cexpr += ';'if i < len(self.projections)-1 else ''
compound = compound and has_groupby and self.datasource.rec not in self.group_node.referenced compound = compound and has_groupby and self.datasource.rec not in self.group_node.referenced
cols.append(ColRef(f'{disp_varname}[{i}]', 'generic', self.out_table, 0, None, cname, i, compound=compound)) cols.append(ColRef(cname, expr.toCExpr(f'decays<decltype({absname})>')(0), self.out_table, 0, None, cname, i, compound=compound))
self.out_table.add_cols(cols, False) self.out_table.add_cols(cols, False)
cexpr += ')'
if has_groupby: if has_groupby:
self.group_node.finalize(cexpr, disp_varname) create_table(self, self.out_table)
self.group_node.finalize(cexprs, self.out_table)
else: else:
self.emit(f'auto {disp_varname} = {cexpr};') create_table(self, self.out_table, cexpr = cexprs)
self.datasource.group_node = None self.datasource.group_node = None
has_orderby = 'orderby' in node has_orderby = 'orderby' in node
@ -122,12 +118,8 @@ class projection(ast_node):
self.emit_no_ln(f"{f'{disp_varname}:+' if flatten else ''}(") self.emit_no_ln(f"{f'{disp_varname}:+' if flatten else ''}(")
if self.disp or has_orderby: if self.disp or has_orderby:
if len(self.projections) > 1: self.emit(f'print(*{self.out_table.cxt_name});')
self.emit_no_ln(f"{'+' if self.inv else ''}{disp_varname}")
else:
self.emit_no_ln(f'print({disp_varname});')
if flatten:
self.emit_no_ln(f'{disp_varname}')
if has_orderby: if has_orderby:
self.emit(f')[{orderby_node.view}]') self.emit(f')[{orderby_node.view}]')
else: else:

@ -1,10 +1,40 @@
from xmlrpc.client import Boolean from xmlrpc.client import Boolean
from engine.ast import ColRef, TableInfo, View, ast_node from engine.ast import ColRef, TableInfo, View, ast_node, Context
from engine.utils import base62uuid from engine.utils import base62uuid
from engine.expr import expr from engine.expr import expr
class scan(ast_node): class scan(ast_node):
name = 'scan' name = 'scan'
def __init__(self, parent: "ast_node", node, size = None, context: Context = None):
self.type = type
self.size = size
super().__init__(parent, node, context)
def init(self, _):
self.datasource = self.context.datasource
self.start = ''
self.body = ''
self.end = '}'
self.filter = None
scan_vars = set(s.it_var for s in self.context.scans)
self.it_ver = 'i' + base62uuid(2)
while(self.it_ver in scan_vars):
self.it_ver = 'i' + base62uuid(6)
self.parent.context.scans.append(self)
def produce(self, node):
if type(node) is ColRef:
if self.size is None:
self.start += f'for (auto& {self.it_ver} : {node.reference()}) {{\n'
else:
self.start += f"for (uint32_t {self.it_ver} = 0; {self.it_ver} < {node.reference()}.size; ++{self.it_ver}){{\\n"
elif type(node) is str:
self.start+= f'for(auto& {self.it_ver} : {node}) {{\n'
else:
self.start += f"for (uint32_t {self.it_ver} = 0; {self.it_ver} < {self.size}; ++{self.it_ver}){{\n"
def add(self, stmt):
self.body+=stmt + '\n'
def finalize(self):
self.context.remove_scan(self, self.start + self.body + self.end)
class filter(ast_node): class filter(ast_node):
name = 'filter' name = 'filter'

@ -0,0 +1,168 @@
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
<PropertyGroup Label="Globals">
<VCProjectVersion>16.0</VCProjectVersion>
<Keyword>Win32Proj</Keyword>
<ProjectGuid>{8081fdaa-4d13-4b7a-adb2-8224af7f1c81}</ProjectGuid>
<RootNamespace>Project1</RootNamespace>
<WindowsTargetPlatformVersion>10.0</WindowsTargetPlatformVersion>
<ProjectName>msc-plugin</ProjectName>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
<ConfigurationType>DynamicLibrary</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v143</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="Configuration">
<ConfigurationType>DynamicLibrary</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v143</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
<ConfigurationType>DynamicLibrary</ConfigurationType>
<UseDebugLibraries>true</UseDebugLibraries>
<PlatformToolset>v143</PlatformToolset>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
<ConfigurationType>DynamicLibrary</ConfigurationType>
<UseDebugLibraries>false</UseDebugLibraries>
<PlatformToolset>v143</PlatformToolset>
<WholeProgramOptimization>true</WholeProgramOptimization>
<CharacterSet>Unicode</CharacterSet>
</PropertyGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
<ImportGroup Label="ExtensionSettings">
</ImportGroup>
<ImportGroup Label="Shared">
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<ImportGroup Label="PropertySheets" Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
</ImportGroup>
<PropertyGroup Label="UserMacros" />
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<TargetExt />
<TargetName>dll.so</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<TargetExt />
<TargetName>dll.so</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<TargetExt />
<TargetName>dll.so</TargetName>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<TargetExt />
<TargetName>dll.so</TargetName>
</PropertyGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<SDLCheck>true</SDLCheck>
<PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<OutputFile>$(ProjectDir)\..\dll.so</OutputFile>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<SDLCheck>true</SDLCheck>
<PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
<GenerateDebugInformation>true</GenerateDebugInformation>
<OutputFile>$(ProjectDir)\..\dll.so</OutputFile>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<SDLCheck>true</SDLCheck>
<PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<GenerateDebugInformation>true</GenerateDebugInformation>
<OutputFile>$(ProjectDir)\..\dll.so</OutputFile>
</Link>
</ItemDefinitionGroup>
<ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
<ClCompile>
<WarningLevel>Level3</WarningLevel>
<FunctionLevelLinking>true</FunctionLevelLinking>
<IntrinsicFunctions>true</IntrinsicFunctions>
<SDLCheck>true</SDLCheck>
<PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode>
</ClCompile>
<Link>
<SubSystem>Console</SubSystem>
<EnableCOMDATFolding>true</EnableCOMDATFolding>
<OptimizeReferences>true</OptimizeReferences>
<GenerateDebugInformation>true</GenerateDebugInformation>
<OutputFile>$(ProjectDir)\..\dll.so</OutputFile>
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="..\out.cpp" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\csv.h" />
<ClInclude Include="..\server\aggregations.h" />
<ClInclude Include="..\server\gc.hpp" />
<ClInclude Include="..\server\hasher.h" />
<ClInclude Include="..\server\libaquery.h" />
<ClInclude Include="..\server\table.h" />
<ClInclude Include="..\server\types.h" />
<ClInclude Include="..\server\utils.h" />
<ClInclude Include="..\server\vector_type.hpp" />
<ClInclude Include="..\server\winhelper.h" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets">
</ImportGroup>
</Project>

@ -1,11 +1,15 @@
#include "./server/libaquery.h" #include "./server/libaquery.h"
extern "C" int dllmain(Context* cxt){ #include "./server/aggregations.h"
auto stocks = new TableInfo("stocks", 2);
extern "C" int __DLLEXPORT__ dllmain(Context* cxt) {
using namespace std;
using namespace types;
auto stocks = new TableInfo<int,int>("stocks", 2);
cxt->tables.insert({"stocks", stocks}); cxt->tables.insert({"stocks", stocks});
stocks->colrefs[0].ty = types::AINT;
stocks->colrefs[1].ty = types::AINT;
auto& stocks_timestamp = *(ColRef<int> *)(&stocks->colrefs[0]); auto& stocks_timestamp = *(ColRef<int> *)(&stocks->colrefs[0]);
auto& stocks_price = *(ColRef<int> *)(&stocks->colrefs[1]); auto& stocks_price = *(ColRef<int> *)(&stocks->colrefs[1]);
stocks_timestamp.init();
stocks_price.init();
stocks_timestamp.emplace_back(1); stocks_timestamp.emplace_back(1);
stocks_price.emplace_back(15); stocks_price.emplace_back(15);
stocks_timestamp.emplace_back(2); stocks_timestamp.emplace_back(2);
@ -38,9 +42,19 @@ stocks_timestamp.emplace_back(15);
stocks_price.emplace_back(2); stocks_price.emplace_back(2);
stocks_timestamp.emplace_back(16); stocks_timestamp.emplace_back(16);
stocks_price.emplace_back(5); stocks_price.emplace_back(5);
auto d2PxTIVW = (max((stocks_price-min(stocks_timestamp)))); auto out_6gPn = new TableInfo<decays<decltype(max((stocks_price[0]-min(stocks_timestamp[0]))))>>("out_6gPn", 1);
print(d2PxTIVW); cxt->tables.insert({"out_6gPn", out_6gPn});
auto d2dVVnjL = (max((stocks_price-mins(stocks_price)))); auto& out_6gPn_maxstockspriceminstockstimestamp = *(ColRef<decays<decltype(max((stocks_price[0]-min(stocks_timestamp[0]))))>> *)(&out_6gPn->colrefs[0]);
print(d2dVVnjL); out_6gPn_maxstockspriceminstockstimestamp.init();
out_6gPn_maxstockspriceminstockstimestamp = max((stocks_price-min(stocks_timestamp)));
print(*out_6gPn);
auto out_7a2d = new TableInfo<decays<decltype(max((stocks_price[0]-mins(stocks_price[0]))))>>("out_7a2d", 1);
cxt->tables.insert({"out_7a2d", out_7a2d});
auto& out_7a2d_maxstockspriceminsstocksprice = *(ColRef<decays<decltype(max((stocks_price[0]-mins(stocks_price[0]))))>> *)(&out_7a2d->colrefs[0]);
out_7a2d_maxstockspriceminsstocksprice.init();
out_7a2d_maxstockspriceminsstocksprice = max((stocks_price-mins(stocks_price)));
print(*out_7a2d);
return 0; return 0;
} }

@ -1,3 +1,4 @@
from concurrent.futures import thread
import re import re
import time import time
import aquery_parser as parser import aquery_parser as parser
@ -8,12 +9,35 @@ import sys
import os import os
from engine.utils import base62uuid from engine.utils import base62uuid
import atexit import atexit
try:
os.remove('server.bin')
except Exception as e:
print(type(e), e)
subprocess.call(['make', 'server.bin'])
cleanup = True
def rm(): def rm():
if cleanup:
mm.seek(0,os.SEEK_SET)
mm.write(b'\x00\x00')
mm.flush()
time.sleep(.001)
server.kill()
files = os.listdir('.') files = os.listdir('.')
for f in files: for f in files:
if f.endswith('.shm'): if f.endswith('.shm'):
os.remove(f) os.remove(f)
mm.close()
cleanup = False
def proc_alive(pid):
try:
os.kill(pid, 0)
except OSError:
return False
else:
return True
atexit.register(rm) atexit.register(rm)
shm = base62uuid() shm = base62uuid()
@ -35,7 +59,8 @@ else:
mm = mmap.mmap(0, 2, shm) mm = mmap.mmap(0, 2, shm)
mm.write(b'\x01\x00') mm.write(b'\x01\x00')
mm.flush() mm.flush()
subprocess.Popen(["./server.bin", shm]) server = subprocess.Popen(["./server.bin", shm])
test_parser = True test_parser = True
# code to test parser # code to test parser
@ -64,6 +89,8 @@ keep = False
cxt = None cxt = None
while test_parser: while test_parser:
try: try:
if not proc_alive(server.pid):
server = subprocess.Popen(["./server.bin", shm])
q = input() q = input()
if q == 'exec': if q == 'exec':
if not keep or cxt is None: if not keep or cxt is None:
@ -90,6 +117,8 @@ while test_parser:
elif q == 'keep': elif q == 'keep':
keep = not keep keep = not keep
continue continue
elif q=='format' or q == 'fmt':
subprocess.call(['clang-format', 'out.cpp'])
elif q == 'exit': elif q == 'exit':
break break
elif q == 'r': elif q == 'r':
@ -108,9 +137,8 @@ while test_parser:
continue continue
stmts = parser.parse(q) stmts = parser.parse(q)
print(stmts) print(stmts)
except (ValueError) as e: except (ValueError, FileNotFoundError) as e:
rm()
print(type(e), e) print(type(e), e)
mm.close() rm()
handle.close()
os.remove(shm)

@ -7,4 +7,4 @@ FIELDS TERMINATED BY ","
SELECT sum(c), b, d SELECT sum(c), b, d
FROM test FROM test
group by a,b,d group by a,b,d
order by d DESC, b ASC -- order by d DESC, b ASC

@ -0,0 +1,106 @@
#pragma once
#include "types.h"
#include <utility>
#include <limits>
#include <deque>
template <class T, template<typename ...> class VT>
size_t count(const VT<T>& v) {
return v.size;
}
template <class T>
constexpr static inline size_t count(const T&) { return 1; }
// TODO: Specializations for dt/str/none
template<class T, template<typename ...> class VT>
types::GetLongType<T> sum(const VT<T>& v) {
types::GetLongType<T> ret = 0;
for (const auto& _v : v)
ret += _v;
return ret;
}
template<class T, template<typename ...> class VT>
types::GetFPType<T> avg(const VT<T>& v) {
return static_cast<types::GetFPType<T>>(
sum<T>(v) / static_cast<long double>(v.size));
}
template <class T, template<typename ...> class VT>
T max(const VT<T>& v) {
T max_v = std::numeric_limits<T>::min();
for (const auto& _v : v)
max_v = max_v > _v ? max_v : _v;
return max_v;
}
template <class T, template<typename ...> class VT>
T min(const VT<T>& v) {
T min_v = std::numeric_limits<T>::max();
for (const auto& _v : v)
min_v = min_v < _v ? min_v : _v;
return min_v;
}
template<class T, template<typename ...> class VT>
VT<T> mins(const VT<T>& arr) {
const int& len = arr.size;
std::deque<std::pair<T, uint32_t>> cache;
VT<T> ret(len);
T min = std::numeric_limits<T>::max();
for (int i = 0; i < len; ++i) {
if (arr[i] < min)
min = arr[i];
ret[i] = min;
}
return ret;
}
template<class T, template<typename ...> class VT>
VT<T> maxs(const VT<T>& arr) {
const int& len = arr.size;
VT<T> ret(len);
T max = std::numeric_limits<T>::min();
for (int i = 0; i < len; ++i) {
if (arr[i] > max)
max = arr[i];
ret[i] = max;
}
return ret;
}
template<class T, template<typename ...> class VT>
VT<T> minw(const VT<T>& arr, uint32_t w) {
const int& len = arr.size;
VT<T> ret(len);
std::deque<std::pair<T, uint32_t>> cache;
for (int i = 0; i < len; ++i) {
if (!cache.empty() && cache.front().second == i - w) cache.pop_front();
while (!cache.empty() && cache.back().first > arr[i]) cache.pop_back();
cache.push_back({ arr[i], i });
ret[i] = cache.front().first;
}
return ret;
}
template<class T, template<typename ...> class VT>
VT<T> maxw(const VT<T>& arr, uint32_t w) {
const int& len = arr.size;
VT<T> ret(len);
std::deque<std::pair<T, uint32_t>> cache;
for (int i = 0; i < len; ++i) {
if (!cache.empty() && cache.front().second == i - w) cache.pop_front();
while (!cache.empty() && cache.back().first > arr[i]) cache.pop_back();
cache.push_back({ arr[i], i });
arr[i] = cache.front().first;
}
return ret;
}
template <class T> constexpr inline T max(const T& v) { return v; }
template <class T> constexpr inline T min(const T& v) { return v; }
template <class T> constexpr inline T avg(const T& v) { return v; }
template <class T> constexpr inline T sum(const T& v) { return v; }
template <class T> constexpr inline T maxw(const T& v, uint32_t) { return v; }
template <class T> constexpr inline T minw(const T& v, uint32_t) { return v; }
template <class T> constexpr inline T avgw(const T& v, uint32_t) { return v; }
template <class T> constexpr inline T sumw(const T& v, uint32_t) { return v; }
template <class T> constexpr inline T maxs(const T& v) { return v; }
template <class T> constexpr inline T mins(const T& v) { return v; }
template <class T> constexpr inline T avgs(const T& v) { return v; }
template <class T> constexpr inline T sums(const T& v) { return v; }

@ -0,0 +1,20 @@
#pragma once
#include <functional>
#include <tuple>
template <class ...Types>
struct hasher {
template <size_t i = 0> typename std::enable_if< i == sizeof...(Types),
size_t>::type hashi(const std::tuple<Types...>& record) const {
return 0;
}
template <size_t i = 0> typename std::enable_if< i < sizeof ...(Types),
size_t>::type hashi(const std::tuple<Types...>& record) const {
using current_type = typename std::decay<typename std::tuple_element<i, std::tuple<Types...>>::type>::type;
return std::hash<current_type>()(std::get<i>(record)) ^ hashi<i+1>(record);
}
size_t operator()(const std::tuple<Types...>& record) const {
return hashi(record);
}
};

@ -3,8 +3,16 @@
#include "table.h" #include "table.h"
#include <unordered_map> #include <unordered_map>
struct Context{ struct Context{
std::unordered_map<const char*, void*> tables; std::unordered_map<const char*, void*> tables;
std::unordered_map<const char*, uColRef *> cols; std::unordered_map<const char*, uColRef *> cols;
}; };
#ifdef _MSC_VER
#define __DLLEXPORT__ __declspec(dllexport) __stdcall
#else
#define __DLLEXPORT__
#endif
#endif #endif

@ -1,4 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<packages>
<package id="Microsoft.Windows.CppWinRT" version="2.0.210806.1" targetFramework="native" />
</packages>

@ -3,6 +3,7 @@
#include <string> #include <string>
//#include <thread> //#include <thread>
#include <chrono> #include <chrono>
#include "libaquery.h" #include "libaquery.h"
#ifdef _WIN32 #ifdef _WIN32
#include "winhelper.h" #include "winhelper.h"
@ -32,6 +33,7 @@ struct thread_context{
void daemon(thread_context* c) { void daemon(thread_context* c) {
} }
#include "aggregations.h"
typedef int (*code_snippet)(void*); typedef int (*code_snippet)(void*);
int _main(); int _main();
int main(int argc, char** argv) { int main(int argc, char** argv) {
@ -51,7 +53,7 @@ int main(int argc, char** argv) {
printf("running: %s\n", running? "true":"false"); printf("running: %s\n", running? "true":"false");
printf("ready: %s\n", ready? "true":"false"); printf("ready: %s\n", ready? "true":"false");
while (running) { while (running) {
std::this_thread::sleep_for(1us); std::this_thread::sleep_for(100us);
if(ready){ if(ready){
printf("running: %s\n", running? "true":"false"); printf("running: %s\n", running? "true":"false");
printf("ready: %s\n", ready? "true":"false"); printf("ready: %s\n", ready? "true":"false");
@ -75,72 +77,26 @@ int main(int argc, char** argv) {
} }
int _main() int _main()
{ {
using string = std::string;
using std::cout;
using std::thread;
void* handle = dlopen("dll.so", RTLD_LAZY);
io::CSVReader<4> in("../test.csv");
in.read_header(io::ignore_extra_column, "a", "b", "c", "d");
int a, b, cc, d;
string tp, tpdf, st;
while (in.read_row(a, b, cc, d))
cout << a<< ' ' << b <<' ' << cc <<' ' << d << '\n';
//code_snippet c = reinterpret_cast<code_snippet>(dlsym(handle, "dlmain"));
//printf("%d", c(0));
//dlclose(handle);
vector_type<double> t1{ 1.2, 3.4, .2, 1e-5, 1, 3, 5, 4, 5};
vector_type<int> t2{ 2, 4, 3, 5, 0, 2, 6, 1, 2};
printf("x: ");
const auto& ret = t1 - t2;
for (const auto& x : ret) {
printf("%lf ", x);
}
puts("");
//vector_type<int> t;
//t = 1;
//t.emplace_back(2);
//print(t);
//return 0;
void* handle = dlopen("dll.so", RTLD_LAZY);
printf("handle: %x\n", handle);
Context* cxt = new Context(); Context* cxt = new Context();
auto stocks = TableInfo<int, int>("stocks", 2); if (handle) {
cxt->tables.insert(std::make_pair("stocks", &stocks)); printf("inner\n");
code_snippet c = reinterpret_cast<code_snippet>(dlsym(handle, "dllmain"));
auto& stocks_0 = get<0>(stocks); printf("routine: %x\n", c);
auto& stocks_1 = get<1>(stocks); if (c) {
printf("inner\n");
stocks_0.init(); printf("return: %d\n", c(cxt));
stocks_1.init(); }
dlclose(handle);
}
stocks_0.emplace_back(1);
stocks_1.emplace_back(15);
stocks_0.emplace_back(2);
stocks_1.emplace_back(19);
stocks_0.emplace_back(3);
stocks_1.emplace_back(16);
stocks_0.emplace_back(4);
stocks_1.emplace_back(17);
stocks_0.emplace_back(5);
stocks_1.emplace_back(15);
stocks_0.emplace_back(6);
stocks_1.emplace_back(13);
stocks_0.emplace_back(7);
stocks_1.emplace_back(5);
stocks_0.emplace_back(8);
stocks_1.emplace_back(8);
stocks_0.emplace_back(9);
stocks_1.emplace_back(7);
stocks_0.emplace_back(10);
stocks_1.emplace_back(13);
stocks_0.emplace_back(11);
stocks_1.emplace_back(11);
stocks_0.emplace_back(12);
stocks_1.emplace_back(14);
stocks_0.emplace_back(13);
stocks_1.emplace_back(10);
stocks_0.emplace_back(14);
stocks_1.emplace_back(5);
stocks_0.emplace_back(15);
stocks_1.emplace_back(2);
stocks_0.emplace_back(16);
stocks_1.emplace_back(5);
printf("%d\n", max(stocks_0 - mins(stocks_1)));
return 0; return 0;
} }

@ -4,6 +4,11 @@ Microsoft Visual Studio Solution File, Format Version 12.00
VisualStudioVersion = 17.2.32210.308 VisualStudioVersion = 17.2.32210.308
MinimumVisualStudioVersion = 10.0.40219.1 MinimumVisualStudioVersion = 10.0.40219.1
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "server", "server.vcxproj", "{031352C2-AFBB-45AA-9518-DBC1F9EF2AF3}" Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "server", "server.vcxproj", "{031352C2-AFBB-45AA-9518-DBC1F9EF2AF3}"
ProjectSection(ProjectDependencies) = postProject
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81} = {8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}
EndProjectSection
EndProject
Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "Project1", "..\Project1\Project1.vcxproj", "{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}"
EndProject EndProject
Global Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution GlobalSection(SolutionConfigurationPlatforms) = preSolution
@ -21,6 +26,14 @@ Global
{031352C2-AFBB-45AA-9518-DBC1F9EF2AF3}.Release|x64.Build.0 = Release|x64 {031352C2-AFBB-45AA-9518-DBC1F9EF2AF3}.Release|x64.Build.0 = Release|x64
{031352C2-AFBB-45AA-9518-DBC1F9EF2AF3}.Release|x86.ActiveCfg = Release|Win32 {031352C2-AFBB-45AA-9518-DBC1F9EF2AF3}.Release|x86.ActiveCfg = Release|Win32
{031352C2-AFBB-45AA-9518-DBC1F9EF2AF3}.Release|x86.Build.0 = Release|Win32 {031352C2-AFBB-45AA-9518-DBC1F9EF2AF3}.Release|x86.Build.0 = Release|Win32
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}.Debug|x64.ActiveCfg = Debug|x64
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}.Debug|x64.Build.0 = Debug|x64
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}.Debug|x86.ActiveCfg = Debug|Win32
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}.Debug|x86.Build.0 = Debug|Win32
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}.Release|x64.ActiveCfg = Release|x64
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}.Release|x64.Build.0 = Release|x64
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}.Release|x86.ActiveCfg = Release|Win32
{8081FDAA-4D13-4B7A-ADB2-8224AF7F1C81}.Release|x86.Build.0 = Release|Win32
EndGlobalSection EndGlobalSection
GlobalSection(SolutionProperties) = preSolution GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE HideSolutionNode = FALSE

@ -1,6 +1,5 @@
<?xml version="1.0" encoding="utf-8"?> <?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003"> <Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<Import Project="packages\Microsoft.Windows.CppWinRT.2.0.210806.1\build\native\Microsoft.Windows.CppWinRT.props" Condition="Exists('packages\Microsoft.Windows.CppWinRT.2.0.210806.1\build\native\Microsoft.Windows.CppWinRT.props')" />
<ItemGroup Label="ProjectConfigurations"> <ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32"> <ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration> <Configuration>Debug</Configuration>
@ -90,6 +89,8 @@
<PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions> <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode> <ConformanceMode>true</ConformanceMode>
<LanguageStandard>stdcpp20</LanguageStandard> <LanguageStandard>stdcpp20</LanguageStandard>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
<EnableParallelCodeGeneration>true</EnableParallelCodeGeneration>
</ClCompile> </ClCompile>
<Link> <Link>
<SubSystem>Console</SubSystem> <SubSystem>Console</SubSystem>
@ -105,6 +106,8 @@
<PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions> <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode> <ConformanceMode>true</ConformanceMode>
<LanguageStandard>stdcpp20</LanguageStandard> <LanguageStandard>stdcpp20</LanguageStandard>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
<EnableParallelCodeGeneration>true</EnableParallelCodeGeneration>
</ClCompile> </ClCompile>
<Link> <Link>
<SubSystem>Console</SubSystem> <SubSystem>Console</SubSystem>
@ -120,6 +123,8 @@
<PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions> <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode> <ConformanceMode>true</ConformanceMode>
<LanguageStandard>stdcpp20</LanguageStandard> <LanguageStandard>stdcpp20</LanguageStandard>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
<EnableParallelCodeGeneration>true</EnableParallelCodeGeneration>
</ClCompile> </ClCompile>
<Link> <Link>
<SubSystem>Console</SubSystem> <SubSystem>Console</SubSystem>
@ -135,6 +140,8 @@
<PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions> <PreprocessorDefinitions>_CRT_SECURE_NO_WARNINGS;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
<ConformanceMode>true</ConformanceMode> <ConformanceMode>true</ConformanceMode>
<LanguageStandard>stdcpp20</LanguageStandard> <LanguageStandard>stdcpp20</LanguageStandard>
<MultiProcessorCompilation>true</MultiProcessorCompilation>
<EnableParallelCodeGeneration>true</EnableParallelCodeGeneration>
</ClCompile> </ClCompile>
<Link> <Link>
<SubSystem>Console</SubSystem> <SubSystem>Console</SubSystem>
@ -144,35 +151,34 @@
</Link> </Link>
</ItemDefinitionGroup> </ItemDefinitionGroup>
<ItemGroup> <ItemGroup>
<Media Include="..\out.cpp" />
<ClCompile Include="table.cpp" /> <ClCompile Include="table.cpp" />
<ClCompile Include="server.cpp" /> <ClCompile Include="server.cpp" />
<ClCompile Include="types.cpp" /> <ClCompile Include="types.cpp" />
<ClCompile Include="utils.cpp" />
<ClCompile Include="vector_type.cpp" /> <ClCompile Include="vector_type.cpp" />
<ClCompile Include="winhelper.cpp" /> <ClCompile Include="winhelper.cpp" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>
<ClInclude Include="csv.h" /> <None Include="cpp.hint" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="..\csv.h" />
<ClInclude Include="aggregations.h" />
<ClInclude Include="gc.hpp" /> <ClInclude Include="gc.hpp" />
<ClInclude Include="table.h" /> <ClInclude Include="hasher.h" />
<ClInclude Include="libaquery.h" /> <ClInclude Include="libaquery.h" />
<ClInclude Include="table.h" />
<ClInclude Include="types.h" /> <ClInclude Include="types.h" />
<ClInclude Include="utils.h" /> <ClInclude Include="utils.h" />
<ClInclude Include="vector_type.hpp" /> <ClInclude Include="vector_type.hpp" />
<ClInclude Include="winhelper.h" /> <ClInclude Include="winhelper.h" />
</ItemGroup> </ItemGroup>
<ItemGroup>
<None Include="cpp.hint" />
<None Include="packages.config" />
</ItemGroup>
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" /> <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
<ImportGroup Label="ExtensionTargets"> <ImportGroup Label="ExtensionTargets" />
<Import Project="packages\Microsoft.Windows.CppWinRT.2.0.210806.1\build\native\Microsoft.Windows.CppWinRT.targets" Condition="Exists('packages\Microsoft.Windows.CppWinRT.2.0.210806.1\build\native\Microsoft.Windows.CppWinRT.targets')" />
</ImportGroup>
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild"> <Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
<PropertyGroup> <PropertyGroup>
<ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText> <ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
</PropertyGroup> </PropertyGroup>
<Error Condition="!Exists('packages\Microsoft.Windows.CppWinRT.2.0.210806.1\build\native\Microsoft.Windows.CppWinRT.props')" Text="$([System.String]::Format('$(ErrorText)', 'packages\Microsoft.Windows.CppWinRT.2.0.210806.1\build\native\Microsoft.Windows.CppWinRT.props'))" />
<Error Condition="!Exists('packages\Microsoft.Windows.CppWinRT.2.0.210806.1\build\native\Microsoft.Windows.CppWinRT.targets')" Text="$([System.String]::Format('$(ErrorText)', 'packages\Microsoft.Windows.CppWinRT.2.0.210806.1\build\native\Microsoft.Windows.CppWinRT.targets'))" />
</Target> </Target>
</Project> </Project>

@ -3,7 +3,7 @@
#include "types.h" #include "types.h"
#include "vector_type.hpp" #include "vector_type.hpp"
#include <iostream>
template <typename T> template <typename T>
class vector_type; class vector_type;
template <> template <>
@ -18,22 +18,64 @@ namespace types {
struct Coercion; struct Coercion;
} }
#endif #endif
template<typename _Ty>
class ColView;
template<typename _Ty> template<typename _Ty>
class ColRef : public vector_type<_Ty> class ColRef : public vector_type<_Ty>
{ {
public: public:
const char* name; const char* name;
types::Type_t ty = types::ERROR; types::Type_t ty = types::ERROR;
ColRef(uint32_t size, const char* name = "") : vector_type<_Ty>(size), name(name) {} ColRef(const uint32_t& size, const char* name = "") : vector_type<_Ty>(size), name(name) {}
ColRef(const char* name) : name(name) {} ColRef(const char* name) : name(name) {}
void init() { ty = types::Types<_Ty>::getType(); this->size = this->capacity = 0; this->container = 0; } void init() { ty = types::Types<_Ty>::getType(); this->size = this->capacity = 0; this->container = 0; }
ColRef(const char* name, types::Type_t ty) : name(name), ty(ty) {} ColRef(const char* name, types::Type_t ty) : name(name), ty(ty) {}
using vector_type<_Ty>::operator[];
using vector_type<_Ty>::operator=;
ColView<_Ty> operator [](const vector_type<uint32_t>&idxs) const {
return ColView<_Ty>(*this, idxs);
}
template<typename T> template<typename T>
ColRef<T> scast(); ColRef<T> scast();
}; };
template<typename _Ty>
class ColView {
public:
const vector_type<uint32_t>& idxs;
const ColRef<_Ty>& orig;
const uint32_t& size;
ColView(const ColRef<_Ty>& orig, const vector_type<uint32_t>& idxs) : orig(orig), idxs(idxs), size(idxs.size) {}
ColView(const ColView<_Ty>& orig, const vector_type<uint32_t>& idxs) : orig(orig.orig), idxs(idxs), size(idxs.size) {
for (uint32_t i = 0; i < size; ++i)
idxs[i] = orig.idxs[idxs[i]];
}
_Ty& operator [](const uint32_t& i) const {
return orig[idxs[i]];
}
struct Iterator_t {
const uint32_t* val;
const ColRef<_Ty>& orig;
constexpr Iterator_t(const uint32_t* val, const ColRef<_Ty>& orig) noexcept : val(val), orig(orig) {}
_Ty& operator*() { return orig[*val]; }
bool operator != (const Iterator_t& rhs) { return rhs.val != val; }
Iterator_t& operator++ () {
++val;
return *this;
}
Iterator_t operator++ (int) {
Iterator_t tmp = *this;
++val;
return tmp;
}
};
Iterator_t begin() const {
return Iterator_t(idxs.begin(), orig);
}
Iterator_t end() const {
return Iterator_t(idxs.end(), orig);
}
};
template<typename _Ty> template<typename _Ty>
template<typename T> template<typename T>
@ -47,10 +89,27 @@ template<class ...Types>
struct TableInfo { struct TableInfo {
const char* name; const char* name;
ColRef<void>* colrefs; ColRef<void>* colrefs;
uint32_t n_cols, n_rows; uint32_t n_cols;
//void print(); void print(const char* __restrict sep, const char* __restrict end) const;
typedef std::tuple<Types...> tuple_type;
template <size_t j = 0>
typename std::enable_if<j == sizeof...(Types) - 1, void>::type print_impl(const uint32_t& i, const char* __restrict sep = " ") const;
template <size_t j = 0>
typename std::enable_if<j < sizeof...(Types) - 1, void>::type print_impl(const uint32_t& i, const char* __restrict sep = " ") const;
template <size_t ...Idxs>
struct GetTypes {
typedef typename std::tuple<typename std::tuple_element<Idxs, tuple_type>::type ...> type;
};
template <size_t ...Idxs>
using getRecordType = typename GetTypes<Idxs...>::type;
TableInfo(const char* name, uint32_t n_cols); TableInfo(const char* name, uint32_t n_cols);
}; };
template <size_t _Index, class... _Types>
constexpr inline ColRef<std::tuple_element_t<_Index, std::tuple<_Types...>>>& get(const TableInfo<_Types...>& table) noexcept {
return *(ColRef<std::tuple_element_t<_Index, std::tuple<_Types...>>> *) & (table.colrefs[_Index]);
}
template <class T> template <class T>
constexpr static inline bool is_vector(const ColRef<T>&) { constexpr static inline bool is_vector(const ColRef<T>&) {
return true; return true;
@ -68,100 +127,110 @@ TableInfo<Types...>::TableInfo(const char* name, uint32_t n_cols) : name(name),
this->colrefs = (ColRef<void>*)malloc(sizeof(ColRef<void>) * n_cols); this->colrefs = (ColRef<void>*)malloc(sizeof(ColRef<void>) * n_cols);
} }
template <class T1, class T2> template <class ...Types>
ColRef<typename types::Coercion<T1, T2>::type> operator -(const ColRef<T1>& lhs, const ColRef<T2>& rhs) { template <size_t j>
auto ret = ColRef<typename types::Coercion<T1, T2>::type>(lhs.size, ""); inline typename std::enable_if<j == sizeof...(Types) - 1, void>::type
TableInfo<Types ...>::print_impl(const uint32_t& i, const char* __restrict sep) const {
std::cout << (get<j>(*this))[i];
}
template<class ...Types>
template<size_t j>
inline typename std::enable_if<j < sizeof...(Types) - 1, void>::type
TableInfo<Types...>::print_impl(const uint32_t& i, const char* __restrict sep) const
{
std::cout << (get<j>(*this))[i] << sep;
print_impl<j+1>(i, sep);
}
template<class ...Types>
inline void TableInfo<Types...>::print(const char* __restrict sep, const char* __restrict end) const {
int n_rows = 0;
if (n_cols > 0 && colrefs[0].size > 0)
n_rows = colrefs[0].size;
for (int i = 0; i < n_rows; ++i) {
print_impl(i);
std::cout << end;
}
}
template <class T1, class T2, template<typename ...> class VT>
VT<typename types::Coercion<T1, T2>::type> operator -(const VT<T1>& lhs, const VT<T2>& rhs) {
auto ret = VT<typename types::Coercion<T1, T2>::type>(lhs.size, "");
for (int i = 0; i < lhs.size; ++i) for (int i = 0; i < lhs.size; ++i)
ret.container[i] = lhs.container[i] - rhs.container[i]; ret.container[i] = lhs.container[i] - rhs.container[i];
return ret; return ret;
} }
template <class T1, class T2> template <class T1, class T2, template<typename ...> class VT>
ColRef<typename types::Coercion<T1, T2>::type> operator -(const ColRef<T1>& lhs, const T2& rhs) { VT<typename types::Coercion<T1, T2>::type> operator -(const VT<T1>& lhs, const T2& rhs) {
auto ret = ColRef<typename types::Coercion<T1, T2>::type>(lhs.size, ""); auto ret = VT<typename types::Coercion<T1, T2>::type>(lhs.size, "");
for (int i = 0; i < lhs.size; ++i) for (int i = 0; i < lhs.size; ++i)
ret.container[i] = lhs.container[i] - rhs; ret.container[i] = lhs.container[i] - rhs;
return ret; return ret;
} }
template <class T1, class T2, template<typename ...> class VT>
template <size_t _Index, class... _Types> VT<typename types::Coercion<T1, T2>::type> operator +(const VT<T1>& lhs, const VT<T2>& rhs) {
constexpr ColRef<std::tuple_element_t<_Index, std::tuple<_Types...>>>& get(TableInfo<_Types...>& table) noexcept { auto ret = VT<typename types::Coercion<T1, T2>::type>(lhs.size, "");
return *(ColRef< std::tuple_element_t<_Index, std::tuple<_Types...>>> *) &(table.colrefs[_Index]); for (int i = 0; i < lhs.size; ++i)
} ret.container[i] = lhs.container[i] + rhs.container[i];
//void TableInfo::print()
//{
// for (int i = 0; i < n_rows; ++i) {
// for (int j = 0; j < n_cols; ++j) // TODO: Deal with date/time
// printf(types::printf_str[colrefs[j].ty], colrefs[j].get(i, colrefs[j].ty));
// puts("");
// }
//}
template<class T>
ColRef<T> mins(const ColRef<T>& arr) {
const int& len = arr.size;
std::deque<std::pair<T, uint32_t>> cache;
ColRef<T> ret(len);
T min = std::numeric_limits<T>::max();
for (int i = 0; i < len; ++i) {
if (arr[i] < min)
min = arr[i];
ret[i] = min;
}
return ret; return ret;
} }
template<class T> template <class T1, class T2, template<typename ...> class VT>
ColRef<T> maxs(const ColRef<T>& arr) { VT<typename types::Coercion<T1, T2>::type> operator +(const VT<T1>& lhs, const T2& rhs) {
const int& len = arr.size; auto ret = VT<typename types::Coercion<T1, T2>::type>(lhs.size, "");
ColRef<T> ret(len); for (int i = 0; i < lhs.size; ++i)
T max = std::numeric_limits<T>::min(); ret.container[i] = lhs.container[i] + rhs;
for (int i = 0; i < len; ++i) {
if (arr[i] > max)
max = arr[i];
ret[i] = max;
}
return ret; return ret;
} }
template <class T1, class T2, template<typename ...> class VT>
template<class T> VT<typename types::Coercion<T1, T2>::type> operator *(const VT<T1>& lhs, const VT<T2>& rhs) {
ColRef<T> minw(const ColRef<T>& arr, uint32_t w) { auto ret = VT<typename types::Coercion<T1, T2>::type>(lhs.size, "");
const int& len = arr.size; for (int i = 0; i < lhs.size; ++i)
ColRef<T> ret(len); ret.container[i] = lhs.container[i] * rhs.container[i];
std::deque<std::pair<T, uint32_t>> cache; return ret;
for (int i = 0; i < len; ++i) {
if (!cache.empty() && cache.front().second == i - w) cache.pop_front();
while (!cache.empty() && cache.back().first > arr[i]) cache.pop_back();
cache.push_back({ arr[i], i });
ret[i] = cache.front().first;
} }
template <class T1, class T2, template<typename ...> class VT>
VT<typename types::Coercion<T1, T2>::type> operator *(const VT<T1>& lhs, const T2& rhs) {
auto ret = VT<typename types::Coercion<T1, T2>::type>(lhs.size, "");
for (int i = 0; i < lhs.size; ++i)
ret.container[i] = lhs.container[i] * rhs;
return ret; return ret;
} }
template <class T1, class T2, template<typename ...> class VT>
template<class T> VT<typename types::Coercion<T1, T2>::type> operator /(const VT<T1>& lhs, const VT<T2>& rhs) {
ColRef<T> maxw(const ColRef<T>& arr, uint32_t w) { auto ret = VT<typename types::Coercion<T1, T2>::type>(lhs.size, "");
const int& len = arr.size; for (int i = 0; i < lhs.size; ++i)
ColRef<T> ret(len); ret.container[i] = lhs.container[i] / rhs.container[i];
std::deque<std::pair<T, uint32_t>> cache; return ret;
for (int i = 0; i < len; ++i) {
if (!cache.empty() && cache.front().second == i - w) cache.pop_front();
while (!cache.empty() && cache.back().first > arr[i]) cache.pop_back();
cache.push_back({ arr[i], i });
arr[i] = cache.front().first;
} }
template <class T1, class T2, template<typename ...> class VT>
VT<typename types::Coercion<T1, T2>::type> operator /(const VT<T1>& lhs, const T2& rhs) {
auto ret = VT<typename types::Coercion<T1, T2>::type>(lhs.size, "");
for (int i = 0; i < lhs.size; ++i)
ret.container[i] = lhs.container[i] / rhs;
return ret; return ret;
} }
template <class ...Types>
void print(const TableInfo<Types...>& v, const char* delimiter = " ", const char* endline = "\n") {
v.print(delimiter, endline);
}
template <class T> template <class T>
void print(const T& v) { void print(const T& v, const char* delimiter = " ") {
printf(types::printf_str[types::Types<T>::getType()], v); printf(types::printf_str[types::Types<T>::getType()], v);
} }
template <class T> template <class T>
void print(const ColRef<T>& v, const char* delimiter) { void inline print_impl(const T& v, const char* delimiter, const char* endline) {
for (const auto& vi : v) { for (const auto& vi : v) {
print(vi); print(vi);
puts(delimiter); printf("%s", delimiter);
} }
printf("%s", endline);
} }
template <class T, template<typename> class VT>
typename std::enable_if<!std::is_same<VT<T>, TableInfo<T>>::value>::type
print(const VT<T>& v, const char* delimiter = " ", const char* endline = "\n") {
print_impl(v, delimiter, endline);
}
#endif #endif

@ -5,6 +5,7 @@
#include <cstdint> #include <cstdint>
#include <type_traits> #include <type_traits>
#include <string> #include <string>
#ifdef _MSC_VER #ifdef _MSC_VER
#define __restrict__ __restrict #define __restrict__ __restrict
#endif #endif
@ -72,13 +73,51 @@ namespace types {
}; };
#define __Eq(x) (sizeof(T) == sizeof(x)) #define __Eq(x) (sizeof(T) == sizeof(x))
template<class T> template<class T>
struct GetFPType { struct GetFPTypeImpl {
using type = Cond(__Eq(float), float, Cond(__Eq(double), double, long double)); using type = Cond(__Eq(float), float, Cond(__Eq(double), double, long double));
}; };
template<class T> template<class T>
struct GetLongType using GetFPType = typename GetFPTypeImpl<T>::type;
{ template<class T>
struct GetLongTypeImpl {
using type = Cond(_U(T), unsigned long long, Cond(Fp(T), long double, long long)); using type = Cond(_U(T), unsigned long long, Cond(Fp(T), long double, long long));
}; };
template<class T>
using GetLongType = typename GetLongTypeImpl<T>::type;
} }
#define getT(i, t) std::tuple_element_t<i, std::tuple<t...>>
template <template<typename ...> class T, typename ...Types>
struct applyTemplates {
using type = T<Types...>;
};
template <class lT, template <typename ...> class rT>
struct transTypes_s;
template <template<typename ...> class lT, typename ...T, template<typename ...> class rT>
struct transTypes_s<lT<T...>, rT> {
using type = rT<T...>;
};
// static_assert(std::is_same<transTypes<std::tuple<int, float>, std::unordered_map>, std::unordered_map<int, float>>::value);
template <class lT, template <typename ...> class rT>
using transTypes = typename transTypes_s<lT, rT>::type;
template <class ...Types>
struct record_types {};
template <class ...Types>
using record = std::tuple<Types...>;
template <class T>
struct decayS {
using type = typename std::decay<T>::type;
};
template<template<typename ...> class T, typename ...Types>
struct decayS <T<Types...>>{
using type = T<typename std::decay<Types>::type ...>;
};
template <class T>
using decays = typename decayS<T>::type;
#endif // !_TYPES_H #endif // !_TYPES_H

@ -0,0 +1,18 @@
#include "utils.h"
#include <random>
#include <chrono>
using std::string;
string base62uuid(int l = 8) {
using namespace std;
constexpr static const char* base62alp = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
static mt19937_64 engine(chrono::system_clock::now().time_since_epoch().count());
static uniform_int_distribution<uint64_t> u(0x10000, 0xfffff);
uint64_t uuid = (u(engine) << 32ull) + (chrono::system_clock::now().time_since_epoch().count() & 0xffffffff);
printf("%llx\n", uuid);
string ret;
while (uuid && l-- >= 0) {
ret = string("") + base62alp[uuid % 62] + ret;
uuid /= 62;
}
return ret;
}

@ -1,18 +1,18 @@
#pragma once #pragma once
#include <string> #include <string>
#include <ctime> #include <ctime>
#include <random>
using std::string; template<int cnt, int begin = 0, int interval = 1>
string base62uuid(int l = 8) { struct const_range {
constexpr static const char* base62alp = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"; int arr[cnt];
static mt19937_64 engine(chrono::system_clock::now().time_since_epoch().count()); constexpr const_range() {
static uniform_int_distribution<uint64_t> u(0x100000000, 0xfffffffff); for (int i = begin, n = 0; n < cnt; ++n, i += interval)
uint64_t uuid = (u(engine)<<16ull) + (time(0)&0xffff); arr[n] = i;
printf("%lx\n", uuid);
string ret;
while (uuid && l-- >= 0){
ret = string("") + base62alp[uuid % 62] + ret;
uuid /= 62;
} }
return ret; const int* begin() const {
return arr;
} }
const int* end() const {
return arr + cnt;
}
};

@ -41,7 +41,7 @@ public:
_Ty* container; _Ty* container;
uint32_t size, capacity; uint32_t size, capacity;
typedef _Ty* iterator_t; typedef _Ty* iterator_t;
vector_type(uint32_t size) : size(size), capacity(size) { vector_type(const uint32_t& size) : size(size), capacity(size) {
container = (_Ty*)malloc(size * sizeof(_Ty)); container = (_Ty*)malloc(size * sizeof(_Ty));
} }
constexpr vector_type(std::initializer_list<_Ty> _l) { constexpr vector_type(std::initializer_list<_Ty> _l) {
@ -51,13 +51,22 @@ public:
*(_container++) = l; *(_container++) = l;
} }
} }
constexpr vector_type() noexcept {}; constexpr vector_type() noexcept : size(0), capacity(0), container(0) {};
constexpr vector_type(vector_type<_Ty>& vt) noexcept { constexpr vector_type(vector_type<_Ty>& vt) noexcept {
_copy(vt); _copy(vt);
} }
constexpr vector_type(vector_type<_Ty>&& vt) noexcept { constexpr vector_type(vector_type<_Ty>&& vt) noexcept {
_move(std::move(vt)); _move(std::move(vt));
} }
vector_type<_Ty> operator =(const _Ty& vt) {
if (!container) {
container = (_Ty*)malloc(sizeof(_Ty));
capacity = 1;
}
size = 1;
container[0] = vt;
return *this;
}
vector_type<_Ty> operator =(vector_type<_Ty>& vt) { vector_type<_Ty> operator =(vector_type<_Ty>& vt) {
_copy(vt); _copy(vt);
return *this; return *this;
@ -102,7 +111,7 @@ public:
return curr; return curr;
} }
_Ty& operator[](const std::size_t _i) const { _Ty& operator[](const uint32_t _i) const {
return container[_i]; return container[_i];
} }
@ -240,42 +249,11 @@ public:
} }
constexpr vector_type() : size(0), capacity(0), container(0) {}; constexpr vector_type() : size(0), capacity(0), container(0) {};
void *get(uint32_t i, types::Type_t atype){ void *get(uint32_t i, types::Type_t atype){
return static_cast<void*>(static_cast<char*>(container) + (i * types::AType_sizes[atype])); return static_cast<void*>(static_cast<char*>(container) + (i * types::AType_sizes[atype]));
} }
}; void operator[](const uint32_t& i) {
#pragma pack(pop)
// TODO: Specializations for dt/str/none
template<class T>
types::GetLongType<T> sum(const vector_type<T> &v) {
types::GetLongType<T> ret = 0;
for (const auto& _v : v)
ret += _v;
return ret;
}
template<class T>
types::GetFPType<T> avg(const vector_type<T> &v) {
return static_cast<types::GetFPType<T>>(
sum<T>(v)/static_cast<long double>(v.size));
}
template <class T>
T max(const vector_type<T>& v) {
T max_v = std::numeric_limits<T>::min();
for (const auto& _v : v)
max_v = max_v > _v ? max_v : _v;
return max_v;
}
template <class T>
T min(const vector_type<T>& v) {
T min_v = std::numeric_limits<T>::max();
for (const auto& _v : v)
min_v = min_v < _v ? min_v : _v;
return min_v;
} }
};
#endif #endif

@ -9,7 +9,7 @@ void* dlopen(const char* lib, int)
void* dlsym(void* handle, const char* proc) void* dlsym(void* handle, const char* proc)
{ {
return GetProcAddress(static_cast<HMODULE>(handle), proc); return reinterpret_cast<void*>(GetProcAddress(static_cast<HMODULE>(handle), proc));
} }
int dlclose(void* handle) int dlclose(void* handle)

Loading…
Cancel
Save