Macro benchmarks currently focus only on query runtimes, in addition to that, we should track and measure:
- Memory usage - could be either using DHAT or jemalloc
- IO (latency and bytes read)
- Separate hot and cold runs, purging OS caches in between queries and measure tracking each individually.
- Hardware setup (machine, disk, os, etc) to make it easier to analyze historical data or compare the benchmarks between hardware setups
- potentially perf counters? instruction counters?
Macro benchmarks currently focus only on query runtimes, in addition to that, we should track and measure: