Linux performance observation tools map: http://qqibrow.github.io/performance-profiling-with-systemtap/

perf

PMU = performance monitor unit, each of the CPU vendors such as Arm, Intel, and AMD have detailed technical reference manual for the available PMU events on their platform

This perf instruction sudo perf top -p [pid] shows a list of functions with CPU usage percentage but I probably need to compare cycles spent in the two likely bottleneck locations to see if one is crazy high.

Specify the frequency and delay: perf top -F100 -d10 -p$(pgrep -d, core-cpu1)

Links

https://perf.wiki.kernel.org/index.php/Tutorial

valgrind

By default, Massif uses "instructions executed" as the unit of time, so the time column shows time(i) but you can change the units to bytes with --time-unit=B (or to real time with ms).

Stack profiling can be turned on with the --stacks=yes option.