rr should open all PMUs

[pcc]

google/benchmark#2123

rr is affected by the same issue and should open all PMUs (or at least the PMU for the pinned CPU).

Symptom:

[FATAL src/PerfCounters.cc:519:check_working_counters() errno: EDOM] 
Got 0 branch events, expected at least 500.

The hardware performance counter seems to not be working on cpu 13
Check that hardware performance counters are working by running
  perf stat -e apple_blizzard_pmu/r90/ true
and checking that it reports a nonzero number of events.
If performance counters seem to be working with 'perf', file an
rr issue, otherwise check your hardware/OS/VM configuration. Also
check that other software is not using performance counters on
this CPU.
=== Start rr backtrace:
rr() [0x4bb430]
rr() [0x4bb7ac]
rr(_ZN2rr12PerfCounters5startEPNS_4TaskEl+0x80c) [0x4e8b98]
rr(_ZN2rr4Task16resume_executionENS_13ResumeRequestENS_11WaitRequestENS_12TicksRequestEi+0x5ac) [0x5b91ac]
rr(_ZN2rr13RecordSession11record_stepEv+0x3b4) [0x50d120]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0xd44) [0x4fbbf0]
rr(main+0x170) [0x431d10]
/lib64/libc.so.6(+0x2621c) [0xffff6107621c]
/lib64/libc.so.6(__libc_start_main+0x9c) [0xffff610762fc]
rr(_start+0x30) [0x433070]
=== End rr backtrace

[khuey]

Is this specific to Apple's PMUs or is this a general Linux thing?

[pcc]

My understanding is that this can happen on any system where there are multiple CPU PMUs, not just Apple's. It might even be possible on x86 as some systems have started to implement heterogeneous cores. @captain5050 to confirm.

[yuyichao]

The intention of the code was to open the PMU for the core the thread is pinned to. Maybe that breaks?

Otoh, I feel like the PMU for the E cores simply stops working anymore and I’m 80% I’ve verified it with perf directly…

[yuyichao]

Sorry I misspoke, the perf counter is working but rr isn't on the E core anymore. I've verified that one of the old commit I had left arund 7428b21 works just fine but master (b32fc69) doesn't anymore.

I may or may not have time in the next few days to bisect.

[captain5050]

My understanding is that this can happen on any system where there are multiple CPU PMUs, not just Apple's. It might even be possible on x86 as some systems have started to implement heterogeneous cores. @captain5050 to confirm.

On x86 the p-core is /sys/bus/event_source/devices/cpu_core with a cpus file that lists which CPUs the events work on. The e_core is cpu_atom and you may have a cpu_lowpower. Pinning a thread to a core and opening the event for that core should avoid errors.

[yuyichao]

On x86 the p-core is /sys/bus/event_source/devices/cpu_core with a cpus file that lists which CPUs the events work on. The e_core is cpu_atom and you may have a cpu_lowpower. Pinning a thread to a core and opening the event for that core should avoid errors.

Aarch64 also has interface for similar info and it was certainly working before.

Bisected to 739c0d9 . Given that it's a commit for x86, I suspect that it might accidentally broke arm and x86 might be fine (I don't have hardware to test x86 hybrid system).