[BUG][ARL] no FW_READY at boot - ROM status 0x50000005, ROM error 0x0

[kv2019i]

Describe the bug
FW build from latest SOF main does not boot on ARL system.

Last know good commit 98a20ce (tag: releases/mtl/v3.1.1, origin/mtl-003-drop-stable)

No FW_READY message is receive by kernel, FW state is FW_ENTERED, so ROM
has completed.

To Reproduce
Build firmware and boot system.

Reproduction Rate
100%

Expected behavior
FW should boot.

Impact
Showstopper

Environment

1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
   • Kernel: -
   • SOF: v2.7.1, main, latest mtl-006-drop-stable
2. Name of the topology file
   • Topology: sof-hda-generic.tplg
3. Name of the platform(s) on which the bug is observed.
   • Platform: arl

Logs

[ 3863.060288] snd_sof_intel_hda_common:mtl_dsp_pre_fw_run: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1000]=0x1010000 successful
[ 3863.061636] snd_sof_intel_hda_common:mtl_dsp_pre_fw_run: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1d1c]=0x8151 successful
[ 3863.061957] snd_sof_intel_hda_common:hda_dsp_stream_hw_params: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1c0]=0x40000 successful
[ 3863.061994] snd_sof_intel_hda_common:hda_dsp_stream_hw_params: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1c0]=0x40000 successful
[ 3863.062005] snd_sof_intel_hda_common:hda_dsp_stream_setup_bdl: sof-audio-pci-intel-mtl 0000:80:1f.3: period_bytes:0x0
[ 3863.062007] snd_sof_intel_hda_common:hda_dsp_stream_setup_bdl: sof-audio-pci-intel-mtl 0000:80:1f.3: periods:1
[ 3863.062077] snd_sof_intel_hda_common:hda_dsp_cl_boot_firmware: sof-audio-pci-intel-mtl 0000:80:1f.3: Attempting iteration 0 of Core En/ROM load...
[ 3863.063409] snd_sof_intel_hda_common:mtl_dsp_core_power_up: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x178d04]=0x2000101 successful
[ 3863.063415] snd_sof_intel_hda_common:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:80:1f.3: Primary core power up successful
[ 3863.063428] snd_sof_intel_hda_common:mtl_dsp_cl_init: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x73214]=0x80000000 successful
[ 3863.063444] snd_sof_intel_hda_common:mtl_enable_interrupts: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1800]=0x41 successful
[ 3863.063457] snd_sof_intel_hda_common:mtl_enable_interrupts: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1140]=0x1 successful
[ 3863.348296] snd_sof_intel_hda_common:hda_cl_copy_fw: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x180000]=0x5 successful
[ 3863.348313] snd_sof_intel_hda_common:hda_dsp_stream_trigger: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1c0]=0x140000 successful
[ 3863.348321] snd_sof_intel_hda_common:hda_dsp_cl_boot_firmware: sof-audio-pci-intel-mtl 0000:80:1f.3: Firmware download successful, booting...
[ 3865.359486] sof-audio-pci-intel-mtl 0000:80:1f.3: ------------[ DSP dump start ]------------
[ 3865.368023] sof-audio-pci-intel-mtl 0000:80:1f.3: Firmware boot failure due to timeout
[ 3865.375997] sof-audio-pci-intel-mtl 0000:80:1f.3: fw_state: SOF_FW_BOOT_IN_PROGRESS (3)
[ 3865.384202] sof-audio-pci-intel-mtl 0000:80:1f.3: ROM status: 0x5, ROM error: 0x0
[ 3865.391742] sof-audio-pci-intel-mtl 0000:80:1f.3: ROM debug status: 0x50000005, ROM debug error: 0x0
[ 3865.401054] sof-audio-pci-intel-mtl 0000:80:1f.3: ROM feature bit enabled
[ 3865.407895] sof-audio-pci-intel-mtl 0000:80:1f.3: ------------[ DSP dump end ]------------
[ 3865.416219] sof-audio-pci-intel-mtl 0000:80:1f.3: error: failed to boot DSP firmware -5
[ 3865.424834] snd_sof_intel_hda_common:mtl_enable_sdw_irq: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x114c]=0x0 successful
[ 3865.426263] snd_sof_intel_hda_common:mtl_dsp_core_power_down: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x178d04]=0x2000000 successful
[ 3865.427448] snd_sof_intel_hda_common:mtl_power_down_dsp: sof-audio-pci-intel-mtl 0000:80:1f.3: FW Poll Status: reg[0x1000]=0x0 successful
[ 3865.428042] sof-audio-pci-intel-mtl 0000:80:1f.3: error: sof_probe_work failed err: -5

[kv2019i]

FYI @arunt1 @mengdonglin @ujfalusi

[lgirdwood]

Can we extend the boot timeout and check the status, i.e. are we running on WOVCLK instead of HPCLK
adding @abonislawski @mwasko

[kv2019i]

First bisect results:

• v2.4 still work
• v2.5 fails to boot

[kv2019i]

Closing as invalid, incorrect configuration targetted.

[kv2019i]

Reopening, the Zephyr soc code has support for dynamic lookup of the core count, so the mtl built FW is expected to boot.

[abonislawski]

Worth to check WOVCRO commits on 006 zephyr