Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sys-net cannot be started (shuts down very soon) after installation of 4.2.2 and 4.2.3 #9495

Open
sjvudp opened this issue Oct 7, 2024 · 8 comments
Labels
affects-4.2 This issue affects Qubes OS 4.2. C: other hardware support needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists.

Comments

@sjvudp
Copy link

sjvudp commented Oct 7, 2024

How to file a helpful issue

Qubes OS release

4.2.3

Brief summary

I installed Qubes OS 4.2.2 on an external NVME SSD connected via USB (I had used the same device successfully for n older Qubes OS on a different computer). After reboot there was an error displayed.

grafik

Installation finished, but sys-net would shutdown itself rather quickly, and without sys-net no qube can be started. Obviously anything fixable through a downloadable update would not work.

Maybe it has to do with some disk that cannot be mounted, like this:

grafik

Maybe this message causes it, but I don't know:

grafik

I suspect that it might also be CPU-related, as the Laptop (Lenovo ThinkPad P16 Gen2) has an Intel i9-13980HX CPU.
As other Linux shows 32 CPUs, and the machine has 96GB RAM, it would be a nice machine to run Qubes OS, I guess.

Here are some more details:

grafik

After that I connected the NVME device to the other computer that ran Qubes OS before, but there also sys-net won't start, so maybe it's not really a CPU issue.
The only difference was that I had to repartition the NVME for GPT (before it was MBR) as the new laptop does not support "legacy boot" any more.

After that I downloaded the latest 4.2.3 installation medium, installed Qubes OSagain, but the result was the very same: sys-net terminates rather quickly.

Steps to reproduce

Install Qubes OS

Expected behavior

Qubes OS starts after installation

Actual behavior

Qubes OS is unusable after installation.

@sjvudp sjvudp added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists. labels Oct 7, 2024
@andrewdavidwong
Copy link
Member

Duplicate of #9440?

@andrewdavidwong andrewdavidwong added C: other hardware support needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. affects-4.2 This issue affects Qubes OS 4.2. labels Oct 8, 2024
@sjvudp
Copy link
Author

sjvudp commented Oct 8, 2024

Duplicate of #9440?

Well, I read that, but an incorrect system time causing qubes to boot would be a very odd bug. Besides that, the BIOS clock may be off up to 2 hours (UTC vs. CEST).

@marmarek
Copy link
Member

marmarek commented Oct 8, 2024

2h is not a problem; several weeks/months is

@sjvudp
Copy link
Author

sjvudp commented Oct 8, 2024

What type of log (if any) would help to diagnose (and more importantly: fix) the problem?

@sjvudp
Copy link
Author

sjvudp commented Oct 9, 2024

OK, I managed to mount the disk in my openSUSE system, so I can provide some more details.

Inspecting "rpm -qa --last", the oldest package was installed at "Mon Oct 7 20:33:40 202" (that last on "Mon Oct 7 20:39:43 2024"), so the time/date should not be an issue.

Inspecting the journal for unusual messages, I found these:

Oct 07 20:42:47 localhost kernel: pci 10000:e0:06.0: Failed to add - passthrough or MSI/MSI-X might fail!
Oct 07 20:42:47 localhost kernel: ucsi_acpi USBC000:00: UCSI_GET_PDOS failed (-95)
Oct 07 20:42:47 localhost kernel: nouveau 0000:01:00.0: unknown chipset (197000a1)

( I have a AD107GLM [RTX 2000 Ada Generation Laptop GPU] , BTW)

Oct 07 20:42:48 localhost kernel: tmpfs: Unsupported parameter 'huge'
Oct 07 20:43:13 localhost kernel: Bluetooth: hci0: Malformed MSFT vendor event: 0x02
Oct 07 20:43:14 localhost fedora-dmraid-activation[4858]: /lib/systemd/fedora-dmraid-activation: line 5: /etc/init.d/functions: No such file or directory
Oct 07 20:43:14 localhost fedora-dmraid-activation[4860]: /lib/systemd/fedora-dmraid-activation: line 9: strstr: command not found
Oct 07 20:43:14 localhost fedora-dmraid-activation[4863]: Removed "/etc/systemd/system/sysinit.target.wants/dmraid-activation.service".

Oct 07 20:43:20 localhost anaconda[7346]: anaconda: ui.gui.hubs: Initialization controller for hub InitialSetupMainHub expected but missing.

Oct 07 20:47:49 dom0 sudo[8085]: PAM unable to dlopen(/usr/lib64/security/pam_sss.so): /usr/lib64/security/pam_sss.so: cannot open shared object file: No such file or directory
Oct 07 20:47:49 dom0 sudo[8085]: PAM adding faulty module: /usr/lib64/security/pam_sss.so
Oct 07 20:52:09 dom0 kernel: xen-blkback: backend/vbd/1/51712: using 2 queues, protocol 1 (x86_64-abi) persistent grants
Oct 07 20:52:09 dom0 kernel: xen-blkback: backend/vbd/1/51728: using 2 queues, protocol 1 (x86_64-abi) persistent grants
Oct 07 20:52:09 dom0 kernel: xen-blkback: backend/vbd/1/51744: using 2 queues, protocol 1 (x86_64-abi) persistent grants
Oct 07 20:52:09 dom0 kernel: xen-blkback: backend/vbd/1/51760: using 2 queues, protocol 1 (x86_64-abi) persistent grants
Oct 07 20:52:10 dom0 runuser[8953]: pam_unix(runuser:session): session closed for user master
Oct 07 20:52:10 dom0 audit[8953]: USER_END pid=8953 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_unix acct="master" exe="/usr/sbin/runuser" hostname=? addr=? terminal=? res=success'
Oct 07 20:52:10 dom0 audit[8953]: CRED_DISP pid=8953 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam_rootok acct="master" exe="/usr/sbin/runuser" hostname=? addr=? terminal=? res=success'
Oct 07 20:52:10 dom0 qubesd[5000]: vm.debian-12-xfce: Start failed: Cannot connect to qrexec agent for 60 seconds, see /var/log/xen/console/guest-debian-12-xfce.log for details
Oct 07 21:11:35 dom0 kernel: xen-blkback: backend/vbd/5/51712: using 2 queues, protocol 1 (x86_64-abi) persistent grants
Oct 07 21:11:35 dom0 kernel: xen-blkback: backend/vbd/5/51728: using 2 queues, protocol 1 (x86_64-abi) persistent grants
Oct 07 21:11:35 dom0 kernel: xen-blkback: backend/vbd/5/51744: using 2 queues, protocol 1 (x86_64-abi) persistent grants
Oct 07 21:11:35 dom0 kernel: xen-blkback: backend/vbd/5/51760: using 2 queues, protocol 1 (x86_64-abi) persistent grants
Oct 07 21:11:36 dom0 libvirtd[5033]: libvirt version: 8.9.0, package: 7.fc37 (Unknown, 2024-05-01-19:12:11, )
Oct 07 21:11:36 dom0 libvirtd[5033]: hostname: localhost
Oct 07 21:11:36 dom0 libvirtd[5033]: internal error: libxenlight failed to create new domain 'sys-firewall'
Oct 07 21:11:36 dom0 qubesd[5000]: vm.sys-firewall: Start failed: internal error: libxenlight failed to create new domain 'sys-firewall'

Messages in /var/log/libvirt/libxl/libxl-driver.log are:

2024-10-07 19:11:36.011+0000: libxl: libxl_device.c:1200:device_backend_callback: Domain 7:unable to add device with path /local/domain/5/backend/vif/7/0
2024-10-07 19:11:36.011+0000: libxl: libxl_create.c:2000:domcreate_attach_devices: Domain 7:unable to add vif devices

In /var/log/xen/console/guest-sys-net-dm.log I found:
...

[2024-10-07 21:11:32] from-unix: {"QMP": {"version": {"qemu": {"micro": 2, "minor": 1, "major": 8}, "package": ""}, "capabilities": ["oob"]}}
[2024-10-07 21:11:32] 
[2024-10-07 21:11:32] wrote 110 bytes to vchan
[2024-10-07 21:11:32] from-vchan: ^A{"execute":"qmp_capabilities","id":2020372737}
[2024-10-07 21:11:32] 
[2024-10-07 21:11:32] from-unix: {"return": {}, "id": 2020372737}
[2024-10-07 21:11:32] 
[2024-10-07 21:11:32] wrote 34 bytes to vchan
[2024-10-07 21:11:32] from-vchan: {"execute":"device_add","id":2020372736,"arguments":{"driver":"xen-pci-passthrough","id":"pci-pt-00_14.3","hostaddr":"0000:00:14.3"}}
[2024-10-07 21:11:32] 
[2024-10-07 21:11:32] [00:06.0] xen_pt_realize: Assigning real physical device 00:14.3 to devfn 0x30
[2024-10-07 21:11:32] [00:06.0] xen_pt_register_regions: IO region 0 registered (size=0x00004000 base_addr=0x6256634000 type: 0x4)
[2024-10-07 21:11:32] [00:06.0] xen_pt_config_reg_init: Offset 0x0010 mismatch! Emulated=0x0000, host=0x56634004, syncing to 0x56634004.
[2024-10-07 21:11:32] [00:06.0] xen_pt_config_reg_init: Offset 0x0014 mismatch! Emulated=0x0000, host=0x0062, syncing to 0x0062.
[2024-10-07 21:11:32] [00:06.0] xen_pt_config_reg_init: Offset 0x00ca mismatch! Emulated=0x0000, host=0x0023, syncing to 0x0023.
[2024-10-07 21:11:32] [00:06.0] xen_pt_pm_ctrl_reg_init_off: PCI power management control passthrough is off
[2024-10-07 21:11:32] [00:06.0] xen_pt_config_reg_init: Offset 0x00d2 mismatch! Emulated=0x0000, host=0x0080, syncing to 0x0080.
[2024-10-07 21:11:32] [00:06.0] xen_pt_config_reg_init: Offset 0x0044 mismatch! Emulated=0x0000, host=0x10000ec0, syncing to 0x0ec0.
[2024-10-07 21:11:32] [00:06.0] xen_pt_config_reg_init: Offset 0x004a mismatch! Emulated=0x0000, host=0x0010, syncing to 0x0010.
[2024-10-07 21:11:32] [00:06.0] xen_pt_msix_init: get MSI-X table BAR base 0x6256634000
[2024-10-07 21:11:32] [00:06.0] xen_pt_config_reg_init: Offset 0x0082 mismatch! Emulated=0x0000, host=0x000f, syncing to 0x000f.
[2024-10-07 21:11:32] [00:06.0] xen_pt_pci_intx: intx=3
[2024-10-07 21:11:32] [00:06.0] xen_pt_realize: Real physical device 00:14.3 registered successfully
[2024-10-07 21:11:32] 
[2024-10-07 21:11:32] ==== Press enter for shell ====

...

In var/log/xen/console/guest-sys-net.log I found:
...

[2024-10-07 21:11:35] [    0.960624] Run /init as init process
[2024-10-07 21:11:35] Qubes initramfs script here:
[2024-10-07 21:11:35] [    0.964147] Invalid max_queues (4), will use default max: 2.
[2024-10-07 21:11:35] [    0.970856] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-10-07 21:11:35] [    0.973361] blkfront: xvdb: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-10-07 21:11:35] [    0.975636] blkfront: xvdc: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-10-07 21:11:35] [    0.977760] blkfront: xvdd: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-10-07 21:11:35] Waiting for /dev/xvda* devices...
[2024-10-07 21:11:35] Qubes: Doing R/W setup for TemplateVM...
[2024-10-07 21:11:35] [    1.352305]  xvdc: xvdc1 xvdc3
[2024-10-07 21:11:35] Setting up swapspace version 1, size = 1073737728 bytes
[2024-10-07 21:11:35] UUID=28eee439-a52f-4676-aecb-6ff3a8eae73d
[2024-10-07 21:11:35] Qubes: done.
[2024-10-07 21:11:35] mount: mounting /dev/mapper/dmroot on /sysroot failed: Invalid argument
[2024-10-07 21:11:35] Waiting for /dev/xvdd device...
[2024-10-07 21:11:35] [    1.383884] EXT4-fs (xvdd): mounting ext3 file system using the ext4 subsystem
[2024-10-07 21:11:35] [    1.386592] EXT4-fs (xvdd): mounted filesystem 212ddd52-f4ea-4d93-9ad8-d88854e31cc4 ro with ordered data mode. Quota mode: none.
[2024-10-07 21:11:35] mount: mounting none on /sysroot/lib/modules failed: No such file or directory
[2024-10-07 21:11:35] [    1.397507] EXT4-fs (xvdd): unmounting filesystem 212ddd52-f4ea-4d93-9ad8-d88854e31cc4.
[2024-10-07 21:11:35] mount: can't read '/proc/mounts': No such file or directory
[2024-10-07 21:11:35] BusyBox v1.36.0 (2023-01-10 00:00:00 UTC) multi-call binary.
[2024-10-07 21:11:35] 
[2024-10-07 21:11:35] Usage: switch_root [-c CONSOLE_DEV] NEW_ROOT NEW_INIT [ARGS]
[2024-10-07 21:11:35] 
[2024-10-07 21:11:35] Free initramfs and switch to another root fs:
[2024-10-07 21:11:35] chroot to NEW_ROOT, delete all in /, move NEW_ROOT to /,
[2024-10-07 21:11:35] execute NEW_INIT. PID must be 1. NEW_ROOT must be a mountpoint.
[2024-10-07 21:11:35] 
[2024-10-07 21:11:35]   -c DEV  Reopen stdio to DEV after switch
[2024-10-07 21:11:35] [    1.404139] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[2024-10-07 21:11:35] [    1.404160] CPU: 1 PID: 1 Comm: switch_root Not tainted 6.6.48-1.qubes.fc37.x86_64 #1
[2024-10-07 21:11:35] [    1.404179] Hardware name: Xen HVM domU, BIOS 4.17.4 07/20/2024
[2024-10-07 21:11:35] [    1.404195] Call Trace:
[2024-10-07 21:11:35] [    1.404205]  <TASK>
[2024-10-07 21:11:35] [    1.404213]  dump_stack_lvl+0x47/0x60
[2024-10-07 21:11:35] [    1.404229]  panic+0x345/0x360
[2024-10-07 21:11:35] [    1.404243]  do_exit+0x4ea/0x510
[2024-10-07 21:11:35] [    1.404255]  ? do_syscall_64+0x65/0x80
[2024-10-07 21:11:35] [    1.404268]  do_group_exit+0x31/0x80
[2024-10-07 21:11:35] [    1.404279]  __x64_sys_exit_group+0x18/0x20
[2024-10-07 21:11:35] [    1.404289]  do_syscall_64+0x59/0x80
[2024-10-07 21:11:35] [    1.404300]  ? syscall_exit_to_user_mode+0x22/0x40
[2024-10-07 21:11:35] [    1.404316]  ? do_syscall_64+0x65/0x80
[2024-10-07 21:11:35] [    1.404327]  ? do_user_addr_fault+0x323/0x630
[2024-10-07 21:11:35] [    1.404341]  ? exc_page_fault+0x77/0x170
[2024-10-07 21:11:35] [    1.404353]  entry_SYSCALL_64_after_hwframe+0x78/0xe2
[2024-10-07 21:11:35] [    1.404368] RIP: 0033:0x423f5e
[2024-10-07 21:11:35] [    1.404380] Code: c3 12 00 48 8b 35 6a c3 12 00 48 8b 05 7b c3 12 00 e9 7c fe ff ff 66 0f 1f 44 00 00 f3 0f 1e fa 48 63 ff b8 e7 00 00 00 0f 05 <ba> 3c 00 00 00 0f 1f 44 00 00 48 89 d0 0f 05 eb f9 90 41 56 41 55

...

/var/log/xen/qmp-proxy-sys-net-dm.log is empty.

The VG looks like this:

# vgs
  VG         #PV #LV #SN Attr   VSize   VFree 
  qubes_dom0   1  36   0 wz--n- 159.98g 20.00m

The LVs look like this:

# lvs
  LV                                               VG         Attr       LSize   Pool      Origin                                           Data%  Meta%  Move Log Cpy%Sync Convert
  root                                             qubes_dom0 Vwi-aotz--  20.00g root-pool                                                  23.08                                  
  root-pool                                        qubes_dom0 twi-aotz--  20.00g                                                            23.08  18.34                           
  swap                                             qubes_dom0 -wi-a-----   4.00g                                                                                                   
  vm                                               qubes_dom0 Vwi-a-tz-- 135.81g vm-pool                                                    0.75                                   
  vm-anon-whonix-private                           qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-debian-12-xfce-private                        qubes_dom0 Vwi-a-tz--   2.00g vm-pool   vm-debian-12-xfce-private-1728327130-back        0.00                                   
  vm-debian-12-xfce-private-1728327130-back        qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-debian-12-xfce-root                           qubes_dom0 Vwi-a-tz--  20.00g vm-pool   vm-debian-12-xfce-root-1728327130-back           25.50                                  
  vm-debian-12-xfce-root-1728327126-back           qubes_dom0 Vwi-a-tz--  10.00g vm-pool                                                    0.00                                   
  vm-debian-12-xfce-root-1728327130-back           qubes_dom0 Vwi-a-tz--  20.00g vm-pool                                                    25.50                                  
  vm-default-dvm-private                           qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-default-mgmt-dvm-private                      qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-fedora-40-xfce-private                        qubes_dom0 Vwi-a-tz--   2.00g vm-pool   vm-fedora-40-xfce-private-1728327631-back        0.00                                   
  vm-fedora-40-xfce-private-1728327631-back        qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-fedora-40-xfce-root                           qubes_dom0 Vwi-a-tz--  20.00g vm-pool   vm-fedora-40-xfce-root-1728327631-back           25.64                                  
  vm-fedora-40-xfce-root-1728327627-back           qubes_dom0 Vwi-a-tz--  10.00g vm-pool                                                    0.00                                   
  vm-fedora-40-xfce-root-1728327631-back           qubes_dom0 Vwi-a-tz--  20.00g vm-pool                                                    25.64                                  
  vm-personal-private                              qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-pool                                          qubes_dom0 twi-aotz-- 135.81g                                                            11.92  13.91                           
  vm-sys-net-private                               qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-sys-net-private-1728329588-back               qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-sys-whonix-private                            qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-untrusted-private                             qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-vault-private                                 qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-whonix-gateway-17-private                     qubes_dom0 Vwi-a-tz--   2.00g vm-pool   vm-whonix-gateway-17-private-1728327871-back     0.00                                   
  vm-whonix-gateway-17-private-1728327871-back     qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-whonix-gateway-17-root                        qubes_dom0 Vwi-a-tz--  20.00g vm-pool   vm-whonix-gateway-17-root-1728327871-back        9.82                                   
  vm-whonix-gateway-17-root-1728327867-back        qubes_dom0 Vwi-a-tz--  10.00g vm-pool                                                    0.00                                   
  vm-whonix-gateway-17-root-1728327871-back        qubes_dom0 Vwi-a-tz--  20.00g vm-pool                                                    9.82                                   
  vm-whonix-workstation-17-dvm-private             qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-whonix-workstation-17-private                 qubes_dom0 Vwi-a-tz--   2.00g vm-pool   vm-whonix-workstation-17-private-1728328200-back 0.00                                   
  vm-whonix-workstation-17-private-1728328200-back qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   
  vm-whonix-workstation-17-root                    qubes_dom0 Vwi-a-tz--  20.00g vm-pool   vm-whonix-workstation-17-root-1728328200-back    14.91                                  
  vm-whonix-workstation-17-root-1728328196-back    qubes_dom0 Vwi-a-tz--  10.00g vm-pool                                                    0.00                                   
  vm-whonix-workstation-17-root-1728328200-back    qubes_dom0 Vwi-a-tz--  20.00g vm-pool                                                    14.91                                  
  vm-work-private                                  qubes_dom0 Vwi-a-tz--   2.00g vm-pool                                                    0.00                                   

The failure messages for the Debian-12 were like this:
...

2024-10-07 20:52:09] [    0.598369] xenbus_probe_frontend: Device with no driver: device/vbd/51712
[2024-10-07 20:52:09] [    0.598386] xenbus_probe_frontend: Device with no driver: device/vbd/51728
[2024-10-07 20:52:09] [    0.598400] xenbus_probe_frontend: Device with no driver: device/vbd/51744
[2024-10-07 20:52:09] [    0.598414] xenbus_probe_frontend: Device with no driver: device/vbd/51760
[2024-10-07 20:52:09] [    0.598662] hid_bpf: error while preloading HID BPF dispatcher: -22
[2024-10-07 20:52:09] [    0.598669] RAS: Correctable Errors collector initialized.
[2024-10-07 20:52:09] [    0.598719] clk: Disabling unused clocks
[2024-10-07 20:52:09] [    0.600247] Freeing unused decrypted memory: 2028K
[2024-10-07 20:52:09] [    0.601723] Freeing unused kernel image (initmem) memory: 5140K
[2024-10-07 20:52:09] [    0.601741] Write protecting the kernel read-only data: 28672k
[2024-10-07 20:52:09] [    0.602344] Freeing unused kernel image (rodata/data gap) memory: 1420K
[2024-10-07 20:52:09] [    0.602365] Run /init as init process
[2024-10-07 20:52:09] Qubes initramfs script here:
[2024-10-07 20:52:09] [    0.605546] Invalid max_queues (4), will use default max: 2.
[2024-10-07 20:52:09] [    0.611318] blkfront: xvda: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-10-07 20:52:09] [    0.614226] blkfront: xvdb: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-10-07 20:52:09] [    0.616127] blkfront: xvdc: flush diskcache: enabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-10-07 20:52:09] [    0.618229] blkfront: xvdd: barrier or flush: disabled; persistent grants: enabled; indirect descriptors: enabled; bounce buffer: enabled
[2024-10-07 20:52:09] Waiting for /dev/xvda* devices...
[2024-10-07 20:52:09] Qubes: Doing R/W setup for TemplateVM...
[2024-10-07 20:52:09] [    0.987498]  xvdc: xvdc1 xvdc3
[2024-10-07 20:52:09] Setting up swapspace version 1, size = 1073737728 bytes
[2024-10-07 20:52:09] UUID=4f21df50-5882-4562-b052-0967a24220b0
[2024-10-07 20:52:09] Qubes: done.
[2024-10-07 20:52:09] mount: mounting /dev/mapper/dmroot on /sysroot failed: Invalid argument
[2024-10-07 20:52:09] Waiting for /dev/xvdd device...
[2024-10-07 20:52:09] [    1.017700] EXT4-fs (xvdd): mounting ext3 file system using the ext4 subsystem
[2024-10-07 20:52:09] [    1.025302] EXT4-fs (xvdd): mounted filesystem 212ddd52-f4ea-4d93-9ad8-d88854e31cc4 ro with ordered data mode. Quota mode: none.
[2024-10-07 20:52:09] mount: mounting none on /sysroot/lib/modules failed: No such file or directory
[2024-10-07 20:52:09] [    1.034338] EXT4-fs (xvdd): unmounting filesystem 212ddd52-f4ea-4d93-9ad8-d88854e31cc4.
[2024-10-07 20:52:09] mount: can't read '/proc/mounts': No such file or directory
[2024-10-07 20:52:09] BusyBox v1.36.0 (2023-01-10 00:00:00 UTC) multi-call binary.
[2024-10-07 20:52:09] 
[2024-10-07 20:52:09] Usage: switch_root [-c CONSOLE_DEV] NEW_ROOT NEW_INIT [ARGS]
[2024-10-07 20:52:09] 
[2024-10-07 20:52:09] Free initramfs and switch to another root fs:
[2024-10-07 20:52:09] chroot to NEW_ROOT, delete all in /, move NEW_ROOT to /,
[2024-10-07 20:52:09] execute NEW_INIT. PID must be 1. NEW_ROOT must be a mountpoint.
[2024-10-07 20:52:09] 
[2024-10-07 20:52:09]   -c DEV  Reopen stdio to DEV after switch
[2024-10-07 20:52:09] [    1.042803] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[2024-10-07 20:52:09] [    1.042829] CPU: 0 PID: 1 Comm: switch_root Not tainted 6.6.48-1.qubes.fc37.x86_64 #1
[2024-10-07 20:52:09] [    1.042856] Call Trace:
[2024-10-07 20:52:09] [    1.042871]  <TASK>
[2024-10-07 20:52:09] [    1.042882]  dump_stack_lvl+0x47/0x60
[2024-10-07 20:52:09] [    1.042902]  panic+0x345/0x360
[2024-10-07 20:52:09] [    1.042921]  do_exit+0x4ea/0x510
[2024-10-07 20:52:09] [    1.042937]  ? vfs_write+0x23b/0x420
[2024-10-07 20:52:09] [    1.042955]  do_group_exit+0x31/0x80
[2024-10-07 20:52:09] [    1.042970]  __x64_sys_exit_group+0x18/0x20
[2024-10-07 20:52:09] [    1.042985]  do_syscall_64+0x59/0x80
[2024-10-07 20:52:09] [    1.043002]  ? ksys_write+0x6f/0xf0
[2024-10-07 20:52:09] [    1.043017]  ? syscall_exit_to_user_mode+0x22/0x40
[2024-10-07 20:52:09] [    1.043038]  ? do_syscall_64+0x65/0x80
[2024-10-07 20:52:09] [    1.043053]  ? xen_sched_clock+0x15/0x30
[2024-10-07 20:52:09] [    1.043068]  ? sched_clock+0x10/0x30
[2024-10-07 20:52:09] [    1.043084]  ? sched_clock_cpu+0xf/0x190
[2024-10-07 20:52:09] [    1.043100]  ? irqtime_account_irq+0x40/0xc0
[2024-10-07 20:52:09] [    1.043119]  ? __irq_exit_rcu+0x4b/0xd0
[2024-10-07 20:52:09] [    1.043135]  ? sysvec_xen_hvm_callback+0x3e/0x90
[2024-10-07 20:52:09] [    1.043154]  entry_SYSCALL_64_after_hwframe+0x78/0xe2
[2024-10-07 20:52:09] [    1.043174] RIP: 0033:0x423f5e
[2024-10-07 20:52:09] [    1.043189] Code: c3 12 00 48 8b 35 6a c3 12 00 48 8b 05 7b c3 12 00 e9 7c fe ff ff 66 0f 1f 44 00 00 f3 0f 1e fa 48 63 ff b8 e7 00 00 00 0f 05 <ba> 3c 00 00 00 0f 1f 44 00 00 48 89 d0 0f 05 eb f9 90 41 56 41 55

...

@marmarek
Copy link
Member

marmarek commented Oct 9, 2024

[2024-10-07 20:52:09] mount: mounting /dev/mapper/dmroot on /sysroot failed: Invalid argument

Interesting, this looks like the template root filesystem is broken/empty. But on lvs I see:

vm-fedora-40-xfce-root qubes_dom0 Vwi-a-tz-- 20.00g vm-pool vm-fedora-40-xfce-root-1728327631-back 25.64

so, it isn't empty. Could it be maybe #4974 ? What fdisk -l on the dom0 disk says regarding sector size?

Oct 07 20:42:47 localhost kernel: pci 10000:e0:06.0: Failed to add - passthrough or MSI/MSI-X might fail!

The message about MSI/MSI-X is not the interesting part. The device BDF is - 0x10000 is not a valid PCI domain. Is it maybe some built-in Intel RAID (more details) ? It isn't necessarily the source of the problems, but I'd still recommend disabling it in the firmware setup menu.

@sjvudp
Copy link
Author

sjvudp commented Oct 10, 2024

[2024-10-07 20:52:09] mount: mounting /dev/mapper/dmroot on /sysroot failed: Invalid argument

What fdisk -l on the dom0 disk says regarding sector size?

First smartctl outputs:
...

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

...

Fdisk reports for sda (NVME):

Disk /dev/sda: 238.47 GiB, 256060514304 bytes, 500118192 sectors
Disk model: 6GB SSD         
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9CB9F2ED-E3FF-46F5-AFE9-C126267C20F6

Device         Start       End   Sectors  Size Type
/dev/sda1       2048   1050623   1048576  512M EFI System
/dev/sda2    1050624   3147775   2097152    1G Linux filesystem
/dev/sda3    3147776 338692095 335544320  160G Linux reserved
/dev/sda4  338692096 500117503 161425408   77G Linux reserved

And for the Fedora LV:

GPT PMBR size mismatch (41943039 != 5242879) will be corrected by write.
Disk /dev/qubes_dom0/vm-fedora-40-xfce-root: 20 GiB, 21474836480 bytes, 5242880 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device                                  Boot Start     End Sectors Size Id Type
/dev/qubes_dom0/vm-fedora-40-xfce-root1          1 5242879 5242879  20G ee GPT

Partition 1 does not start on physical sector boundary.

Besides the smartctl output the above is from Dom0, and here is what the openSUSE kernel (6.11.1-lp155.2.g3bf25fe-default) says for sda:

Oct 10 21:21:13 localhost kernel: sd 0:0:0:0: [sda] Spinning up disk...
Oct 10 21:21:14 localhost kernel: sd 0:0:0:0: [sda] 500118192 512-byte logical blocks: (256 GB/238 GiB)
Oct 10 21:21:14 localhost kernel: sd 0:0:0:0: [sda] 4096-byte physical blocks
Oct 10 21:21:14 localhost kernel: sd 0:0:0:0: [sda] Write Protect is off
Oct 10 21:21:14 localhost kernel: sd 0:0:0:0: [sda] Mode Sense: 5f 00 00 08
Oct 10 21:21:14 localhost kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 10 21:21:14 localhost kernel: sd 0:0:0:0: [sda] Preferred minimum I/O size 4096 bytes
Oct 10 21:21:14 localhost kernel: sd 0:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of preferred minimum block size (4096 bytes)

And just in case: The NVME module is in an Icy Box "IB-1817Ma-C31" using a Jmicron USB bridge (ID 152d:0562).

@marmarek
Copy link
Member

GPT PMBR size mismatch (41943039 != 5242879) will be corrected by write.
Disk /dev/qubes_dom0/vm-fedora-40-xfce-root: 20 GiB, 21474836480 bytes, 5242880 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes

Yes, this looks exactly like #4974

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-4.2 This issue affects Qubes OS 4.2. C: other hardware support needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists.
Projects
None yet
Development

No branches or pull requests

3 participants