Page MenuHomeSolus

Stuck in emergency mode after regular Solus reboot
Closed, ResolvedPublic

Description

My Solus ship stranded today. After a reboot it enters emergency mode which I have not been able to exit. I went to the Solus Help Center and looked up Boot Rescue and found the exact same issue someone else described on StackExchange.

I remember having lots of browser tabs open and when trying to save a file it didn't work. I then decided to reboot to free the system from too much used memory and then the system got locked up in emergency mode. I ran journalctl and it appears that fsck failed with exit status 4. The issue is virtually identical to the one described on StackExchange.

How can I get my system up and running again?

Event Timeline

Check if the UUID changed of your disk to what is inside the fsck file

feuerstein added a comment.EditedDec 5 2018, 9:44 AM

@Girtablulu How would I do this? I use Solus because it's easy to use but never had to deal with the emergency mode before.

  • Chroot into your system via this guide Link
  • inside the chrooted system and run
    • df -h (see which /dev/sdx your root disk is)
    • blkid /dev/sdx
    • cat /etc/fstab (and check if the numbers match)
    • if not, change the number via nano /etc/fstab

reboot

feuerstein added a comment.EditedDec 6 2018, 1:04 PM

@Girtablulu Thank you for trying to help me with this but I already cannot pass the first issue as I don't understand the instructions for boot rescue given in the Help Center. All commands I entered are not working in emergency mode but most likely I'm not sure with which one to proceed. For example I started with sudo su but it says: command not found.

You could not mount your partition via a LiveISO?

Girtablulu triaged this task as Needs More Info priority.Dec 7 2018, 11:27 PM
Girtablulu edited projects, added Software; removed Lacks Project.

@Girtablulu I'm afraid that I will lose all my data if I run a LiveISO. Could you walk me through the steps I have to take? I can have a LiveISO ready tomorrow.

You won't loose anything if you boot up a LiveISO as long as you aren't reinstalling anything, follow this Guide for instruction how to chroot into your system.

@Girtablulu I don't understand the instructions in the Help Center. I booted the Live ISO and typed sudo su. Not sure how to proceed. Do I need to replace the mkdir /target command with something and if yes, with what? I think my system uses Grub.

This comment was removed by feuerstein.

I found out that I'm using EFI and I followed all steps - at least I think - however, the sudo clr-boot-manager update command failed with the following error: [Error] cbm (src/bootman/update.c:L270): Cannot determine currently running kernel. I tried following the instructions via https://www.reddit.com/r/SolusProject/comments/9gc4gk/help_with_fresh_installation/ by sudo eopkg dr Solus, sudo eopkg rr Solus, and sudo eopkg ar Solus https://mirrors.rit.edu/solus/packages/shannon/eopkg-index.xml.xz and got the following error: No repository found. Automatically adding Solus stable. Repo already present with name Solus and same URL. Removing first. Repo Solus added to system. Updating repository: Solus. Solus repository could not be reached. Removing Solus from system. Is there anyone who could give a hint what seems to be the problem?

feuerstein renamed this task from Solus boots into emergency mode to clr-boot-manager update cannot determine currently running kernel after chrooting to resolve emergency boot mode.Jan 2 2019, 2:46 PM
Lorien added a subscriber: Lorien.Jan 2 2019, 2:52 PM
This comment was removed by Lorien.


This is the latest series of commands. I should add that I am a fairly conservative user and always used the stable repositories and didn't install any software outside what is available in the software center.

Lorien added a comment.EditedJan 2 2019, 3:02 PM

Edit< Posted while you provided the picture above, which gives the answer...

What did you type when mounting the EFI System partition with this command?: mount /dev/sdX# /target/boot

Did you type?: mount /dev/sda1 /target/boot

After doing these commands:

mount --bind /proc /target/proc
mount --bind /dev /target/dev
mount --bind /sys /target/sys

Did you remember to chroot /target before doing the sudo clr-boot-manager update

Thanks for your help. Yes, I first typed mount /dev/sda3/target, then mount /dev/sda1/target/boot and then the three mount --bind ones, and after that chroot /target. Not sure why - as in the screenshot - the Solus repository could not be reached and clr-boot-manager cannot determine the currently running kernel.

Apparently, this happened before: https://getsol.us/forums/viewtopic.php?t=10871
@sunnyflunk You replied to this message in the forums. Does that mean this is perfectly normal and the sudo clr-boot-manager update command was successful despite the error?

Just exit chroot, the rebooted and I am still stuck in emergency mode.

Lorien added a comment.EditedJan 2 2019, 3:26 PM

So if you run: CBM_DEBUG=1 clr-boot-manager update do you get the same result as the user gets here?: https://getsol.us/forums/viewtopic.php?t=10871#p34756

Sorry feuerstein i am just a normal Solus user, not an expert in Solus or Linux at all, to me it looks like you have done everything right. I guess you have to wait for assistence from one of the Solus devs.

feuerstein renamed this task from clr-boot-manager update cannot determine currently running kernel after chrooting to resolve emergency boot mode to Stuck in emergency mode after regular Solus reboot.Jan 2 2019, 3:27 PM


It looks like more or less the same?

Lorien added a comment.EditedJan 2 2019, 4:24 PM

It looks like your system has 2 kernels installed: 4.18.5-90 and 4.18.16-97

Have you tried booting from both kernels? And is it exectly the same problem with both of them ?

When you restart the pc, you can spam the keybords "space bar" (tab it twice a second or something like that) to get a bootmenu on screen, from there you can choose what kernel you want to boot of the two installed...

feuerstein added a comment.EditedJan 2 2019, 5:25 PM

Interesting. Is this related to the rollback functionality of clr-boot-manager so that there are always automatically two kernels installed? I hit the "space bar" several times and booted into Solus-current-4.18.16-96 first and then tried with Solus-current-4.18.16-97 but both lead to emergency mode.

Lorien added a comment.EditedJan 2 2019, 9:33 PM

Now that you know how to Chroot into your system from a liveusb, have you tried the original suggestion by Girtablulu?:

https://dev.getsol.us/T7284#137682

I followed the suggestions by @Girtablulu but I'm not sure what to look for or how to interpret the results I've got. This is how it looks like:


Shall I proceed by just typing nano /etc/fstab now?

Everything looks fine, please run

sudo eopkg check | grep Broken | awk '{print $4}' | xargs sudo eopkg it --reinstall

Thanks, I got this result now:


How shall I proceed?

@Girtablulu After the previous terminal command I exited chroot and rebooted but it still boots to emergency mode. Is there a command to add system.base and to update the repositories while in chroot?

Girtablulu added a comment.EditedJan 3 2019, 11:24 PM

What the hell?

Can you give me the output of

Eopkg list-repo (eopkg lr)

Can you re run the command of above? Maybe throw a

Eopkg ur (eopkg update-repo) in before executing the command above, now.it shouldn't showbthe message

It looks like the repo link doesn't work? Wifi is working well and I can open other websites from the live iso.

yea known issue if you connect over wifi while chroot, either you connect you system with a cable to the internet or you do

nano /etc/resolv.conf

and add

nameserver <ipofyourrouter>

after this you should be able to connect

I connect with a cable and it is still the same issue. Is the repo mirror down or an alternative one I can try?

The output of eopkg list-repo is https://mirrors.rit.edu/solus/packages/shannon/eopkg-index.xml.xz
While the output of eopkg ur points to the .../eopkg-index.xml.xz.sha1sum
Is this correct?
When opening mirrors.rit.edu/solus/packages/shannon through a web browser there's three options: .xml.sha1sum, .xml.xz, and .xml.xz.sha1sum?

In my terminal window it says root@solus / #
I guess I should be correctly in chroot?

Please run

Ping -c 3 www.google.com

To check if you really have internet connection, if not please do steps explained above and add your router ip

@Girtablulu Thanks, there's indeed no internet connection when in chroot. I typed nano /etc/resolv.conf but don't know how to save the addition of the nameserver with the IP. How can I do that?

This is how it looks like:

Girtablulu added a comment.EditedJan 4 2019, 12:18 PM

Ctrl+o, Ctrl+x

And without the <>

Thanks a lot! Internet is working now in chroot. I ran sudo eopkg check | grep Broken | awk '{print $4}' | xargs sudo eopkg it --reinstall again and got another system error "25, inappropriate ioctl for device":

Can you try sudo eopkg up please?

I ran sudo eopkg up and it upated nearly 1.1 GB. This is quite a lot and I remember having kept my system always up to do. Not sure why it is so much to update. It looks like some were skipped:


I exited chroot, rebooted and it still boots into emergency mode.

well it clearly shows you were somehow not fully updated and no wonder breaks your system, chroot back in and run the

sudo eopkg check | grep Broken | awk '{print $4}' | xargs sudo eopkg it --reinstall it should be able now to fix stuff

if it's act up again of you can do it manually with

eopkg check | grep Broken

eopkg it --reinstall packagename

It's a lot of updates because you haven't been fully up to date in some time. Kernel 4.18 hasn't been in use for months now. Updates aren't automatic on Solus. You need to be running them from the Software Center periodically.

Can you make sure you mount the /boot partition (/dev/sda1) and then run sudo CBM_DEBUG=1 clr-boot-manager update?

I see, sudo eopkg check | grep Broken | awk '{print $4}' | xargs sudo eopkg it --reinstall worked better than before but still some items were skipped as seen in the screenshot. @Girtablulu

At which point exactly shall I run sudo CBM_DEBUG=1 clr-boot-manager update? @DataDrake

My steps so far were (in exactly this order):

sudo su
mkdir /target
mount /dev/sda3 /target
mount /dev/sda1 /target/boot
mount --bind /proc /target/proc
mount --bind /dev /target/dev
mount --bind /sys /target/sys
chroot /target

Then:
ping -c 3 www.google.com ---- Internet is working in chroot via cable

Then:
sudo CBM_DEBUG=1 clr-boot-manager update

Then:
sudo eopkg check | grep Broken | awk '{print $4}' | xargs sudo eopkg it --reinstall

Then:
exit
reboot

Finally:
Removed Live ISO and system booted into emergency mode again.

Is there a possibility that I've mounted the wrong partition? Is it correct to do all these steps when chrooting into the system after each reboot?

Could you please post the error?


This is the emergency boot message I get.

I typed journalctl and looked through most of the lines. The error messages that stood out were two systemd related messages in bold:

And two systemd-fsck related messages in red:


Apologies for the low quality of the pictures. I hope its readable. The error looks similar to the one described here: https://unix.stackexchange.com/questions/412948/solus-distro-is-not-booting-anymore

When I check my previous screenshots/pictures, it looks that udev and systemd configurations were skipped?

The systemd-related messages are fine, it's the fsck ones that concern me.

Can you do a fsck /dev/sda3 from the emergency recovery terminal where you can journalctl?


I'll go ahead and try to fix it? Will this be permanent or only for the session? I've never had to deal with fsck before and like to be careful...

I clicked y. It says:
Inode 3670134 was part of the orphaned inode list. FIXED.
Deleted inode 4194399 has zero dtime. Fix<y>?

Ah sorry, yeah you should be able to say y to all of them. Alternatively, fsck -a /dev/sda3 will try to do so automatically (and safely).


Filesystem is modified and I rebooted and it worked! Thank you all for helping me in this process @Lorien @Girtablulu and @DataDrake
I will do a backup and see if the next couple of updates and reboots work. As I understand changes with fsck are not permanent but only for one session? Is this correct? Are there any more things I should do to ensure a safe system or can I help find out what caused this problem? Apparently there was another Solus user with the same problem?

fsck changes are permanent. Best guess is that you either had some partial updates that messed things up or filesystem corruption. Either way, we push out updates on Fridays so try to not be more than a week or two behind :)

feuerstein closed this task as Resolved.Jan 4 2019, 5:24 PM
feuerstein claimed this task.

Will do :) Learnt a lot in the process and it was worth the effort. Thanks again!

Lorien awarded a token.Jan 4 2019, 5:42 PM

I seem to have gotten myself into this predicament now, however after attempting to fix it via the means outlined in the Solus Help Center (except for updating the system, as it was already mostly up-to-date (just plasma-desktop-branding not being in the index, and I do not have that installed in any case), plus I was unable to get networking to work inside the chroot), I had managed to fix the Systemd errors, however not the fact that the system is unable to switch-root.

Attached is the rdsosreport.

moriel5 added a comment.EditedApr 14 2020, 1:12 PM

Update: I have managed to get networking to work via the chroot, and update the system, however there is no change unfortunately.

Update 2: This is quite late (I did this about two weeks ago), however I decided to reinstall Solus, using the chance to test out UEFI on my laptop, now that I have upgraded my desktop (Asrock H97M, i5-4570, and a 500GB Seagate SATA3 HDD (CPU and HDD pulled from an old Dell miniPC that I found in the electronics recycling bin half a year ago), and finally running UEFI, without any firmware bugs (there is a hardware fault, though that fault is shared by all Asrock motherboards from that time, due to lack of proper QA at the time)), so my issue is now irrelevant (though now that I am also using UEFI on my laptop, the firmware bugs are in full force, due to broken ACPI tables in the firmware, that's my punishment for having bought a Lenovo IdeaPad back in the day, garbage quality at premium prices).