-
-
Save roommen/18cd78d07b0fbc962de4e79c1d468f92 to your computer and use it in GitHub Desktop.
| #!/bin/bash | |
| sudo yum install gcc -y | |
| sudo yum install openssl-devel -y | |
| sudo yum install zlib-devel -y | |
| sudo yum install mlocate -y | |
| sudo yum install autoconf -y | |
| wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.1p1.tar.gz | |
| tar zxvf openssh-9.1p1.tar.gz | |
| cd openssh-9.1p1 && ./configure && make && sudo make install |
I'm seeing the timeout/flapping on a fairly clean EC2 instance using your June 1st script:
systemd[1]: sshd.service start operation timed out. Terminating.
systemd[1]: Failed to start OpenSSH server daemon.
systemd[1]: Unit sshd.service entered failed state.
systemd[1]: sshd.service failed.
systemd[1]: sshd.service holdoff time over, scheduling restart.
systemd[1]: Stopped OpenSSH server daemon.
systemd[1]: Starting OpenSSH server daemon...
sshd[29999]: Server listening on 0.0.0.0 port 22.
sshd[29999]: Server listening on :: port 22.
systemd[1]: sshd.service start operation timed out. Terminating.
systemd[1]: Failed to start OpenSSH server daemon.
systemd[1]: Unit sshd.service entered failed state.
systemd[1]: sshd.service failed.
journalctl -r /usr/sbin/sshd also has sshd[28676]: User ec2-user not allowed because account is locked, which is interesting as that's the account I've used for several weeks since I started the instance. Running sudo passwd ec2-user to set a password and then sudo passwd -u ec2-user to unlock the account lets me login when I catch the service in the middle restart.
Edit: updated info on ec2-user locked acount
Yeah, it's the same error logs I got
It works 👍
I'm seeing the timeout/flapping on a fairly clean EC2 instance using your June 1st script:
systemd[1]: sshd.service start operation timed out. Terminating. systemd[1]: Failed to start OpenSSH server daemon. systemd[1]: Unit sshd.service entered failed state. systemd[1]: sshd.service failed. systemd[1]: sshd.service holdoff time over, scheduling restart. systemd[1]: Stopped OpenSSH server daemon. systemd[1]: Starting OpenSSH server daemon... sshd[29999]: Server listening on 0.0.0.0 port 22. sshd[29999]: Server listening on :: port 22. systemd[1]: sshd.service start operation timed out. Terminating. systemd[1]: Failed to start OpenSSH server daemon. systemd[1]: Unit sshd.service entered failed state. systemd[1]: sshd.service failed.
journalctl -r /usr/sbin/sshdalso hassshd[28676]: User ec2-user not allowed because account is locked, which is interesting as that's the account I've used for several weeks since I started the instance. Runningsudo passwd ec2-userto set a password and thensudo passwd -u ec2-userto unlock the account lets me login when I catch the service in the middle restart.Edit: updated info on ec2-user locked acount
followed this to resolve the issue
@samaddico, with that hint I found https://unix.stackexchange.com/a/313159. It all makes sense now. The built from raw source binary does not have systemd support build in, so it never notifies systemd that sshd successfully started causing the flapping/loop. It looks like another option might be to change the unit/service file, but I don't know the (security) ramifications of that trade-off.
@samaddico, with that hint I found https://unix.stackexchange.com/a/313159. It all makes sense now. The built from raw source binary does not have systemd support build in, so it never notifies systemd that sshd successfully started causing the flapping/loop. It looks like another option might be to change the unit/service file, but I don't know the (security) ramifications of that trade-off.
Good catch !
Based on the link provided by @samaddico , this is what I did on Amazon Linux 2
(note I included the line numbers because it may help - they should be similar, if not exactly the same, for other users)
# install pre-reqs and get software
sudo yum install -y gcc openssl-devel zlib-devel mlocate autoconf
wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.3p2.tar.gz
tar zxvf openssh-9.3p2.tar.gz
cd openssh-9.3p2
./configure --prefix=/usr
# edit sshd.c
1. add the following marked line:
127 #include "sk-api.h"
128 #include "srclimit.h"
129 #include "dh.h"
130 #include <systemd/sd-daemon.h> <---- this line
2. add the following marked lines:
2097 /* Signal systemd that we are ready to accept connections */ <---- this line
2098 sd_notify (0, "READY=1"); <---- this line
2099
2100 /* Accept a connection and return in a forked child */
2101 server_accept_loop(&sock_in, &sock_out,
2102 &newsock, config_s);
# install devel library
sudo yum install -y systemd-devel
# update the LIBS variable in the Makefile
51 #LIBS=-ldl -lutil -lresolv
52 LIBS =-lcrypto -ldl -lutil -lz -lcrypt -lresolv -lsystemd
# run make, make install
make
sudo make install
# restart sshd
systemctl restart sshd.service
# check sshd status
systemctl status sshd.service
Thanks for this @sttuartt, following your answer and editing the sshd.c file with the indicated lines made my sshd.service work again, now I can restart it without any problem.
I'm using Amazon Linux 2
Based on the link provided by @samaddico , this is what I did on Amazon Linux 2
(note I included the line numbers because it may help - they should be similar, if not exactly the same, for other users)
# install pre-reqs and get software sudo yum install -y gcc openssl-devel zlib-devel mlocate autoconf wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.3p2.tar.gz tar zxvf openssh-9.3p2.tar.gz cd openssh-9.3p2 ./configure --prefix=/usr # edit sshd.c 1. add the following marked line: 127 #include "sk-api.h" 128 #include "srclimit.h" 129 #include "dh.h" 130 #include <systemd/sd-daemon.h> <---- this line 2. add the following marked lines: 2097 /* Signal systemd that we are ready to accept connections */ <---- this line 2098 sd_notify (0, "READY=1"); <---- this line 2099 2100 /* Accept a connection and return in a forked child */ 2101 server_accept_loop(&sock_in, &sock_out, 2102 &newsock, config_s); # install devel library sudo yum install -y systemd-devel # update the LIBS variable in the Makefile 51 #LIBS=-ldl -lutil -lresolv 52 LIBS =-lcrypto -ldl -lutil -lz -lcrypt -lresolv -lsystemd # run make, make install make sudo make install # restart sshd systemctl restart sshd.service # check sshd status systemctl status sshd.service
It seems that this works exactly for this openssh-9.3p2 version, because I tried this with a previous version (openssh-9.1p1) and was unsuccessful when compiling.
Thank you, @softlberton ! Just a quick note for others, you need to sudo the systemctl calls.
Based on the link provided by @samaddico , this is what I did on Amazon Linux 2
(note I included the line numbers because it may help - they should be similar, if not exactly the same, for other users)
# install pre-reqs and get software sudo yum install -y gcc openssl-devel zlib-devel mlocate autoconf wget https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.3p2.tar.gz tar zxvf openssh-9.3p2.tar.gz cd openssh-9.3p2 ./configure --prefix=/usr # edit sshd.c 1. add the following marked line: 127 #include "sk-api.h" 128 #include "srclimit.h" 129 #include "dh.h" 130 #include <systemd/sd-daemon.h> <---- this line 2. add the following marked lines: 2097 /* Signal systemd that we are ready to accept connections */ <---- this line 2098 sd_notify (0, "READY=1"); <---- this line 2099 2100 /* Accept a connection and return in a forked child */ 2101 server_accept_loop(&sock_in, &sock_out, 2102 &newsock, config_s); # install devel library sudo yum install -y systemd-devel # update the LIBS variable in the Makefile 51 #LIBS=-ldl -lutil -lresolv 52 LIBS =-lcrypto -ldl -lutil -lz -lcrypt -lresolv -lsystemd # run make, make install make sudo make install # restart sshd systemctl restart sshd.service # check sshd status systemctl status sshd.service
So awesome @brianantonelli, many thanks! BTW, this solution works well with OpenSSH v9.5p1 too.
To facilitate subsequent use by others, you can use the following code directly:
sed -i '129a\#include <systemd/sd-daemon.h>' sshd.c
sed -i '2095a\ /* Signal systemd that we are ready to accept connections */' sshd.c
sed -i '2096a\ sd_notify (0, "READY=1");' sshd.c
When I try the configure command, it eventually fails with:
Checking OpenSSL library version... configure: error: OpenSSL >= 1.1.1 required (have "100020bf (OpenSSL 1.0.2k-fips 26 Jan 2017)")
OpenSSL via yum for Amazon Linux 2 only lets me install 1.0.2.
Any ideas?
Use:
yum install -y openssl11 openssl11-devel
I was able to upgrade to 9.6p1 with that.
Both versions are installed. Executing this:
openssl11 version
returns 1.1.1
I literally just did the yum install above and ./configure and everything following worked.
ssh -V afterwards gave me:
OpenSSH_9.6p1, OpenSSL 1.1.1g FIPS 21 Apr 2020
Ah well, I am not having any luck. I even renamed openssl to openssl.orig and openssl11 to openssl so that when openssl is run, it uses the newer version. The configure command still returns:
checking for openssl... /usr/bin/openssl
checking for openssl/opensslv.h... yes
checking OpenSSL header version... 100020bf (OpenSSL 1.0.2k 26 Jan 2017)
checking for OpenSSL_version... no
checking for OpenSSL_version_num... no
checking OpenSSL library version... configure: error: OpenSSL >= 1.1.1 required (have "100020bf (OpenSSL 1.0.2k-fips 26 Jan 2017)")
I did have errors trying to install openssl11-devel as it said that openssl-devel was needed for openssl 1.0.2. I told it to ignore that to get it to install.
After a period of research, here are some conclusions(which may be useful to you).
Checking configure help doc
./configure --help |grep 'ssl'
--without-openssl Disable use of OpenSSL; use only limited internal crypto **EXPERIMENTAL**
--with-ssl-dir=PATH Specify path to OpenSSL installation
--without-openssl-header-check Disable OpenSSL version consistency check
--with-ssl-engine Enable OpenSSL (hardware) ENGINE supportAs you can see, here is a param --with-ssl-dir=PATH that can be used for specify openssl path.
Dive into openssl11
# Install yum-utils
sudo yum install -y yum-utils
# Download the rpm package
sudo yumdownloader openssl11 openssl11-devel
# Check out the package
rpm -qpl openssl11-1.1.1g-12.amzn2.0.20.x86_64.rpm
#> /usr/bin/make-dummy-cert
#> /usr/bin/openssl11
#> /usr/bin/renew-dummy-cert
#> /usr/share/doc/openssl11-1.1.1g
#> /usr/share/doc/openssl11-1.1.1g/FAQ
#> /usr/share/doc/openssl11-1.1.1g/Makefile.certificate
#> /usr/share/doc/openssl11-1.1.1g/NEWS
#> /usr/share/doc/openssl11-1.1.1g/README
#> /usr/share/doc/openssl11-1.1.1g/README.FIPS
#> /usr/share/licenses/openssl11-1.1.1g
#> /usr/share/licenses/openssl11-1.1.1g/LICENSE
#> /usr/share/man/man1/openssl11.1.gz
rpm -qpl openssl11-devel-1.1.1g-12.amzn2.0.20.x86_64.rpm
#> /usr/include/openssl
#> /usr/include/openssl/aes.h
#> /usr/include/openssl/asn1.h
#> /usr/include/openssl/asn1_mac.h
#> /usr/include/openssl/asn1err.h
#> ....The path /usr/include/openssl is the path that you should fill into above.
Conclusion
Run before: sudo yum install -y gcc openssl11 openssl11-devel zlib-devel mlocate autoconf
Try: ./configure --with-ssl-dir=/usr/include/openssl
BTW, The above method has been verified to work on a brand new AL2 system.
If your colleague also decided to follow these instructions on your day-off, and now finds themselves in a situation where the OpenSSH server is almost inaccessible anymore for new connections (e.g. ssh your-server-ip returns with error "kex_exhcnage_identification: read: Connection reset by peer"), follow my instructions to restore everything back.
Assuming you (or your coworker) still have an open SSH connection and can run commands in the shell:
- Set a secure password for the root user using
sudo passwd(you can use tools like pwgen to generate a strong password). - Log into the AWS console, navigate to your EC2 instance, select it, then go to Actions → Monitor and troubleshoot → EC2 serial console → Connect
- Press Enter a few times in the large black rectangle, and you'll get a login prompt. Type root, press Enter, and then enter the password generated in step 1
- Stop the SSH server and terminate all connections by running
systemctl stop sshd; killall sshd - Navigate to the openssh server source code directory, most likely it would be
cd /home/ec2-user/openssh-9.1p1 - Run
make uninstallto clean up all the files installed by the previous "make install" command (please avoid doing this on any non-personal machines in the future). - After step 6, you'll no longer have SSH client & SSH server on your system, even though the system package manager still thinks OpenSSH is installed. Reinstall the SSH-related packages to restore everything:
yum reinstall openssh openssh-server openssh-client - Finally, start the SSH server again by running
systemctl start sshd, and check if you can connect to the server from the server itself byssh localhost.
P.S. If upgrading is necessary to make npm work (due to the unsupported option "accept-now"), you can use this workaround by setting an environment variable: GIT_SSH_COMMAND=ssh npm i ... (Source)
I am use centOS 7.6, it should like this
./configure --with-ssl-dir=/usr/local/openssl
smh this broke my clones
sed -i '129a\#include <systemd/sd-daemon.h>' sshd.c sed -i '2095a\ /* Signal systemd that we are ready to accept connections */' sshd.c sed -i '2096a\ sd_notify (0, "READY=1");' sshd.c
Hi everyone, I also updated OpenSSH on EC2 this week. After performing the above steps,
systemctl restart sshd executes successfully but the client cannot connect via ssh.
I think there may be some problem with the original ec2 sshkey after the update?
Has anyone else encountered a similar situation and fixed it?
Hope to get the answer... QQ
I haven't seen this happen with sshd before, but to get more info on it, you should see what kind of logs / error messages sshd is outputting. Try
sudo journalctl -u sshd.serviceand look for error messages.