Bash cheatsheet: Common Bash tasks

Bash cheatsheet.md

Bash Cheatsheet

Bash scripting cheatsheet: https://github.com/dylanaraps/pure-bash-bible#loop-over-files-and-directories

Table of Contents

• Bash Cheatsheet
  • File operations:
    • Searching for di.xml files on the command line
    • Searching for strings inside specific files
    • Find all files ending with .sql
    • Find all files greater than 700MB
    • Find and replace using sed
    • Extract the first/last N characters from a string
    • Trim the last N characters from a string
    • Extract part of a file into a new file
  • Exclude files from tarball:
  • Find and replace email addresses in a file:
  • List size of each file and folder with largest first:
    • Create a symbolic link to a file
    • Create a symbolic link to a folder
  • cURL
    • Make a POST request
    • Add custom HTTP headers
  • SSH
    • Generate an SSH key
      • Flags
    • Verify the fingerprint of an SSH key
    • Setup connection defaults using an SSH config file:
    • Copying a file using scp
    • Troubleshooting
      • Pushing to git works on the CLI, but not in PhpStorm
  • Tailing files
  • Mounting drives
  • nano
  • Ubuntu: Fix 'unathenticated sources' error

File operations:

Searching for di.xml files on the command line

$ find vendor/magento/ -name di.xml | xargs grep ValidationStateInterface
vendor/magento/magento2-base/app/etc/di.xml:    <preference for="Magento\Framework\Config\ValidationStateInterface" type="Magento\Framework\App\Arguments\ValidationState"></preference>

Searching for strings inside specific files

E.g. To find the string Google API inside acl.xml files only:

$ find vendor/magento/ -name 'acl.xml' -exec grep -i -P 'Google API' '{}' +
vendor/magento//module-google-analytics/etc/acl.xml: <resource id="Magento_GoogleAnalytics::google" title="Google API" translate="title" />

Note that grep uses the Basic RegEx Engine (BRE) by default, however, PHP uses the Perl Compatible RegEx Engine (PCRE) by default. So, depending on the complexity of the RegEx pattern, RegEx patterns that work in PHP or PhpStorm may not produce the same results (or, indeed, any results at all) when executed on the command line grep's default settings. To tell grep to use the PCRE engine, you will need to pass the -P flag to grep:

# With default (Basic) RegEx engine:
$ grep -r --color=auto '\\(.+?)\\Service\\(V\d+)+(\\.+)Interface' vendor/magento/
# (... no results)

# With -P (PCRE) RegEx engine (note the -P flag):
$ grep -r -P --color=auto '\\(.+?)\\Service\\(V\d+)+(\\.+)Interface' vendor/magento/
vendor/magento/magento2-base/CHANGELOG.md:  * Moved the authorization services according to the new directory format: was \Magento\Authz\Service\AuthorizationV1Interface, became \Magento\Authz\Service\V1\AuthorizationInterface
vendor/magento/magento2-base/dev/tests/static/testsuite/Magento/Test/Legacy/_files/obsolete_methods.php:        'Magento\Tax\Service\V1\TaxRuleServiceInterface',
vendor/magento/magento2-base/dev/tests/static/testsuite/Magento/Test/Legacy/_files/obsolete_classes.php:        'Magento\Customer\Service\V1\CustomerCurrentServiceInterface',
vendor/magento/magento2-base/dev/tests/static/testsuite/Magento/Test/Legacy/_files/obsolete_classes.php:        'Magento\Customer\Service\V1\CustomerAddressCurrentServiceInterface',

Find all files ending with .sql

find <path> -type <file type flag, f for ordinary files> -name <filename expression>
find . -type f -name *.sql

Find all files greater than 700MB

find <path> -size <size expression>
find / -size +700M

Find and replace using sed

sed -i -e 's/findthis/replacewiththis/g' inthisfile.sql

-i: Edit files in place

Extract the first/last N characters from a string

STRING=/var/www/vhosts/something
OUTPUT=${STRING: -9}
echo $OUTPUT # Outputs 'something'
OUTPUT=${STRING: +9}
echo $OUTPUT # Outputs '/var/www/'

Note the single semi-colon.

Trim the last N characters from a string

STRING=/var/www/vhosts/something
OUTPUT=${STRING:: -9}
echo $OUTPUT # Outputs '/var/www/vhosts/'

Note the two semi-colons.

Extract part of a file into a new file

sed -n '1,100p' source.txt > target.txt

Where 1 and 100 are the starting and finishing line numbers of the section to extract from the source.txt document

Exclude files from tarball:

tar -jcf media.tar.bz2 --exclude=media/captcha/* --exclude=media/catalog/category/cache/* --exclude=media/catalog/product/cache/* --exclude=media/*.tar media

Find and replace email addresses in a file:

The backslash-numbers (e.g. \1) in the right-hand side of the sed expression represent the capture classes of the regex.

sed -i -r "s/([a-z0-9_\.-]+)@([a-z0-9_-]+)(\.[a-z]{2,3})(\.[a-z]{2})?/\1@\2\3.DEBUGMODE/gi" m2_dfl_customer_entity.sql
# Before:
[email protected]
[email protected]
[email protected]
# After:
[email protected]
[email protected]
[email protected]

List size of each file and folder with largest first:

du -shc * | sort -hr

Create a symbolic link to a file

ln -s <source> <target>
ln -s /etc/php5/mods-available/mcrypt.ini /etc/php5/fpm/conf.d/20-mcrypt.ini

Create a symbolic link to a folder

ln -s <source>
ln -s /var/Documents/

ls -lah 
# lrwxrwxrwx  1 simon simon   15 Nov 11 09:36 Documents -> /var/Documents/

cURL

Make a POST request

curl --request POST http://example.com/index.php/rest/default/V1/integration/admin/token -d '{"username":"admin","password":"password123"}' -H "Content-Type:application/json"

Add custom HTTP headers

curl http://magento2-sample-modules.localhost.com/index.php/rest/default/V1/integration/admin/token -H "Content-Type:application/json"

Download and save to a file (whilst following redirects)

-L, --location: Follow redirects (redirects to the URL in the 'Location' header)
-o FILE: Saves the resource to FILE

curl -L -o pub/media/feed/pricefeed/import/stockm.csv http://www.example.com/media/feed/pricefeed/import/stockm.csv

SSH

Generate an SSH key

# Create the ~/.ssh directory if it doesn't already exist
$ mkdir ~/.ssh/ && cd ~/.ssh/
$ ssh-keygen -t rsa -b 4096 -C "jenkins-ci"

Flags

• -t The cryptographic standard to use (rsa or dsa)
• -b Key length. Larger is better.
• -C A human-readable comment to identify the key

Verify the fingerprint of an SSH key

Run this command before attempting to SSH into a new server:

$ ssh -o FingerprintHash=sha256 github.com
The authenticity of host 'github.com (18.205.93.0)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)?

The fingerprint should be the same as when you attempt to connect using your public/private key combination:

$ git clone [email protected]:ProcessEight/magento-scripts.git
Cloning into 'magento-scripts'...
The authenticity of host 'github.com (140.82.118.4)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,140.82.118.4' (RSA) to the list of known hosts.
# git clone output continues...

Setup connection defaults using an SSH config file:

Create the file ~/.ssh/config. Changes take effect on save.

Host bitbucket.org
 # Tell SSH to use this private key when connecting to 'bitbucket.org'
 IdentityFile ~/.ssh/simon-xps-15-9570
 # There are plenty of other options too:
 # Port 1234
 # User apollo

You can also create aliases, so you don't have to type out the full command every time:

Host eg
  HostName example.com # Could also be an IP address
  IdentityFile ~/.ssh/id_rsa.pub
  User ubuntu

Using the config above:

ssh eg

...is equivalent to typing...

ssh -i ~/.ssh/id_rsa.pub [email protected]

See https://www.digitalocean.com/community/tutorials/how-to-configure-custom-connection-options-for-your-ssh-client for more options.

Copying a file using scp

# From local machine to remote server:
scp [-i ~/.ssh/public.key] /path/to/local/file_original [username@][hostname]:/path/to/remote/file_copy

# From a remote server to local machine:
scp [username@][hostname]:/path/to/remote/file_original /path/to/local/file_copy

# From a remote server to another remote server:
scp [username@][hostname]:/path/to/remote_1/file_original [username@][hostname]:/path/to/remote_2/file_copy

[hostname] can be an IP address or a domain name.

Use the -i (for identity_file) flag to login to the remote server with an SSL cert. Use the -P (for port) flag to specify a custom port number.

Troubleshooting

Pushing to git works on the CLI, but not in PhpStorm

• In PhpStorm settings, verify that a token has been created to allow PhpStorm to access your GitHub account
• If that didn't work, you may need to create a config file in your .ssh directory to tell it to use the specific SSH keys you added to your account:

Host github.com
 IdentityFile ~/.ssh/project8_ubuntu_github_rsa

Host gist.github.com
 IdentityFile ~/.ssh/project8_ubuntu_github_rsa

ssh: Could not resolve hostname bitbucket.org: Temporary failure in name resolution

When trying to clone a repository, this error may occur:

$ git clone [email protected]:elementarydigital/magento2-module-faqs.git magento2-module-faqs
Cloning into 'magento2-module-faqs'...
ssh: Could not resolve hostname bitbucket.org: Temporary failure in name resolution
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Try disconnecting from Wi-fi and using ethernet connection instead.

ssh_exchange_identification: Connection closed by remote host

The server may be overloaded. Wait and try again later.

Tailing files

Tail log file from a specific point to the end of the file:

# Find the line first using grep:
grep -n 'feedparser\/stock\.csv' var/log/feedparser.log
$ 593525:2014-09-10T12:34:21+01:00 DEBUG (7): Processing log 6: /var/www/vhosts/homehardware.co.uk/htdocs/var/feedparser/stock.csv

# Use the + switch to specify the line number to start from:
$ tail -n +593525 var/log/feedparser.log

Mounting drives

mount -t "ntfs" -o "uhelper=udisks2,nodev,nosuid,uid=1000,gid=1000,dmask=0077,fmask=0177" "/dev/sdg1" "/media/simon/WAMP1"

To mount an 'unclean' drive, use the read only (ro) option:

mount -t "ntfs" -o "ro,uhelper=udisks2,nodev,nosuid,uid=1000,gid=1000,dmask=0077,fmask=0177" "/dev/sdh1" "/media/simon/Data"

nano

# Delete line: Ctrl + K
# Jump to top of file: Alt + \
# Jump to bottom of file: Alt + /

Ubuntu: Fix 'unathenticated sources' error

http://askubuntu.com/questions/85641/how-do-i-deal-with-unauthenticated-sources-errors-in-the-software-center

Run apt-get update to get the keys which are the problem. Then:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys YOURKEYNUMBERHERE