Command Line Tips 2

By Ku Wee Kiat, Research Computing, NUS Information Technology

 

In the previous article » Data Manipulation and More with the Command Line we covered some bash basics like pipes and variables. We also covered bash utilities that are useful for data manipulation.

In this installment of Command Line tips and tricks we will go back to the basics:

1. Keyboard Shortcuts
2. Pathname Expansion

As HPC users, we often work in the Linux terminal. We need to be productive in a pure text environment where the only interface you can use to control it is the keyboard. What you enter is what you get, if you do not execute the command to list a directory’s contents, you will not know what is inside the directory. So how can we do the things in the most productive way?  Yes, by using shortcuts!

Keyboard Shortcuts

Working with the current line

Besides using arrow keys to move around the current line, there are a few simple shortcuts to do things faster.

Do note that the shortcuts are not case sensitive. E.g.: Ctrl + A => hold Ctrl key and ‘a’ key together.

Action	Shortcut
Go to start of line	Ctrl + A
Go to end of line	Ctrl + E
Go to next word	Alt + F
Go to previous word	Alt + B

Deleting Text

Action	Shortcut
Delete current character	Ctrl + D
Delete previous word	Ctrl + W
Delete next word	Alt + D

Cutting and Pasting

We have covered navigation and deletion; it is time now for one of our favourites: Cut & Paste.

Action	Shortcut
Cut current word before cursor	Ctrl + W
Cut from cursor to end of line	Ctrl + K
Cut from cursor to start of line	Ctrl + U
Paste the cut buffer at current position	Ctrl + Y

Shell Expansions

What are shell expansions? They are symbols that can be expanded to a longer string or command. Let’s take a look at some examples:-

The Tilde ~

The tilde is useful when you want to reference your home directory. It is the equivalent to the full path of your home directory.

In NUS HPC, this would be:

~ == /home/svu/e0123456

Rather than typing out the full path to your home directory or any file or folder in your home directory, you can replace it with the tilde.

$ ls ~ # rather than $ ls /home/svu/e0123456   $ cat ~/my_text_file.txt # rather than $ cat /home/svu/e0123456/my_text_file.txt   $ ls ~/kitti_dataset # rather than $ ls /home/svu/e0123456/kitti_dataset

How many keypresses did the tilde save you?

Pathname Expansion

This allows you to write a short path pattern and it will expand to a list of files and directories.

Using *

The wildcard character * matches any string. It can be used as a prefix or a suffix.

For example, if you want to list all json files in a directory: ls my_directory/*.json will give you all json files in my_directory.

If you have a directory with files that have prefixes: train, test, val and you just want all the train files: ls dataset_directory/train*

All train files that are jpg format: ls dataset_directory/train*.jpg

With the following directory structure

abc@123a:/media/docker_mnt/mask_dataset$ tree                          .                                                             ├── README.dataset.txt                                                     ├── README.txt                                                                     ├── test                                                     ├── 0_Concern-In-China-As-Mystery-Virus-Spreads_jpg.rf.3135dfc5feab288d76a4ccfd22dfc5bf.jpg ├── 1224331650_g_400-w_g_jpg.rf.b816f49e2d84044fc997a8cbd55c347d.jpg                         ├── 15391513324714o1n0r10n6_jpg.rf.a91fbc7be8a94ed3c48d2e4b35bd53bb.jpg           ├── 15391513329330sooq10859_jpg.rf.89c8524c2096175fa2c728e5d73f1c28.jpg ├── 1579924271_jpg.rf.be5b27c2b2801bccc191e6dbd9bfccca.jpg         ├── _annotations.coco.json                                         ├── w1240-p16x9-fa978043deff83fed485af12d16e39c61398fc30_jpg ├── train │   ├── 000_1ov3n5_0_jpeg.rf.a23f1c89491779996f4519858277a4e0.jpg │   ├── 0002526673_jpg.rf.2c547115c01c1b5a10c6d467551f0cae.jpg │   ├── 000b7b75-1600_jpg.rf.5d7117e8571505dbfe49e2f737089ea0.jpg │   ├── 001_1024_jpeg.rf.f915f4689737658b59732df33fdbee22.jpg │   ├── 003_1024_jpeg.rf.bc025d99896124ddce6fa706b4a5c7b3.jpg │   └──  004_1024_jpeg.rf.763d99dadfb9fe5f3f72ff3cc34f88b6.jpg └── valid                                                         ├── 000_1OC3DT_jpg.rf.0570a1f4ab79e324496c311456b6b1f8.jpg     ├── 0009S6815V3PEU1N-C123-F4_jpg.rf.6b0befd018e6308b6029a17a2ee08c48.jpg     ├── 002_1024_jpeg.rf.b8c41940cb8dd4fa61e92892345859ad.jpg                 ├── 0109-00176-096b1_jpg.rf.c55de2d88f34d1217acdc245fd8292fa.jpg          └── 022814asiatodaymasks_960x540_jpg.rf.20ad64bc5ec9eed694d9f559b9aba31d.jpg

We can see that the test directory is empty and that there are jpg files outside of either test/train/valid directories. Let’s assume that the stray jpg files belongs to the test directory and we need to move it back to where it belongs. Pause for a moment to think of what would be the appropriate command using path expansion.

mv *.jpg test/

We can also use * in the following way to list all jpg files in all sub directories:

$ ls */*.jpg val/pic1.jpg   val/pic3.jpg   val/pic5.jpg  val/pic7.jpg val/pic2.jpg   val/pic4.jpg   val/pic6.jpg  train/1.jpg test/2.jpg

Braces

Brace expansion allows the shell to generate multiple strings based on the tokens in the curly braces.

For example, to quickly create train/test/val directories:

$ mkdir ~/my_dataset/{train,test,val} $ ls ~/my_dataset train   test   val

We can also do the following:-

$ mkdir ~/my_dataset/{train,test,val}_set $ ls ~/my_dataset train_set   test_set   val_set

It can also be used to create files/folders with sequential (numeric) names.

$ touch ~/dataset/test/sounds/noise-{1..5}.wav $ ls ~/dataset/test/sounds/ noise-1.wav noise-2.wav noise-3.wav noise-4.wav noise-5.wav

$ touch ~/test/class_{1..10..2} $ ls ~/test class_1 class_3 class_5 class_7 class_9

It most definitely can be used for listing files as well.

$ ls ~/dataset/test/{*.jpg,*.png} 01.jpg  02.png  02.jpg  03.png

We hope these tips and tricks covered in this article will increase your productivity in the HPC environment. These skills are useful whenever and wherever you are as long as you need to work in the UNIX terminal.