Command Line Tips 2
By Ku Wee Kiat, Research Computing, NUS Information Technology
In the previous article » Data Manipulation and More with the Command Line we covered some bash basics like pipes and variables. We also covered bash utilities that are useful for data manipulation.
In this installment of Command Line tips and tricks we will go back to the basics:
- Keyboard Shortcuts
- Pathname Expansion
As HPC users, we often work in the Linux terminal. We need to be productive in a pure text environment where the only interface you can use to control it is the keyboard. What you enter is what you get, if you do not execute the command to list a directory’s contents, you will not know what is inside the directory. So how can we do the things in the most productive way? Yes, by using shortcuts!
Keyboard Shortcuts
Working with the current line
Besides using arrow keys to move around the current line, there are a few simple shortcuts to do things faster.
Do note that the shortcuts are not case sensitive. E.g.: Ctrl + A => hold Ctrl key and ‘a’ key together.
|
Action |
Shortcut |
|
Go to start of line |
Ctrl + A |
|
Go to end of line |
Ctrl + E |
|
Go to next word |
Alt + F |
|
Go to previous word |
Alt + B |
Deleting Text
|
Action |
Shortcut |
|
Delete current character |
Ctrl + D |
|
Delete previous word |
Ctrl + W |
|
Delete next word |
Alt + D |
Cutting and Pasting
We have covered navigation and deletion; it is time now for one of our favourites: Cut & Paste.
|
Action |
Shortcut |
|
Cut current word before cursor |
Ctrl + W |
|
Cut from cursor to end of line |
Ctrl + K |
|
Cut from cursor to start of line |
Ctrl + U |
|
Paste the cut buffer at current position |
Ctrl + Y |
Shell Expansions
What are shell expansions? They are symbols that can be expanded to a longer string or command. Let’s take a look at some examples:-
The Tilde ~
The tilde is useful when you want to reference your home directory. It is the equivalent to the full path of your home directory.
In NUS HPC, this would be:
|
~ == /home/svu/e0123456 |
Rather than typing out the full path to your home directory or any file or folder in your home directory, you can replace it with the tilde.
|
$ ls ~ # rather than $ ls /home/svu/e0123456
$ cat ~/my_text_file.txt # rather than $ cat /home/svu/e0123456/my_text_file.txt
$ ls ~/kitti_dataset # rather than $ ls /home/svu/e0123456/kitti_dataset |
How many keypresses did the tilde save you?
Pathname Expansion
This allows you to write a short path pattern and it will expand to a list of files and directories.
Using *
The wildcard character * matches any string. It can be used as a prefix or a suffix.
For example, if you want to list all json files in a directory: ls my_directory/*.json will give you all json files in my_directory.
If you have a directory with files that have prefixes: train, test, val and you just want all the train files: ls dataset_directory/train*
All train files that are jpg format: ls dataset_directory/train*.jpg
With the following directory structure
|
abc@123a:/media/docker_mnt/mask_dataset$ tree . ├── README.dataset.txt ├── README.txt ├── test ├── 0_Concern-In-China-As-Mystery-Virus-Spreads_jpg.rf.3135dfc5feab288d76a4ccfd22dfc5bf.jpg ├── 1224331650_g_400-w_g_jpg.rf.b816f49e2d84044fc997a8cbd55c347d.jpg ├── 15391513324714o1n0r10n6_jpg.rf.a91fbc7be8a94ed3c48d2e4b35bd53bb.jpg ├── 15391513329330sooq10859_jpg.rf.89c8524c2096175fa2c728e5d73f1c28.jpg ├── 1579924271_jpg.rf.be5b27c2b2801bccc191e6dbd9bfccca.jpg ├── _annotations.coco.json ├── w1240-p16x9-fa978043deff83fed485af12d16e39c61398fc30_jpg ├── train │ ├── 000_1ov3n5_0_jpeg.rf.a23f1c89491779996f4519858277a4e0.jpg │ ├── 0002526673_jpg.rf.2c547115c01c1b5a10c6d467551f0cae.jpg │ ├── 000b7b75-1600_jpg.rf.5d7117e8571505dbfe49e2f737089ea0.jpg │ ├── 001_1024_jpeg.rf.f915f4689737658b59732df33fdbee22.jpg │ ├── 003_1024_jpeg.rf.bc025d99896124ddce6fa706b4a5c7b3.jpg │ └── 004_1024_jpeg.rf.763d99dadfb9fe5f3f72ff3cc34f88b6.jpg └── valid ├── 000_1OC3DT_jpg.rf.0570a1f4ab79e324496c311456b6b1f8.jpg ├── 0009S6815V3PEU1N-C123-F4_jpg.rf.6b0befd018e6308b6029a17a2ee08c48.jpg ├── 002_1024_jpeg.rf.b8c41940cb8dd4fa61e92892345859ad.jpg ├── 0109-00176-096b1_jpg.rf.c55de2d88f34d1217acdc245fd8292fa.jpg └── 022814asiatodaymasks_960x540_jpg.rf.20ad64bc5ec9eed694d9f559b9aba31d.jpg |
We can see that the test directory is empty and that there are jpg files outside of either test/train/valid directories. Let’s assume that the stray jpg files belongs to the test directory and we need to move it back to where it belongs. Pause for a moment to think of what would be the appropriate command using path expansion.
|
mv *.jpg test/ |
We can also use * in the following way to list all jpg files in all sub directories:
|
$ ls */*.jpg val/pic1.jpg val/pic3.jpg val/pic5.jpg val/pic7.jpg val/pic2.jpg val/pic4.jpg val/pic6.jpg train/1.jpg test/2.jpg |
Braces
Brace expansion allows the shell to generate multiple strings based on the tokens in the curly braces.
For example, to quickly create train/test/val directories:
|
$ mkdir ~/my_dataset/{train,test,val} $ ls ~/my_dataset train test val |
We can also do the following:-
|
$ mkdir ~/my_dataset/{train,test,val}_set $ ls ~/my_dataset train_set test_set val_set |
It can also be used to create files/folders with sequential (numeric) names.
|
$ touch ~/dataset/test/sounds/noise-{1..5}.wav $ ls ~/dataset/test/sounds/ noise-1.wav noise-2.wav noise-3.wav noise-4.wav noise-5.wav |
|
$ touch ~/test/class_{1..10..2} $ ls ~/test class_1 class_3 class_5 class_7 class_9 |
It most definitely can be used for listing files as well.
|
$ ls ~/dataset/test/{*.jpg,*.png} 01.jpg 02.png 02.jpg 03.png |
We hope these tips and tricks covered in this article will increase your productivity in the HPC environment. These skills are useful whenever and wherever you are as long as you need to work in the UNIX terminal.