hcwang

Down to the earth

0%

[Missing Semester in CS] Shell scripting and tools

Posted on Edited on In

After going through the basic bash commands in the previous class, we will explore more in putting all sort of commands together into a bash script file.

Brief

There are a bunch of must-knows for writing the scripts introduced in the class. Here is the complete list to check if you all understand:

  • variables assignment (NO SPACES!)
  • difference between strings enclosed with ' and "
  • control flows: if, while, case, for
  • function definition, inputs, return value and some special variables.
  • && and || operators
  • get the outputs from a command
  • testing condition, also mentioned in my last post.
  • wildcards(foo*) and curly brace(foo.{jpg,png}, foo{a..h}, foo{1..100}) for variables.
  • shellcheck

I would add the artical for some helpful bash settings. However I guess finding your own convention of writing / debugging bash script would be even helpful by adopting those conventions that fit for you.

Besides the techniques, several practical use cases are discussed, which might make our experience in writing bash less painful.

  • find how to use command
  • find files
  • find / match codes
  • find previous used shell commands
  • faster directory navigation

Exercises

  1. Read man ls and write an ls command that lists files in the following manner

    • Includes all files, including hidden files

    • Sizes are listed in human readable format (e.g. 454M instead of 454279954)

    • Files are ordered by recency

    • Output is colorized

      A sample output would look like this

      1
      2
      3
      4
      5
      -rw-r--r--   1 user group 1.1M Jan 14 09:53 baz
      drwxr-xr-x   5 user group  160 Jan 14 09:53 .
      -rw-r--r--   1 user group  514 Jan 14 06:42 bar
      -rw-r--r--   1 user group 106M Jan 13 12:12 foo
      drwx------+ 47 user group 1.5K Jan 12 18:08 ..
1
$ ls -lath --color=auto

This is an example which requires us to understand more options from ls. In short, -l give LONG format, -a specifies to show ALL. -t means showing in TIME (recency) order. -h equals to --human-readable. --color=auto colorizes the output.
Usually the default in Ubuntu System, ls already refers to ls --color=auto for better visualization using aliasing.

  1. Write bash functions marco and polo that do the following.
    Whenever you execute marco the current working directory should be saved in some manner, then when you execute polo, no matter what directory you are in, polo should cd you back to the directory where you executed marco.
    For ease of debugging you can write the code in a file marco.sh and (re)load the definitions to your shell by executing source marco.sh.
1
2
3
4
5
6
7
8
# bash variable setting and usage
marco() {
    export MARCO=$(pwd)
}

polo() {
    cd "$MARCO" # or cd ${MARCO}
}

  1. Say you have a command that fails rarely. In order to debug it you need to capture its output but it can be time consuming to get a failure run.
    Write a bash script that runs the following script until it fails and captures its standard output and error streams to files and prints everything at the end.
    Bonus points if you can also report how many runs it took for the script to fail.

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    #!/usr/bin/env bash

    n=$(( RANDOM % 100 ))

    if [[ n -eq 42 ]]; then
       echo "Something went wrong"
       >&2 echo "The error was using magic numbers"
       exit 1
    fi

    echo "Everything went according to plan"
1
2
3
4
5
6
7
8
9
10
11
#!/usr/bin/env bash

count=0
until [[ "$?" -ne 0 ]];
do
  count=$((count+1))
  ./random.sh &> out.txt
done

echo "found error after $count runs"
cat out.txt

Some points interesting to check:

n=$(( RANDOM % 100 ))
Check math context about how to do arithmetic calculations in bash script.

if [[ n -eq 42 ]]
Check Test for different test conditions syntax in bash.

"$?" Refers the exit code from last command.
Check Special Variables in bash.

  1. As we covered in lecture find‘s -exec can be very powerful for performing operations over the files we are searching for.
    However, what if we want to do something with all the files, like creating a zip file?
    As you have seen so far commands will take input from both arguments and STDIN.
    When piping commands, we are connecting STDOUT to STDIN, but some commands like tar take inputs from arguments.
    To bridge this disconnect there’s the xargs command which will execute a command using STDIN as arguments.
    For example ls | xargs rm will delete the files in the current directory.

    Your task is to write a command that recursively finds all HTML files in the folder and makes a zip with them. Note that your command should work even if the files have spaces (hint: check -d flag for xargs)

    1
    2
    $ find . -type f -name "*.html" | xargs -d '\n'  tar -cvzf
      archive.tar.gz

  2. (Advanced) Write a command or script to recursively find the most recently modified file in a directory. More generally, can you list all files by recency?

Here is one of the answer from here.

1
2
#!/bin/bash
find $1 -type f -exec stat --format '%Y :%y %n' "{}" \; | sort -nr | cut -d: -f2- | head -n1

Notes:

sort -nr only sorted by lexical order, but because now the lexical order matches the time order, we can use in this way.