Files - Bash Notes 3

The UNIX system treats everything as a file; even the command terminal is associated with a device file. Information can be written to the terminal by writing to this file, for example:

# 输出到当前终端
echo hoho > /dev/tty

# bash环境，发送到连接在ttys001端口的终端上
echo hoho > /dev/ttys001

dd

Used to generate files of a specified size, commonly used for disk read/write speed testing.

// 生成测试文件test.data
// 填充\0，大小为10M
dd if=/dev/zero of=test.data bs=1024 count=10k

// 写硬盘测速
// 对磁盘进行连续写入，不使用内存缓冲区，每次写入8k的数据，总共写入4k次，产生32M大小的文件
dd bs=8k count=4k if=/dev/zero of=test.data conv=fdatasync

// 读硬盘测速
dd if=test.data of=/dev/null bs=4k

if is the input file, of is the output file, and bs is the block size for writing. /dev/zero is a special device used to generate an infinite stream of \0 characters, and dev/null is a null device where any input is discarded.

P.S. On Mac, the conv parameter does not have the fdatasync and fsync options.

comm

The comm command is used to compare text files to achieve a diff. However, the input must be sorted files, so it is generally used in combination with sort:

# sort命令的-o选项表示输出到文件，这里直接替掉原文件
sort a.txt -o a.txt; sort b.txt -o b.txt
# 做diff
comm a.txt b.txt

P.S. The -o- option of wget looks interesting, meaning "Get as a file and print the result on STDOUT."

It produces three columns of results:

# a有b没有的 b有a没有的 ab都有的
    a差b       b差a     a交b

With these three columns, file a and file b can be restored (the sorted versions; the state before sorting cannot be recovered), for example a = (a minus b) union (a intersection b).

The -1/-2/-3 options suppress the specified columns. These options must appear before the input files, for example:

# 去掉第三列，不输出a交b
comm -3 a.txt b.txt

# 把ab的差异合并到1列，结果是ab有差异的行
comm -3 a.txt b.txt | sed $'s/\t//g'

Special attention: The $ operator before the regular expression is used for value evaluation. Without it, the tab character won't be matched, similar to IFS=$'\n'.

P.S. On Mac, in-place replacement with sed -i is quite troublesome; a backup filename must be specified (though it can be an empty string).

diff

Used to generate difference files, for example:

# u选项输出更常见的格式，写入diff.txt
diff -u a.txt b.txt > ab.diff

Compares files line by line to determine which lines were added and which were deleted (modifications are treated as adding new lines and deleting old ones). The result is similar to:

--- a.txt    2017-03-15 10:50:34.000000000 +0800
+++ b.txt   2017-03-19 16:56:14.000000000 +0800
@@ -1,6 +1,11 @@
+
+end yaya
 data
-end
 is
-line
 no
+line
 this
+
+newend
+line

Using the diff result to patch a or b can yield b or a:

# 给a打补丁
patch -p1 a.txt < ab.diff
# ab的内容相同
md5 a.txt; md5 b.txt

To restore the original content, patch it again:

# 补丁回来
patch -p1 a.txt < ab.diff

You can also perform a diff on folders:

# N不存在的文件视为空文件，a所有文件视为文本文件，r递归比较子目录
diff -Naur data files

mkdir

Used to create folders. If it already exists, it returns an error:

mkdir: bak: File exists

Generally, you need to check:

# 如果不存在的话，创建
if [ ! -e path ]; then mkdir $path; fi

For deep paths, checking level by level is tedious; you can simply discard the error:

mkdir ./dir1 2>/dev/null
mkdir ./dir1/dir2 2>/dev/null
mkdir ./dir1/dir2/dir3 2>/dev/null

A simpler approach is:

mkdir -p ./dir1/dir2/dir3

This ignores existing directories and creates those that are missing.

File Permissions

Commonly categorized into three types:

• user: The owner of the file

• group: The user group

• others: Users other than the owner and the group

The file permission format listed by ls -l is: File type (1 char) user permissions (3 chars) group permissions (3 chars) others permissions (3 chars)

File types are as follows:

- Regular file
d Directory
c Character device
b Block device
l Symbolic link
s Socket
p Pipe

The subsequent three sets of permissions each have four possible values: -rwx, representing None/Read/Write/Execute respectively.

P.S. If file permissions are ----------, it means no users except root have permissions to handle the file; it cannot be read, written, or executed.

Additionally, there are three special permissions that appear in the x position:

• setuid: Allows users to execute the file with the owner's permissions, e.g., -rws------

• setgid: Allows users to execute the file with the group's permissions, e.g., ----rws---

• sticky bit: Only the user who created the directory has permission to delete the files within it; other users cannot delete them even if they have write permissions, e.g., -------rwt

Note: Both s and t have uppercase and lowercase versions. Lowercase s or t indicates that the x (execute) permission is also present, while uppercase S or T indicates it is not.

Modifying Permissions

The chmod command, for example:

# 设置权限
chmod u=rwx g=rwx o=rwx test.sh
# 添加user执行权限
chmod u+x test.sh
# 添加所有执行权限
chmod a+x test.sh
# 删除user权限
chmod u-x test.sh

You can also set permissions using numerical values:

# 等价于u=rwx g=rwx o=rwx
chmod 777 test.sh

777 represents the decimal equivalent of the binary representation for the three sets of rwx. For instance, r-- is 4 (100).

P.S. This is generally called an octal value (since values 0 to 7 do not exceed 8?), but a binary explanation is actually more reasonable.

Setting the three special permissions also uses chmod:

# setuid, setgid, sticky bit
chmod u+s
chmod g+s
chmod o+t

When setting numerically, add a set of sst before the three standard sets. For example, the special permission in chmod 2777 test.sh is 2 (010), where -s- represents the setgid permission.

The command to modify ownership is chown:

# 设置属于user1用户，属于staff组
chown user1.staff test.sh

Generally used in conjunction with the setuid permission:

# root组root用户
chown root.root bomb.sh
# 如朕亲临
chown u+s bomb.sh

touch

touch it: if it exists, update the timestamp; if not, create an empty file:

# 存���的话，更新所有时间戳为当前时间
touch test.sh
# 存在的话，只更新访问时间
touch -a test.sh
# 存在的话，只更新修改时间
touch -m test.sh

head/tail

View only a specific part of the file content, for example:

# 看前10行
head test.sh
# 看前3行
head -n 3 test.sh
# 不看后10行
head -n -10 test.sh

# 看后10行
tail test.sh
# 看后3行
tail -n 3 test.sh
# 不看前90行（输出第91行到结束）
seq 100 | tail -n +91

P.S. On Mac, the -n parameter cannot be negative; it will throw an error: head: illegal line count -- -10.

ls: List Directories Only

There are three methods:

# d选项最简洁
ls -d */
# F添上类型标识，筛选/结尾的
ls -F | grep "/$"
# l选项结果以权限开头，权限以类型开头，筛选d开头的
ls -l | grep "^d" | awk '{print $9"/"}'

You can also use find:

# 按文件类型查找
find . -type d -maxdepth 1 -mindepth 1 -print

Note the depth range to search only the next level.

Path Switching

cd - (cd $OLDPWD) is often used to return to the last working directory. In fact, there is something even more powerful:

# 与cd作用看似相同
pushd /tmp
# 查看历史路径栈（-v编号）
dirs -v
# 返回上一次工作路径
pushd
# 返回上上一次...
pushd +1
# 返回上上上一次...
pushd +2

pushd is used to switch the working directory by swapping the top record of the stack with a specified record, but the length of the history stack remains the same. If you need to delete a record, use popd:

# 返回上一次工作路径，并删除当前路径
popd
# 删除上上一次工作路径
popd +1
# 清空历史栈（只保留当前记录）
dirs -c

+N and -N indicate the direction: +N counts from the top of the stack (0123...), and -N counts from the bottom.

P.S. Note that in a certain version of oh-my-zsh, the +/- directions are reversed, while bash is normal.

wc

Counts lines, words, and characters; often used for simple code statistics:

# 输出行数、单词数、字符数
wc test.sh
# 取行数
wc -l test.sh | awk '{print $1}'
# 取单词数
wc -w test.sh | awk '{print $1}'
# 取字符数
wc -c test.sh | awk '{print $1}'

P.S. The word counting feature is quite weak; any string separated by spaces is counted as a word, and it does not distinguish punctuation.