shell笔记

[uniquejava]

shell

变量与变量展开

hello=world
echo $hello
hello="i am world"
echo $hello
echo hello world
echo -n "enter your name:" (-n去掉行尾的回车)
read name
echo $name

1. $是取变量的值，所以总是出现在=号的右边
2. =号前后不能有任何空格
3. =号后面的字符串如果不包含空格可以不加""
4. echo后面字符串即使包含空格也可不加"", 如果要正常输出变量中的换行符，则必须加双引号
5. shell视所有变量为字符串类型.
6. 单引号不展开变量, 仅原样输出字符串, 双引号会展开变量.
7. 强制展开变量的办法：嵌套使用单引号，或者整体使用双引号或者拼字符串
   $ sed -n '1 c$HOME' package.json(不会展开变量)
   $HOME
   $ sed -n "1 c$HOME" package.json(整体使用双引号)
   /home/cyper
   $ sed -n '1 c'$HOME'' package.json(嵌套使用单引号)
   /home/cyper
   $ sed -n '1 c'$HOME package.json(拼字符串)
   /home/cyper

输出重定向

3个特殊的文件描述符0 stdin, 1 stdout, 2 stderr.

格式:command FILE_DESCRIPTOR>outputFileName

foo 1>ok.txt 2>error.txt

因为foo命令不存在，所以出错信息输出到了error.txt中，注意到同时生成了一个空的ok.txt文件, 其中标准输出重定向1>可以简写成>

怎么将正常的及错误的消息输出到同一文件foo 1>ok.txt 2>&1， 即使用&1来表示文件描述符为1对应的输出文件，在这里它们同时指向ok.txt.

引号

'原样输出
"会先做expansion

参数

$HOME 打印出/home/cyper
$0: shell script的名字
$#: 参数个数
$$: 运行脚本时的process ID
$1,$2, ...传递给脚本的参数
$*: 所有参数使用$IFS分隔(默认为空格）
$@: 同$*但总是使用空格分隔
$?: 命令的退出状态,0成功，非0失败
PS1,PS2: 命令提示符

快速测试

 $ IFS=’‘
 $ set foo bar bam
 $ echo “$@“
 foo bar bam
 $ echo “$*“
 foobarbam
 $ unset IFS
 $ echo “$*“
 foo bar bam

条件测试

if test -f fred
then
...
elif test -d fred; then
...
else
...
fi

或

if [ -f fred.c ]
then
...
fi

或

if [ -f fred.c ]; then
...
fi

[]和其中包含的测试条件必须用空格隔开

条件比较

string comparison	result
str1 = str2	string equal
str1 != str2	string not equal
-n str	string is not null
-z str	string is null(an empty string)
num1 -eq num2	exprs are equal
eq, ne, gt, ge, lt, le	...
! expr1	is false?
-d file	is directory?
-e file	exist?
-f file	is file?
-r file	is readable?
-w file	is writable?
-x file	is executable?
-g file	if set-group-id is set on file?
-u file	if set-user-id is set on file?

You may be wondering what the set-group-id and set-user-id (also known as set-gid and set-uid) bits are. The set-uid bit gives a program the permissions of its owner, rather than its user, while the set-gid bit gives a program the permissions of its group.The bits are set with chmod,using the s and g options.The set-gid and set-uid flags have no effect on files containing shell scripts, only on executable binary files.

特殊情况

if [ $timeofday = "yes" ]这段可能有问题, 当timeofday为空的时候if [ = "yes"]语法不合规,最好写成if [ "$timeofday" = "yes" ]

for

for i in foo bar 43
do
 echo $i
done

或

for i in $(ls f*.sh); do
 lpr $file
done

while

echo "Enter password"
read trythis
while [ "$trythis" != "secret" ]; do
 echo "sorry, try again"
 read trythis
done

util

who查看当前系统有哪个登录用户, 比如who|grep cyper返回0, 而who|grep green返回1, 这里我们只关心script的执行情况(0 or 1), 不关心stdout.

# 如果参数1对应的用户没有出现, 就等待60s后再次检测
until who | grep "$1" > /dev/null
do
 sleep 60
done

# 响铃
echo -e '\a'
echo "**** $1 has just logged in ****"

exit 0

case

基本结构

case "$x" in
 pattern | pattern2 ) statements;
 * ) echo "default case";
esac

例子：

#!/bin/sh
echo "Enter your name?"
read timeofday
case "$timeofday" in
 yes | y | Yes | YES )
 echo "Good morning"
 echo "It's cool"
 ;
 [nN]*)
 echo "Good afternoon"
 ;
 *)
 echo "sorry"
 exit 1
 ;
esac

exit 0

AND

打印出hello in else

touch file_one
rm -rf file_two
if [ -f file_one ] && echo "hello" && [ -f file_two ] && echo " there"
then
 echo "in if"
else
 echo "in else"
fi

OR

打印出hello in if

rm -f file_one
if [ -f file_one ] || echo "hello" || echo "there"
then
 echo "in if"
else
 echo "in else"
fi

Statement Blocks

可以用AND/OR选择性的执行语句块

get_confirm &&{
 echo "hello"
 echo "dome some other thing"
}

braces and brackets

braces()中的command在subshell中执行，在期间修改的env var不会影响current shell(比如执行了cd命令).在brackets{}中包含的command在当前shell中执行.

set -e

遇到错误即退出, 不继续执行后续脚本

set -o noclobber

打开此选项是为了防止使用重定向符>意外覆盖文件，如果要强制覆盖，使用>|.

Here document

用来快速提供多行输入，常用于交互式命令比如telnet, ftp.

command << terminaor [input ...]
或者
command <<- terminator [input ...]

每次读一行，直到读到某行是terminator时结束,<<-会把input中每行前面的tabs去掉。
登录到一个远程机器并把某文件mail给本机。

$ telnet << END
> open 190.100.2.1
> gregl password
> mail -s "remote userlogs" gregl@190.100.2.2 < /var/log/userlog
> quit
> END

substitution

文件名替换: 使用w*,?,[a-z],
命令替换: $(command), $(< filename)

* 0或多个字符
? 单个字符
[a-z0-9] 范围内单个字符
[!0-9] 范围外单个字符，和正则^不一样
$(command) 执行command并把输出结果作为其它命令的参数
$(< filename) 将filename的内容作为其它命令的参数
echo '*' 如果需要转递特殊字符做为参数，需要使用引号或\以防止该符号被shell解释.
echo \* 同上
echo "*" 同上

Date

DATE=`date +%Y-%m-%d`
$(date +%F)
DATE=`date +%Y-%m-%d:%H:%M:%S`

其中%Y-%m-%d <==> %F, 输出2017-04-26

[uniquejava]

查看端口占用

macOS: https://stackoverflow.com/questions/4421633/who-is-listening-on-a-given-tcp-port-on-mac-os-x

lsof -n -i4TCP:$PORT | grep LISTEN
lsof -n -iTCP:$PORT | grep LISTEN
lsof -n -i:$PORT | grep LISTEN

试了最后一个很好用. 难记的命令我都习惯建个alias, 如何给alias传参数, 请看下回分解.

给alias传参数

完美的方案如下:

alias blah='function _blah(){echo "First: $1"; echo "Second: $2"};_blah'

见: https://stackoverflow.com/questions/941338/how-to-pass-command-line-arguments-to-a-shell-alias

然后

alias port='function _blah(){lsof -n -i:$1 | grep LISTEN};_blah'

哈哈, 查看8080被哪个进程占用. port 8080 既可, 爽!.......

取文件名和extension

First, get file name without the path:

filename=$(basename "$fullfile")
extension="${filename##*.}"
filename="${filename%.*}"

Alternatively, you can focus on the last '/' of the path instead of the '.' which should work even if you have unpredictable file extensions:

filename="${fullfile##*/}"

#是删除之后匹配的pattern的意思, %是取之后匹配的pattern. ## 和 %% 是最长match, 单个字符是最少match.

见: https://stackoverflow.com/questions/965053/extract-filename-and-extension-in-bash

把逗号替换成换行

sed 's/old/new/g' input.txt > output.txt

方法一

sed 's/,/\
/g'

方法二

tr , '\n' < file

方法三

Use $'string'

You need a backslash-escaped literal newline to get to sed. In bash at least, $'' strings will replace \n with a real newline, but then you have to double the backslash that sed will see to escape the newline, e.g.

echo "a,b" | sed -e $'s/,/\\\n/g'

https://stackoverflow.com/questions/10748453/replace-comma-with-newline-in-sed

判断字符串是否包含substring

You need to interpolate the $testseq variable with one of the following ways:

`$file == *_"$testseq"_*` (here $testseq considered as a fixed string)

`$file == *_${testseq}_*` (here $testseq considered as a pattern).

file="JetConst_reco_allconst_4j2t.png"
testseq="gen"

case "$file" in
 *_${testseq}_*) echo 'True' ;
 *) echo 'False' ;
esac

see https://unix.stackexchange.com/questions/370889/test-if-a-string-contains-a-substring

[uniquejava]

遍历文件中的每一行

不能简单用for x in $(cat file), 因为默认的分隔符为空格或换行, 如果行内有空格会变成多行输出.
解决办法: https://unix.stackexchange.com/questions/7011/how-to-loop-over-the-lines-of-a-file

#!/bin/bash

IFS=$'\n' # make newlines the only separator
set -f # disable globbing
for i in $(cat < "$1"); do
 echo "tester: $i"
done

find

cd /etc/httpd
find ./ -type f -name "*.conf" -exec grep -i 'hello'{} \;


1. 查找所有".h"文件

find /PATH -name "*.h"

2. 查找所有".h"文件中的含有"helloworld"字符串的文件

find /PATH -name "*.h" -exec grep -in "helloworld"{} \;

find /PATH -name "*.h" | xargs grep -in "helloworld"

3. 查找所有".h"和".c"文件中的含有"helloworld"字符串的文件

find /PATH /( -name "*.h" -or -name "*.c" /) -exec grep -in "helloworld"{} \;

4. 查找非备份文件中的含有"helloworld"字符串的文件

find /PATH /( -not -name "*~" /) -exec grep -in "helloworld"{} \;

注：/PATH为查找路径，默认为当前路径。带-exec参数时必须以\;结尾，否则会提示“find: 遗漏“-exec”的参数”。

grep

实战一:

find . -type f -name "*.properties" | xargs grep "error.hello.world"

http://www.cnblogs.com/end/archive/2012/02/21/2360965.html

实战二(搜索并显示文件名):

只显示文件名
find . -name "*.pat" -exec grep -in "?|たら|たい"{} -l \;

显示文件名和行号
find . -name "*.pat" -exec grep -in "?|たら|たい"{} -H \;

判断

[ -e file ] || mkdir file #文件file不存在则mkdir file
test -e file || mkdir file #文件file不存在则mkdir file
ls *head.csv &>/dev/null || (echo "diff directory is damaged!" && exit 1)

查看系统运行时间

$ uptime
 09:24:05 up 15 days, 6:44, 2 users, load average: 0.02, 0.02, 0.05
$ date -d "$(awk -F. '{print $1}' /proc/uptime) second ago" +"%Y-%m-%d %H:%M:%S"
2018-02-25 02:39:19
$ cat /proc/uptime| awk -F. '{run_days=$1 / 86400;run_hour=($1 % 86400)/3600;run_minute=($1 % 3600)/60;run_second=$1 % 60;printf("系统已 运行：%d天%d时%d分%d秒",run_days,run_hour,run_minute,run_second)}'
系统已运行：15天6时45分37秒

数学计算

http://faculty.salina.k-state.edu/tim/unix_sg/bash/math.html
https://unix.stackexchange.com/questions/55069/how-to-add-arithmetic-variables-in-a-script

i=1
a=$i+1 #不可以
let b=$i+1 #可以计算
c=`expr $i + 1` #可以
d=$(($i+1)) #可以
((e=$i+1)) #可以
f=(($i + 1)) #不可以

rgb2hex

rgb2hex(){
 for var in "$@"
 do
 printf '%x' "$var";
 done
 printf '\n'
}

base64加密字符串

echo -n 'input' | openssl base64

[uniquejava]

取得脚本文件所在的目录及parent目录

cd "$( dirname "${BASH_SOURCE[0]}" )"/..

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" # 脚本所在目录
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )"/.. && pwd )" # 父目录

递归删除文件

search_dir=/Users/xx/xxxx/data/master_config2

function remove(){
 echo "removing $1"

 if [ -d "$1" ];then
 rm -rf $1 2> /dev/null
 else
 rm $1 2> /dev/null
 fi

}

for entry in "$search_dir"/*.indexservice
do
 if [ -d "$entry" ];then
 echo "* $entry"
 remove $entry/.git
 remove $entry/wes_db*.db
 remove $entry/wes_config
 remove $entry/resource/pattern/wes_deploy_version.pat
 fi
done

[uniquejava]

timezone

方法一
https://stackoverflow.com/a/12523283/2497876

方法二：
https://serverfault.com/a/925410

我的：

unameOut="$(uname -s)"
case "${unameOut}" in
 Linux*) machine=Linux;
 Darwin*) machine=Mac;
 *) echo "Unsupported OS:${unameOut}"; exit 0;
esac

echo "Running Xxx $TAG on $unameOut"

if [ "$machine" = "Mac" ]; then
 MAC_ADDR=$(ifconfig | grep "ether*" | tr -d ' ' | tr -d '\t' | cut -c 6-42 | head -1)
 OLSONTZ=`readlink /etc/localtime | sed "s/\/var\/db\/timezone\/zoneinfo\///"`
else
 MAC_ADDR=$(cat /sys/class/net/$(ip route show default | awk '/default/{print $5}')/address)
 OLSONTZ=`readlink /etc/localtime | sed "s/\.\.//" | sed "s/\/usr\/share\/zoneinfo\///"`
fi

echo
echo "Current time zone: $OLSONTZ"

Sed

查找替换

OLD="x.y.z=/hello/cyper"
NEW="x.y.z=/hello/green"
sed "s $OLD $NEW " application.properties > hello.properties

直接操作文件本身 (使用-i)

sed -i "s $OLD $NEW " application.properties

改变echo的文字颜色

function build_be(){
 echo
 echo -e "\033[1;34m> Building back end...\033[0m"
 echo

见: How to change the output color of echo in Linux

调用其它bash脚本

#! /bin/bash
set -e

cd "$( dirname "${BASH_SOURCE[0]}" )"/..
echo Basedir: `pwd`
echo

echo -n "Please enter version number(example: 1.0.0): "
read version_number

if [ -z "$version_number" ] ;then
 echo User cancelled.
 echo
 exit 0
fi


`pwd`/scripts/build_xxx.sh $version_number

接受参数

# release mode
if [[ $# -gt 0 ]]; then
 echo "> Update version # as $1"
 echo "$1" > ./xxx-api/src/main/resources/version.txt
 build_fe && build_be
 exit
fi

判断OS

https://stackoverflow.com/questions/3466166/how-to-check-if-running-in-cygwin-mac-or-linux

取mac地址

https://stackoverflow.com/questions/23828413/get-mac-address-using-shell-script

[uniquejava]

字符串比较startsWith endsWith

有两种方式 if 和 case, case更短， case大法👌

for i in $(git branch)
if [[ $i == abc ]] || [[ $i=def* ]];then
git branch -D $i
fi


for i in $(git branch)
case $i in abc | def*)
git branch -D $i
esac

见： In Bash, how can I check if a string begins with some value?