linux Bash脚本:从文件中读取30000条记录并并行运行多个进程

yqlxgs2m  于 2022-11-02  发布在  Linux
关注(0)|答案(3)|浏览(148)

txt,超过30000条记录。所有记录都是一个行,是这样的IP:

192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
192.168.0.6
192.168.0.7
192.168.0.8
192.168.0.9
192.168.0.10

我读取了bash脚本中的每一行,需要运行如下所示的curl:

while IFS= read -r line || [[ -n "$line" ]]; do
    #check_site "$line"
    resp=$(curl -i -m1 http://$line 2>&1)
    echo "$resp" | grep -Eo "$ok" > /dev/null

    if [ $? -ne 0 ]; then
        #echo -e "failed: $line" >> "${logfile}"
        echo -e "Command: curl -i -m1 http://$line 2>&1" >> "${outfile}"
        echo -e "failed: $line:\n\n \"$resp\"\n\n" >> "${outfile}"
        echo "$line" >> "${faillog}"
    fi
done < "${FILE}"

是否有方法可以在我的文件中同时运行多行以减少执行时间?

c9x0cxw0

c9x0cxw01#

Xargs就可以了。维基百科

lyfkaqu1

lyfkaqu12#

本文描述了解决并行执行的方法,它可以帮助您:Parallel execution in Bash
文章中的示例:


# !/bin/bash

RANDOM=10
JOBS_COUNTER=0
MAX_CHILDREN=10
MY_PID=$$

for i in {1..100}
do
    echo Cycle counter: $i
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    while [ $JOBS_COUNTER -ge $MAX_CHILDREN ]
    do
        JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
        echo Jobs counter: $JOBS_COUNTER
        sleep 1
    done
    sleep $(($RANDOM % 30)) &
done
echo Finishing children ...

# wait for children here

while [ $JOBS_COUNTER -gt 1 ]
do
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    echo Jobs counter: $JOBS_COUNTER
    sleep 1
done
echo Done
gdrx4gfi

gdrx4gfi3#

我用这样的方法求解多进程:


# export variable to be used into function

export outlog="/tmp/out.log"
export faillog="/tmp/fail.log"
export ok="(curl: \(7\) Failed to connect to)" # acceptable responses

# create function:

check_site() {
  ip=$1
  resp=$(curl -i -m1 http://$ip 2>&1)
  echo "$resp" | grep -Eo "$ok" > /dev/null

  if [ $? -ne 0 ]; then
    echo -e "Command: curl -i -m1 http://$ip 2>&1" >> "${outlog}"
    echo -e "Block failed: $ip:\n\n \"$resp\"\n\n" >> "${outlog}"
    echo "$ip" >> "${faillog}"
  fi
}

# call the function:

export -f check_site
parallel -j 252 -a "${FILE}" check_site

相关问题