我尝试在airflow中运行docker container。
然而,由于env问题,DockerOperator无法导入。因此我尝试使用BashOperator运行Docker容器。
这是一个示例运算符,我正在尝试运行
run_model_op = BashOperator(
task_id=f'run_model_GPU{gpu}',
env={"CUDA_VISIBLE_DEVICES":gpu},
bash_command='docker images',
append_env=True,
dag=dag,
)
最后的图看起来像这样,在节点中使用多个GPU并行运行
start - run_model_op(gpu0)- run_model_op_3(gpu0)...
- run_model_op_1(gpu1)- run_model_op_2(gpu1)...
...
但是,运行此dag会引发此错误
[2023-04-07, 22:29:19 KST] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'docker images']
[2023-04-07, 22:29:19 KST] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/operators/bash.py", line 185, in execute
result = self.subprocess_hook.run_command(
File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/hooks/subprocess.py", line 76, in run_command
self.sub_process = Popen(
File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/subprocess.py", line 858, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/subprocess.py", line 1626, in _execute_child
env_list.append(k + b'=' + os.fsencode(v))
File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/os.py", line 806, in fsencode
filename = fspath(filename) # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not int
有什么问题吗?先谢谢你。
1条答案
按热度按时间iezvtpos1#
问题是你的env变量CUDA_VISIBLE_DEVICES的值应该是str,而不是int。用str方法 Package 它就足够了: