有没有办法在airflow dag中运行docker命令?

mrfwxfqh  于 2023-04-11  发布在  Docker
关注(0)|答案(1)|浏览(122)

我尝试在airflow中运行docker container。
然而,由于env问题,DockerOperator无法导入。因此我尝试使用BashOperator运行Docker容器。
这是一个示例运算符,我正在尝试运行

run_model_op = BashOperator(
                task_id=f'run_model_GPU{gpu}',
                env={"CUDA_VISIBLE_DEVICES":gpu},
                bash_command='docker images',
                append_env=True,
                dag=dag,
            )

最后的图看起来像这样,在节点中使用多个GPU并行运行

start - run_model_op(gpu0)- run_model_op_3(gpu0)...  
      - run_model_op_1(gpu1)- run_model_op_2(gpu1)...  
      ...

但是,运行此dag会引发此错误

[2023-04-07, 22:29:19 KST] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'docker images']
[2023-04-07, 22:29:19 KST] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/operators/bash.py", line 185, in execute
    result = self.subprocess_hook.run_command(
  File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/hooks/subprocess.py", line 76, in run_command
    self.sub_process = Popen(
  File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/subprocess.py", line 1626, in _execute_child
    env_list.append(k + b'=' + os.fsencode(v))
  File "/home/airflow/anaconda3/envs/airflow/lib/python3.8/os.py", line 806, in fsencode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not int

有什么问题吗?先谢谢你。

iezvtpos

iezvtpos1#

问题是你的env变量CUDA_VISIBLE_DEVICES的值应该是str,而不是int。用str方法 Package 它就足够了:

...
env={"CUDA_VISIBLE_DEVICES":str(gpu)},
...

相关问题