jupyterlab容器无法连接到端口9092上的kafka localhost服务器

我有docker compose包含一个kafka服务器、一个zookeeper、spark和一个jupyter-lab。这是我的jupyter-lab dockerfile代码

ARG debian_buster_image_tag=8-jre-slim
FROM openjdk:${debian_buster_image_tag}
RUN mkdir -p ${shared_workspace} && \
    apt-get update -y && \
    apt-get install -y python3 && \
    ln -s /usr/bin/python3 /usr/bin/python && \
    rm -rf /var/lib/apt/lists/* && \
    apt-get install -y python3-pip && \
    pip3 install pyspark==3.0.1 jupyterlab==2.1.5

这是我的docker-compose.yml代码

version: "3.6"
volumes:
  shared-workspace:
    name: "hadoop-distributed-file-system"
    driver: local
services:
  jupyterlab:
    image: jupyterlab
    container_name: jupyterlab
    ports:
      - 8888:8888
    volumes:
      - shared-workspace:/opt/workspace

  spark-master:
    image: spark-master
    container_name: spark-master
    ports:
      - 8080:8080
      - 7077:7077
    volumes:
      - shared-workspace:/opt/workspace
  spark-worker-1:
    image: spark-worker
    container_name: spark-worker-1
    environment:
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=512m
    ports:
      - 8081:8081
    volumes:
      - shared-workspace:/opt/workspace
    depends_on:
      - spark-master
  spark-worker-2:
    image: spark-worker
    container_name: spark-worker-2
    environment:
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=512m
    ports:
      - 8082:8081
    volumes:
      - shared-workspace:/opt/workspace
    depends_on:
      - spark-master

  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    expose:
    - "2181"

  kafka:
    image: wurstmeister/kafka:latest
    depends_on:
    - zookeeper
    ports:
    - "9092:9092"
    expose:
    - "9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

在jupyter实验室我无法进入kafka:9093, localhost:9092 by sparksession.readstream.format（“kafka”）.option（“kafka.bootstrap.servers”，”kafka:9093“”…，显示错误：

20/12/03 16:10:23 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4b4422bc-2a6a-456a-b750-4fba8300328c-37706672-driver-0-1, groupId=spark-kafka-source-4b4422bc-2a6a-456a-b750-4fba8300328c-37706672-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

我还安装了telnet通过jupyterlab的bash检查连接，我可以telnet到一些服务器，比如 telnet www.cyberciti.biz 80 ，但无法连接到localhost:9092. 请帮帮我。

根据@onecricketeer的说法，我认为我之前的答案是一条红鲱鱼。
所以，我查了你的 Dockerfile 以及 docker-compose.yaml 而且，这对我很有效。

ARG debian_buster_image_tag=8-jre-slim
FROM openjdk:${debian_buster_image_tag}

RUN apt-get update -y && \
    apt-get install -y python3 python3-pip && \
    rm -rf /var/lib/apt/lists/* && \
    pip3 install pyspark==3.0.1 jupyterlab==2.1.5

ENTRYPOINT [ "jupyter","notebook","--allow-root","--ip=0.0.0.0"]

笔记
远离的 ${shared_workspace} 因为这看起来 /opt/workspace 重新排序 apt-get install s
补充 ENTRYPOINT 跑 jupyter-notebook --allow-root -ip=0.0.0.0 这将公开上的笔记本服务 localhost:8888 但需要一个令牌。
调整了合成：

services:
  jupyterlab:
    image: jupyterlab
    build:
      context: .
      dockerfile: ./Dockerfile
    container_name: jupyterlab
    ports:
      - 8888:8888
    volumes:
      - shared-workspace:/opt/workspace

然后：

docker-compose build && docker-compose up

以及 docker-compose ps :

Name                      Command               State                            Ports                          
------------------------------------------------------------------------------------------------------------------------
65130144_kafka_1       start-kafka.sh                   Up      0.0.0.0:9092->9092/tcp, 9093/tcp                        
65130144_zookeeper_1   /bin/sh -c /usr/sbin/sshd  ...   Up      2181/tcp, 22/tcp, 2888/tcp, 3888/tcp                    
jupyterlab             jupyter notebook --allow-r ...   Up      0.0.0.0:8888->8888/tcp                                  
spark-master           /bin/bash /master.sh             Up      6066/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
spark-worker-1         /bin/bash /worker.sh             Up      0.0.0.0:8081->8081/tcp                                  
spark-worker-2         /bin/bash /worker.sh             Up      0.0.0.0:8082->8081/tcp

你可能想要笔记本的代币，用吧 docker-compose logs jupyterlab :

Attaching to jupyterlab
jupyterlab        | [I 17:57:08.412 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
jupyterlab        | [I 17:57:08.659 NotebookApp] JupyterLab extension loaded from /usr/local/lib/python3.7/dist-packages/jupyterlab
jupyterlab        | [I 17:57:08.659 NotebookApp] JupyterLab application directory is /usr/local/share/jupyter/lab
jupyterlab        | [I 17:57:08.662 NotebookApp] Serving notebooks from local directory: /
jupyterlab        | [I 17:57:08.662 NotebookApp] Jupyter Notebook 6.1.5 is running at:
jupyterlab        | [I 17:57:08.662 NotebookApp] http://1b77e2b01d7b:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
jupyterlab        | [I 17:57:08.662 NotebookApp]  or http://127.0.0.1:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
jupyterlab        | [I 17:57:08.662 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
jupyterlab        | [W 17:57:08.665 NotebookApp] No web browser found: could not locate runnable browser.
jupyterlab        | [C 17:57:08.666 NotebookApp] 
jupyterlab        |     
jupyterlab        |     To access the notebook, open this file in a browser:
jupyterlab        |         file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
jupyterlab        |     Or copy and paste one of these URLs:
jupyterlab        |         http://1b77e2b01d7b:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
jupyterlab        |      or http://127.0.0.1:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457

使用第二个链接（您需要使用生成的任何令牌），创建一个笔记本并添加：

from pyspark.sql import SparkSession;

spark = SparkSession \
    .builder \
    .appName("test") \
    .getOrCreate()

spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9093")

然后运行它：

<pyspark.sql.streaming.DataStreamReader at 0x7fbf8defe208>

似乎是（！？）好啊。

我怀疑 jupyter-lab 以及 jupyterlab 在配置中是相同的。
dockerfile不运行（否 CMD 或者 ENTRYPOINT )所以我也假设你在里面使用了一个shell。这很好只是不寻常。
有几件事 kafka 服务（注意，我只对Kafka略知一二）：
在 kafka 您必须将其引用为 localhost 你有 INSIDE://kafka:9093 但这有两个问题。在服务内部，它应该将自己引用为 localhost 在服务区内我怀疑它在港口 :8082 声明 expose: - "9093" 不是功能性的，而是记录片，除非你的图像也在无声地曝光，否则是不正确的 :9093 ; 这实际上没有暴露任何东西，因此可能是不正确的。

jupyterlab容器无法连接到端口9092上的kafka localhost服务器

1条答案

相关问题

热门标签

最新问答