多个mysql kubernetes pods具有crashloopbackoff状态

uinbv5nw  于 2021-06-21  发布在  Mysql
关注(0)|答案(2)|浏览(578)

我在kubernetes集群上部署wordpress+mysql应用程序时遇到了一个问题。
使用时 HorizontalPodAutoscaler 自动缩放我的 wordpress 以及 wordpress-mysql 在部署时，它对 wordpress 一个但不是那个 wordpress-mysql 一个。
事实上，当创建多个mysql pod时，有些会进入 CrashLoopBackOff 状态：

$ kubectl get all -n wordpress
NAME                                 READY     STATUS             RESTARTS   AGE
po/wordpress-3874566264-7031k        1/1       Running            0          16h
po/wordpress-mysql-898811424-2bdnn   0/1       CrashLoopBackOff   6          16h
po/wordpress-mysql-898811424-dxj92   1/1       Running            146        16h
po/wordpress-mysql-898811424-vs29j   0/1       CrashLoopBackOff   148        16h

NAME                  CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
svc/wordpress         10.254.121.190   10.0.0.13     80:30003/TCP   16h
svc/wordpress-mysql   None             <none>        3306/TCP       16h

NAME                  REFERENCE                    TARGETS               MINPODS   MAXPODS   REPLICAS   AGE
hpa/wordpress         Deployment/wordpress         28% / 80%, 0% / 80%   1         10        1          16h
hpa/wordpress-mysql   Deployment/wordpress-mysql   90% / 80%, 0% / 80%   1         10        3          16h

NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/wordpress         1         1         1            1           16h
deploy/wordpress-mysql   3         3         3            1           16h

NAME                           DESIRED   CURRENT   READY     AGE
rs/wordpress-3874566264        1         1         1         16h
rs/wordpress-mysql-898811424   3         3         1         16h

当我查看他们的日志时，我发现：

$ kubectl logs -p wordpress-mysql-898811424-2bdnn -n wordpress
2018-09-12 08:04:12 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-09-12 08:04:12 0 [Note] mysqld (mysqld 5.6.41) starting as process 436 ...
2018-09-12 08:04:12 436 [Note] Plugin 'FEDERATED' is disabled.
2018-09-12 08:04:12 436 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-09-12 08:04:12 436 [Note] InnoDB: The InnoDB memory heap is disabled
2018-09-12 08:04:12 436 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-09-12 08:04:12 436 [Note] InnoDB: Memory barrier is not used
2018-09-12 08:04:12 436 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-09-12 08:04:12 436 [Note] InnoDB: Using Linux native AIO
2018-09-12 08:04:12 436 [Note] InnoDB: Using CPU crc32 instructions
2018-09-12 08:04:12 436 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-09-12 08:04:12 436 [Note] InnoDB: Completed initialization of buffer pool
2018-09-12 08:04:12 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:12 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:12 436 [Note] InnoDB: Retrying to lock the first data file
2018-09-12 08:04:13 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:13 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:14 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:14 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:15 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
[...]
2018-09-12 08:05:51 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:05:52 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:05:52 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:05:52 436 [Note] InnoDB: Unable to open the first data file
2018-09-12 08:05:52 7f57a329f5c0  InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2018-09-12 08:05:52 436 [ERROR] InnoDB: Can't open './ibdata1'
2018-09-12 08:05:52 436 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' init function returned error.
2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-09-12 08:05:52 436 [ERROR] Unknown/unsupported storage engine: InnoDB
2018-09-12 08:05:52 436 [ERROR] Aborting

2018-09-12 08:05:52 436 [Note] Binlog end
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'partition'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_METRICS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCKS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_TRX'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'BLACKHOLE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'ARCHIVE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MRG_MYISAM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MyISAM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MEMORY'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'CSV'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'sha256_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_old_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_native_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'binlog'
2018-09-12 08:05:52 436 [Note] mysqld: Shutdown complete

所以这可能很正常，因为每个mysql pod都试图访问 ./ibdata1 同时，我的问题是：有没有可能让几个mysql pod共享相同的数据？如果答案是肯定的，那么我应该如何避免这些恼人的错误呢？
如果你需要一些其他信息，只要告诉我，我会编辑我的帖子。
提前感谢您的帮助！

sqlmysqlinnodbkubernetes

2条答案

按热度按时间
zxlwwiss

zxlwwiss1#

The port name is the same as any pod within your cluster. 
ex:

containers:
  - image: mysql:5.6
    name: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mysql-pass
          key: password
    ports:
    - containerPort: 3306
      name: mysql-wiki
    volumeMounts:
    - name: mysql-persistent-storage
      mountPath: /var/lib/mysql
  volumes:
  - name: mysql-persistent-storage
    persistentVolumeClaim:
      claimName: mysql-pv-claim
赞(0）分享  回复(0）举报  2021-06-21
vulvrdjw

vulvrdjw2#

所以这可能很正常，因为每个mysql pod都试图同时访问./ibdata1
是的，如果您尝试这样做（您没有提供清单），那么这正是您拥有crashloopbackoff状态的原因。首先启动的instace将锁定它，随后所有的操作都将失败。
... 正在尝试同时访问./ibdata1。。。有没有可能让几个mysql pod共享相同的数据？
如果我们在两个独立的mysql示例上讨论相同的数据文件夹（非常相同的持久性卷，或者hostpath或nfs共享…）——那么不，不是真的，并且出于许多原因，这是不可取的。
如果您需要多个mysql示例（进程、容器或pod）共享相同的数据（而不是数据文件夹！）您需要使用复制（与读取副本或任何…），其中每个示例都有自己的数据文件夹结构，但它们以某种方式在它们之间同步数据。下面是一个mysql单主拓扑的例子，其中多个从机在kubernetes官方文档上运行异步复制。请注意，这不是一个生产设置，也不是ha设置，只是一个简单复制场景的示例，让您有一个想法。
现在，有一些简单的问题：你确定你不能用一个mysql示例来处理负载，而这个mysql示例正在为几个wordpress示例提供服务吗？你是不是在试着做ha设置？因为每个问题的答案都需要不同的方法和架构决策，而不是“将pod的数量从1增加到1”。

赞(0）分享  回复(0）举报  2021-06-21
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com