我在kubernetes集群上部署wordpress+mysql应用程序时遇到了一个问题。
使用时 HorizontalPodAutoscaler
自动缩放我的 wordpress
以及 wordpress-mysql
在部署时,它对 wordpress
一个但不是那个 wordpress-mysql
一个。
事实上,当创建多个mysql pod时,有些会进入 CrashLoopBackOff
状态:
$ kubectl get all -n wordpress
NAME READY STATUS RESTARTS AGE
po/wordpress-3874566264-7031k 1/1 Running 0 16h
po/wordpress-mysql-898811424-2bdnn 0/1 CrashLoopBackOff 6 16h
po/wordpress-mysql-898811424-dxj92 1/1 Running 146 16h
po/wordpress-mysql-898811424-vs29j 0/1 CrashLoopBackOff 148 16h
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/wordpress 10.254.121.190 10.0.0.13 80:30003/TCP 16h
svc/wordpress-mysql None <none> 3306/TCP 16h
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa/wordpress Deployment/wordpress 28% / 80%, 0% / 80% 1 10 1 16h
hpa/wordpress-mysql Deployment/wordpress-mysql 90% / 80%, 0% / 80% 1 10 3 16h
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deploy/wordpress 1 1 1 1 16h
deploy/wordpress-mysql 3 3 3 1 16h
NAME DESIRED CURRENT READY AGE
rs/wordpress-3874566264 1 1 1 16h
rs/wordpress-mysql-898811424 3 3 1 16h
当我查看他们的日志时,我发现:
$ kubectl logs -p wordpress-mysql-898811424-2bdnn -n wordpress
2018-09-12 08:04:12 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-09-12 08:04:12 0 [Note] mysqld (mysqld 5.6.41) starting as process 436 ...
2018-09-12 08:04:12 436 [Note] Plugin 'FEDERATED' is disabled.
2018-09-12 08:04:12 436 [Note] InnoDB: Using atomics to ref count buffer pool pages
2018-09-12 08:04:12 436 [Note] InnoDB: The InnoDB memory heap is disabled
2018-09-12 08:04:12 436 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-09-12 08:04:12 436 [Note] InnoDB: Memory barrier is not used
2018-09-12 08:04:12 436 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-09-12 08:04:12 436 [Note] InnoDB: Using Linux native AIO
2018-09-12 08:04:12 436 [Note] InnoDB: Using CPU crc32 instructions
2018-09-12 08:04:12 436 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2018-09-12 08:04:12 436 [Note] InnoDB: Completed initialization of buffer pool
2018-09-12 08:04:12 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:12 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:12 436 [Note] InnoDB: Retrying to lock the first data file
2018-09-12 08:04:13 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:13 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:14 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:04:14 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:04:15 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
[...]
2018-09-12 08:05:51 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:05:52 436 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
2018-09-12 08:05:52 436 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2018-09-12 08:05:52 436 [Note] InnoDB: Unable to open the first data file
2018-09-12 08:05:52 7f57a329f5c0 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/operating-system-error-codes.html
2018-09-12 08:05:52 436 [ERROR] InnoDB: Can't open './ibdata1'
2018-09-12 08:05:52 436 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' init function returned error.
2018-09-12 08:05:52 436 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2018-09-12 08:05:52 436 [ERROR] Unknown/unsupported storage engine: InnoDB
2018-09-12 08:05:52 436 [ERROR] Aborting
2018-09-12 08:05:52 436 [Note] Binlog end
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'partition'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_METRICS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMPMEM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_CMP'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_LOCKS'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'INNODB_TRX'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'BLACKHOLE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'ARCHIVE'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MRG_MYISAM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MyISAM'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'MEMORY'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'CSV'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'sha256_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_old_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'mysql_native_password'
2018-09-12 08:05:52 436 [Note] Shutting down plugin 'binlog'
2018-09-12 08:05:52 436 [Note] mysqld: Shutdown complete
所以这可能很正常,因为每个mysql pod都试图访问 ./ibdata1
同时,我的问题是:有没有可能让几个mysql pod共享相同的数据?如果答案是肯定的,那么我应该如何避免这些恼人的错误呢?
如果你需要一些其他信息,只要告诉我,我会编辑我的帖子。
提前感谢您的帮助!
2条答案
按热度按时间zxlwwiss1#
vulvrdjw2#
所以这可能很正常,因为每个mysql pod都试图同时访问./ibdata1
是的,如果您尝试这样做(您没有提供清单),那么这正是您拥有crashloopbackoff状态的原因。首先启动的instace将锁定它,随后所有的操作都将失败。
... 正在尝试同时访问./ibdata1。。。有没有可能让几个mysql pod共享相同的数据?
如果我们在两个独立的mysql示例上讨论相同的数据文件夹(非常相同的持久性卷,或者hostpath或nfs共享…)——那么不,不是真的,并且出于许多原因,这是不可取的。
如果您需要多个mysql示例(进程、容器或pod)共享相同的数据(而不是数据文件夹!)您需要使用复制(与读取副本或任何…),其中每个示例都有自己的数据文件夹结构,但它们以某种方式在它们之间同步数据。下面是一个mysql单主拓扑的例子,其中多个从机在kubernetes官方文档上运行异步复制。请注意,这不是一个生产设置,也不是ha设置,只是一个简单复制场景的示例,让您有一个想法。
现在,有一些简单的问题:你确定你不能用一个mysql示例来处理负载,而这个mysql示例正在为几个wordpress示例提供服务吗?你是不是在试着做ha设置?因为每个问题的答案都需要不同的方法和架构决策,而不是“将pod的数量从1增加到1”。