elasticsearch Monstache未从MongoDB启动当前数据的首次同步，并一直等待变更流上的事件

3duebb1j  于 2023-01-29  发布在  ElasticSearch
关注(0)|答案(1)|浏览(165)

这是我第一次用小胡子。
事实上，我已经把我的基础设施从内部迁移到了云端，现在我正在使用mongoDB Atlas和AWS opensearch。
我已经在awsec 2示例上安装了monstache，并对其进行了良好的配置。一切看起来都很正常，monstache连接到了Elasticsearch和MongoDB，但它正在索引已经迁移到Elasticsearch中的mongoDB atlas的文档。它一直在等待我的集合/索引上的事件，如下所示

[ec2-user@ip-172-31-1-200 ~]$ journalctl -u monstache.service -f
-- Logs begin at Wed 2022-11-09 10:22:04 UTC. --
Jan 26 08:54:00 ip-172-31-1-200.eu-west-3.compute.internal systemd[1]: Starting monstache sync service...
Jan 26 08:54:00 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:00 Started monstache version 6.1.0
Jan 26 08:54:00 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:00 Successfully connected to MongoDB version 4.4.18
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Successfully connected to Elasticsearch version 7.10.2
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal systemd[1]: Started monstache sync service.
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Joined cluster HA
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Starting work for cluster HA
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Listening for events
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Watching changes on collection wtlive.myuser
Jan 26 08:54:01 ip-172-31-1-200.eu-west-3.compute.internal monstache[27813]: INFO 2023/01/26 08:54:01 Resuming from timestamp {T:1674723241 I:1}

我真的应该在mongoDB集合上启动一个写操作来让monstache开始同步吗？为什么它不开始同步mongoDB中的当前数据呢？
我的elasticsearch仍然显示0文档计数，而mongoDB中的集合充满了文档。

[ec2-user@ip-172-31-0-5 ~]$ curl --insecure -u es-appuser https://vpc-wtlive-domain-staging-om2cbdeex4qk6trkdrcb3dg4vm.eu-west-3.es.amazonaws.com/_cat/indices?v
Enter host password for user 'es-appuser':
health status index                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   wtlive.myuser         YzqLx9_uTZ2qFVjFF2CMag   1   1          0            0       416b           208b
green  open   .opendistro_security          Jb1fLqGjRd2vvluoX-ZgKw   1   1          9            4      129kb         64.4kb
green  open   .kibana_1                     v6WdqQDvSN2L16EZTXxuHQ   1   1         30            2     70.4kb         33.4kb
green  open   .kibana_252235597_esappuser_1 OY1bbDGvTqK8oEgwopzbhQ   1   1          1            0     10.1kb            5kb
[ec2-user@ip-172-31-0-5 ~]$

下面是我的monstache配置：

[ec2-user@ip-172-31-1-200 ~]$ cat monstache/wtlive_pipeline.toml
enable-http-server = true
http-server-addr = ":8888"
#direct-read-namespaces = ["wtlive.myuser"]
change-stream-namespaces = ["wtlive.myuser"]
namespace-regex = '^wtlive.myuser$'
cluster-name="HA"
resume = true
replay = false
resume-write-unsafe = false
exit-after-direct-reads = false
elasticsearch-user = "es-appuser"
elasticsearch-password = "9V#xxxxxx"
elasticsearch-urls = ["https://vpc-wtlive-domain-staging-om2cbdeek6trkdrcb3dg4vm.eu-west-3.es.amazonaws.com"]
mongo-url = "mongodb://admin:VYn7ZD4CHDh8@wtlive-dedicated-shard-00-00.ynxpn.mongodb.net:27017,wtlive-dedicated-shard-00-01.ynxpn.mongodb.net:27017,wtlive-dedicated-shard-00-02.ynxpn.mongodb.net:27017/?tls=true&replicaSet=atlas-lmkye1-shard-0&authSource=admin&retryWrites=true&w=majority&tlsCAFile=/home/ec2-user/mongodb-ca.pem"

#[logs]
#info = "/home/ec2-user/logs/monstache/info.log"
#error = "/home/ec2-user/logs/monstache/error.log"
#warn = "/home/ec2-user/logs/monstache/warn.log"

#[[mapping]]
#namespace = "wtlive.myuser"
#index = "wtlive.myuser"

[[pipeline]]
namespace = "wtlive.myuser"
script = """
module.exports = function(ns, changeStream) {
  if (changeStream) {
    return [
      {
        $project: {
          _id: 1,
      operationType : 1,
       clusterTime : 1,
       documentKey : 1,
       to : 1,
           updateDescription : 1,
       txnNumber : 1,
       lsid :  1,
          "fullDocument._id": 1,
          "fullDocument.created": 1,
          "fullDocument.lastVisit": 1,
          "fullDocument.verified": 1,
          "fullDocument.device.locale": "$fullDocument.device.locale",
          "fullDocument.device.country": "$fullDocument.device.country",
          "fullDocument.device.tz": "$fullDocument.device.tz",
          "fullDocument.device.latLonCountry": "$fullDocument.device.latLonCountry",
          "fullDocument.details.firstname": "$fullDocument._details.firstname",
          "fullDocument.details.gender": "$fullDocument._details.gender",
          "fullDocument.details.category": "$fullDocument._details.category",
          "fullDocument.details.dob": "$fullDocument._details.dob",
          "fullDocument.details.lookingFor": "$fullDocument._details.lookingFor",
          "fullDocument.details.height": "$fullDocument._details.height",
          "fullDocument.details.weight": "$fullDocument._details.weight",
          "fullDocument.details.cigarette": "$fullDocument._details.cigarette",
          "fullDocument.details.categorizedBy": "$fullDocument._details.categorizedBy",
          "fullDocument.details.origin": "$fullDocument._details.origin",
          "fullDocument.details.city": "$fullDocument._details.city",
          "fullDocument.details.country": "$fullDocument._details.country",
          "fullDocument.lifeSkills.educationLevel": "$fullDocument._lifeSkills.educationLevel",
          "fullDocument.lifeSkills.pets": "$fullDocument._lifeSkills.pets",
          "fullDocument.lifeSkills.religion": "$fullDocument._lifeSkills.religion",
          "fullDocument.loveLife.children": "$fullDocument._loveLife.children",
          "fullDocument.loveLife.relationType": "$fullDocument._loveLife.relationType",
          "fullDocument.searchCriteria": "$fullDocument._searchCriteria",
          "fullDocument.blocked" : 1,
          "fullDocument.capping" : 1,
          "fullDocument.fillingScore" : 1,
          "fullDocument.viewed" : 1,
          "fullDocument.likes" : 1,
          "fullDocument.matches" : 1,
          "fullDocument.blacklisted" : 1,
          "fullDocument.uploadsList._id" : 1,
          "fullDocument.uploadsList.status" : 1,
          "fullDocument.uploadsList.url" : 1,
          "fullDocument.uploadsList.position" : 1,
          "fullDocument.uploadsList.imageSet" : 1,
          "fullDocument.location" : 1,
          "fullDocument.searchZone" : 1,
          "fullDocument.locationPoint" : "$fullDocument.location.coordinates",
          "fullDocument.selfieDateUpload" : 1,
       "ns": 1

        }
      }
    ]
  } else {
    return [
        {
        $project: {
          _id: 1,
          "_id": 1,
          "created": 1,
          "lastVisit": 1,
          "verified": 1,
          "device.locale": "$device.locale",
          "device.country": "$device.country",
          "device.tz": "$device.tz",
          "device.latLonCountry": "$device.latLonCountry",
          "details.firstname": "$_details.firstname",
          "details.gender": "$_details.gender",
          "details.category": "$_details.category",
          "details.dob": "$_details.dob",
          "details.lookingFor": "$_details.lookingFor",
          "details.height": "$_details.height",
          "details.weight": "$_details.weight",
          "details.cigarette": "$_details.cigarette",
          "details.categorizedBy": "$_details.categorizedBy",
          "details.origin": "$_details.origin",
          "details.city": "$_details.city",
          "details.country": "$_details.country",
          "lifeSkills.educationLevel": "$_lifeSkills.educationLevel",
          "lifeSkills.pets": "$_lifeSkills.pets",
          "lifeSkills.religion": "$_lifeSkills.religion",
          "loveLife.children": "$_loveLife.children",
          "loveLife.relationType": "$_loveLife.relationType",
          "searchCriteria": "$_searchCriteria",
          "blocked" : 1,
          "capping" : 1,
          "fillingScore" : 1,
          "viewed" : 1,
          "likes" : 1,
          "matches" : 1,
          "blacklisted" : 1,
          "uploadsList._id" : 1,
          "uploadsList.status" : 1,
          "uploadsList.url" : 1,
          "uploadsList.position" : 1,
          "uploadsList.imageSet" : 1,
          "location" : 1,
          "searchZone" : 1,
          "selfieDateUpload" : 1,
          "locationPoint" : "$location.coordinates"
        }
      }
    ]
  }
}
"""

有什么问题吗？接下来我该怎么做？

elasticsearch

1条答案

按热度按时间
w8ntj3qf

w8ntj3qf1#

通过取消注解#direct-read-namespaces = ["wtlive.myuser"]行，monstache现在可以进行初始同步，一切都很顺利，我将注解掉aigain并在初始同步后重启monstache服务，以避免从头开始重新同步。

赞(0）分享  回复(0）举报  2023-01-29
我来回答

相关问题

    查看更多

    热门标签

    更多
    JavaquerypythonNode开发语言requestUtil数据库Table后端算法LoggerMessageElementParser

    最新问答

    更多
    • 蜀ICP备2022004421号-2 NULL123 https://www.null123.com © All rights reserved
    • 本站内容来源互联网,如果侵犯您的权益请联系我们删除, 联系方式：448109455@qq.com