mysql8与elasticsearch7.3.2数据同步,使用docker容器

我就是我 2022-12-30 12:57 9阅读 0赞

# 一、需求软件及相关说明 #

<table> 
 <tbody> 
 <tr> 
 <td> 需求软件 </td> 
 <td> 来源容器镜像 </td> 
 <td> 是否改造 </td> 
 <td> 改造后镜像 </td> 
 <td> 支持特别说明 </td> 
 <td> 备注 </td> 
 </tr> 
 <tr> 
 <td> Mysql8.0 </td> 
 <td> mysql:8.0 </td> 
 <td> 否 </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 </tr> 
 <tr> 
 <td> elasticsearch-head </td> 
 <td> mobz/elasticsearch-head:5-alpine </td> 
 <td> 是 </td> 
 <td> registry.cn-shanghai.aliyuncs.com/qlongit/elasticsearch-head:v1.0 &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> 拉取最近代码修改而成，原有的代码无法显示数据 </td> 
 </tr> 
 <tr> 
 <td> Elasticsearch7.3.2 </td> 
 <td> Elasticsearch:7.3.2 </td> 
 <td> 否 </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> 安全性考虑升级到较高版本7.3.2 但是此版本不支持一个索引下多个类型，支持一个索引下单个类型 </td> 
 </tr> 
 <tr> 
 <td> go-mysql-elasticsearch v2.1 </td> 
 <td> mysql:8.0 </td> 
 <td> 是 </td> 
 <td> registry.cn-shanghai.aliyuncs.com/qlongit/go-mysql-elasticsearch:v2.1 </td> 
 <td> 支持mysql8.0版本 </td> 
 <td> 基于mysql8.0做mysqldump的客户端，使用mariadb最新的客户端版本是不兼容mysql8.0服务端的 </td> 
 </tr> 
 <tr> 
 <td> go-mysql-elasticsearch v1 </td> 
 <td> https://github.com/siddontang/go-mysql-elasticsearch </td> 
 <td> 是 </td> 
 <td> registry.cn-shanghai.aliyuncs.com/qlongit/go-mysql-elasticsearch:v1 </td> 
 <td> 支持mysql5.7以下版本 </td> 
 <td> 因为Dockerfile文件构建镜像时go的模块文件下载一直报错，所以改用国内的下载源 </td> 
 </tr> 
 <tr> 
 <td> kibana </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> 待有时间再上线这个 </td> 
 </tr> 
 <tr> 
 <td> Java服务端调用 </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> &nbsp; </td> 
 <td> 使用jest方式调用 </td> 
 </tr> 
 </tbody> 
</table>

# 二、自己制作的容器镜像上传地址 #

目前都在阿里云上

# 三、docker-compose的编排脚本 #

## 3.1、编排脚本 ##

version: '2.2'

services:

es01:

image: elasticsearch:7.3.2

container\_name: es01

environment:

- node.name=es01

- discovery.seed\_hosts=es02

- cluster.initial\_master\_nodes=es01,es02

- cluster.name=docker-cluster

- bootstrap.memory\_lock=true

- "ES\_JAVA\_OPTS=-Xms512m -Xmx512m"

ulimits:

memlock:

soft: -1

hard: -1

volumes:

- esdata01:/usr/share/elasticsearch/data

- ~/elastic/elastic/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

ports:

- 9200:9200

networks:

- esnet

es02:

image: elasticsearch:7.3.2

container\_name: es02

environment:

- node.name=es02

- discovery.seed\_hosts=es01

- cluster.initial\_master\_nodes=es01,es02

- cluster.name=docker-cluster

- bootstrap.memory\_lock=true

- "ES\_JAVA\_OPTS=-Xms512m -Xmx512m"

ulimits:

memlock:

soft: -1

hard: -1

volumes:

- esdata02:/usr/share/elasticsearch/data

- ~/elastic/elastic/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml

networks:

- esnet

eshead:

image: elasticsearch-head:v1.0

container\_name: elasticsearch-head

ports:

- 9100:9100

links:

- es01:es

command:

- sh

- -c

- npm run start

networks:

- esnet

mysql8:

image: mysql:8.0

container\_name: mysql8

ports:

- 13306:3306

environment:

- MYSQL\_ROOT\_PASSWORD=\*\*\*\*\*

- MYSQL\_DATABASE=\*\*\*\*

- MYSQL\_USER=\*\*\*\*\*

- MYSQL\_PASSWORD=\*\*\*\*\*\*

volumes:

- ~/elastic/mysql8/data:/var/lib/mysql

networks:

- esnet

gomysqles:

image: go-mysql-es:v2.1

container\_name: go-mysql-es

volumes:

- ~/elastic/go-mysql-elasticsearch/var:/goworkerspace/bin/var

- ~/elastic/go-mysql-elasticsearch/etc/river.toml:/goworkerspace/bin/etc/river.toml

networks:

- esnet

links:

- es01:es

- mysql8:mysql

depends\_on:

- es01

- es02

- mysql8

volumes:

esdata01:

driver: local

esdata02:

driver: local

networks:

esnet:

其中一些敏感的数据库账号与密码需要自行调整，还要镜像调整为上面阿里云镜像

docker login --username=postmaster@newfdf.com registry.cn-shanghai.aliyuncs.com

docker pull registry.cn-shanghai.aliyuncs.com/qlongit/go-mysql-elasticsearch:v2.1

关于一些外挂的一些配置文件看下面几章节

## 3.2、elasticsearch配置文件如下： ##

~/elastic/elastic/elasticsearch.yml

配置文件位置视系统环境调整

cluster.name: "docker-cluster"

network.host: 0.0.0.0

http.cors.enabled: true

http.cors.allow-origin: "\*"

## 3.3、go-mysql-elasticsearch的配置文件 ##

~/elastic/go-mysql-elasticsearch/etc/river.toml

\# MySQL address, user and password

\# user must have replication privilege in MySQL.

my\_addr = "mysql:3306"

my\_user = "\*\*\*"

my\_pass = "\*\*\*\*\*"

my\_charset = "utf8"

\# Set true when elasticsearch use https

\#es\_https = false

\# Elasticsearch address

es\_addr = "es:9200"

\# Elasticsearch user and password, maybe set by shield, nginx, or x-pack

es\_user = ""

es\_pass = ""

\# Path to store data, like master.info, if not set or empty,

\# we must use this to support breakpoint resume syncing.

\# TODO: support other storage, like etcd.

data\_dir = "./var"

\# Inner Http status address

stat\_addr = "127.0.0.1:12800"

\# pseudo server id like a slave

server\_id = 1001

\# mysql or mariadb

flavor = "mysql"

\#flavor = "mariadb"

\# mysqldump execution path

\# if not set or empty, ignore mysqldump.

mysqldump = "mysqldump"

\# if we have no privilege to use mysqldump with --master-data,

\# we must skip it.

\#skip\_master\_data = false

\# minimal items to be inserted in one bulk

bulk\_size = 128

\# force flush the pending requests if we don't have enough items >= bulk\_size

flush\_bulk\_time = "200ms"

\# Ignore table without primary key

skip\_no\_pk\_table = false

\# MySQL data source

\[\[source\]\]

schema = "sakila"

\# Only below tables will be synced into Elasticsearch.

\# "t\_\[0-9\]\{4\}" is a wildcard table format, you can use it if you have many sub tables, like table\_0000 - table\_1023

\# I don't think it is necessary to sync all tables in a database.

tables = \["sougoulog","film"\]

\# Below is for special rule mapping

\# Very simple example

\# desc t;

\# +-------+--------------+------+-----+---------+-------+

\# +-------+--------------+------+-----+---------+-------+

\# | id    | int(11)      | NO   | PRI | NULL    |       |

\# +-------+--------------+------+-----+---------+-------+

\# The table \`t\` will be synced to ES index \`test\` and type \`t\`.

\[\[rule\]\]

schema = "sakila"

table = "sougoulog"

index = "sakila\_sougoulog"

type = "sougoulog"

\[\[rule\]\]

schema = "sakila"

table = "film"

index = "sakila\_film"

type = "film"

如果是es6以下版本支持一个index下多个type，

在es7.3.2版本是不支持一个index多个type的，只能是一个index一个type。

这个视个人情况来配置

# 四、关于go-mysql-elasticsearch不支持DDL操作同步解决方案 #

## 4.1、删除做了ddl操作的index ##

## 4.2、新开一个docker镜像 ##

新开的只有原来的表，重新mysqldump操作，需要调整配置文件与删除日志存储点文件实现

## 4.3、销毁新开的，原来的就会重新按binlog来同步 ##

# 五、关于go-mysql-elasticsearch运行过久不同步的问题 #

这个大家都遇到过，建立一个定时任务重启docker即可。

真正解决，目前要看作者有无时间研究处理。本人对于go也才刚刚接触

\#\# 标题

mysql8与elasticsearch7.3.2数据同步,使用docker容器

发表评论取消回复

还没有评论，来说两句吧...

相关阅读

相关在docker容器中安装Elasticsearch分词器并同步数据

相关 elasticsearch与mysql数据同步

相关使用logstash同步mysql数据到elasticsearch

相关 mysql8与elasticsearch7.3.2数据同步，使用docker容器

相关 docker部署logstash实现mysql数据同步至elasticsearch

相关 mysql8与elasticsearch7.3.2数据同步,使用docker容器

相关 ElasticSearch--与MySQL同步

相关 elasticsearch使用river同步mysql数据

相关 elasticsearch与MySQL数据同步

相关 docker容器与系统时间同步