Skip to content

v0.5.0

Compare
Choose a tag to compare
@medcl medcl released this 19 Dec 08:57
· 11 commits to master since this release
d6402a7
  • Add buffer_count to control memory consumed by ESM
  • Speedup, several performance optimization

A 3 nodes cluster(3 * c5d.4xlarge, 16C,32GB,10Gbps)

./esm -s https://localhost:8000 -d https://localhost:8000 -x kibana_sample_data_logs -y logs-test -m elastic:medcl123 -n elastic:medcl123  --regenerate_id --repeat_times=5
./esm -s https://localhost:8000 -d https://localhost:8000 -x logs-test -y logs-test1 -m elastic:medcl123 -n elastic:medcl123  --regenerate_id  -w 20 --sliced_scroll_size=20 -b 20 --repeat_times=500
./esm -s https://localhost:8000 -d https://localhost:8000 -x logs-test1 -y logs-test -m elastic:medcl123 -n elastic:medcl123  --regenerate_id  -w 20 --sliced_scroll_size=20 -b 20 --repeat_times=500

[12-19 06:29:40] [INF] [main.go:537,main] data migration finished.
root@ip-172-31-13-181:/tmp# ./esm -s https://localhost:8000 -d https://localhost:8000 -x logs1kw -y logs122 -m elastic:medcl123 -n elastic:medcl123 -w 40 --sliced_scroll_size=60 -b 5 --buffer_count=2000000  --regenerate_id
[12-19 06:31:20] [INF] [main.go:506,main] start data migration..
Scroll 10064570 / 10064570 [============================================] 100.00% 55s
Bulk 10062602 / 10064570 [=============================================]  99.98% 55s
[12-19 06:32:15] [INF] [main.go:537,main] data migration finished.

Migrate 10,000,000 documents within 1 minute (Nginx log generated from kibana_sample_data_logs).