ElasticSearch集群部署和使用
2018年01月30日


Installation

参考文档:

https://www.elastic.co/guide/index.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html#_installation

https://www.elastic.co/guide/en/elasticsearch/reference/6.1/zip-targz.html


准备:

Elasticsearch requires at least Java 8. Specifically as of this writing, it is recommended that you use the Oracle JDK version 1.8.0_131.

    java -version

    echo $JAVA_HOME

部署系统:Linux(CentOS)

去官方网站下载 https://www.elastic.co

例如:https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.2.tar.gz

解压:

tar -xvf elasticsearch-6.1.2.tar.gz


前台启动:./bin/elasticsearch

后台启动:./bin/elasticsearch -d -p ./bin/pid


启动报错:

[2018-01-30T03:51:53,031][ERROR][o.e.b.Bootstrap          ] [esnode03] node validation exception

[1] bootstrap checks failed

[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]


解决方案:进行系统参数优化

这可以临时设置:sysctl -w vm.max_map_count=655350

或者在/etc/sysctl.conf下永久设置vm.max_map_count。

vm.max_map_count=655350直接写到/etc/sysctl.conf中,然后执行sysctl -p

查看设置

cat /proc/sys/vm/max_map_count


或者执行下面的脚本:

#!/bin/sh
#
# 系统参数优化,Execute Only Once
#

export PATH=$PATH:/sbin

sudo sysctl -w vm.extra_free_kbytes=3000000
sudo sysctl -w vm.min_free_kbytes=1000000
sudo sysctl -w vm.overcommit_memory=1
sudo sysctl -w vm.drop_caches=1
sudo sysctl -w vm.zone_reclaim_mode=0
sudo sysctl -w vm.max_map_count=655360
sudo sysctl -w vm.dirty_background_ratio=50
sudo sysctl -w vm.dirty_ratio=50
sudo sysctl -w vm.dirty_writeback_centisecs=360000
sudo sysctl -w vm.page-cluster=3
sudo sysctl -w vm.swappiness=0
sudo sysctl -w vm.max_map_count=655350

echo 'ulimit -n 655350' >> /etc/profile
echo '*       hard    nofile  655350' >> /etc/security/limits.conf
echo '*       hard    memlock  unlimited' >> /etc/security/limits.conf
echo '*       soft    memlock  unlimited' >> /etc/security/limits.conf

关于系统参数设置,官方文档有详细说明:

https://www.elastic.co/guide/en/elasticsearch/reference/6.1/setting-system-settings.html

https://www.elastic.co/guide/en/elasticsearch/reference/6.1/max-number-of-threads.html


集群配置

配置示例如下:

# ======================== Elasticsearch Configuration =========================
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: escluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
node.name: esnode03
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
path.data: /home/admin/esdata/data
#
# Path to log files:
#
#path.logs: /path/to/logs
path.logs: /home/admin/esdata/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
network.host: 192.168.11.242
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["192.168.11.242", "192.168.11.249", "192.168.11.250"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

不懂的配置,比如minimum_master_nodes,可以参考官方说明,或者百度:

https://www.elastic.co/guide/en/elasticsearch/reference/6.1/important-settings.html


测试

查看节点状态:

curl -XGET 'http://192.168.11.242:9200/_cat/nodes?pretty'

192.168.11.242 17 41 0 0.00 0.01 0.05 mdi * esnode03
192.168.11.250  9 20 0 0.00 0.01 0.03 mdi - esnode02
192.168.11.249 14 20 0 0.00 0.01 0.03 mdi - esnode01

查询更多支持的命令:

curl -XGET 'http://192.168.11.242:9200/_cat?pretty' 

/_cat/allocation
/_cat/shards
/_cat/shards/{index}
/_cat/master
/_cat/nodes
/_cat/tasks
/_cat/indices
/_cat/indices/{index}
/_cat/segments
/_cat/segments/{index}
/_cat/count
/_cat/count/{index}
/_cat/recovery
/_cat/recovery/{index}
/_cat/health
/_cat/pending_tasks
/_cat/aliases
/_cat/aliases/{alias}
/_cat/thread_pool
/_cat/thread_pool/{thread_pools}
/_cat/plugins
/_cat/fielddata
/_cat/fielddata/{fields}
/_cat/nodeattrs
/_cat/repositories
/_cat/snapshots/{repository}
/_cat/templates

查看集群状态;

curl -XGET http://192.168.11.242:9200/_cluster/health?pretty


API文档参见:

https://www.elastic.co/guide/en/elasticsearch/reference/6.1/cat-health.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.1/cluster-health.html  


关闭

jps获取进程pid,然后kill -SIGTERM pid


性能测试

官方提供了测试工具和测试方法:

https://www.elastic.co/guide/en/elasticsearch/reference/6.1/testing-framework.html



生产部署,优化建议

参见:http://blog.csdn.net/thomas0yang/article/details/55518105


集群原理:http://blog.csdn.net/zwgdft/article/details/54585644



安装 elasticsearch Head 插件,方便查看es里面的数据

参见:https://www.cnblogs.com/aubin/p/8018081.html

官方地址:https://github.com/mobz/elasticsearch-head

注意配置:

vim elasticsearch.yml

http.cors.enabled: true            # elasticsearch中启用
CORShttp.cors.allow-origin: "*"    # 允许访问的IP地址段,* 为所有IP都