Elastic search Kibana

 

Installation and Configuration of ElasticSearch
Pre-requisites

sudo apt-get update

sudo apt-get install openjdk-8-jdk

Manual ElK Stack Installation steps

  1. Download and install public signing key wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

  2. Install apt-transport-https package sudo apt-get install apt-transport-https -y

  3. Save directory definitions echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

  4. Update and Install elasticsearch sudo apt-get update && sudo apt-get install elasticsearch && sudo apt-get install logstash && sudo apt-get install kibana

  5. Configure elasticsearch sudo su nano /etc/elasticsearch/elasticsearch.yml

change cluster name cluster.name: demo-elk

give the cluster a descriptive name node.name: elk-1

change network binding network.host: 0.0.0.0

setup discovery.type as single node discovery.type: single-node

  1. Enable elasticsearch service sudo systemctl enable elasticsearch

  2. Start Elasticsearch service sudo systemctl start elasticsearch

  3. Validate Elasticsearch cluster health curl -XGET http://localhost:9200/_cluster/health?pretty

Expected Result:

{ "cluster_name" : "moc-uat-applications", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 0, "active_shards" : 0, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

  1. Configure kibana nano /etc/kibana/kibana.yml

uncomment server.port and server.port: 5601

change server.host in server.host: "0.0.0.0" or your private ip of server

change server.name server.name: "demo-kibana"

uncomment elasticsearch.host elasticsearch.hosts: ["http://localhost:9200"] or your private ip of server

  1. Enable kibana service sudo systemctl enable kibana

  2. Start Kibana service sudo systemctl start kibana


About "Filebeat"

  • File beat is a lightweight shipper for forwarding and centralizing log data.

  • Installed as an agent on your servers, File beat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or log stash for indexing.

  • It has TWO components

    • Harvester - It's responsible for reading the content of a single file. The harvester reads each file, line by line, and sends the content to the output.
    • Input - An input is responsible for managing the harvesters and finding all sources to read from.

Configure Kibana logging

  1. Edit, sudo nano /etc/kibana/kibana.yml

Installation and configuration of Filebeat

  1. Download and install public signing key wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

  2. Install apt-transport-https package sudo apt-get install apt-transport-https -y

  3. Save directory definitions echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

  4. Update sudo apt-get update

  5. Install filebeat sudo apt-get install filebeat

  6. Configure filebeat (Navigate and open the file /etc/filebeat/filebeat.yml) i. Live reloading reload.enabled: true reload.period: 10s

ii. Setup dashboards setup.dashboards.enabled: true

iii. Kibana setup host: "<server_IP>:5601" // By default kibana will run under 5601 port

In the set iv, we can configure output to either logstash or elasticsearch. As of now configuring filebeat with Elasticsearch

iv. elasticsearch output hosts: "<server_IP>:9200" protocol: "http" username: "elastic" password: "<generated password, or we can change the password from the kibana UI>"

v. Set the path to send the logs and save

inset these lines in the .yml file at the last
logging.to_files: true 

logging.files: path: /var/log/filebeat 

name: filebeat 

keepfiles: 7 

permissions: 0644


Configure X-Pack Security

  1. Stop kibana sudo systemctl stop kibana

  2. Stop elasticsearch sudo systemctl stop elasticsearch

  3. Enable xpack in elasticsearch.yml sudo nano /etc/elasticsearch/elasticsearch.yml xpack.security.enabled: true

Note: After doing above configuration, we need to start both kibana and elasticsearch services.

  1. Setup default user passwords cd /usr/share/elasticsearch/bin

sudo ./elasticsearch-setup-passwords --help

sudo ./elasticsearch-setup-passwords auto

  1. Check healthy check curl -XGET http://localhost:9200/_cluster/health?pretty

We will get the output as,

{ "error" : { "root_cause" : [ { "type" : "security_exception", "reason" : "missing authentication credentials for REST request [/_cluster/health?pretty]", "header" : { "WWW-Authenticate" : "Basic realm="security" charset="UTF-8"" } } ], "type" : "security_exception", "reason" : "missing authentication credentials for REST request [/_cluster/health?pretty]", "header" : { "WWW-Authenticate" : "Basic realm="security" charset="UTF-8"" } }, "status" : 401 }

Pass credentials from curl, curl -XGET -u elastic:password http://localhost:9200/_cluster/health?pretty

{ "cluster_name" : "moc-cam-production-applications", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 11, "active_shards" : 11, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 91.66666666666666 }

  1. Add the default username in kibana sudo nano /etc/kibana/kibana.yml

elasticsearch.username: "kibana" elasticsearch.password: "new_password"

Login to kibana portal using elastic user


It is required if you use in the organization 

About "metricbeat"

  • It's a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on the server.

  • Modules are standard configurations that can be enabled to monitor following ---> Apache HTTPD servers ---> System(CPU Load, memory), etc

METRIC BEAT - MODULES AND METRICSET

  • A Metricbeat module defines the basic logic for collecting data from a specific service, such as apache http webserver and so on.
  • A metric set is the part of the module that fetches and structures the data for example system module collects
Installation and configuration of Metricbeat on ELK Server
  1. Enable Monitoring on Elasticsearch Cluster

cd /etc/elasticsearch

sudo nano /etc/elasticsearch/elasticsearch.yml

xpack.monitoring.collection.enabled: true

xpack.monitoring.elasticsearch.collection.enabled: true

  1. Restart elasticsearch service

sudo systemctl restart elasticsearch

  1. Install Metricbeat

sudo apt-get install metricbeat

  1. Configure Metribeat

sudo su cd /etc/metricbeat

nano metricbeat.yml

i. Live reloading reload.enabled: true reload.period: 10s

ii. Name and tag name: <"Some Name"> tags: <"Some Name">

iii. Setup dashboards setup.dashboards.enabled: true

In the set iv, we can configure output to either logstash or elasticsearch. As of now configuring filebeat with Elasticsearch

iv. elasticsearch output hosts: "<server_IP>:9200" protocol: "http" username: "elastic" password: "<generated password, or we can change the password from the kibana UI>"

v. Set the path to send the
logging.to_files: true logging.files: path: /var/log/metricbeat name: metricbeat keepfiles: 7 permissions: 0644

vi. enable interal collection of metricbeat monitoring.enabled: true monitoring.elasticsearch

  1. Start Metricbeat sudo systemctl start metricbeat

  2. enable elasticsearch-xpack sudo metricbeat modules enable elasticsearch-xpack

nano elasticsearch-xpack.yml username: remote_monitoring_user password: remote_monitoring_user_password

  1. enable kibana-xpack sudo metricbeat modules enable kibana-xpack

nano kibana-xpack.yml

username: remote_monitoring_user password: remote_monitoring_user_password

  1. enable beat-xpack sudo metricbeat modules enable beat-xpack

  2. disable system module sudo metricbeat modules disable system



Instllation and configuration of Metricbeat on WebServer

  1. Download and install public signing key wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

  2. Install apt-transport-https package sudo apt-get install apt-transport-https -y

  3. Save directory definitions echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

  4. Update and Install metricbeat sudo apt-get update sudo apt-get install metricbeat

Configure Metribeat

sudo su cd /etc/metricbeat

nano metricbeat.yml

### live monitoring section 
reload.enabled: true
reload.period: 10s
### name and tag
name: webserver
Tags: demo-web
### enable kibana dashboards
Setup.dashboards.enabled: true
host: <<ip_address_of_Kibana_server>>:5601
### output configuration
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["<<ip_address_of_elastic_server>>:9200"]
# Protocol - either `http` (default) or `https`.
protocol: "https"
#api_key: "id:api_key"
username: "elastic"      ####  elasticsuperuser
password: "password".    #### elastic users password

logging

logging.level: debug

logging.selectors: ["*"]

logging.to_files: true
logging.files:
path: /var/log/metricbeat
name: metricbeat
keepfiles: 7
permissions: 0644
## enable interal collection of metricbeat
monitoring.enabled: true
monitoring.elasticsearch

Comments

New comments are not allowed.

Popular posts from this blog

Installing metabase on ec2 linux server as a service file.

 Metabase installation   log in with root user # sudo su  Install the java on linux server # sudo amazon-linux-extras install java-openjdk11  Creating a directory  # mkdir ~/metabase Download a metabase.jar file and move to directory to download metabase.jar # wget https://metabase.com/start/oss/jar    # mv /home/ec2-user/metabase.jar ~/metabase Change directory to metabase # cd ~/metabase Now Run the cammand to run the jar file # java -jar metabase.jar To make the jar file into service, these are the following steps   # sudo groupadd -r metabase # sudo useradd -r -s /bin/false -g metabase metabase # pwd # sudo chown -R metabase:metabase /home/ec2-user/metabase/metabase/directoy # sudo touch /var/log/metabase.log  # sudo chown syslog:adm /var/log/metabase.log  # sudo touch /etc/default/metabase  # sudo chmod 640 /etc/default/metabase Once the folder and file is made just copy and past the folloing steps in the file  # vi /etc/systemd/system/metabase.service [Unit] Description=Metabase
Read more

Launching EC2 using Terraform | Installing Grafana and promotues using Terraform.

  Launching EC2 instance using Terraform: The set of files used to describe infrastructure in Terraform is known as a Terraform  configuration . You will write your first configuration to define a single AWS EC2 instance. Each Terraform configuration must be in its own working directory. Create a directory for your configuration. $ mkdir learn-terraform-aws-instance Copy Change into the directory. $ cd learn-terraform-aws-instance Copy Create a file to define your infrastructure. $ touch main.tf Copy Open  main.tf  in your text editor, paste in the configuration below, and save the file. Tip:  The AMI ID used in this configuration is specific to the  us-west-2  region. If you would like to use a different region, see the  Troubleshooting section  for guidance. terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 3.27" } } required_version = ">= 0.14.9" } provider "aws"
Read more