If you are using Elasticsearch as a database to store data from various sources, you are going to need to a way to prune the indices before they end up filling your drive. Using Curator is one way to go about this task.
The following instructions are based on a Ubuntu LTS 16.04 install with Elasticsearch 5.6 locally installed on the same machine.
- First, Add the source to your apt repository list –
|
1 |
sudo echo "deb [arch=amd64] http://packages.elastic.co/curator/5/debian9 stable main" >> /etc/apt/sources/curator.list |
- Update your package listing and install the curator package
|
1 |
sudo apt-get update && sudo apt-get install elasticsearch-curator |
The Curator application uses the popular YAML format as a basis for it’s configuration files. We will need to create two files at a minimum to get going.
- First, create a blank file for the base configuration –
|
1 |
touch ~/.curator/curator.yml |
Paste in the following basic configuration (Assuming your Elasticsearch sever is on the same machine as curator)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: logformat: default blacklist: ['elasticsearch', 'urllib3'] ~ |
Then, create another “action file” that holds the action we want to run. Here, I’m using a slightly modified action from an example that will delete any indice older than 30 days that starts with “netflow-“. Modify this file to suit your needs. We will call it “delete_indices.yml”
|
1 |
touch ~/delete_indices.yml |
Contents of file
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
actions: 1: action: delete_indices description: >- Delete indices older than 30 days (based on index name), for netflow- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True disable_action: false filters: - filtertype: pattern kind: prefix value: netflow- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 30 |
We can then run the configuration file with the –dry-run parameter to simulate the actions taken. Remove –dry-run when you are ready to run it for real!
|
1 2 3 4 5 6 7 |
➜ ~ sudo curator ./delete_indice.yml --dry-run 2017-10-30 14:28:42,528 INFO Preparing Action ID: 1, "delete_indices" 2017-10-30 14:28:42,536 INFO Trying Action ID: 1, "delete_indices": Delete indices older than 30 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. 2017-10-30 14:28:42,582 INFO DRY-RUN MODE. No changes will be made. 2017-10-30 14:28:42,582 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices". 2017-10-30 14:28:42,582 INFO Action ID: 1, "delete_indices" completed. 2017-10-30 14:28:42,582 INFO Job completed. |
To run this on a schedule, simply add it to crontab. This will run it daily at midnight. Open your crontab with
|
1 |
crontab -e |
Add these lines –
|
1 |
00 00 * * * /usr/bin/curator ~/delete_indice.yml |
Good post
After this got stuck in FileBeat 401 unauthorization with aws Elasticsearch.
Got help from
https://learningsubway.com/filebeat-401-unauthorized-error-with-aws-elasticsearch/