Using Curator to prune Elasticsearch Indices

If you are using Elasticsearch as a database to store data from various sources, you are going to need to a way to prune the indices before they end up filling your drive. Using Curator is one way to go about this task.

The following instructions are based on a Ubuntu LTS 16.04 install with Elasticsearch 5.6 locally installed on the same machine.

  • First, Add the source to your apt repository list –

  • Update your package listing and install the curator package

The Curator application uses the popular YAML format as a basis for it’s configuration files. We will need to create two files at a minimum to get going.

  • First, create a blank file for the base configuration –

Paste in the following basic configuration (Assuming your Elasticsearch sever is on the same machine as curator)

Then, create another “action file” that holds the action we want to run. Here, I’m using a slightly modified action from an example that will delete any indice older than 30 days that starts with “netflow-“. Modify this file to suit your needs. We will call it “delete_indices.yml”

Contents of file

We can then run the configuration file with the –dry-run parameter to simulate the actions taken. Remove –dry-run when you are ready to run it for real!

To run this on a schedule, simply add it to crontab. This will run it daily at midnight. Open your crontab with

Add these lines –

Windows VMWare vCenter 6.0 to 6.5 VCSA Migration Issues

I encountered a few issues earlier today when trying to use the VMWare migration tool to move from a Windows vCenter 6.0 to a VCSA 6.5 appliance.

The migration first failed with a error that was helpfully detailed by another person here

If you were monitoring the source vCenter server, during the export process, one would notice that an error pops up while compressing the source data. The error presented is generated from Windows creating an archive (zip file), the error reads: β€œThe compressed (zipped) folder is invalid or corrupted.”. The entire migration process halts until you dismiss this message, with the entire migration ultimately failing (at first it appears to continue, but ultimately fails).

The fix for this error is centered around removing VMWare update manager before attempting the migration again. Unfortunately, there are two common errors I encountered trying to accomplish that uninstall process —

These errors are fairly cryptic, but you can resolve it by carefully reading the vminst.log file generated by the uninstaller (Located in %temp%). We can see the following lines –

The uninstall process is looking for two things that don’t exist on the local system, a firewall rule and a system DSN (ODBC connector used to connect to the VUM database)

The resolution is as follows –

  • Create temporary firewall rule that matches the uninstall script (run from admin command prompt)
    • add rule name=”vmware-update-temp” dir=in localport=8084 protocol=TCP action=allow
  • Create temporary 32bit system DSN that matches the uninstall script name “VMware vSphere Update Manager” on your old vCenter/VUM server.
  • Disable VUM Database in your SQL server (Right click > set offline)

 

After completing the above steps, re-run the migration wizard and your 6.0 > 6.5 VCSA migration should complete without errors!