Introduction

Managing the lifecycle of data stored in AWS OpenSearch is critical for maintaining performance and controlling costs. For logs and other time-series data, it's often necessary to delete old data after a certain period. In this blog post, we’ll walk through how to set up an Index State Management (ISM) policy to automatically delete data older than 60 days, how to automate this process for new indices using ISM templates, and how to set up monitoring and alerts to ensure everything is working as expected.

Prerequisites

Before we start, ensure you have:

• An AWS OpenSearch domain with Index State Management (ISM) enabled.
• Access to OpenSearch Dashboards or the ability to interact with OpenSearch via its API.
• The necessary permissions to manage ISM policies and index settings.

Step 1: Create an ISM Policy

The first step is to create an ISM policy that defines the conditions under which indices should be deleted. In our case, we want to delete indices older than 60 days.

PUT _plugins/_ism/policies/delete-old-indices
{
  "policy": {
    "description": "Delete indices older than 60 days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "60d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ]
      }
    ]
  }
}

This policy sets up a lifecycle where indices automatically transition to a delete state once they are 60 days old.

Step 2: Apply the ISM Policy to Your Index

Next, you need to apply this policy to your existing index (in this case, <your-log-group-name>). Here's how you can attach the policy:

POST _plugins/_ism/add/<your-log-group-name>
{
  "policy_id": "delete-old-indices"
}

This command attaches the delete-old-indices policy to the <your-log-group-name> index.

Step 3: Automate Policy Application for New Indices Using ISM Templates

If your environment regularly creates new indices (e.g., daily log indices), manually applying the ISM policy to each new index can be cumbersome. ISM templates allow you to automate this process by automatically applying the ISM policy to any new index that matches a specified pattern.

How ISM Templates Work

ISM templates match newly created indices against specified patterns and automatically apply the designated ISM policy to those indices. This ensures that your data lifecycle management policies are consistently enforced across all relevant indices without manual intervention.

Creating an ISM Template

Here's how to create an ISM template:

PUT _plugins/_ism/templates
{
  "ism_templates": [
    {
      "index_patterns": ["logs-*"],
      "priority": 100,
      "policy_id": "delete-old-indices"
    }
  ]
}

• index_patterns: Specifies which indices the template will apply to. For example, "logs-*" would apply to any index with a name that starts with "logs-".
• priority: If multiple templates could apply to the same index, the one with the highest priority is used.
• policy_id: The ID of the ISM policy to apply to indices matching the pattern.

With this template, any new index created with a name starting with logs- will automatically have the delete-old-indices policy applied, ensuring that data older than 60 days is deleted.

Step 4: Verify Policy Application

To ensure that the policy has been successfully applied, you can use the following command:

GET _plugins/_ism/explain/<your-log-group-name>

The response should indicate that the policy is active and managing your index:

{
  "<your-log-group-name>": {
    "index.plugins.index_state_management.policy_id": "delete-old-indices",
    "index.opendistro.index_state_management.policy_id": "delete-old-indices",
    "index": "<your-log-group-name>",
    "index_uuid": "fvujhJe8T1SimSzni6_eTg",
    "policy_id": "delete-old-indices",
    "enabled": true
  },
  "total_managed_indices": 1
}

Step 5: Monitor Policy Execution and Set Up Alerts

Now that the policy is in place, you need to monitor its execution to ensure that data older than 60 days is being deleted. Additionally, setting up alerts can help you stay informed about the status of your indices and ISM policies.

Method 1: Check Index Lifecycle Status

You can regularly check the lifecycle status of your index:

GET _plugins/_ism/explain/<your-log-group-name>

This will show the current state of the index and whether it has transitioned to the delete state.

Method 2: Monitor Document Count

Another approach is to monitor the document count in your index. A reduction in document count after 60 days indicates that the policy is working:

GET <your-log-group-name>/_count

Method 3: Review OpenSearch Logs

Check your OpenSearch logs for actions related to index deletions:

• Log Analysis: Look for entries that indicate indices have been deleted according to the ISM policy.

Setting Up Alerts

To automate monitoring and receive alerts when something goes wrong, you can set up alerting mechanisms in OpenSearch. For example:

1. Create Monitor: Create a monitor in OpenSearch Dashboards that queries the state of your indices and checks for conditions like "index not deleted after 60 days."
2. Set Up Alerts: Configure alerts that trigger based on the monitor's findings. For instance, if an index has not been deleted as expected, an alert can be sent via email, Slack, or another notification service.
3. Automate Checks: Schedule these monitors to run at regular intervals (e.g., daily) to ensure continuous oversight of your ISM policies.

Conclusion

By following these steps, you can effectively manage the lifecycle of your data in AWS OpenSearch. Automating the deletion of old data not only helps with performance but also ensures that your storage costs remain under control. With ISM templates, you can automate the application of these policies to new indices, and by setting up monitoring and alerts, you can ensure that your data retention policies are always enforced as intended.