library
turbot/aws_thrifty

Detect & correct EMR clusters idle for more than 30 mins

EMR clusters which are live but not currently running tasks should be reviewed and checked whether the cluster has been idle for more than 30 minutes. It is ideal to delete such clusters for cost optimization.

This pipeline detects EMR clusters idle for more than 30 mins and then either sends a notification or attempts to perform a predefined corrective action.

Getting Started

This control will work out-of-the-box with some sensible defaults (configurable via variables).

You should be able to simply run the following command in your terminal:

flowpipe pipeline run detect_and_correct_emr_clusters_idle_30_mins

You should now receive notification messages for the detections in your configured notifier.

However, you may want to actually perform an action against these resources beyond a simple notification.

Interactive Decisions

Through the use of an Input Step, you can make a decision on how to handle each detected item.

In order to acheieve this, you will need to have an instance of Flowpipe Server running:

flowpipe server --mod-location=/path/to/mod

or if the current working directory contains the mod, simply:

flowpipe server

You can then run the command below:

flowpipe pipeline run detect_and_correct_emr_clusters_idle_30_mins --host local --arg='approvers=["default"]'

This will prompt for an action for each detected resource and then attempt to perform the chosen action upon receipt of input.

You can also decide to bypass asking for decision and just automatically apply the same action against all detections.

Automatic Actioning

You can automatically apply a specific action without the need for running a Flowpipe Server and asking for a decision by setting the default_action parameter:

flowpipe pipeline run detect_and_correct_emr_clusters_idle_30_mins --arg='default_action="terminate_cluster"'

However; if you have configured a non-empty list for your approvers variable, you will need to override it as below:

flowpipe pipeline run detect_and_correct_emr_clusters_idle_30_mins --arg='approvers=[]' --arg='default_action="terminate_cluster"'

This will attempt to apply the action to every detected item, if you're happy with this approach you could have this occur mmore frequently by either scheduling the command by yourself or enabling the associated Query Trigger.

Params

NameTypeRequiredDescriptionDefault
notifierstringYesThe name of the notifier to use for sending notification messages.default
notification_levelstringYesThe verbosity level of notification messages to send. Valid options are 'verbose', 'info', 'error'.info
approverslist of stringYesList of notifiers to be used for obtaining action/approval decisions.
[]
default_actionstringYesThe default action to use for the detected item, used if no input is provided.notify
enabled_actionslist of stringYesThe list of enabled actions to provide to approvers for selection.
[
"skip",
"delete_cluster"
]
databasestringYesDatabase connection string.postgres://steampipe@localhost:9193/steampipe

Outputs

This pipeline has no outputs.