Pipeline: Detect & correct Dataproc clusters without autoscaling

Detect & correct Dataproc clusters withouy autoscaling

Overview

Dataproc clusters can be costly to run, especially if they're not being used efficiently. Clusters with autoscaling disabled should be reviewed to determine if they're still required.

This pipeline detects Dataproc clusters with autoscaling disabled and then either sends a notification or attempts to perform a predefined corrective action.

Getting Started

This control will work out-of-the-box with some sensible defaults (configurable via variables).

You should be able to simply run the following command in your terminal:

flowpipe pipeline run detect_and_correct_dataproc_clusters_autoscaling_disabled

By default, Flowpipe runs in wizard mode and prompts directly in the terminal for a decision on the action(s) to take for each detected resource.

However, you can run Flowpipe in server mode with external integrations, allowing it to prompt for input via http, slack, teams, etc.

Alternatively, you can choose to configure and run in other modes:

Notify: Provides detections without taking any corrective action.
Automatic: Performs corrective actions automatically without user intervention.

Run the pipeline

To run this pipeline from your terminal:

flowpipe pipeline run gcp_thrifty.pipeline.detect_and_correct_dataproc_clusters_without_autoscaling

Use this pipeline

To call this pipeline from your pipeline, use a step:

step "pipeline" "step_name" {
  pipeline = gcp_thrifty.pipeline.detect_and_correct_dataproc_clusters_without_autoscaling  
}

Params

Name	Type	Required	Description	Default
database	connection.steampipe	Yes	Database connection string.	`connection.steampipe.default`
notifier	notifier	Yes	The name of the notifier to use for sending notification messages.	`notifier.default`
notification_level	string `info`, `verbose`, `error`	Yes	The verbosity level of notification messages to send.	`info`
approvers	list(notifier)	Yes	List of notifiers to be used for obtaining action/approval decisions.	`notifier.default`
default_action	string `notify`, `skip`, `delete_dataproc_cluster`	Yes	The default action to use for the detected item, used if no input is provided.	`notify`
enabled_actions	list(string) `skip`, `delete_dataproc_cluster`	Yes	The list of enabled actions to provide to approvers for selection.	`[ "skip", "delete_dataproc_cluster" ]`

Outputs

This pipeline has no outputs.