Pipeline: Correct Dataproc clusters without autoscaling

Correct Dataproc cluster without autoscaling

Overview

Dataproc clusters can be costly to run, especially if they're not being used to their full potential. Clusters with autoscaling disabled should be reviewed to determine if they're still required.

This pipeline allows you to specify a single cluster and then either send a notification or attempt to perform a predefined corrective action.

Whilst it is possible to utilise this pipeline standalone, it is usually called from either:

Run the pipeline

To run this pipeline from your terminal:

flowpipe pipeline run gcp_thrifty.pipeline.correct_dataproc_clusters_without_autoscaling \
  --arg 'items=<list(object({
      title    = string
      conn     = string
      name     = string
      location = string
      project  = string
    }))>'

Use this pipeline

To call this pipeline from your pipeline, use a step:

step "pipeline" "step_name" {
  pipeline = gcp_thrifty.pipeline.correct_dataproc_clusters_without_autoscaling  
  args = {
    items = <list(object({
      title    = string
      conn     = string
      name     = string
      location = string
      project  = string
    }))>
  }
}

Triggers

Detect & correct Dataproc clusters without autoscaling

Params

Name	Type	Required	Description	Default
items	list(object({ title = string conn = string name = string location = string project = string }))	Yes		-
notifier	notifier	Yes	The name of the notifier to use for sending notification messages.	`notifier.default`
notification_level	string `info`, `verbose`, `error`	Yes	The verbosity level of notification messages to send.	`info`
approvers	list(notifier)	Yes	List of notifiers to be used for obtaining action/approval decisions.	`notifier.default`
default_action	string `notify`, `skip`, `delete_dataproc_cluster`	Yes	The default action to use for the detected item, used if no input is provided.	`notify`
enabled_actions	list(string) `skip`, `delete_dataproc_cluster`	Yes	The list of enabled actions to provide to approvers for selection.	`[ "skip", "delete_dataproc_cluster" ]`

Outputs

This pipeline has no outputs.