Detect & correct HDInsight clusters exceeding max age

Overview

HDInsight clusters can be quite costly to retain, it is also likely that after a certain point in time they're no longer required and should be cleaned up to prevent further costs.

This pipeline allows you to specify a collection of HDInsight clusters and then either send notifications or attempt to perform a predefined corrective action upon the collection.

Getting Started

By default, this trigger is disabled, however it can be configured by setting the below variables

hdinsight_clusters_exceeding_max_age_trigger_enabled should be set to true as the default is false.
hdinsight_clusters_exceeding_max_age_trigger_schedule should be set to your desired running schedule
hdinsight_clusters_exceeding_max_age_default_action should be set to your desired action (i.e. "notify" for notifications or "delete_cluster" to delete the HDInsight cluster).

Then starting the server:

flowpipe server

or if you've set the variables in a .fpvars file:

flowpipe server --var-file=/path/to/your.fpvars

Pipeline

Correct HDInsight clusters exceeding max age

Query

select
  concat(
    c.id,
    ' [',
    c.resource_group,
    '/',
    c.subscription_id,
    ']'
  ) as title,
  c.name,
  c.resource_group,
  c.subscription_id,
  c.sp_connection_name as conn
from
  azure_hdinsight_cluster as c
  join azure_resource as r on lower(c.id) = lower(r.id)
  join azure_subscription as sub on sub.subscription_id = c.subscription_id
where
  date_part('day', now() - created_time) > 90;

Schedule