Skip to content

sqlxpert/lights-off-aws

Repository files navigation

Lights Off!

Ever forget to turn the lights off? Now you can:

  • Stop EC2 instances and RDS/Aurora databases temporarily by tagging them with cron schedules, to cut AWS costs. Schedules (not references to schedules) go directly in the tags.

  • Trigger AWS Backup with cron schedules in resource tags.

  • Delete expensive infrastructure temporarily by tagging your own CloudFormation stacks with cron schedules.

  • Easily deploy this tool to multiple AWS accounts and regions.

Jump to: Quick StartTagsSchedulesMulti-Account, Multi-RegionSecurity


🔒 Software supply chain security is on everyone's mind. This tool's two Lambda functions share one Python source file that's short enough to read (750 lines total). I made GitHub releases immutable as of v3.6.0 (2026-04-06). AWS manages patching of the stock Lambda runtime, which provides the Python standard library and the AWS software development kit (boto, boto3).

AWS's Instance Scheduler, the closest competing tool, has well over 10,000 lines of Python spread across more than 100 files. As of 2026-04-05, the latest release was still mutable: v3.2.1 (2026-03-27). Instance Scheduler depends on numerous Python modules and npm packages. It helps itself to permission to modify and stop any EC2 instance and delete any RDS snapshot. It also sends data to AWS. Instance Scheduler is powerful, and I have tremendous respect for its authors, but you'd need your own expert to run it securely.

Click to view the Lights Off architecture diagram:

An Event Bridge Scheduler rule triggers the 'Find' Amazon Web Services Lambda function every 10 minutes. The function calls 'describe' methods, checks the resource records returned for tag keys such as 'sched-start', and uses regular expressions to check the tag values for day, hour, and minute terms. Current day and time elements are inserted into the regular expressions using 'strftime'. If there is a match, the function sends a message to a Simple Queue Service queue. The 'Do' function, triggered in response, checks whether the message has expired. If not, this function calls the method indicated by the message attributes, passing the message body for the parameters. If the request is successful or a known exception occurs and it is not okay to re-try, the function is done. If an unknown exception occurs, the message remains in the operation queue, becoming visibile again after 90 seconds. After 3 tries, a message goes from the operation queue to the error (dead letter) queue.

Lights Off addresses Cloud Efficiency Hub report CER-0096: Missing Scheduled Shutdown for Non-Production EC2 Instances, and more!

Quick Start

  1. Log in to the AWS Console as an administrator.

  2. Tag a running, non-essential EC2 instance with:

    • sched-stop : d=_ H:M=11:30 , replacing 11:30 with the current UTC time + 20 minutes, rounded upward to :00, :10, :20, :30, :40, or :50.
  3. Install Lights Off using CloudFormation or Terraform.

    • CloudFormation
      Easy

      Create a CloudFormation stack.

      Select "Upload a template file", then select "Choose file" and navigate to a locally-saved copy of cloudformation/lights_off_aws.yaml [right-click to save as...].

      On the next page, set:

      • Stack name: LightsOff
    • Terraform

      Check that you have at least:

      Add the following child module to your existing root module:

      module "lights_off" {
        source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform?ref=v3.6.0"
        # Reference a specific version from github.com/sqlxpert/lights-off-aws/releases
        # Check that the release is immutable!
      }

      Have Terraform download the module's source code. Review the plan before typing yes to allow Terraform to proceed with applying the changes.

      terraform init
      terraform apply
  4. Wait for resource creation to complete.

    If there is an "UnreservedConcurrentExecution" error...

    Request that Service Quotas → AWS services → AWS Lambda → Concurrent executions be increased. The default is 1000 .

    Lights Off needs 1 unit for a time-critical function. New AWS accounts start with a quota of 10, but Lambda always holds back 10, which leaves 0 available! Within a given AWS account, the quota is set separately for each region.

  5. After about 20 minutes, check whether the EC2 instance is stopped. Restart it and delete the sched-stop tag.

Jump to: Extra SetupMulti-Account, Multi-Region

Tag Keys (Operations)

sched-stop sched-hibernate sched-backup
sched-start
EC2:
Instance → Image (AMI)
EBS Volume → Snapshot
RDS and Aurora:
Database Instance → Snapshot
Database Cluster → Snapshot

Whether a database operation is at the cluster or instance level depends on your choice of Aurora or RDS, and for RDS, on your database's configuration.

Tag Values (Schedules)

Work Week Examples

These cover Monday to Friday daytime work hours, 07:30 to 19:30, year-round (see time zone converter).

Locations Hours Saved sched-start sched-stop
USA Mainland > 50% u=1 u=2 u=3 u=4 u=5 H:M=11:30 u=2 u=3 u=4 u=5 u=6 H:M=03:30
North America
(from Hawaii
to Newfoundland)
> 40% u=1 u=2 u=3 u=4 u=5 H:M=10:00 u=2 u=3 u=4 u=5 u=6 H:M=05:30
Europe > 50% u=1 u=2 u=3 u=4 u=5 H:M=04:30 u=1 u=2 u=3 u=4 u=5 H:M=19:30
India > 60% u=1 u=2 u=3 u=4 u=5 H:M=02:00 u=1 u=2 u=3 u=4 u=5 H:M=14:00
North America
+ Europe
> 20% u=1 H:M=04:30 u=6 H:M=05:30
North America
+ Europe
+ India
> 20% u=1 H:M=02:00 u=6 H:M=05:30
Europe
+ India
> 40% u=1 u=2 u=3 u=4 u=5 H:M=02:00 u=1 u=2 u=3 u=4 u=5 H:M=19:30

Stopping an RDS or Aurora Database Longer than 7 Days

To stop a database indefinitely...

RDS and Aurora automatically start stopped databases after 7 days. Install github.com/sqlxpert/step-stay-stopped-aws-rds-aurora to re-stop them automatically, or set a once-a-week sched-start and add days to sched-stop :

Locations sched-start sched-stop
USA Mainland uTH:M=6T02:30 d=_ H:M=03:30
North America (from Hawaii to Newfoundland) uTH:M=6T04:30 d=_ H:M=05:30
Europe uTH:M=5T18:30 d=_ H:M=19:30
India uTH:M=5T13:00 d=_ H:M=14:00
North America + Europe uTH:M=6T04:30 ⚠ u=6 u=7 H:M=05:30
North America + Europe + India uTH:M=6T04:30 ⚠ u=6 u=7 H:M=05:30
Europe + India uTH:M=5T18:30 d=_ H:M=19:30
  • If the database usually takes longer than 1 hour to start, change sched-start to an earlier time (and to the preceding weekday, if necessary).
  • For most time zone combinations, sched-stop can be made daily. If a database takes longer than usual to start, another stop attempt will occur the next day. If you start the database manually, it will be stopped automatically at the end of the day.
  • ⚠ For North America + Europe [+ India], stop attempts will occur only on weekends. If you start the database manually, stop it manually when you are finished.
  • Set the database's weekly maintenance window to a time period when the database will be running.

Rules

  • Coordinated Universal Time (UTC)
  • 24-hour clock
  • Days before times, hours before minutes
  • The day, the hour and the minute must all be resolved
  • Multiple operations on the same resource at the same time are all canceled

Space was chosen as the separator and underscore, as the wildcard, because RDS does not allow commas or asterisks.

Single Terms

Type Literal Values (strftime) Wildcard
Day of month d=01 ... d=31 d=_
Day of week (ISO 8601) u=1 (Monday) ... u=7 (Sunday)
Hour H=00 ... H=23 H=_
Minute (multiple of 10) M=00 , M=10 , M=20 , M=30 , M=40 , M=50

Compound Terms

Type Note Literal Values
Once a day d=_ or d=NN or u=N first! H:M=00:00 ... H:M=23:50
Once a week uTH:M=1T00:00 ... uTH:M=7T23:50
Once a month dTH:M=01T00:00 ... dTH:M=31T23:50

Backup Examples

sched-backup Description
d=01 d=15 H=03 H=19 M=00 Traditional cron: 1st and 15th days of the month, at 03:00 and 19:00
d=_ H:M=03:00 H=_ M=15 M=45 Every day, at 03:00 plus every hour at 15 and 45 minutes after the hour
dTH:M=01T00:00 Start of month (instead of end of month)
dTH:M=01T03:00 uTH:M=5T19:00 d=_ H=11 M=15 1st day of the month at 03:00, plus Friday at 19:00, plus every day at 11:15

Extra Setup

Starting EC2 Instances with Encrypted EBS Volumes

In most cases, you can use the sched-start tag without setup.

If you use custom KMS encryption keys from a different AWS account...

The sched-start tag works for EC2 instances with EBS volumes if:

  • Your EBS volumes are unencrypted, or
  • You use the default, AWS-managed aws/ebs encryption key, or
  • You use custom keys in the same AWS account as each EC2 instance, the key policies contain the default "Enable IAM User Permissions" statement, and they do not contain "Deny" statements.

Because your custom keys are in a different AWS account than your EC2 instances, you must add a statement like the following to the key policies:

    {
      "Sid": "LightsOffEc2StartInstancesWithEncryptedEbsVolumes",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "kms:CreateGrant",
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:PrincipalOrgPaths": "o-ORG_ID/r-ROOT_ID/ou-PARENT_ORG_UNIT_ID/*"
        },
        "ArnLike": {
          "aws:PrincipalArn": "arn:aws:iam::ACCOUNT:role/*LightsOff*-DoLambdaFnRole-*"
        },
        "StringLike": {
          "kms:ViaService": "ec2.*.amazonaws.com"
        },
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    }
  • One account: Delete the entire "ForAnyValue:StringLike" section and replace ACCOUNT with the account number of the AWS account in which you have installed Lights Off.

  • AWS Organizations: Replace ACCOUNT with * and o-ORG_ID , r-ROOT_ID , and ou-PARENT_ORG_UNIT_ID with the identifiers of your organization, your organization root, and the organizational unit in which you have installed Lights Off. /* at the end of this organization path stands for child OUs, if any. Do not use a path less specific than "o-ORG_ID/*" .

If an EC2 instance does not start as scheduled, a KMS key permissions error is possible.

Making Backups

You can use the sched-backup tag with minimal setup if you work in a small number of regions and/or AWS accounts. Use the AWS Console to view the list of AWS Backup vaults one time in each AWS account and region. Make one backup in each AWS account (AWS Backup → My account → Dashboard → On-demand backup). If you use custom KMS keys, they must be in the same AWS account as the disks and databases encrypted with them.

If you work across many regions and/or AWS accounts...

Because you want to use the sched-backup tag in a complex AWS environment, you must address the following AWS Backup requirements:

  1. Vault

    AWS Backup creates the Default vault the first time you open the list of vaults in a given AWS account and region, using the AWS Console. Otherwise, see Backup vault creation and AWS::Backup::BackupVault or aws_backup_vault . Update the BackupVaultName CloudFormation stack parameter if necessary.

  2. Vault policy

    If you have added "Deny" statements, be sure that DoLambdaFnRole still has access.

  3. Backup role

    AWS Backup creates AWSBackupDefaultServiceRole the first time you make a backup in a given AWS account using the AWS Console (AWS Backup → My account → Dashboard → On-demand backup). Otherwise, see Default service role for AWS Backup. Update the BackupRoleName parameter if necessary.

  4. KMS key policies

    AWSBackupDefaultServiceRole works if:

    • Your EBS volumes and RDS/Aurora databases are unencrypted, or
    • You use the default, AWS-managed aws/ebs and aws/rds encryption keys, or
    • You use custom keys in the same AWS account as each disk and database, the key policies contain the default "Enable IAM User Permissions" statement, and they do not contain "Deny" statements.

    If your custom keys are in a different AWS account than your disks and databases, you must modify the key policies. See Encryption for backups in AWS Backup, How EBS uses KMS, Overview of encrypting RDS resources, and Key policies in KMS.

If no backup jobs appear in AWS Backup, or if jobs do not start, a permissions problem is likely.

Hidden Policies

Service and resource control policies (SCPs and RCPs), permissions boundaries, and session policies can interfere with the installation or usage of Lights Off. Check with your AWS administrator!

Accessing Backups

Goal Services
List backups AWS Backup
View underlying images and/or snapshots EC2 and RDS
Restore (create new resources from) backups EC2 and RDS, or AWS Backup
Delete backups AWS Backup

AWS Backup copies resource tags to backups. Lights Off adds sched-time to indicate when the backup was scheduled to occur, in ISO 8601 basic format (example: 20241231T1400Z).

On/Off Switch

  • You can toggle the Enable parameter of your Lights Off CloudFormation stack, CloudFormation StackSet, or Terraform module.
  • While Enable is false, scheduled operations do not happen; they are skipped permanently.

Logging and Monitoring

  1. Check the LightsOff CloudWatch log group.

    • Log entries are JSON objects.
      • Lights Off includes "level" , "type" and "value" keys.
      • Other software components may use different keys.
    • For more data, change the LogLevel parameter.
    • Scrutinize log entries at the ERROR level.
      • All entries with the "stackTrace" key represent unexpected exceptions that require correction. These are unusual.

      • "Find" function log streams: All other entries at the ERROR level require correction.

      • "Do" function log streams: Some other entries at the ERROR level do not require correction.

        What to consider when evaluating errors...

        The state of an AWS resource might change between the "Find" and "Do" steps; this sequence is fundamentally non-atomic. An operation might also be repeated due to queue message delivery logic; operations are idempotent. If a state change is favorable or an operation is repeated, Lights Off logs success responses or expected exceptions (depending on the AWS service) at the INFO level. For RDS database instance start/stop operations, however, expected exceptions are logged at the ERROR level because Lights Off cannot determine whether they represent actual errors or harmless repetition (such as trying to start a database instance that has already been started).

        For complete details, see the technical article Idempotence: Doing It More than Once.

  2. Check the ErrorQueue SQS queue for "Find" and "Do" events that were not delivered, or not fully processed.

  3. Check CloudTrail Event history for the final stages of sched-start and sched-backup operations.

    • CloudTrail events with an "Error code" may indicate permissions problems, typically due to the local security configuration.
    • To see more events, change "Read-only" from false to true .

Why No Built-In Monitoring?

Whether and how to monitor Lights Off...

Two strengths of this tool are its distributed design and its simplicity.

Lights Off operates independently in each region, in each AWS account. Every (region, account) pair has its own log and error queue. Operation does not depend on central resources, other than an optional customer-managed multi-region KMS key. Centralized logging and monitoring would introduce a single point of failure and add complexity, only to duplicate AWS features that can cover all of your applications.

Consider monitoring Lights Off through...

...which might already be configured in your organization.

The fetish for metrics, dashboards, and alerts raises a deeper question. Is it worth paging someone over an unsuccessful EC2 instance stop request? Probably not! If large amounts of money are at stake, you need AWS Budgets. The thresholds, notifications, and actions you define there cover any and all tools, applications, provisioning processes, and people, not just Lights Off.

Backups are worth monitoring, but at the point of consumption, not at the point of creation. No matter how you create backups, you should implement an automated process to restore critical ones (the first hourly backup of the day, for example) and validate the results. AWS Backup restore testing solves this problem.

When your organization becomes large and formal, you will graduate from friendly, locally-managed sched-backup tags to centralized backup plans. Then, you can use AWS Backup Audit Manager to continuously check your resources, your plans, and the presence of your backups. (You will still have to validate the backups.) Consider quitting before your job shifts from making software to making PowerPoint presentations.

Employees, consultants, and vendors who demonstrate pretty dashboards should instead start by asking what is material to you. You should ask whether AWS provides standard ways to gather the information you actually care about.

Advanced Installation

Multi-Account, Multi-Region (CloudFormation StackSet)

For reliability, Lights Off works completely independently in each region, in each AWS account. To deploy to multiple regions and/or AWS accounts,

  1. Delete any standalone Lights Off CloudFormation stacks in the target AWS accounts and regions (including any instances of the basic //terraform module; you will be installing one instance of the //terraform-multi module).

  2. Complete the prerequisites for creating a StackSet with service-managed permissions.

  3. Make sure that the AWS Lambda Concurrent executions quota is sufficient in every target AWS account, in every target region. See the note in Quick Start Step 4.

  4. Install Lights Off as a CloudFormation StackSet, using CloudFormation or Terraform. You must use your AWS organization's management account, or a delegated administrator AWS account.

    • CloudFormation
      Easy

      Create a CloudFormation StackSet.

      Select "Upload a template file", then select "Choose file" and upload a locally-saved copy of cloudformation/lights_off_aws.yaml [right-click to save as...].

      On the next page, set:

      • StackSet name: LightsOff

      On the "Set deployment options" page, under "Accounts", select "Deploy stacks in organizational units". Enter the ou- ID(s). Lights Off will be deployed to all AWS accounts within the organizational unit(s). Next, "Specify Regions".

    • Terraform

      Your module block will now resemble:

      module "lights_off_stackset" {
        source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform-multi?ref=v3.6.0"
        # Reference a specific version from github.com/sqlxpert/lights-off-aws/releases
        # Check that the release is immutable!
      
        lights_off_stackset_regions                 = ["us-east-1", "us-west-2",]
        lights_off_stackset_organizational_unit_ids = ["ou-0123-abcdefg",]
      }

Installation with Terraform

Quick Start Step 3 includes the option to install Lights Off as a Terraform module in one region in one AWS account. This is the basic //terraform module.

The enhanced region support added in v6.0.0 of the Terraform AWS provider makes it possible to deploy resources in multiple regions in one AWS account without configuring a separate provider for each region. Lights Off is compatible because the Terraform module was written for AWS provider v6, the original CloudFormation templates always let CloudFormation assign unique physical names to account-wide, non-regional resources like IAM roles, and the CloudFormation parameters were already region-independent. Your module block will now resemble:

module "lights_off" {
  source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform?ref=v3.6.0"
  # Reference a specific version from github.com/sqlxpert/lights-off-aws/releases
  # Check that the release is immutable!

  for_each          = toset(["us-east-1", "us-west-2",])
  lights_off_region = each.key
}

For installation in multiple AWS accounts (regardless of the number of regions), wrapping a CloudFormation StackSet in HashiCorp Configuration Language remains much easier than configuring Terraform to deploy identical resources in multiple AWS accounts. The Multi-Account, Multi-Region (CloudFormation StackSet) installation instructions include the option to do this using a Terraform module, in Step 4. This is the //terraform-multi module.

Least-Privilege Installation

Least-privilege installation details...

CloudFormation Stack Least-Privilege

You can use a CloudFormation service role to delegate only the privileges needed to create the LightsOff stack. (This is done for you if you use Terraform at Step 3 of the Quick Start.)

First, create the LightsOffPrereq stack from cloudformation/lights_off_aws_prereq.yaml .

Under "Additional settings" → "Stack policy - optional", you can "Upload a file" and select a locally-saved copy of cloudformation/lights_off_aws_prereq_policy.json . The stack policy prevents inadvertent replacement or deletion of the deployment role during stack updates, but it cannot prevent deletion of the entire LightsOffPrereq stack.

Next, when you create the LightsOff stack from cloudformation/lights_off_aws.yaml , set "Permissions - optional" → "IAM role - optional" to LightsOffPrereq-DeploymentRole . If your own privileges are limited, you might need permission to pass the deployment role to CloudFormation. See the LightsOffPrereq-SampleDeploymentRolePassRolePol IAM policy for an example.

CloudFormation StackSet Least-Privilege

For a CloudFormation StackSet, you can use self-managed permissions by copying the inline IAM policy of LightsOffPrereq-DeploymentRole to a customer-managed IAM policy, attaching your policy to AWSCloudFormationStackSetExecutionRole and propagating the policy and the role policy attachment to all target AWS accounts.

Terraform Least-Privilege

If you do not give Terraform full AWS administrative permissions, you must give it permission to:

  • List, describe, get tags for, create, tag, update, untag and delete IAM roles, update the "assume role" (role trust or "resource-based") policy, and put and delete in-line policies

  • Create, tag, describe, update, untag and delete arn:aws:s3:::terraform-* S3 buckets and put, tag, list, get, untag and delete arn:aws:s3:::terraform-*/* S3 objects

  • List, describe, create, tag, update, untag, and delete CloudFormation stacks

  • Set and get CloudFormation stack policies

  • Pass LightsOffPrereq-DeploymentRole-* to CloudFormation

  • List, describe, and get tags for, all data sources. For a list, run:

    grep 'data "' terraform*/*.tf | cut --delimiter=' ' --fields='1,2' | sort | uniq

Open the AWS Service Authorization Reference, go through the list of services on the left, and consult the "Actions" table for each of:

  • AWS Identity and Access Management (IAM)
  • Amazon S3
  • CloudFormation
  • AWS Security Token Service
  • AWS Backup (if you use the sched-backup tag)
  • AWS Key Management Service (if you encrypt the SQS queues and/or the CloudWatch log group with KMS keys)
  • AWS Organizations (if you create a CloudFormation StackSet with the //terraform-multi module)

In most cases, you can scope Terraform's permissions to one workload by regulating resource naming and tagging, and then by using:

Check Service and Resource Control Policies (SCPs and RCPs), as well as resource policies (such as AWS Backup vault policies and KMS key policies).

The basic //terraform module creates the LightsOffPrereq stack, which defines the IAM role that gives CloudFormation the permissions it needs to create the LightsOff stack. Terraform itself does not need the deployment role's permissions.

Security

In accordance with the software license, nothing in this document creates a warranty, an indemnification, an assumption of liability, etc. Use this software at your own risk. You are encouraged to evaluate the source code.

Security Design Goals

Security goals...
  • Least-privilege roles for the AWS Lambda functions that find resources and do scheduled operations. The "Do" function is authorized to perform a small set of operations, and at that, only when a resource has the correct tag key. (AWS Backup creates backups, using a role that you can configure.)
  • A least-privilege queue policy. The operation queue can only consume messages from the "Find" function and produce messages for the "Do" function, or the error queue, if an operation fails. Encryption in transit is required for both queues.
  • Readable IAM policies, broken down into discrete statements by service, resource or principal. Policies are formatted as CloudFormation YAML rather than as native JSON, except when it's necessary to allow insertion of custom, user-specified JSON.
  • Optional encryption at rest with the AWS Key Management System (KMS), for queue message bodies (may contain resource identifiers) and for log entries (may contain resource metadata).
  • No data storage other than in queues and logs, with short or configurable retention periods.
  • Tolerance for clock drift in a distributed system. The "Find" function starts 1 minute into the 10-minute cycle and operation queue entries expire 9 minutes in.
  • An optional CloudFormation service role for least-privilege deployment.

Security Steps You Can Take

Security actions...
  • Only allow trusted people and services to tag AWS resources. A sample service control policy is available.
  • Prevent people who can set the sched-backup tag from deleting backups.
  • Prevent people from modifying components, most of which can be identified by LightsOff in ARNs and in the automatic aws:cloudformation:stack-name tag. Limiting permissions so that the deployment role is necessary for CloudFormation stack modifications is ideal.
  • Prevent people from directly invoking the AWS Lambda functions and from passing the function roles to arbitrary functions.
  • Log infrastructure changes using AWS CloudTrail, and set up alerts.
  • Automatically copy backups to an AWS Backup vault in an isolated account. Lights Off is compatible with my Backup Events utility.
  • Separate production workloads. You might choose not to deploy Lights Off to AWS accounts used for production, or you might add a custom policy to the "Do" function's role, denying authority to stop production resources. See the AttachLocalPolicy parameter.
  • If you use Terraform, do not use it with an AWS access key and do not give it full AWS administrative privileges. Instead, follow AWS's Best practices for using the Terraform AWS Provider: Security best practices. Do the extra work of defining a least-privilege IAM role for deploying each workload. Configure Terraform to assume workload-specific roles. The CloudFormation service role is one element, but achieving least-privilege also requires limiting Terraform's privileges.

Service Control Policy

Protecting schedule tags...

A sample service control policy is available to prevent tampering with Lights Off schedule tags.

This SCP offers two-way protection: roles subject to the SCP can neither remove nor add schedule tags. They cannot change existing schedule tag values, either.

In your AWS Organizations management account, in the region where you manage infrastructure-as-code templates for non-regional resources, create a CloudFormation stack from cloudformation/scp_protect_lights_off_tags.yaml .

Or, reference the equivalent Terraform module:

module "lights_off_scp" {
  source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform-scp?ref=v3.6.0"
  # Reference a specific version from github.com/sqlxpert/lights-off-aws/releases
  # Check that the release is immutable!

  scp_target_ids = [
    "ou-0123-abcdefg",
  ]
}

In either case, specify the number of the account or the ou- ID of the organizational unit that you use for testing SCPs.

Test the SCP before applying it broadly, because it generally reduces existing EC2, EBS, and RDS/Aurora tagging permissions. Human users or automated processes might rely on those permissions. This is especially true of backup restoration, blue/green deployment, and cluster scaling workflows, which might copy tags to new resources.

You will need at least one SCP-exempt role in every AWS account, to manage schedule tags. I recommend IAM Identity Center permission sets. You can customize ScpPrincipalCondition / scp_principal_condition to reference permission set roles.

The SCP works by denying certain tag addition/change and removal requests. It cannot add permissions that have been denied by another SCP, or that were never allowed by a role's attached or inline policies.

SCPs do not affect roles or other IAM principals in the AWS Organizations management account.

Advice

  • Test Lights Off in your own AWS environment. After following the suggestions in the Logging and Monitoring section, please report bugs.

  • Be aware: of charges for S3 (holds CloudFormation templates, even if you use Terraform), EventBridge Scheduler, AWS Lambda, SQS, CloudWatch Logs, KMS, backup storage, and early deletion from cold storage; of the minimum charge when you stop an EC2 instance with a commercial license, or any RDS database; of the resumption of charges when RDS/Aurora restarts a stopped database after 7 days; and of ongoing storage charges and potential public IP address charges while EC2 instances and RDS/Aurora databases are stopped. What have we missed? 💸

  • Test your backups! Are they finishing on-schedule? Can they be restored successfully?

Bonus: Delete and Recreate Expensive Resources on a Schedule

Scheduled CloudFormation stack update details...

Lights Off can delete and recreate many types of expensive AWS infrastructure in your own CloudFormation stacks, based on cron schedules in stack tags.

Deleting AWS Client VPN resources overnight, while developers are asleep, is a sample use case. See 10-minute AWS Client VPN.

To make your own CloudFormation template compatible, see cloudformation/lights_off_aws_bonus_cloudformation_example.yaml .

Not every resource needs to be deleted and recreated; condition the creation of expensive resources on the Enable parameter. In the AWS Client VPN stack, the VPN endpoints and VPC security groups are not deleted, because they do not cost anything. The VPN attachments can be deleted and recreated with no need to reconfigure VPN clients.

Set the sched-set-Enable-true and sched-set-Enable-false tags on your own CloudFormation stack and make sure that the EnableSchedCloudFormationOps parameter is set to true (the default) in your Lights Off CloudFormation stack/StackSet or Terraform module. At the scheduled times, Lights Off will perform a stack update, toggling the value of the Enable parameter to true or false. (Capitalize Enable in the tag keys, to match the parameter name.)

If your tagged stack lacks a CloudFormation service role, Lights Off logs an error of "type" STACK_NEEDS_SERVICE_ROLE in a "Find" log stream in the log. To make scheduled updates possible, you must first perform a stack update in which you attach an IAM role that gives CloudFormation the permissions it needs to manage the resources defined in your stack. See the RoleARN request parameter in the UpdateStack reference.

💡 Because you can attach a different service role by performing a stack update, you may wish to maintain two roles, a less restrictive one that you can use for stack creation, arbitrary modification, and deletion, and a restrictive one that you can leave in place for automated stack updates. The latter role need only support the AWS actions invoked when the stack's Enable parameter changes from true to false and vice versa.

If the status of your tagged stack is other than CREATE_COMPLETE or UPDATE_COMPLETE at the scheduled time, Lights Off logs an error of "type" STACK_STATUS_IRREGULAR in a "Find" log stream, instead of attempting an update that is likely to fail and require a rollback. To resume scheduled stack updates, resolve the underlying template error or permissions error and successfully complete one manual stack update.

The sample service control policy does not cover sched-set-Enable-true and sched-set-Enable-false tags on CloudFormation stacks (or StackSets, whose tags would be copied to member stack instances). Because UpdateStack overwrites the entire set of tags, distinguishing between adding a tag, preserving an existing tag's value, explicitly removing a tag, and removing a tag by removing all tags, requires multiple IAM policy statements for each tag key. Only privileged roles should be allowed to create and update CloudFormation stacks/StackSets, including their tags.

Extensibility

Extensibility details...

Lights Off takes advantage of patterns in boto3, the AWS software development kit (SDK) for Python, and in the underlying AWS API. Adding AWS services, resource types, and operations is easy. For example, supporting Aurora database clusters (RDS database instances were already supported) required adding:

    AWSRsrcType(
      "rds",
      ("DB", "Cluster"),
      {
        ("start", ): {},
        ("stop", ): {},
        ("backup", ): {},
      },
      rsrc_id_key_suffix="Identifier",
      tags_key="TagList",
    )

Given the words DB and Cluster in the resource type name, plus the operation verb start, the sched-start tag key and the start_db_cluster method name are derived mechanically.

If an operation method takes more than just the resource identifier, add a dictionary of static keyword arguments. For complex arguments, sub-class the AWSOp class and override op_kwargs .

The start_backup_job method takes an Amazon Resource Name (ARN), whose format is consistent for all resource types. As long as AWS Backup supports the resource type, there is no extra work to do.

Add statements like the one below to the Identity and Access Management (IAM) policy for the role used by the "Do" AWS Lambda function, to authorize operations. You must of course authorize the role used by the "Find" function to describe (list) resources.

          - Effect: Allow
            Action: rds:StartDBCluster
            Resource: !Sub "arn:${AWS::Partition}:rds:${AWS::Region}:${AWS::AccountId}:cluster:*"
            Condition:
              "Null": { "aws:ResourceTag/sched-start": "false" }

Let me know what resource types you'd like me to add!

Progress

I wrote TagSchedOps, the original version of Lights Off, in 2017, before Systems Manager, Data Lifecycle Manager or AWS Backup existed. Lights Off remains a simple alternative to Systems Manager Automation runbooks for stopping EC2 instances, etc. It is now integrated with AWS Backup, leveraging the security and management benefits (including backup retention lifecycle policies) but offering a simple alternative to backup plans.

Counting Complexity

Year Lambda Python Lines Core CloudFormation YAML Lines Core Terraform HCL Lines
2017 ≈ 775 ≈ 2,140
2022 630 800 ✓
2025 620 ✓ 1,000 ≈ 270
2026 620 1,120 270

Here I report "loc" figures from GitHub. Figures for CloudFormation are net of in-line Lambda Python code. GitHub seems to count non-blank, non-comment lines, for a rough indication of complexity.

In the introduction, I reported total lines of code for my Lambda Python source file, because blank lines and comment lines contribute to the reading experience. To provide an order-of-magnitude comparison of complexity, I counted non-blank, non-comment lines in Instance Scheduler .py files without test in their paths. People would never read all the Python source in AWS's Instance Scheduler, which is my point!

Dedication

This project is dedicated to ej, Marianne and Régis, Ivan, and to the wonderful colleagues whom Paul has worked with over the years. Thank you to Corey for sharing it with the AWS user community in Last Week in AWS newsletter issues 286 (2022-10-03) and 424 (2025-05-27), and to Lee for suggesting the new name.

Licenses

Scope Link Included Copy
Source code files, and source code embedded in documentation files GNU General Public License (GPL) 3.0 LICENSE-CODE.md
Documentation files (including this ReadMe file) GNU Free Documentation License (FDL) 1.3 LICENSE-DOC.md

Copyright Paul Marcelin

Contact: marcelin at cmu.edu (replace "at" with @)