Ever forget to turn the lights off? Now you can:
-
Stop EC2 instances and RDS/Aurora databases temporarily by tagging them with cron schedules, to cut AWS costs. Schedules (not references to schedules) go directly in the tags.
-
Trigger AWS Backup with cron schedules in resource tags.
-
Delete expensive infrastructure temporarily by tagging your own CloudFormation stacks with cron schedules.
-
Easily deploy this tool to multiple AWS accounts and regions.
Jump to: Quick Start • Tags • Schedules • Multi-Account, Multi-Region • Security
🔒 Software supply chain security is on everyone's mind. This tool's two Lambda functions share one Python source file that's short enough to read (750 lines total). I made GitHub releases immutable as of
v3.6.0(2026-04-06). AWS manages patching of the stock Lambda runtime, which provides the Python standard library and the AWS software development kit (boto, boto3).AWS's Instance Scheduler, the closest competing tool, has well over 10,000 lines of Python spread across more than 100 files. As of 2026-04-05, the latest release was still mutable: v3.2.1 (2026-03-27). Instance Scheduler depends on numerous Python modules and npm packages. It helps itself to permission to modify and stop any EC2 instance and delete any RDS snapshot. It also sends data to AWS. Instance Scheduler is powerful, and I have tremendous respect for its authors, but you'd need your own expert to run it securely.
Click to view the Lights Off architecture diagram:
Lights Off addresses Cloud Efficiency Hub report CER-0096: Missing Scheduled Shutdown for Non-Production EC2 Instances, and more!
-
Log in to the AWS Console as an administrator.
-
Tag a running, non-essential EC2 instance with:
sched-stop:d=_ H:M=11:30, replacing 11:30 with the current UTC time + 20 minutes, rounded upward to :00, :10, :20, :30, :40, or :50.
-
Install Lights Off using CloudFormation or Terraform.
-
CloudFormation
Easy ✓Create a CloudFormation stack.
Select "Upload a template file", then select "Choose file" and navigate to a locally-saved copy of cloudformation/lights_off_aws.yaml [right-click to save as...].
On the next page, set:
- Stack name:
LightsOff
- Stack name:
-
Terraform
Check that you have at least:
Add the following child module to your existing root module:
module "lights_off" { source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform?ref=v3.6.0" # Reference a specific version from github.com/sqlxpert/lights-off-aws/releases # Check that the release is immutable! }
Have Terraform download the module's source code. Review the plan before typing
yesto allow Terraform to proceed with applying the changes.terraform init terraform apply
-
-
Wait for resource creation to complete.
If there is an "UnreservedConcurrentExecution" error...
Request that Service Quotas → AWS services → AWS Lambda → Concurrent executions be increased. The default is
1000.Lights Off needs 1 unit for a time-critical function. New AWS accounts start with a quota of 10, but Lambda always holds back 10, which leaves 0 available! Within a given AWS account, the quota is set separately for each region.
-
After about 20 minutes, check whether the EC2 instance is stopped. Restart it and delete the
sched-stoptag.
Jump to: Extra Setup • Multi-Account, Multi-Region
sched-stop |
sched-hibernate |
sched-backup |
|
|---|---|---|---|
sched-start |
|||
| EC2: | |||
| Instance | ✓ | ✓ | → Image (AMI) |
| EBS Volume | → Snapshot | ||
| RDS and Aurora: | |||
| Database Instance | ✓ | → Snapshot | |
| Database Cluster | ✓ | → Snapshot |
Whether a database operation is at the cluster or instance level depends on your choice of Aurora or RDS, and for RDS, on your database's configuration.
These cover Monday to Friday daytime work hours, 07:30 to 19:30, year-round (see time zone converter).
| Locations | Hours Saved | sched-start |
sched-stop |
|---|---|---|---|
| USA Mainland | > 50% | u=1 u=2 u=3 u=4 u=5 H:M=11:30 |
u=2 u=3 u=4 u=5 u=6 H:M=03:30 |
| North America (from Hawaii to Newfoundland) |
> 40% | u=1 u=2 u=3 u=4 u=5 H:M=10:00 |
u=2 u=3 u=4 u=5 u=6 H:M=05:30 |
| Europe | > 50% | u=1 u=2 u=3 u=4 u=5 H:M=04:30 |
u=1 u=2 u=3 u=4 u=5 H:M=19:30 |
| India | > 60% | u=1 u=2 u=3 u=4 u=5 H:M=02:00 |
u=1 u=2 u=3 u=4 u=5 H:M=14:00 |
| North America + Europe |
> 20% | u=1 H:M=04:30 |
u=6 H:M=05:30 |
| North America + Europe + India |
> 20% | u=1 H:M=02:00 |
u=6 H:M=05:30 |
| Europe + India |
> 40% | u=1 u=2 u=3 u=4 u=5 H:M=02:00 |
u=1 u=2 u=3 u=4 u=5 H:M=19:30 |
To stop a database indefinitely...
RDS and Aurora automatically start stopped databases after 7 days. Install
github.com/sqlxpert/step-stay-stopped-aws-rds-aurora
to re-stop them automatically, or set a once-a-week sched-start and add
days to sched-stop :
| Locations | sched-start |
sched-stop |
|---|---|---|
| USA Mainland | uTH:M=6T02:30 |
d=_ H:M=03:30 |
| North America (from Hawaii to Newfoundland) | uTH:M=6T04:30 |
d=_ H:M=05:30 |
| Europe | uTH:M=5T18:30 |
d=_ H:M=19:30 |
| India | uTH:M=5T13:00 |
d=_ H:M=14:00 |
| North America + Europe | uTH:M=6T04:30 |
⚠ u=6 u=7 H:M=05:30 |
| North America + Europe + India | uTH:M=6T04:30 |
⚠ u=6 u=7 H:M=05:30 |
| Europe + India | uTH:M=5T18:30 |
d=_ H:M=19:30 |
- If the database usually takes longer than 1 hour to start, change
sched-startto an earlier time (and to the preceding weekday, if necessary). - For most time zone combinations,
sched-stopcan be made daily. If a database takes longer than usual to start, another stop attempt will occur the next day. If you start the database manually, it will be stopped automatically at the end of the day. - ⚠ For North America + Europe [+ India], stop attempts will occur only on weekends. If you start the database manually, stop it manually when you are finished.
- Set the database's weekly maintenance window to a time period when the database will be running.
- Coordinated Universal Time (UTC)
- 24-hour clock
- Days before times, hours before minutes
- The day, the hour and the minute must all be resolved
- Multiple operations on the same resource at the same time are all canceled
Space was chosen as the separator and underscore, as the wildcard, because RDS does not allow commas or asterisks.
| Type | Literal Values (strftime) | Wildcard |
|---|---|---|
| Day of month | d=01 ... d=31 |
d=_ |
| Day of week (ISO 8601) | u=1 (Monday) ... u=7 (Sunday) |
|
| Hour | H=00 ... H=23 |
H=_ |
| Minute (multiple of 10) | M=00 , M=10 , M=20 , M=30 , M=40 , M=50 |
| Type | Note | Literal Values |
|---|---|---|
| Once a day | d=_ or d=NN or u=N first! |
H:M=00:00 ... H:M=23:50 |
| Once a week | uTH:M=1T00:00 ... uTH:M=7T23:50 |
|
| Once a month | dTH:M=01T00:00 ... dTH:M=31T23:50 |
sched-backup |
Description |
|---|---|
d=01 d=15 H=03 H=19 M=00 |
Traditional cron: 1st and 15th days of the month, at 03:00 and 19:00 |
d=_ H:M=03:00 H=_ M=15 M=45 |
Every day, at 03:00 plus every hour at 15 and 45 minutes after the hour |
dTH:M=01T00:00 |
Start of month (instead of end of month) |
dTH:M=01T03:00 uTH:M=5T19:00 d=_ H=11 M=15 |
1st day of the month at 03:00, plus Friday at 19:00, plus every day at 11:15 |
In most cases, you can use the sched-start tag without setup.
If you use custom KMS encryption keys from a different AWS account...
The sched-start tag works for EC2 instances with EBS volumes if:
- Your EBS volumes are unencrypted, or
- You use the default, AWS-managed
aws/ebsencryption key, or - You use custom keys in the same AWS account as each EC2 instance, the key
policies contain the default
"Enable IAM User Permissions"statement, and they do not contain"Deny"statements.
Because your custom keys are in a different AWS account than your EC2 instances, you must add a statement like the following to the key policies:
{
"Sid": "LightsOffEc2StartInstancesWithEncryptedEbsVolumes",
"Effect": "Allow",
"Principal": "*",
"Action": "kms:CreateGrant",
"Resource": "*",
"Condition": {
"ForAnyValue:StringLike": {
"aws:PrincipalOrgPaths": "o-ORG_ID/r-ROOT_ID/ou-PARENT_ORG_UNIT_ID/*"
},
"ArnLike": {
"aws:PrincipalArn": "arn:aws:iam::ACCOUNT:role/*LightsOff*-DoLambdaFnRole-*"
},
"StringLike": {
"kms:ViaService": "ec2.*.amazonaws.com"
},
"Bool": {
"kms:GrantIsForAWSResource": "true"
}
}
}-
One account: Delete the entire
"ForAnyValue:StringLike"section and replace ACCOUNT with the account number of the AWS account in which you have installed Lights Off. -
AWS Organizations: Replace ACCOUNT with
*and o-ORG_ID , r-ROOT_ID , and ou-PARENT_ORG_UNIT_ID with the identifiers of your organization, your organization root, and the organizational unit in which you have installed Lights Off./*at the end of this organization path stands for child OUs, if any. Do not use a path less specific than"o-ORG_ID/*".
If an EC2 instance does not start as scheduled, a KMS key permissions error is possible.
You can use the sched-backup tag with minimal setup if you work in a small
number of regions and/or AWS accounts. Use the AWS Console to view the
list of AWS Backup vaults
one time in each AWS account and region. Make one backup in each AWS account
(AWS Backup → My account
→ Dashboard → On-demand backup). If you use custom KMS keys, they
must be in the same AWS account as the disks and databases encrypted with
them.
If you work across many regions and/or AWS accounts...
Because you want to use the sched-backup tag in a complex AWS environment,
you must address the following AWS Backup requirements:
-
Vault
AWS Backup creates the
Defaultvault the first time you open the list of vaults in a given AWS account and region, using the AWS Console. Otherwise, see Backup vault creation and AWS::Backup::BackupVault or aws_backup_vault . Update theBackupVaultNameCloudFormation stack parameter if necessary. -
Vault policy
If you have added
"Deny"statements, be sure thatDoLambdaFnRolestill has access. -
Backup role
AWS Backup creates
AWSBackupDefaultServiceRolethe first time you make a backup in a given AWS account using the AWS Console (AWS Backup → My account → Dashboard → On-demand backup). Otherwise, see Default service role for AWS Backup. Update theBackupRoleNameparameter if necessary. -
KMS key policies
AWSBackupDefaultServiceRoleworks if:- Your EBS volumes and RDS/Aurora databases are unencrypted, or
- You use the default, AWS-managed
aws/ebsandaws/rdsencryption keys, or - You use custom keys in the same AWS account as each disk and database,
the key policies contain the default
"Enable IAM User Permissions"statement, and they do not contain"Deny"statements.
If your custom keys are in a different AWS account than your disks and databases, you must modify the key policies. See Encryption for backups in AWS Backup, How EBS uses KMS, Overview of encrypting RDS resources, and Key policies in KMS.
If no backup jobs appear in AWS Backup, or if jobs do not start, a permissions problem is likely.
Hidden Policies
Service and resource control policies (SCPs and RCPs), permissions boundaries, and session policies can interfere with the installation or usage of Lights Off. Check with your AWS administrator!
| Goal | Services |
|---|---|
| List backups | AWS Backup |
| View underlying images and/or snapshots | EC2 and RDS |
| Restore (create new resources from) backups | EC2 and RDS, or AWS Backup |
| Delete backups | AWS Backup |
AWS Backup copies resource tags to backups. Lights Off adds sched-time to
indicate when the backup was scheduled to occur, in
ISO 8601
basic format (example: 20241231T1400Z).
- You can toggle the
Enableparameter of your Lights Off CloudFormation stack, CloudFormation StackSet, or Terraform module. - While Enable is
false, scheduled operations do not happen; they are skipped permanently.
-
Check the LightsOff CloudWatch log group.
- Log entries are JSON objects.
- Lights Off includes
"level","type"and"value"keys. - Other software components may use different keys.
- Lights Off includes
- For more data, change the
LogLevelparameter. - Scrutinize log entries at the
ERRORlevel.-
All entries with the
"stackTrace"key represent unexpected exceptions that require correction. These are unusual. -
"Find" function log streams: All other entries at the
ERRORlevel require correction. -
"Do" function log streams: Some other entries at the
ERRORlevel do not require correction.What to consider when evaluating errors...
The state of an AWS resource might change between the "Find" and "Do" steps; this sequence is fundamentally non-atomic. An operation might also be repeated due to queue message delivery logic; operations are idempotent. If a state change is favorable or an operation is repeated, Lights Off logs success responses or expected exceptions (depending on the AWS service) at the
INFOlevel. For RDS database instance start/stop operations, however, expected exceptions are logged at theERRORlevel because Lights Off cannot determine whether they represent actual errors or harmless repetition (such as trying to start a database instance that has already been started).For complete details, see the technical article Idempotence: Doing It More than Once.
-
- Log entries are JSON objects.
-
Check the
ErrorQueueSQS queue for "Find" and "Do" events that were not delivered, or not fully processed. -
Check CloudTrail Event history for the final stages of
sched-startandsched-backupoperations.- CloudTrail events with an "Error code" may indicate permissions problems, typically due to the local security configuration.
- To see more events, change "Read-only" from
falsetotrue.
Whether and how to monitor Lights Off...
Two strengths of this tool are its distributed design and its simplicity.
Lights Off operates independently in each region, in each AWS account. Every (region, account) pair has its own log and error queue. Operation does not depend on central resources, other than an optional customer-managed multi-region KMS key. Centralized logging and monitoring would introduce a single point of failure and add complexity, only to duplicate AWS features that can cover all of your applications.
Consider monitoring Lights Off through...
...which might already be configured in your organization.
The fetish for metrics, dashboards, and alerts raises a deeper question. Is it worth paging someone over an unsuccessful EC2 instance stop request? Probably not! If large amounts of money are at stake, you need AWS Budgets. The thresholds, notifications, and actions you define there cover any and all tools, applications, provisioning processes, and people, not just Lights Off.
Backups are worth monitoring, but at the point of consumption, not at the point of creation. No matter how you create backups, you should implement an automated process to restore critical ones (the first hourly backup of the day, for example) and validate the results. AWS Backup restore testing solves this problem.
When your organization becomes large and formal, you will graduate from
friendly, locally-managed sched-backup tags to centralized
backup plans.
Then, you can use
AWS Backup Audit Manager
to continuously check your resources, your plans, and the presence of your
backups. (You will still have to validate the backups.) Consider quitting
before your job shifts from making software to making PowerPoint presentations.
Employees, consultants, and vendors who demonstrate pretty dashboards should instead start by asking what is material to you. You should ask whether AWS provides standard ways to gather the information you actually care about.
For reliability, Lights Off works completely independently in each region, in each AWS account. To deploy to multiple regions and/or AWS accounts,
-
Delete any standalone Lights Off CloudFormation stacks in the target AWS accounts and regions (including any instances of the basic
//terraformmodule; you will be installing one instance of the//terraform-multimodule). -
Complete the prerequisites for creating a StackSet with service-managed permissions.
-
Make sure that the AWS Lambda
Concurrent executionsquota is sufficient in every target AWS account, in every target region. See the note in Quick Start Step 4. -
Install Lights Off as a CloudFormation StackSet, using CloudFormation or Terraform. You must use your AWS organization's management account, or a delegated administrator AWS account.
-
CloudFormation
Easy ✓Create a CloudFormation StackSet.
Select "Upload a template file", then select "Choose file" and upload a locally-saved copy of cloudformation/lights_off_aws.yaml [right-click to save as...].
On the next page, set:
- StackSet name:
LightsOff
On the "Set deployment options" page, under "Accounts", select "Deploy stacks in organizational units". Enter the
ou-ID(s). Lights Off will be deployed to all AWS accounts within the organizational unit(s). Next, "Specify Regions". - StackSet name:
-
Terraform
Your module block will now resemble:
module "lights_off_stackset" { source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform-multi?ref=v3.6.0" # Reference a specific version from github.com/sqlxpert/lights-off-aws/releases # Check that the release is immutable! lights_off_stackset_regions = ["us-east-1", "us-west-2",] lights_off_stackset_organizational_unit_ids = ["ou-0123-abcdefg",] }
-
Quick Start
Step 3 includes the option to install Lights Off as a Terraform module in
one region in one AWS account. This is the basic //terraform module.
The enhanced region support added in v6.0.0 of the Terraform AWS provider makes it possible to deploy resources in multiple regions in one AWS account without configuring a separate provider for each region. Lights Off is compatible because the Terraform module was written for AWS provider v6, the original CloudFormation templates always let CloudFormation assign unique physical names to account-wide, non-regional resources like IAM roles, and the CloudFormation parameters were already region-independent. Your module block will now resemble:
module "lights_off" {
source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform?ref=v3.6.0"
# Reference a specific version from github.com/sqlxpert/lights-off-aws/releases
# Check that the release is immutable!
for_each = toset(["us-east-1", "us-west-2",])
lights_off_region = each.key
}For installation in multiple AWS accounts (regardless of the number of
regions), wrapping a CloudFormation StackSet in HashiCorp Configuration
Language remains much easier than configuring Terraform to deploy identical
resources in multiple AWS accounts. The
Multi-Account, Multi-Region (CloudFormation StackSet)
installation instructions include the option to do this using a Terraform
module, in Step 4. This is the //terraform-multi module.
Least-privilege installation details...
You can use a
CloudFormation service role
to delegate only the privileges needed to create the LightsOff stack. (This
is done for you if you use Terraform at Step 3 of the
Quick Start.)
First, create the LightsOffPrereq stack from
cloudformation/lights_off_aws_prereq.yaml .
Under "Additional settings" → "Stack policy - optional", you can "Upload a
file" and select a locally-saved copy of
cloudformation/lights_off_aws_prereq_policy.json .
The stack policy prevents inadvertent replacement or deletion of the deployment
role during stack updates, but it cannot prevent deletion of the entire
LightsOffPrereq stack.
Next, when you create the LightsOff stack from
cloudformation/lights_off_aws.yaml ,
set "Permissions - optional" → "IAM role - optional" to
LightsOffPrereq-DeploymentRole . If your own privileges are limited, you
might need permission to pass the deployment role to CloudFormation. See the
LightsOffPrereq-SampleDeploymentRolePassRolePol IAM policy for an example.
For a CloudFormation StackSet, you can use
self-managed permissions
by copying the inline IAM policy of LightsOffPrereq-DeploymentRole to a
customer-managed IAM policy, attaching your policy to
AWSCloudFormationStackSetExecutionRole and propagating the policy and the
role policy attachment to all target AWS accounts.
If you do not give Terraform full AWS administrative permissions, you must give it permission to:
-
List, describe, get tags for, create, tag, update, untag and delete IAM roles, update the "assume role" (role trust or "resource-based") policy, and put and delete in-line policies
-
Create, tag, describe, update, untag and delete
arn:aws:s3:::terraform-*S3 buckets and put, tag, list, get, untag and deletearn:aws:s3:::terraform-*/*S3 objects -
List, describe, create, tag, update, untag, and delete CloudFormation stacks
-
Set and get CloudFormation stack policies
-
Pass
LightsOffPrereq-DeploymentRole-*to CloudFormation -
List, describe, and get tags for, all
datasources. For a list, run:grep 'data "' terraform*/*.tf | cut --delimiter=' ' --fields='1,2' | sort | uniq
Open the AWS Service Authorization Reference, go through the list of services on the left, and consult the "Actions" table for each of:
AWS Identity and Access Management (IAM)Amazon S3CloudFormationAWS Security Token ServiceAWS Backup(if you use thesched-backuptag)AWS Key Management Service(if you encrypt the SQS queues and/or the CloudWatch log group with KMS keys)AWS Organizations(if you create a CloudFormation StackSet with the//terraform-multimodule)
In most cases, you can scope Terraform's permissions to one workload by regulating resource naming and tagging, and then by using:
- ARN patterns in
Resourcelists - ARN patterns in
Conditionentries - Request tag and then resource tag
Conditionentries
Check Service and Resource Control Policies (SCPs and RCPs), as well as resource policies (such as AWS Backup vault policies and KMS key policies).
The basic //terraform module creates the LightsOffPrereq stack, which
defines the IAM role that gives CloudFormation the permissions it needs to
create the LightsOff stack. Terraform itself does not need the deployment
role's permissions.
In accordance with the software license, nothing in this document creates a warranty, an indemnification, an assumption of liability, etc. Use this software at your own risk. You are encouraged to evaluate the source code.
Security goals...
- Least-privilege roles for the AWS Lambda functions that find resources and do scheduled operations. The "Do" function is authorized to perform a small set of operations, and at that, only when a resource has the correct tag key. (AWS Backup creates backups, using a role that you can configure.)
- A least-privilege queue policy. The operation queue can only consume messages from the "Find" function and produce messages for the "Do" function, or the error queue, if an operation fails. Encryption in transit is required for both queues.
- Readable IAM policies, broken down into discrete statements by service, resource or principal. Policies are formatted as CloudFormation YAML rather than as native JSON, except when it's necessary to allow insertion of custom, user-specified JSON.
- Optional encryption at rest with the AWS Key Management System (KMS), for queue message bodies (may contain resource identifiers) and for log entries (may contain resource metadata).
- No data storage other than in queues and logs, with short or configurable retention periods.
- Tolerance for clock drift in a distributed system. The "Find" function starts 1 minute into the 10-minute cycle and operation queue entries expire 9 minutes in.
- An optional CloudFormation service role for least-privilege deployment.
Security actions...
- Only allow trusted people and services to tag AWS resources. A sample service control policy is available.
- Prevent people who can set the
sched-backuptag from deleting backups. - Prevent people from modifying components, most of which can be identified by
LightsOffin ARNs and in the automaticaws:cloudformation:stack-nametag. Limiting permissions so that the deployment role is necessary for CloudFormation stack modifications is ideal. - Prevent people from directly invoking the AWS Lambda functions and from passing the function roles to arbitrary functions.
- Log infrastructure changes using AWS CloudTrail, and set up alerts.
- Automatically copy backups to an AWS Backup vault in an isolated account. Lights Off is compatible with my Backup Events utility.
- Separate production workloads. You might choose not to deploy Lights Off to
AWS accounts used for production, or you might add a custom policy to the
"Do" function's role, denying authority to stop production resources. See the
AttachLocalPolicyparameter. - If you use Terraform, do not use it with an AWS access key and do not give it full AWS administrative privileges. Instead, follow AWS's Best practices for using the Terraform AWS Provider: Security best practices. Do the extra work of defining a least-privilege IAM role for deploying each workload. Configure Terraform to assume workload-specific roles. The CloudFormation service role is one element, but achieving least-privilege also requires limiting Terraform's privileges.
Protecting schedule tags...
A sample service control policy is available to prevent tampering with Lights Off schedule tags.
This SCP offers two-way protection: roles subject to the SCP can neither remove nor add schedule tags. They cannot change existing schedule tag values, either.
In your AWS Organizations management account, in the region where you manage infrastructure-as-code templates for non-regional resources, create a CloudFormation stack from cloudformation/scp_protect_lights_off_tags.yaml .
Or, reference the equivalent Terraform module:
module "lights_off_scp" {
source = "git::https://github.com/sqlxpert/lights-off-aws.git//terraform-scp?ref=v3.6.0"
# Reference a specific version from github.com/sqlxpert/lights-off-aws/releases
# Check that the release is immutable!
scp_target_ids = [
"ou-0123-abcdefg",
]
}In either case, specify the number of the account or the ou- ID of the
organizational unit that you use for testing SCPs.
Test the SCP before applying it broadly, because it generally reduces existing EC2, EBS, and RDS/Aurora tagging permissions. Human users or automated processes might rely on those permissions. This is especially true of backup restoration, blue/green deployment, and cluster scaling workflows, which might copy tags to new resources.
You will need at least one SCP-exempt role in every AWS account, to manage
schedule tags. I recommend
IAM Identity Center permission sets.
You can customize ScpPrincipalCondition / scp_principal_condition to
reference permission set roles.
The SCP works by denying certain tag addition/change and removal requests. It cannot add permissions that have been denied by another SCP, or that were never allowed by a role's attached or inline policies.
SCPs do not affect roles or other IAM principals in the AWS Organizations management account.
-
Test Lights Off in your own AWS environment. After following the suggestions in the Logging and Monitoring section, please report bugs.
-
Be aware: of charges for S3 (holds CloudFormation templates, even if you use Terraform), EventBridge Scheduler, AWS Lambda, SQS, CloudWatch Logs, KMS, backup storage, and early deletion from cold storage; of the minimum charge when you stop an EC2 instance with a commercial license, or any RDS database; of the resumption of charges when RDS/Aurora restarts a stopped database after 7 days; and of ongoing storage charges and potential public IP address charges while EC2 instances and RDS/Aurora databases are stopped. What have we missed? 💸
-
Test your backups! Are they finishing on-schedule? Can they be restored successfully?
Scheduled CloudFormation stack update details...
Lights Off can delete and recreate many types of expensive AWS infrastructure in your own CloudFormation stacks, based on cron schedules in stack tags.
Deleting AWS Client VPN resources overnight, while developers are asleep, is a sample use case. See 10-minute AWS Client VPN.
To make your own CloudFormation template compatible, see cloudformation/lights_off_aws_bonus_cloudformation_example.yaml .
Not every resource needs to be deleted and recreated; condition the creation
of expensive resources on the Enable parameter. In the AWS Client VPN
stack, the VPN endpoints and VPC security groups are not deleted, because they
do not cost anything. The VPN attachments can be deleted and recreated with no
need to reconfigure VPN clients.
Set the sched-set-Enable-true and sched-set-Enable-false tags on
your own CloudFormation stack and make sure that the
EnableSchedCloudFormationOps parameter is set to true (the default) in your
Lights Off CloudFormation stack/StackSet or Terraform module. At the scheduled
times, Lights Off will perform a stack update, toggling the value of the
Enable parameter to true or false. (Capitalize Enable in the tag keys,
to match the parameter name.)
If your tagged stack lacks a CloudFormation service role, Lights Off logs an
error of "type" STACK_NEEDS_SERVICE_ROLE in a "Find" log stream in the
log.
To make scheduled updates possible, you must first perform a stack update in
which you attach an IAM role that gives CloudFormation the permissions it
needs to manage the resources defined in your stack. See the RoleARN request
parameter in the
UpdateStack reference.
💡 Because you can attach a different service role by performing a stack
update, you may wish to maintain two roles, a less restrictive one that you can
use for stack creation, arbitrary modification, and deletion, and a restrictive
one that you can leave in place for automated stack updates. The latter role
need only support the AWS actions invoked when the stack's Enable parameter
changes from true to false and vice versa.
If the status of your tagged stack is other than CREATE_COMPLETE or
UPDATE_COMPLETE at the scheduled time, Lights Off logs an error of "type"
STACK_STATUS_IRREGULAR in a "Find" log stream, instead of attempting an
update that is likely to fail and require a rollback. To resume scheduled stack
updates, resolve the underlying template error or permissions error and
successfully complete one manual stack update.
The sample
service control policy
does not cover sched-set-Enable-true and sched-set-Enable-false tags on
CloudFormation stacks (or StackSets, whose tags would be copied to member
stack instances). Because
UpdateStack
overwrites the entire set of tags, distinguishing between adding a tag,
preserving an existing tag's value, explicitly removing a tag, and removing a
tag by removing all tags, requires multiple IAM policy statements for each tag
key. Only privileged roles should be allowed to create and update
CloudFormation stacks/StackSets, including their tags.
Extensibility details...
Lights Off takes advantage of patterns in boto3, the AWS software development kit (SDK) for Python, and in the underlying AWS API. Adding AWS services, resource types, and operations is easy. For example, supporting Aurora database clusters (RDS database instances were already supported) required adding:
AWSRsrcType(
"rds",
("DB", "Cluster"),
{
("start", ): {},
("stop", ): {},
("backup", ): {},
},
rsrc_id_key_suffix="Identifier",
tags_key="TagList",
)Given the words DB and Cluster in the resource type name, plus the
operation verb start, the sched-start tag key and the start_db_cluster
method name are derived mechanically.
If an operation method takes more than just the resource identifier, add a
dictionary of static keyword arguments. For complex arguments, sub-class the
AWSOp class and override op_kwargs .
The start_backup_job method takes an Amazon Resource Name (ARN), whose
format is consistent for all resource types. As long as AWS Backup supports
the resource type, there is no extra work to do.
Add statements like the one below to the Identity and Access Management (IAM) policy for the role used by the "Do" AWS Lambda function, to authorize operations. You must of course authorize the role used by the "Find" function to describe (list) resources.
- Effect: Allow
Action: rds:StartDBCluster
Resource: !Sub "arn:${AWS::Partition}:rds:${AWS::Region}:${AWS::AccountId}:cluster:*"
Condition:
"Null": { "aws:ResourceTag/sched-start": "false" }Let me know what resource types you'd like me to add!
I wrote TagSchedOps, the original version of Lights Off, in 2017, before Systems Manager, Data Lifecycle Manager or AWS Backup existed. Lights Off remains a simple alternative to Systems Manager Automation runbooks for stopping EC2 instances, etc. It is now integrated with AWS Backup, leveraging the security and management benefits (including backup retention lifecycle policies) but offering a simple alternative to backup plans.
| Year | Lambda Python Lines | Core CloudFormation YAML Lines | Core Terraform HCL Lines |
|---|---|---|---|
| 2017 | ≈ 775 | ≈ 2,140 | |
| 2022 | 630 | 800 ✓ | |
| 2025 | 620 ✓ | 1,000 | ≈ 270 |
| 2026 | 620 | 1,120 | 270 |
Here I report "loc" figures from GitHub. Figures for CloudFormation are net of in-line Lambda Python code. GitHub seems to count non-blank, non-comment lines, for a rough indication of complexity.
In the introduction, I reported total lines of code for my Lambda Python
source file, because blank lines and comment lines contribute to the reading
experience. To provide an order-of-magnitude comparison of complexity, I
counted non-blank, non-comment lines in Instance Scheduler .py files without
test in their paths. People would never read all the Python source in AWS's
Instance Scheduler, which is my point!
This project is dedicated to ej, Marianne and Régis, Ivan, and to the wonderful colleagues whom Paul has worked with over the years. Thank you to Corey for sharing it with the AWS user community in Last Week in AWS newsletter issues 286 (2022-10-03) and 424 (2025-05-27), and to Lee for suggesting the new name.
| Scope | Link | Included Copy |
|---|---|---|
| Source code files, and source code embedded in documentation files | GNU General Public License (GPL) 3.0 | LICENSE-CODE.md |
| Documentation files (including this ReadMe file) | GNU Free Documentation License (FDL) 1.3 | LICENSE-DOC.md |
Copyright Paul Marcelin
Contact: marcelin at cmu.edu (replace "at" with @)