AWS S3 Lifecycle delete rules

By putting a cleanup Lifecycle rule in place on your S3 buckets, you may be able to potentially save costs and increase LIST performance.

"Incomplete Multipart Uploads – S3’s multipart upload feature accelerates the uploading of large objects by allowing you to split them up into logical parts that can be uploaded in parallel.  If you initiate a multipart upload but never finish it, the in-progress upload occupies some storage space and will incur storage charges. However, these uploads are not visible when you list the contents of a bucket and (until today’s release) had to be explicitly removed.

Expired Object Delete Markers – S3’s versioning feature allows you to preserve, retrieve, and restore every version of every object stored in a versioned bucket. When you delete a versioned object, a delete marker is created. If all previous versions of the object subsequently expire, an expired object delete marker is left. These markers do not incur storage charges. However, removing unneeded delete markers can improve the performance of S3’s LIST operation."

Source: https://aws.amazon.com/blogs/aws/s3-lifecycle-management-update-support-for-multipart-uploads-and-delete-markers/

To add a cleanup Lifecycle rule:

• Log into the Amazon S3 web console

• Select your S3 bucket

• Select Management

• Select Add lifecycle rule

• Enter a name such as 

'Delete incomplete multipart upload and Delete previous versions'

• Skip Transitions for now

Transitions allow you to move storage to slower locations at a reduced cost

https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html

• Expiration

• Delete Previous versions after 365 days

You can choose shorter periods such as 7 days or 30 days if you don’’t have a use case for retrieving prior S3 versions.

You will still have the current version, which is usually all you want, but deleting previous versions can help with costs and S3 LIST performance.

• Clean up incomplete multipart uploads after 7 days

If you do not have any automated processes that may re-try uploads, you could choose 1 day

• Review

Agree to the 'scary' this applies to all objects in bucket

Note, if you have S3 objects (uploads) which require different policies, you may find it easier to manage by creating a S3 bucket per policy.

You now have some basic cleanup of your S3 bucket(s) configured.

-End of Document-

Thanks for reading