Aaron VanSledright

Tag: AWS Boto3 Tutorial

  • Streamline Your S3 Management: How to Count Small Files Locally with Python

    I recently came across a need to count objects in an S3 bucket that were of particular size. There isn’t a native way to do this within the S3 console so I decided to script it out using Python and the AWS SDK. You can see all of the code on my GitHub.

    The script is very easy to utilize. The logic is as follows:

    Step 1: Setup

    First, the script initializes a client for the S3 service using Boto3. This setup requires your AWS credentials configured, which the script uses to authenticate requests to your S3 buckets.

    Step 2: Input Parameters

    The script accepts two command-line arguments:

    • The name of the S3 bucket.
    • The maximum file size (in bytes) for which you want to count the files.

    Step 3: List and Count

    Using Boto3’s paginator, the script efficiently handles large datasets by fetching lists of objects in batches. It iterates through each object in the specified bucket:

    • It checks if the object’s size is less than or equal to the size limit you specified.
    • It increments a count for each file that meets the criteria.

    Step 4: Error Handling

    If the script encounters issues accessing the bucket or if an API call fails, it catches the exception and prints an error message. This helps in debugging and ensures you understand why a particular operation failed.

    Step 5: Output

    Finally, the script outputs the total count of files that meet the size criteria. This result can be used directly in reports, further automation scripts, or just for informational purposes.

    Usage:

    python count_small_files.py my-example-bucket 1048576

    Replace my-example-bucket with your bucket name and 1048576 with the maximum file size in bytes (1 MB in this example). This command will tell you how many files in my-example-bucket are 1 MB or smaller.

    This Python script is a practical tool for anyone looking to manage S3 storage more efficiently. By running this script, you can quickly gather insights into your data distribution, helping you make informed decisions about storage management and optimization.

    Stay tuned for more insights on managing cloud services and enhancing your productivity through automation. Don’t forget to subscribe for more updates and tutorials!

  • Securing AWS S3 Objects with Python: Implementing SSE-S3 Encryption

    In the cloud-native world, data security is paramount, and securing Amazon Web Services (AWS) S3 storage is a critical task for any developer. In this article, we dive into a Python script designed to ensure that all your S3 objects are encrypted using Server-Side Encryption with S3-Managed Keys (SSE-S3). This method provides robust security by encrypting S3 objects at the server level using keys managed by S3.

    Understanding the Python Script

    Using the code located at: https://github.com/avansledright/s3-object-re-encryption we have a good framework for re-encrypting our objects.

    The script utilizes the boto3 library, a Python SDK for AWS, enabling developers to integrate their applications with AWS services directly. It includes functions to list objects in an S3 bucket, check their encryption status, and apply SSE-S3 encryption if necessary.

    Key Functions:

    1. Listing Objects: Retrieves all objects within a specified bucket and prefix, managing pagination to handle large datasets.
    2. Checking Encryption: Examines if each object is encrypted with SSE-S3 by accessing its metadata.
    3. Applying Encryption: Updates objects not encrypted with SSE-S3, ensuring all data is securely encrypted using copy_object with the ServerSideEncryption parameter.

    Why Encrypt with SSE-S3?

    Encrypting your S3 objects with SSE-S3 ensures that data is automatically encrypted before being saved to disk and decrypted when accessed. This happens transparently, allowing you to secure your data without modifying your application code.

    Running the Script

    The script is executed via the command line, where users specify the S3 bucket and prefix. It then processes each object, ensuring encryption standards meet organizational and compliance requirements.

    Expanding the Script

    While this script provides a basic framework for S3 encryption, it can be expanded with additional error handling, logging, and perhaps integration into a larger AWS security auditing tool.

    AWS developers looking to enhance their application security will find this script a valuable starting point for implementing standard security practices within their S3 environments. By automating the encryption process, developers can ensure consistency and security across all stored data.

    For those who manage sensitive or regulated data in AWS, applying SSE-S3 encryption programmatically can help meet legal and compliance obligations while providing peace of mind about data security.

    If you find this article helpful please share it with your friends!