S3 Credentials: A Complete Guide to Creating, Managing, and Securing Your Access

Amazon Simple Storage Service (S3) is the backbone of many data pipelines, analytics workloads, and backup solutions. The gateway to S3 is a set of security credentials—an Access Key ID and a Secret Access Key. This article walks you through the entire lifecycle of S3 credentials, from creation to best‑practice security, and shows how they integrate with platforms such as Azure Databricks and Unity Catalog.

Why S3 Credentials Matter

S3 credentials authenticate every request you make to an S3 bucket. Without them, you cannot:

Because these keys grant programmatic access, protecting them is as important as protecting a password.

Step‑by‑Step: Creating S3 Access Keys

1. Sign in to the AWS Management Console

Navigate to the IAM (Identity and Access Management) console. If you do not have an IAM user, create one first; never use your root account for everyday tasks.

2. Create a New Access Key

  1. In the left navigation pane, select Users and click the user name that needs S3 access.
  2. Open the Security credentials tab.
  3. Click Create access key. Choose “Programmatic access” to generate an Access Key ID and Secret Access Key.
  4. Download the CSV file or copy the keys immediately—AWS does not show the secret again.

3. Verify Permissions

Attach an appropriate IAM policy. For a write‑only scenario, use a custom policy such as:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::your-bucket", "arn:aws:s3:::your-bucket/*" ] } ] }

Save the policy and attach it to the user or role.

Integrating S3 Credentials with Azure Databricks

In a recent video tutorial, Shreel Shah, a Scale Solutions Engineer at Databricks, demonstrates how to configure Azure Databricks to read from an S3 bucket using Unity Catalog. The steps are:

  1. Store the Access Key ID and Secret Access Key in Azure Key Vault.
  2. Mount the S3 bucket in Databricks using the dbutils.fs.mount command, referencing the secret scope.
  3. Configure Unity Catalog to recognize the external storage location, enabling fine‑grained data governance.