Download a file securely from GCS on an untrusted system
The Problem
We publish some of our build artifacts to Google Cloud Storage, and users need to download these to the target installation system. But, this target system is not always trusted and can have shared local users, so we don't want to store long-lived credentials.
As a user, I can download the artifact on my (secure) laptop and transfer it to the target system. But, the artifact can be large (several GBs). So, downloading and uploading again makes it cumbersome and slow.
Option 1: use gcloud CLI on the target system
Log in to the target system, install gcloud CLI, authenticate, and then download the file:
$ gcloud storage cp gs://$BUCKET/$FILE ./
This has two problems:
- The user must install (and maybe update) gcloud CLI on the target system.
- The user needs to store their credentials on the target system. These credentials have full access to whatever resources the user has. So, it's a huge security risk, especially if we don't trust the target system.
To mitigate (2), the user can log out of gcloud CLI after downloading. But, this is a manual step they might miss.
Option 2: use gcloud CLI with a service account
This is a variation of the above solution - we log in using a service account instead of the user account. This service account can have restricted access to only the resources needed.
$ gcloud iam service-accounts create $SA_NAME \
--description="Service Account for downloading artifacts"
$ gsutil iam ch \
serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com:roles/storage.objectViewer \
gs://$BUCKET
This partially mitigates problem (2) above. If the user forgets to log out of gcloud CLI, the damage will be restricted to the resources accessible by the service account.
Option 3: Short-lived access token
Gcloud CLI supports creating short-lived credentials for the end-user account or any service account.
This credential can be used to download the artifact using wget with an authorization header - no need to install gcloud CLI.
Here's a small script that asks for the auth token as input, parses various GCS bucket URL formats, and downloads the requested artifact directly using wget:
#!/bin/bash
# Download artifact from GCS bucket
set -e
echo -e "====> Run \`gcloud auth print-access-token\` on a system where you've setup gcloud to get access token\n"
read -r -p "Enter access token: " StorageAccessToken
read -r -p "Enter GCS artifact URL: " ArtifactURL
if [[ "${ArtifactURL:0:33}" == "https://console.cloud.google.com/" ]]; then
BucketAndFile="${ArtifactURL#*https://console.cloud.google.com/storage/browser/_details/}"
elif [[ "${ArtifactURL:0:33}" == "https://storage.cloud.google.com/" ]]; then
BucketAndFile="${ArtifactURL#*https://storage.cloud.google.com/}"
elif [[ "${ArtifactURL:0:5}" == "gs://" ]]; then
BucketAndFile="${ArtifactURL#*gs://}"
else
echo "Invalid GCS artifact URL"
exit 1
fi
StorageBucket="${BucketAndFile%%/*}"
StorageFile="${BucketAndFile#*/}"
StorageFileEscaped=$(echo "${StorageFile}" | sed 's/\//%2F/g')
OutputFileName="${StorageFile##*/}"
echo -e "\n====> Downloading gs://${StorageBucket}/${StorageFile} to ${OutputFileName}\n"
wget -O "${OutputFileName}" --header="Authorization: Bearer ${StorageAccessToken}" \
"https://storage.googleapis.com/storage/v1/b/${StorageBucket}/o/${StorageFileEscaped}?alt=media"
Option 4: Signed URLs
Google Cloud Storage also supports signed URLs - which give time-limited access to a specific Cloud Storage resource. Anyone possessing the signed URL can use it while it's active without any further credentials. This fits our use case brilliantly.
To do this, first we need to give ourselves the iam.serviceAccountTokenCreator
role so that we can impersonate a service account.
$ gcloud iam service-accounts add-iam-policy-binding \
$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
--member=$MY_EMAIL \
--role=roles/iam.serviceAccountTokenCreator
Then, we can generate a signed URL:
$ gcloud config set auth/impersonate_service_account \
$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com
$ gsutil signurl -u -r $REGION -d 10m gs://$BUCKET/$FILE
$ gcloud config unset auth/impersonate_service_account
And we can use wget to download the artifact from this URL without any further authentication.
Interactions