While working on a small PoC recently, there was a need to transfer a good amount of data between Azure Blob Storage and AWS S3 storage. The transfer could happen either from Azure blob to S3 or vice versa and what was required is a good tool to do it. On searching for options, found the following ways to achieve this. Wanted to share this work because I had to gather it from various sources.

Azure Blob to AWS S3 - Use Azure CLI 

It was good to see that Azure CLI already has this in-built feature to fetch from or transfer data to S3. A third-party tool is not needed. One can start the Azure console from their account and use the given command to perform the transfer.

// Set the environment variables

$env:AWS_ACCESS_KEY_ID = "<your aws access key id> "

$env:AWS_SECRET_ACCESS_KEY = "<your aws secret>"

// Execute the copy command (for fetching data from s3)

azcopy cp "<s3 file url>" "<azure blob url>" 

Example - 

azcopy cp "https://poc.s3.ap-south-1.amazonaws.com/my_bucket/myfile.csv" "https://poc.blob.core.windows.net/my_folder?st=2020-01-15T09%3A33&se=2020-01-23T09%3A33%3A00Z&sp=racwdl&sv=2018-03-28&sr=c&sig=JrFd82k9DdBXfJz5DUn%" 

// Execute the copy command (to transfer data to s3)

azcopy cp "<azure blob url>" "<s3 file url>" 

Example -

azcopy cp "https://poc.blob.core.windows.net/my_folder?st=2020-01-15T09%3A33&se=2020-01-23T09%3A33%3A00Z&sp=racwdl&sv=2018-03-28&sr=c&sig=JrFd82k9DdBXfJz5DUn%"  "https://poc.s3.ap-south-1.amazonaws.com/my_bucket/myfile.csv" 

AWS S3 to Azure - Use a third-party tool 

S3 does not have in-built capabilities to transfer data to or from Azure Blob. However, there is a third-party node-js application available on github which does the job pretty well. This app can be installed on an EC2 instance or a local system and then used as instructed here. The steps to use this app are:

  • Install node-js
  • Download and execute the app with the appropriate options/parameters from the CLI
  • CLI options are mentioned in the link