While working on a small PoC recently, there was a need to transfer a good amount of data between Azure Blob Storage and AWS S3 storage. The transfer could happen either from Azure blob to S3 or vice versa and what was required is a good tool to do it. On searching for options, found the following ways to achieve this. Wanted to share this work because I had to gather it from various sources.
Azure Blob to AWS S3 - Use Azure CLI
It was good to see that Azure CLI already has this in-built feature to fetch from or transfer data to S3. A third-party tool is not needed. One can start the Azure console from their account and use the given command to perform the transfer.
// Set the environment variables
$env:AWS_ACCESS_KEY_ID = "<your aws access key id> "
$env:AWS_SECRET_ACCESS_KEY = "<your aws secret>"
// Execute the copy command (for fetching data from s3)
azcopy cp "<s3 file url>" "<azure blob url>"
azcopy cp "https://poc.s3.ap-south-1.amazonaws.com/my_bucket/myfile.csv" "https://poc.blob.core.windows.net/my_folder?st=2020-01-15T09%3A33&se=2020-01-23T09%3A33%3A00Z&sp=racwdl&sv=2018-03-28&sr=c&sig=JrFd82k9DdBXfJz5DUn%"
// Execute the copy command (to transfer data to s3)
azcopy cp "<azure blob url>" "<s3 file url>"
azcopy cp "https://poc.blob.core.windows.net/my_folder?st=2020-01-15T09%3A33&se=2020-01-23T09%3A33%3A00Z&sp=racwdl&sv=2018-03-28&sr=c&sig=JrFd82k9DdBXfJz5DUn%" "https://poc.s3.ap-south-1.amazonaws.com/my_bucket/myfile.csv"
AWS S3 to Azure - Use a third-party tool
S3 does not have in-built capabilities to transfer data to or from Azure Blob. However, there is a third-party node-js application available on github which does the job pretty well. This app can be installed on an EC2 instance or a local system and then used as instructed here. The steps to use this app are:
- Install node-js
- Download and execute the app with the appropriate options/parameters from the CLI
- CLI options are mentioned in the link