<p>The cURL command to obtain the download link returns a JSON containing the URL. This can be parsed on the command line using <a href="https://stedolan.github.io/jq/" rel="nofollow noreferrer">jq</a> and used to construct a command to download the file. For example, with a list of URLs in a file (one per line) called <em>urls.txt</em>, you could try the following:</p>
<pre><code>while IFS= read -r url ; do
    &gt;&amp;2 echo &quot;Downloading: ${url}&quot;

    curl -sSH @cosmic_header &quot;<span class="math-container">${url}" |
    jq --arg base "$</span>(basename &quot;<span class="math-container">${url}")" -r '"curl -o \($</span>base) \(.url | @sh)&quot;' |
    sh
done &lt; urls.txt
</code></pre>
<p>Note that the above stores the HTTP header in a file called <em>cosmic_header</em>, which can be created using:</p>
<pre><code>echo &quot;Authorization: Basic $(echo 'username:password' | base64)&quot; &gt; cosmic_header
</code></pre>
<p>This avoids exposing the authorization key on the command line when you go to download your list of files, which might be preferable if working in a shared environment.</p>

CC BY-SA 4.0
2023-03-29T01:43:06.747Z
2023-03-29T01:43:06.747Z
Steve
<p>Hi I am downloading the data from a website which requires authentication of email_id and password before the download. They then provide the download link which is used to download dataset. This download link is different every time</p>
<pre><code>### Download Cosmic Files

##First you need to login with your login details and the access key ID may vary with each login ID

##STEP-1
echo &quot;pr@gmail.com:abc123@&quot; | base64
#The above lines gives a authorization key cHIxNEBzYW5

##STEP-2
# get the download link using the authorization key from step 1
#curl -H &quot;Authorization: Basic cHIxNEBzYW5&quot; https://cancer.sanger.ac.uk/cosmic/file_download/GRCh38/cosmic/v97/CosmicSample.tsv.gz

#The above line gives the download link where AWS access key changes everytime like in this case it was https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&amp;Expires=1679591113&amp;Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D

##STEP-3
# Download file with the link from step 2
#curl -o CosmicSample.tsv &quot;https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&amp;Expires=1679591113&amp;Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D&quot;

</code></pre>
<ol>
<li>I want to automate the script first by passing used_id and password as an string</li>
<li>I want to replace the AWS key in the step 3 with a string that is generated from step 2</li>
</ol>
<p>Thank you</p>

<database><bash><curl>
Passing strings for password, email authentication and download link
# Passing strings for password, email authentication and download link

(Asked by: username_0 on 2023-03-28)

Hi I am downloading the data from a website which requires authentication of email_id and password before the download. They then provide the download link which is used to download dataset. This download link is different every time
### Download Cosmic Files

##First you need to login with your login details and the access key ID may vary with each login ID

##STEP-1
echo "pr@gmail.com:abc123@" | base64
#The above lines gives a authorization key cHIxNEBzYW5

##STEP-2
# get the download link using the authorization key from step 1
#curl -H "Authorization: Basic cHIxNEBzYW5" https://cancer.sanger.ac.uk/cosmic/file_download/GRCh38/cosmic/v97/CosmicSample.tsv.gz

#The above line gives the download link where AWS access key changes everytime like in this case it was https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&Expires=1679591113&Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D

##STEP-3
# Download file with the link from step 2
#curl -o CosmicSample.tsv "https://cog.sanger.ac.uk/cosmic/GRCh38/cosmic/v97/CosmicSample.tsv.gz?AWSAccessKeyId=KRV7P7QR9DL41J9EWGA2&Expires=1679591113&Signature=WPjwDQsMoaLtRA2utiGLNrOSlOA%3D"



I want to automate the script first by passing used_id and password as an string
I want to replace the AWS key in the step 3 with a string that is generated from step 2

Thank you

**Tags:** database, bash, curl

---

## 1 answers

### Answer by username_1 on 2023-03-29. Score: 3 | Accepted answer

The cURL command to obtain the download link returns a JSON containing the URL. This can be parsed on the command line using jq and used to construct a command to download the file. For example, with a list of URLs in a file (one per line) called urls.txt, you could try the following:
while IFS= read -r url ; do
    >&2 echo "Downloading: ${url}"

    curl -sSH @cosmic_header "${url}" |
    jq --arg base "$(basename "${url}")" -r '"curl -o \($base) \(.url | @sh)"' |
    sh
done < urls.txt

Note that the above stores the HTTP header in a file called cosmic_header, which can be created using:
echo "Authorization: Basic $(echo 'username:password' | base64)" > cosmic_header

This avoids exposing the authorization key on the command line when you go to download your list of files, which might be preferable if working in a shared environment.


---

