Google cloud storage upload using generic signed url and curl

google cloud storage upload from url
google cloud storage signed url java
google cloud storage signed url javascript
google cloud storage signed url python
google cloud storage public url
how to upload file to google cloud storage using javascript
google cloud storage upload from string
how to upload file to google cloud storage using python

I am trying to put a file into a Google Cloud Storage (GCS) bucket from the command line. At a later stage this shall be used in a deployed script at the user end without any type of user-visible authentication.

So far I generate a signed url like this:

gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/testfile

which will generate something like

https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount@developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D

The next step (at the user end) would be curl uploading the file with a PUT request. Like so:

curl -X PUT --data-binary @testfile 'https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount@developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D'

I can get this to work with an existing file in the bucket and a GET request (for downloading), but it does not seem to work for uploading. curl throws the server's response with error messages like this at me:

<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided.
         Check your Google secret key and signing method.</Message>
<StringToSign>PUT

application/x-www-form-urlencoded
1430963040
/mybucket/testfile</StringToSign>
</Error>

And this makes sense to me, as obviously I am not just making a bare PUT request, but one for a particular file of a specific size, whereas the signature computed by 'gsutil signurl' would not know about these details at the time it is computed.

Somehow I was under the impression (e.g., based on the last usage case described in gsutil signurl documentation and also in the post How to allow anonymous uploads to cloud storage) that it should be possible to generate a generic signed url for uploading purposes and then use it later. Am I just mistaken about this point or is there a way to fix the curl request?

Any thoughts about this are appreciated. However, I'd like this to work with "minimal tools", i.e., ideally shell and curl only, but no other programming languages.

EDIT: Organising one's thoughts by formulating the exact problem is the first step towards the solution. I realise now that

curl -X PUT -T - [request-url] < testfile

does actually solve the immediate problem. However, this means multiple users would write to the same file if they use the same signed url. The documentation suggests you can omit the object name in the creation of the signed url, i.e., use

gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/

This, supposedly, would allow anyone with the resulting signed url to put any object of any type into my bucket. Only I do not get this work, as I don't see how you can then tell GCS which object you are actually writing to.

This was driving me mad too. Turns out it was the binary-file part of the curl command. Try this instead:

curl -X PUT --upload-file me.jpeg $SIGNED_URL

Signed URLs | Cloud Storage, Signed URL makes it possible to generate temporary credentials valid only for a specific end user to securely upload a file. The Google Cloud  Search for Cloud Online Storage Here. Visit our Web Now! Search for Cloud Online Storage on fastquicksearch.com!

If the resource does not specify a single object, you can do so on an individual basis by adding a URL param to the request with the name of the object. For example:

curl -X PUT -T - [request-url]?name=[object-name] < testfile

This surely works with storage/v1, although I have not tried myself with a signed URL yet.

Uploading images directly to Cloud Storage by using Signed URL , A couple days ago a colleague asked if its possible to use Google Cloud Storage Signed URL with Resumable Uploads. SignedURLs are  Get What Is Google Cloud Storage. Get Instant Quality Info at iZito Now!

I encountered the similar problem(403 forbidden). It turned out that my json library, which I use it to marshal each response, would replace & by \u0026 for security concern. So the url may be correct in the program but invalid in client side So I guess that there might be some string encoding bug inside the Signature query string of your url since the signature string is harder to detect error in comparison with my \u0026.

Google Cloud Storage SignedURL + Resumable upload with cURL, GCS SignedURLs do not support multiple file uploads with one URL. Google Cloud Storage SignedURL + Resumable upload with cURL  Simply specify Cloud Storage resources, point to the host storage.googleapis.com, and use Google HMAC credentials in the process of generating the signed URL. Signed URL example The following is an example of a signed URL that was created following the V4 signing process with service account authentication:

Upload/Download files from a browser with GCS Signed URLs and , How do I upload files to Google Cloud Storage using Python? Anyway, here is the raw protocol using curl. create a file of 10⁸ bytes; base64 /dev/urandom | head -c (100000000) > file.txt. 2. Generate the signed URL and exchange it for the upload location URL.

Using Signed URLs - Amazon CloudFront, I am trying to put a file into a Google Cloud Storage (GCS) bucket from the command line. At a later. You may also specify RESUMABLE to create a signed resumable upload start URL. When using a signed URL to start a resumable upload session, you will need to specify the 'x-goog-resumable:start'

Manage Files in Google Cloud Storage With Python, ogrinfo a shapefile in a zip file on the internet: ogrinfo -ro -al -so /vsizip//vsicurl/​https:// Virtual file systems can only be used with GDAL or OGR drivers supporting the /vsicurl/ will try to query directly redirected URLs to Amazon S3 signed URLs in Google Cloud Storage buckets, without prior download of the entire file. How to upload to Google Cloud Storage buckets using CURL Signed URLs are pretty nifty feature given by Google Cloud Platform to let anyone access your cloud storage (bucket or any file in the bucket) without need to sign in. Official documentation gives step by step details as to how to read/write to the bucket using gsutil or through a program.

Comments
  • Did you get any further with this? I am having the same problem, getting access denied.
  • Nothing to report here since those edits. Unfortunately.
  • i got the same problem
  • > The documentation suggests you can omit the object name in the creation of the signed url - Where do you see that? You have to specify the object name.
  • Maybe this helps. I have an appengine Python script which uses a form post to send a file to GCS. The post request is signed. Appengine makes creating a post HTML form very easy, without the need for a local p12 key. More here: github.com/voscausa/appengine-gcs-upload
  • That's interesting, thanks. Unfortunately, I can't get it to work. The storage/v1 approach uses different URLs and the POST method for inserting objects. gsutil signurl on the other hand does not support the POST method (it uses PUT). Just replacing my PUT request URL above with https://www.googleapis.com/storage/v1/b/mybucket/o?GoogleAccessId=...as_before...?name=myobject does not seem to work for me, either.
  • Thanks. I don't think it is a url encoding problem, as the command curl -X PUT -T - [request-url] < testfile does work for me. The HTTP 403 response you got is a strong indicator that that something with the URL is not right and the wrong object or a nonexisting object is requested. Glad to hear you could work out the reason for this and fix it.