Skip to content

Instantly share code, notes, and snippets.

@bewest
Last active May 20, 2018 15:01
Show Gist options
  • Save bewest/6142358 to your computer and use it in GitHub Desktop.
Save bewest/6142358 to your computer and use it in GitHub Desktop.
practice uploading things with curl

uploading files to API

recommended

with urlencoded body, multipart streams

This is an example of how to submit data through post request using both GET params and POST urlencoded parameters, in addition to uploading several more files.

A profile.json is just a demo... could be data2.log, for example.

+ /usr/local/bin/json
+ curl -v -s -X POST -H '' -F '[email protected];type=application/json' -F 'raw_data=@raw_data.log;type=plain/text' -F hello=world -F profile=profile.json -F raw=raw 'http://localhost:5000/api/echo?upload=raw_data.log'
* About to connect() to localhost port 5000 (#0)
*   Trying 127.0.0.1... connected
> POST /api/echo?upload=raw_data.log HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:5000
> Accept: */*
> Content-Length: 1717
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------cdf099a4bb93
> 
< HTTP/1.1 100 Continue
} [data not shown]
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 418
< Date: Fri, 02 Aug 2013 20:47:04 GMT
< Connection: keep-alive
< 
{ [data not shown]
* Connection #0 to host localhost left intact
* Closing connection #0
[
  "hahaha",
  {
    "params": {
      "upload": "raw_data.log",
      "hello": "world",
      "profile": "profile.json",
      "raw": "raw"
    },
    "headers": {
      "user-agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
      "host": "localhost:5000",
      "accept": "*/*",
      "content-length": "1717",
      "expect": "100-continue",
      "content-type": "multipart/form-data; boundary=----------------------------cdf099a4bb93"
    },
    "method": "POST"
  }
]

colophon

Here are the (heavily deduped) commands I used to create this repo. I edited out bunch of practice runs. You can see how the editing history lines up in revisions.

cd src/gist
[email protected]:6142358.git
# https://gist.github.com/6142358.git
git clone $gist curl_uploads
touch practice.sh
chmod a+xr practice.sh 
./practice.sh 
./practice.sh  | json
./practice.sh | tee out.log
cat out.log 
mv out.log baseline-nothing.log
git add baseline-nothing.log
git status
git add practice.sh
git status
git commit  -av
git push
./practice.sh 
git status
git add out.log
git commit  -v
git diff
git show
wc raw_data.log  profile.json | tee practice
git diff
rm practice
git status
./practice.sh 
git commit  -av
git diff
git show
./practice.sh 
git commit  -avm 'with command'
git push
./practice.sh # 7x
cp out.log  some_json.log
git status
git commit  -avm 'some json'
git status
git add some_json.log
git commit  -avm 'some json'
git push
./practice.sh  # 10x
cp out.log all_data.log
git add all_data.log
git commit  -avm 'success'
git push
git show
./practice.sh # 4x
mv out.log  all_data_2.log
git add all_data*
git commit  -av
git push
git show
./practice.sh # 9x
cat profile.json | json
cat profile.json 
vim profile.json # match some proposed suggestions 
./practice.sh 
wc raw_data.log 
ls -alh raw_data.log 
./practice.sh  # 5x
vim raw_data.log 
./practice.sh  # 6x
wc raw_data.log  profile.json | tee -a practice_http_streams.markdown 
./practice.sh  # 4x
git status
git add out.log
git commit  -av 'use form data'
git commit  -avm 'use url encoded form data'
git push
./practice.sh 
mv out.log pure_json.log
git add pure_json.log
git commit  -avm 'as pure json'
git push
./practice.sh # 2x
cp out.log recommended.log
git add recommended.log
git commit  -av
git push
./practice.sh 
git add README.markdown
git commit  -av
git push origin 
git diff
git commit  -avm 'tweak'
git push
history | grep -E "pract|git|log|json|tee|gist" | tee hist
+ curl -v -s -X POST -H '"Content-Type:' 'application/json"' -F '[email protected];type=application/json' -F 'raw_data=@raw_data.log;type=plain/text' 'http://localhost:5000/api/echo?upload=raw_data.log'
+ /usr/local/bin/json
* getaddrinfo(3) failed for application:80
* Couldn't resolve host 'application'
* Closing connection #0
* About to connect() to localhost port 5000 (#0)
* Trying 127.0.0.1... connected
> POST /api/echo?upload=raw_data.log HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:5000
> Accept: */*
> Content-Length: 862
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------76334dc9101c
>
< HTTP/1.1 100 Continue
} [data not shown]
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 364
< Date: Fri, 02 Aug 2013 19:53:44 GMT
< Connection: keep-alive
<
{ [data not shown]
* Connection #0 to host localhost left intact
* Closing connection #0
[
"hahaha",
{
"params": {
"upload": "raw_data.log"
},
"headers": {
"user-agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
"host": "localhost:5000",
"accept": "*/*",
"content-length": "862",
"expect": "100-continue",
"content-type": "multipart/form-data; boundary=----------------------------76334dc9101c"
},
"method": "POST"
}
]
+ /usr/local/bin/json
+ curl -v -s -X POST -H '"Content-Type:' 'application/json"' -F '[email protected];type=application/json' -F 'raw_data=@raw_data.log;type=plain/text' 'http://localhost:5000/api/echo?upload=raw_data.log'
* getaddrinfo(3) failed for application:80
* Couldn't resolve host 'application'
* Closing connection #0
* About to connect() to localhost port 5000 (#0)
* Trying 127.0.0.1... connected
> POST /api/echo?upload=raw_data.log HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:5000
> Accept: */*
> Content-Length: 862
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------b595d197abf0
>
< HTTP/1.1 100 Continue
} [data not shown]
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 364
< Date: Fri, 02 Aug 2013 19:58:18 GMT
< Connection: keep-alive
<
{ [data not shown]
* Connection #0 to host localhost left intact
* Closing connection #0
[
"hahaha",
{
"params": {
"upload": "raw_data.log"
},
"headers": {
"user-agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
"host": "localhost:5000",
"accept": "*/*",
"content-length": "862",
"expect": "100-continue",
"content-type": "multipart/form-data; boundary=----------------------------b595d197abf0"
},
"method": "POST"
}
]
* About to connect() to localhost port 5000 (#0)
* Trying 127.0.0.1... connected
> POST /api/echo?upload HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:5000
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 217
< Date: Fri, 02 Aug 2013 18:56:47 GMT
< Connection: keep-alive
<
{ [data not shown]
* Connection #0 to host localhost left intact
* Closing connection #0
[
"hahaha",
{
"params": {
"upload": ""
},
"headers": {
"user-agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
"host": "localhost:5000",
"accept": "*/*"
},
"method": "POST"
}
]
#!/bin/bash
API="http://localhost:8085/upload/foo?raw=raw_data.log"
profile_upload="-T profile.json"
profile_send="--data-binary @profile.json"
data_upload="-T raw_data.log"
profile_form_data="-F [email protected];type=application/json"
raw_form_data="-F raw_data=@raw_data.log;type=plain/text"
data_send="--data-binary raw=@raw_data.log"
extra_urlencoded_post="-F hello=world -F profile=profile.json -F raw=raw"
# curl follow RFC 2388 with --form which conflicts with sending raw bytes
pure_json="$profile_send"
submit_multipart="$profile_form_data $raw_form_data $extra_urlencoded_post"
METHOD="-X POST"
HEADERS=""
#HEADERS='Content-Type: application/json'
UPLOAD_OPTS="$submit_multipart"
# UPLOAD_OPTS="$profile_upload -F \"data=@raw_data.log;type=plain/text\""
#UPLOAD_OPTS="-F \"data=@raw_data.log;type=plain/text\" -F \"[email protected];type=application/json\""
# -T raw_data.log
# -T profile.json
# -F "data=@raw_data.log;type=plain/text"
# -F "[email protected];type=application/json"
viewer=$(which json || echo "cat")
(
set -x
curl -v -s $METHOD -H "${HEADERS}" $UPLOAD_OPTS $API | $viewer
) 2>&1 | tee out.log
#####
# EOF

create and manipulate curl http streams

edit

[email protected]:6142358.git

# https://gist.github.com/6142358.git

git clone $gist curl_uploads

#####
# EOF

recommended

Using -F causes curl to submit an urlencoded POST body.

curl -X POST   -F "[email protected];type=application/json" -F "data=@work/README;type=plain/text"   localhost:5000/api/echo  | json
[
  "hahaha",
  {
    "params": {},
    "headers": {
      "user-agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
      "host": "localhost:5000",
      "accept": "*/*",
      "content-length": "3327",
      "expect": "100-continue",
      "content-type": "multipart/form-data; boundary=----------------------------8b62ec9fdaca"
    },
    "method": "POST"
  }
]

wc profile.json raw_data.log

wc profile.json raw_data.log
  9  20 274 raw_data.log
 14  20 241 profile.json
 23  40 515 total

wc profile.json raw_data.log

  30   60  825 raw_data.log
  14   20  243 profile.json
  44   80 1068 total
{
"device-metadata":
{ "type": "SomeCGM",
"serial": "233XDH4",
"firmware": "1.7.3"
},
"date-retrieved": "2013-08-01T10:33:03.235145",
"md5": "553bd35a564c5952963ac1a76c1fb27c",
"rawdata": {
"refs": ["data.log"]
}
}
+ /usr/local/bin/json
+ curl -v -s -X POST -H 'Content-Type: application/json' --data-binary @profile.json 'http://localhost:5000/api/echo?upload=raw_data.log'
* About to connect() to localhost port 5000 (#0)
* Trying 127.0.0.1... connected
> POST /api/echo?upload=raw_data.log HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:5000
> Accept: */*
> Content-Type: application/json
> Content-Length: 243
>
} [data not shown]
* upload completely sent off: 243out of 243 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 480
< Date: Fri, 02 Aug 2013 20:22:58 GMT
< Connection: keep-alive
<
{ [data not shown]
* Connection #0 to host localhost left intact
* Closing connection #0
[
"hahaha",
{
"params": {
"upload": "raw_data.log",
"device-metadata": {
"type": "SomeCGM",
"serial": "233XDH4",
"firmware": "1.7.3"
},
"date-retrieved": "2013-08-01T10:33:03.235145",
"md5": "553bd35a564c5952963ac1a76c1fb27c",
"rawdata": {
"refs": [
"data.log"
]
}
},
"headers": {
"user-agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
"host": "localhost:5000",
"accept": "*/*",
"content-type": "application/json",
"content-length": "243"
},
"method": "POST"
}
]
2011-03-09T08:38:01 242
2011-03-08T21:34:27 126
2011-03-08T19:30:37 94
2011-03-08T18:29:12 91
2011-03-08T16:55:07 100
2011-03-08T16:21:07 84
2011-03-08T16:20:21 57
2011-03-08T14:43:02 89
2011-03-08T12:22:34 100
2011-03-08T11:28:39 143
2011-03-09T08:38:01 242
2011-03-08T21:34:27 126
2011-03-08T19:30:37 94
2011-03-08T18:29:12 91
2011-03-08T16:55:07 100
2011-03-08T16:21:07 84
2011-03-08T16:20:21 57
2011-03-08T14:43:02 89
2011-03-08T12:22:34 100
2011-03-08T11:28:39 143
2011-03-09T08:38:01 242
2011-03-08T21:34:27 126
2011-03-08T19:30:37 94
2011-03-08T18:29:12 91
2011-03-08T16:55:07 100
2011-03-08T16:21:07 84
2011-03-08T16:20:21 57
2011-03-08T14:43:02 89
2011-03-08T12:22:34 100
2011-03-08T11:28:39 143
+ /usr/local/bin/json
+ curl -v -s -X POST -H '"content-type:' 'application/json"' --data-binary @profile.json 'http://localhost:5000/api/echo?upload=raw_data.log'
* getaddrinfo(3) failed for application:80
* Couldn't resolve host 'application'
* Closing connection #0
* About to connect() to localhost port 5000 (#0)
* Trying 127.0.0.1... connected
> POST /api/echo?upload=raw_data.log HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:5000
> Accept: */*
> Content-Length: 241
> Content-Type: application/x-www-form-urlencoded
>
} [data not shown]
* upload completely sent off: 241out of 241 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 592
< Date: Fri, 02 Aug 2013 19:41:00 GMT
< Connection: keep-alive
<
{ [data not shown]
* Connection #0 to host localhost left intact
* Closing connection #0
[
"hahaha",
{
"params": {
"upload": "raw_data.log",
"{\n \"device-metadata\":\n { \"type\": \"SomeCGM\",\n \"serial\": \"233XDH4\",\n \"firmware\": \"1.7.3\"\n },\n \"date-retrieved\": \"2013-08-01T10:33:03.235145\",\n \"md5\": \"553bd35a564c5952963ac1a76c1fb27c\",\n \"rawdata\": {\n refs: ": {
"\"data.log\"]\n }\n}\n\n": ""
}
},
"headers": {
"user-agent": "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3",
"host": "localhost:5000",
"accept": "*/*",
"content-length": "241",
"content-type": "application/x-www-form-urlencoded"
},
"method": "POST"
}
]
@bewest
Copy link
Author

bewest commented Aug 5, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment