Skip to content

Instantly share code, notes, and snippets.

@chianingwang

chianingwang/ServerSideCopySLOSegements

Created September 6, 2017 16:58
Show Gist options
  • Save chianingwang/5596cae6b21c6d57ce96e0a0d295e49d to your computer and use it in GitHub Desktop.
Save chianingwang/5596cae6b21c6d57ce96e0a0d295e49d to your computer and use it in GitHub Desktop.
Download ZIP
Raw
ServerSideCopySLOSegements
# demo outline - bbb_sunflower_1080_2min.mp4
# 1. DLO upload to container demo to get Segments
# 2. Server Side Copy Segments
# 3. upload SLO json for copied Segments
# leverage DLO to get
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx upload demo -S 10485760 bbb_sunflower_1080_2min.mp4
bbb_sunflower_1080_2min.mp4 segment 0
bbb_sunflower_1080_2min.mp4 segment 2
bbb_sunflower_1080_2min.mp4 segment 1
bbb_sunflower_1080_2min.mp4
# double check DLO content
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx stat -v demo bbb_sunflower_1080_2min.mp4
URL: https://cloud.swiftstack.com/v1/AUTH_elk/demo/bbb_sunflower_1080_2min.mp4
Auth Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
Account: AUTH_elk
Container: demo
Object: bbb_sunflower_1080_2min.mp4
Content Type: video/mp4
Content Length: 27883167
Last Modified: Wed, 06 Sep 2017 16:09:29 GMT
ETag: "215729ecb16ddc52e240f10d04b7751d"
Manifest: demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/
Meta Mtime: 1501534505.000000
Accept-Ranges: bytes
X-Timestamp: 1504714168.96232
X-Trans-Id: tx7ddee95ece084955b1fb6-0059b01deb
X-Openstack-Request-Id: tx7ddee95ece084955b1fb6-0059b01deb
# double check DLO Segments Content
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx list demo_segments
bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000000
bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000001
bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000002
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx stat -v demo_segments bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000000
URL: https://cloud.swiftstack.com/v1/AUTH_elk/demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000000
Auth Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
Account: AUTH_elk
Container: demo_segments
Object: bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000000
Content Type: application/swiftclient-segment
Content Length: 10485760
Last Modified: Wed, 06 Sep 2017 16:09:24 GMT
ETag: 07b81cd438af5097d584321ab99cdc06
Accept-Ranges: bytes
X-Timestamp: 1504714163.48142
X-Trans-Id: txa3623343e7cc45e48c2d2-0059b02598
X-Openstack-Request-Id: txa3623343e7cc45e48c2d2-0059b02598
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx stat -v demo_segments bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000001
URL: https://cloud.swiftstack.com/v1/AUTH_elk/demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000001
Auth Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
Account: AUTH_elk
Container: demo_segments
Object: bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000001
Content Type: application/swiftclient-segment
Content Length: 10485760
Last Modified: Wed, 06 Sep 2017 16:09:26 GMT
ETag: ff53521cfdfe801ab1e52d0e7fda4969
Accept-Ranges: bytes
X-Timestamp: 1504714165.96129
X-Trans-Id: tx111d5b9e3a4a4c9786c7c-0059b01fb6
X-Openstack-Request-Id: tx111d5b9e3a4a4c9786c7c-0059b01fb6
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx stat -v demo_segments bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000002
URL: https://cloud.swiftstack.com/v1/AUTH_elk/demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000002
Auth Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
Account: AUTH_elk
Container: demo_segments
Object: bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000002
Content Type: application/swiftclient-segment
Content Length: 6911647
Last Modified: Wed, 06 Sep 2017 16:09:24 GMT
ETag: e924eaf1e2171ff355c588fa84da5778
Accept-Ranges: bytes
X-Timestamp: 1504714163.59075
X-Trans-Id: tx576b1d9dec31404b8d606-0059b01fb8
X-Openstack-Request-Id: tx576b1d9dec31404b8d606-0059b01fb8
# create container copy for SLO, container copy_segments for SLO segements
swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx post copy
swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx post copy_segments
# server side copy from demo_segments to copy_segment and rename segment objects
$ curl -i -X PUT https://cloud.swiftstack.com/v1/AUTH_elk/copy_segments/bbb_sunflower_1080_2min.mp4.segement0 -H 'X-Auth-Token:AUTH_tkcc600af9a0b142729afb0e3232a1986f' -H 'X-Copy-From:/demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000000' -H 'Content-Length: 0'
HTTP/1.1 201 Created
Last-Modified: Wed, 06 Sep 2017 16:22:03 GMT
Content-Length: 0
Etag: 07b81cd438af5097d584321ab99cdc06
Content-Type: text/html; charset=UTF-8
X-Copied-From-Account: AUTH_elk
X-Copied-From-Last-Modified: Wed, 06 Sep 2017 16:09:24 GMT
X-Copied-From: demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000000
X-Trans-Id: tx831e50a3d9ef495991100-0059b020aa
X-Openstack-Request-Id: tx831e50a3d9ef495991100-0059b020aa
Date: Wed, 06 Sep 2017 16:22:04 GMT
$ curl -i -X PUT https://cloud.swiftstack.com/v1/AUTH_elk/copy_segments/bbb_sunflower_1080_2min.mp4.segement1 -H 'X-Auth-Token:AUTH_tkcc600af9a0b142729afb0e3232a1986f' -H 'X-Copy-From:/demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000001' -H 'Content-Length: 0'
HTTP/1.1 201 Created
Last-Modified: Wed, 06 Sep 2017 16:22:36 GMT
Content-Length: 0
Etag: ff53521cfdfe801ab1e52d0e7fda4969
Content-Type: text/html; charset=UTF-8
X-Copied-From-Account: AUTH_elk
X-Copied-From-Last-Modified: Wed, 06 Sep 2017 16:09:26 GMT
X-Copied-From: demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000001
X-Trans-Id: tx5e2f287695444f2d97c7d-0059b020cb
X-Openstack-Request-Id: tx5e2f287695444f2d97c7d-0059b020cb
Date: Wed, 06 Sep 2017 16:22:37 GMT
$ curl -i -X PUT https://cloud.swiftstack.com/v1/AUTH_elk/copy_segments/bbb_sunflower_1080_2min.mp4.segement2 -H 'X-Auth-Token:AUTH_tkcc600af9a0b142729afb0e3232a1986f' -H 'X-Copy-From:/demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000002' -H 'Content-Length: 0'
HTTP/1.1 201 Created
Last-Modified: Wed, 06 Sep 2017 16:23:01 GMT
Content-Length: 0
Etag: e924eaf1e2171ff355c588fa84da5778
Content-Type: text/html; charset=UTF-8
X-Copied-From-Account: AUTH_elk
X-Copied-From-Last-Modified: Wed, 06 Sep 2017 16:09:24 GMT
X-Copied-From: demo_segments/bbb_sunflower_1080_2min.mp4/1501534505.000000/27883167/10485760/00000002
X-Trans-Id: tx92c4c019ee9a4506ba823-0059b020e3
X-Openstack-Request-Id: tx92c4c019ee9a4506ba823-0059b020e3
Date: Wed, 06 Sep 2017 16:23:00 GMT
# double check SLO segmetns
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx stat -v copy_segments bbb_sunflower_1080_2min.mp4.segement0
URL: https://cloud.swiftstack.com/v1/AUTH_elk/copy_segments/bbb_sunflower_1080_2min.mp4.segement0
Auth Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
Account: AUTH_elk
Container: copy_segments
Object: bbb_sunflower_1080_2min.mp4.segement0
Content Type: application/swiftclient-segment
Content Length: 10485760
Last Modified: Wed, 06 Sep 2017 16:22:03 GMT
ETag: 07b81cd438af5097d584321ab99cdc06
Accept-Ranges: bytes
X-Timestamp: 1504714922.81131
X-Trans-Id: tx5a69c1729ea14fd88a98f-0059b022aa
X-Openstack-Request-Id: tx5a69c1729ea14fd88a98f-0059b022aa
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx stat -v copy_segments bbb_sunflower_1080_2min.mp4.segement1
URL: https://cloud.swiftstack.com/v1/AUTH_elk/copy_segments/bbb_sunflower_1080_2min.mp4.segement1
Auth Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
Account: AUTH_elk
Container: copy_segments
Object: bbb_sunflower_1080_2min.mp4.segement1
Content Type: application/swiftclient-segment
Content Length: 10485760
Last Modified: Wed, 06 Sep 2017 16:22:36 GMT
ETag: ff53521cfdfe801ab1e52d0e7fda4969
Accept-Ranges: bytes
X-Timestamp: 1504714955.96129
X-Trans-Id: tx9859b71af4f5454580529-0059b022b1
X-Openstack-Request-Id: tx9859b71af4f5454580529-0059b022b1
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx stat -v copy_segments bbb_sunflower_1080_2min.mp4.segement2
URL: https://cloud.swiftstack.com/v1/AUTH_elk/copy_segments/bbb_sunflower_1080_2min.mp4.segement2
Auth Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
Account: AUTH_elk
Container: copy_segments
Object: bbb_sunflower_1080_2min.mp4.segement2
Content Type: application/swiftclient-segment
Content Length: 6911647
Last Modified: Wed, 06 Sep 2017 16:23:01 GMT
ETag: e924eaf1e2171ff355c588fa84da5778
Accept-Ranges: bytes
X-Timestamp: 1504714980.09511
X-Trans-Id: tx6fd3087fd78b42738165f-0059b022b3
X-Openstack-Request-Id: tx6fd3087fd78b42738165f-0059b022b3
# prepare SLO manifest json
$ cat slo.json
[
{
"path": "/copy_segments/bbb_sunflower_1080_2min.mp4.segement0",
"etag": "07b81cd438af5097d584321ab99cdc06",
"size_bytes": 10485760
},
{
"path": "/copy_segments/bbb_sunflower_1080_2min.mp4.segement1",
"etag": "ff53521cfdfe801ab1e52d0e7fda4969",
"size_bytes": 10485760
},
{
"path": "/copy_segments/bbb_sunflower_1080_2min.mp4.segement2",
"etag": "e924eaf1e2171ff355c588fa84da5778",
"size_bytes": 6911647
}
]
# upload SLO manifest
$ curl -v -X PUT -H "Content-Type: video/mp4" -H 'X-Auth-Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f' -H -H 'Cache-Control: no-cache' -d '[{"path":"/copy_segments/bbb_sunflower_1080_2min.mp4.segement0","etag":"07b81cd438af5097d584321ab99cdc06","size_bytes":10485760},{"path":"/copy_segments/bbb_sunflower_1080_2min.mp4.segement1","etag":"ff53521cfdfe801ab1e52d0e7fda4969","size_bytes":10485760},{"path":"/copy_segments/bbb_sunflower_1080_2min.mp4.segement2","etag":"e924eaf1e2171ff355c588fa84da5778","size_bytes":6911647}]' https://cloud.swiftstack.com/v1/AUTH_elk/copy/bbb_sunflower_1080_2min.mp4?multipart-manifest=put
* Rebuilt URL to: Cache-Control: no-cache/
* Port number ended with ' '
* Closing connection -1
curl: (3) Port number ended with ' '
* Trying 192.168.201.82...
* TCP_NODELAY set
* Connected to cloud.swiftstack.com (192.168.201.82) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: *.swiftstack.com
* Server certificate: DigiCert SHA2 Secure Server CA
* Server certificate: DigiCert Global Root CA
> PUT /v1/AUTH_elk/copy/bbb_sunflower_1080_2min.mp4?multipart-manifest=put HTTP/1.1
> Host: cloud.swiftstack.com
> User-Agent: curl/7.54.0
> Accept: */*
> Content-Type: video/mp4
> X-Auth-Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
> Content-Length: 384
>
* upload completely sent off: 384 out of 384 bytes
< HTTP/1.1 201 Created
< Last-Modified: Wed, 06 Sep 2017 16:34:01 GMT
< Content-Length: 0
< Etag: "215729ecb16ddc52e240f10d04b7751d"
< Content-Type: text/html; charset=UTF-8
< X-Trans-Id: txe5aac0c95da444f4919d9-0059b02378
< X-Openstack-Request-Id: txe5aac0c95da444f4919d9-0059b02378
< Date: Wed, 06 Sep 2017 16:34:00 GMT
<
* Connection #0 to host cloud.swiftstack.com left intact
# double check SLO
$ curl -i https://cloud.swiftstack.com/v1/AUTH_elk/copy/bbb_sunflower_1080_2min.mp4?multipart-manifest=get -H "X-Auth-Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f"
HTTP/1.1 200 OK
Content-Length: 680
Accept-Ranges: bytes
Last-Modified: Wed, 06 Sep 2017 16:34:01 GMT
Etag: 80ba317af9f92493cbbffb0c9a7bd6fb
X-Timestamp: 1504715640.31061
X-Static-Large-Object: True
Content-Type: application/json; charset=utf-8
X-Trans-Id: tx7788b3ac1dc8415c8c4a6-0059b02426
X-Openstack-Request-Id: tx7788b3ac1dc8415c8c4a6-0059b02426
Date: Wed, 06 Sep 2017 16:36:54 GMT
[{"hash": "07b81cd438af5097d584321ab99cdc06", "last_modified": "2017-09-06T16:22:03.000000", "bytes": 10485760, "name": "/copy_segments/bbb_sunflower_1080_2min.mp4.segement0", "content_type": "application/swiftclient-segment"}, {"hash": "ff53521cfdfe801ab1e52d0e7fda4969", "last_modified": "2017-09-06T16:22:36.000000", "bytes": 10485760, "name": "/copy_segments/bbb_sunflower_1080_2min.mp4.segement1", "content_type": "application/swiftclient-segment"}, {"hash": "e924eaf1e2171ff355c588fa84da5778", "last_modified": "2017-09-06T16:23:01.000000", "bytes": 6911647, "name": "/copy_segments/bbb_sunflower_1080_2min.mp4.segement2", "content_type": "application/swiftclient-segment"}]
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx stat -v copy bbb_sunflower_1080_2min.mp4
URL: https://cloud.swiftstack.com/v1/AUTH_elk/copy/bbb_sunflower_1080_2min.mp4
Auth Token: AUTH_tkcc600af9a0b142729afb0e3232a1986f
Account: AUTH_elk
Container: copy
Object: bbb_sunflower_1080_2min.mp4
Content Type: video/mp4
Content Length: 27883167
Last Modified: Wed, 06 Sep 2017 16:34:01 GMT
ETag: "215729ecb16ddc52e240f10d04b7751d"
Accept-Ranges: bytes
X-Timestamp: 1504715640.31061
X-Trans-Id: tx47005dbf2e0d4ababa3d6-0059b023b0
X-Static-Large-Object: True
X-Openstack-Request-Id: tx47005dbf2e0d4ababa3d6-0059b023b0
# test download new SLO object
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U elk -K elk download copy bbb_sunflower_1080_2min.mp4
bbb_sunflower_1080_2min.mp4 [auth 0.240s, headers 0.503s, total 3.926s, 7.564 MB/s]
# dobule check md5 hash for new SLO and original upload via DLO
$ md5 bbb_sunflower_1080_2min.mp4
MD5 (bbb_sunflower_1080_2min.mp4) = ee54e072da4a66478a187ba834a0d56a
$ md5 ./demo/bbb_sunflower_1080_2min.mp4
MD5 (./demo/bbb_sunflower_1080_2min.mp4) = ee54e072da4a66478a187ba834a0d56a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment