chianingwang

On Windows many types from multiprocessing need to be picklable example

You can work around this by using an absolute import for OtherCustomClass:

from base_module import OtherCustomClass
I'm not exactly sure why, but it seems that when multiprocessing spawns a new process and imports your __main__, it's not able to handle the implicit relative import you're using with OtherCustomClass. If you explicitly import it from base_module, it works fine. My guess is that the spawned child process is not recognized as being part of the base_module package, so the implicit import fails, but that's just a guess.

Note that you shouldn't be using implicit relative imports anyway (they're altogether removed from Python 3), so switching to an absolute import isn't a bad thing.

Also of note, that doing an explicit relative import works on Python 3.4:

chianingwang

[global]
user = dedup
auth = https://cloud.swiftstack.com/auth/v1.0
key = dedup
insecure = False

# default is SLO
use_slo = False

# default segment_size is 10MB

chianingwang

Johnnys-MacBook-Pro:resource jwang$ python deduper.py ./libclsPyVariableChunk.dll ./slo | python -mjson.tool
[
    {
        "etag": "f27c7cf9be58f29fb4eb6170095f3b29",
        "path": "/3b29/f27c7cf9be58f29fb4eb6170095f3b29",
        "size_bytes": 371985
    },
    {
        "etag": "1cacd9462d54ccafd783122c1af1cb25",
        "path": "/cb25/1cacd9462d54ccafd783122c1af1cb25",

chianingwang

{u'action': u'create_container', u'headers': {}, u'container': u'24d3', u'response_dict': {'status': 201, 'headers': {u'date': u'Thu, 09 Mar 2017 05:41:49 GMT', u'content-length': u'0', u'content-type': u'text/html; charset=UTF-8', u'x-openstack-request-id': u'tx9edd1d7ba9f74509bea73-0058c0eb1d', u'x-trans-id': u'tx9edd1d7ba9f74509bea73-0058c0eb1d'}, 'reason': 'Created', 'response_dicts': [{'status': 201, 'headers': {u'date': u'Thu, 09 Mar 2017 05:41:49 GMT', u'content-length': u'0', u'content-type': u'text/html; charset=UTF-8', u'x-openstack-request-id': u'tx9edd1d7ba9f74509bea73-0058c0eb1d', u'x-trans-id': u'tx9edd1d7ba9f74509bea73-0058c0eb1d'}, 'reason': 'Created'}]}, u'success': True}
{u'status': u'uploaded', u'large_object': False, u'container': u'24d3', u'success': True, u'object': u'fc558eba5928785296b0c77e17e224d3', u'headers': {u'x-object-meta-mtime': u'1489038038.000000'}, u'attempts': 1, u'action': u'upload_object', u'path': '/Users/jwang/Documents/tmp/resource/slo/fc558eba5928785296b0c77e17e224d3'

chianingwang

$ cat elk-server-dpkg.yml
---
#
# Deploying ELK stack server using .deb files
#

- name: ELK server deployment
  hosts: elk-log-replay
  remote_user: ss
  become: yes

chianingwang

# demo outline - bbb_sunflower_1080_2min.mp4
# 1. DLO upload to container demo to get Segments
# 2. Server Side Copy Segments
# 3. upload SLO json for copied Segments

# leverage DLO to get
$ swift -A https://cloud.swiftstack.com/auth/v1.0 -U xxx -K xxx upload demo -S 10485760 bbb_sunflower_1080_2min.mp4
bbb_sunflower_1080_2min.mp4 segment 0
bbb_sunflower_1080_2min.mp4 segment 2
bbb_sunflower_1080_2min.mp4 segment 1

chianingwang

package tcp

import (
	"io"
	"log"
	"net"
	"os"
)

// Progress indicates transfer status

chianingwang

code I modified ( not sure it's correct or not )

===s3_sync/provider_factory.py===

from .sync_s3 import SyncS3
from .sync_swift import SyncSwift

def create_provider(sync_settings, max_conns, per_account=False):
    provider_type = sync_settings.get('protocol', None)
    if not provider_type or provider_type == 's3':

chianingwang

sync_swift.py
==============

...

    def _get_client_factory(self):
        # TODO: support LDAP auth
        # TODO: support v2 auth
        username = self.settings['aws_identity']
        key = self.settings['aws_secret']

chianingwang

# Code Example - PySwiftClientKeyStoneAuth.py

import swiftclient.client as swiftclient

# reference: # https://docs.openstack.org/python-swiftclient/latest/client-api.html

# v3
#_authurl = 'http://172.28.128.43:35357/v3/'
#_auth_version = '3'
#_user = 'admin'