Last active
March 2, 2017 06:16
-
-
Save chianingwang/d0285667e69b9dc853043ebdd3f60639 to your computer and use it in GitHub Desktop.
SBU Windows
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| On Windows many types from multiprocessing need to be picklable example | |
| You can work around this by using an absolute import for OtherCustomClass: | |
| from base_module import OtherCustomClass | |
| I'm not exactly sure why, but it seems that when multiprocessing spawns a new process and imports your __main__, it's not able to handle the implicit relative import you're using with OtherCustomClass. If you explicitly import it from base_module, it works fine. My guess is that the spawned child process is not recognized as being part of the base_module package, so the implicit import fails, but that's just a guess. | |
| Note that you shouldn't be using implicit relative imports anyway (they're altogether removed from Python 3), so switching to an absolute import isn't a bad thing. | |
| Also of note, that doing an explicit relative import works on Python 3.4: | |
| from . import OtherCustomClass | |
| 2017-02-21 22:42:43,176 - bulk-Uploader - WARNING - Create remote container: test_bulk_a | |
| 2017-02-21 22:42:44,398 - bulk-Uploader - INFO - Created Container(post_container) test_bulk_a | |
| 2017-02-21 22:42:44,398 - bulk-Uploader - WARNING - Create remote container: test_bulk_b | |
| 2017-02-21 22:42:44,466 - bulk-Uploader - INFO - Created Container(post_container) test_bulk_b | |
| 2017-02-21 22:42:44,466 - bulk-Uploader - DEBUG - Folder C:\Users\Johnny\Documents\a - file patterns: ['*'] | |
| 2017-02-21 22:42:44,467 - bulk-Uploader - DEBUG - Folder C:\Users\Johnny\Documents\b - file patterns: ['*'] | |
| 2017-02-21 22:42:44,467 - bulk-checker - DEBUG - Checking config file | |
| 2017-02-21 22:42:44,469 - bulk-checker - DEBUG - Folder C:\Users\Johnny\Documents\a - file patterns: ['*'] | |
| 2017-02-21 22:42:44,476 - bulk-checker - DEBUG - Folder C:\Users\Johnny\Documents\b - file patterns: ['*'] | |
| 2017-02-21 22:42:44,575 - bulk-checker - ERROR - Default segemt_size is too big: 10485760 | |
| 2017-02-27 23:59:13,049 - bulk-Uploader - WARNING - Create remote container: test_bulk_a | |
| 2017-02-27 23:59:13,920 - bulk-Uploader - INFO - Created Container(post_container) test_bulk_a | |
| 2017-02-27 23:59:13,920 - bulk-Uploader - WARNING - Create remote container: test_bulk_b | |
| 2017-02-27 23:59:13,989 - bulk-Uploader - INFO - Created Container(post_container) test_bulk_b | |
| 2017-02-27 23:59:13,989 - bulk-Uploader - DEBUG - Folder C:\Users\Johnny\Documents\a - file patterns: ['*'] | |
| 2017-02-27 23:59:13,989 - bulk-Uploader - DEBUG - Folder C:\Users\Johnny\Documents\b - file patterns: ['*'] | |
| 2017-02-27 23:59:13,989 - bulk-checker - DEBUG - Checking config file | |
| 2017-02-27 23:59:13,989 - bulk-checker - DEBUG - Folder C:\Users\Johnny\Documents\a - file patterns: ['*'] | |
| 2017-02-27 23:59:13,989 - bulk-checker - DEBUG - Folder C:\Users\Johnny\Documents\b - file patterns: ['*'] | |
| 2017-02-27 23:59:14,088 - bulk-checker - ERROR - Default segemt_size is too big: 10485760 | |
| C:\Users\Johnny\Documents\GitHub\swift-bulk-uploader>pip install -e ./ | |
| Obtaining file:///C:/Users/Johnny/Documents/GitHub/swift-bulk-uploader | |
| Collecting watchdog (from swift-bulk-uploader==0.0.2) | |
| Collecting python-swiftclient (from swift-bulk-uploader==0.0.2) | |
| Using cached python_swiftclient-3.3.0-py2.py3-none-any.whl | |
| Collecting PyYAML>=3.10 (from watchdog->swift-bulk-uploader==0.0.2) | |
| Downloading PyYAML-3.12-cp27-cp27m-win32.whl (182kB) | |
| 100% |################################| 184kB 860kB/s | |
| Collecting pathtools>=0.1.1 (from watchdog->swift-bulk-uploader==0.0.2) | |
| Collecting argh>=0.24.1 (from watchdog->swift-bulk-uploader==0.0.2) | |
| Using cached argh-0.26.2-py2.py3-none-any.whl | |
| Requirement already satisfied: six>=1.5.2 in c:\python27\lib\site-packages (from python-swiftclient->swift-bulk-uploader==0.0.2) | |
| Collecting futures>=3.0; python_version == "2.7" or python_version == "2.6" (from python-swiftclient->swift-bulk-uploader==0.0.2) | |
| Downloading futures-3.0.5-py2-none-any.whl | |
| Requirement already satisfied: requests>=1.1 in c:\python27\lib\site-packages (from python-swiftclient->swift-bulk-uploader==0.0.2) | |
| Installing collected packages: PyYAML, pathtools, argh, watchdog, futures, python-swiftclient, swift-bulk-uploader | |
| Running setup.py develop for swift-bulk-uploader | |
| Successfully installed PyYAML-3.12 argh-0.26.2 futures-3.0.5 pathtools-0.1.2 python-swiftclient-3.3.0 swift-bulk-uploader watchdog-0.8.3 | |
| C:\Users\Johnny\Documents\GitHub\swift-bulk-uploader>swift-bulk-uploader | |
| usage: swift-bulk-uploader [-h] [-l LOG_FILE] [--log-level LOG_LEVEL] | |
| [-c CONFIG] | |
| optional arguments: | |
| -h, --help show this help message and exit | |
| -l LOG_FILE, --log-file LOG_FILE | |
| Log file | |
| --log-level LOG_LEVEL | |
| Log level | |
| -c CONFIG, --config CONFIG | |
| config file | |
| C:\Users\Johnny\Documents\GitHub\swift-bulk-uploader>swift-bulk-uploader -c folder.conf -l log.txt --log-level DEBUG | |
| Traceback (most recent call last): | |
| File "C:\Python27\Scripts\swift-bulk-uploader-script.py", line 11, in <module> | |
| load_entry_point('swift-bulk-uploader', 'console_scripts', 'swift-bulk-uploader')() | |
| File "c:\users\johnny\documents\github\swift-bulk-uploader\bulk_uploader\bulk_uploader.py", line 128, in main | |
| p_uploader.start() | |
| File "c:\python27\lib\multiprocessing\process.py", line 130, in start | |
| self._popen = Popen(self) | |
| File "c:\python27\lib\multiprocessing\forking.py", line 277, in __init__ | |
| dump(process_obj, to_child, HIGHEST_PROTOCOL) | |
| File "c:\python27\lib\multiprocessing\forking.py", line 199, in dump | |
| ForkingPickler(file, protocol).dump(obj) | |
| File "c:\python27\lib\pickle.py", line 224, in dump | |
| self.save(obj) | |
| File "c:\python27\lib\pickle.py", line 331, in save | |
| self.save_reduce(obj=obj, *rv) | |
| File "c:\python27\lib\pickle.py", line 425, in save_reduce | |
| save(state) | |
| File "c:\python27\lib\pickle.py", line 286, in save | |
| f(self, obj) # Call unbound method with explicit self | |
| File "c:\python27\lib\pickle.py", line 655, in save_dict | |
| self._batch_setitems(obj.iteritems()) | |
| File "c:\python27\lib\pickle.py", line 687, in _batch_setitems | |
| save(v) | |
| File "c:\python27\lib\pickle.py", line 286, in save | |
| f(self, obj) # Call unbound method with explicit self | |
| File "c:\python27\lib\multiprocessing\forking.py", line 67, in dispatcher | |
| self.save_reduce(obj=obj, *rv) | |
| File "c:\python27\lib\pickle.py", line 401, in save_reduce | |
| save(args) | |
| File "c:\python27\lib\pickle.py", line 286, in save | |
| f(self, obj) # Call unbound method with explicit self | |
| File "c:\python27\lib\pickle.py", line 554, in save_tuple | |
| save(element) | |
| File "c:\python27\lib\pickle.py", line 331, in save | |
| self.save_reduce(obj=obj, *rv) | |
| File "c:\python27\lib\pickle.py", line 425, in save_reduce | |
| save(state) | |
| File "c:\python27\lib\pickle.py", line 286, in save | |
| f(self, obj) # Call unbound method with explicit self | |
| File "c:\python27\lib\pickle.py", line 655, in save_dict | |
| self._batch_setitems(obj.iteritems()) | |
| File "c:\python27\lib\pickle.py", line 687, in _batch_setitems | |
| save(v) | |
| File "c:\python27\lib\pickle.py", line 331, in save | |
| self.save_reduce(obj=obj, *rv) | |
| File "c:\python27\lib\pickle.py", line 425, in save_reduce | |
| save(state) | |
| File "c:\python27\lib\pickle.py", line 286, in save | |
| f(self, obj) # Call unbound method with explicit self | |
| File "c:\python27\lib\pickle.py", line 655, in save_dict | |
| self._batch_setitems(obj.iteritems()) | |
| File "c:\python27\lib\pickle.py", line 687, in _batch_setitems | |
| save(v) | |
| File "c:\python27\lib\pickle.py", line 286, in save | |
| f(self, obj) # Call unbound method with explicit self | |
| File "c:\python27\lib\pickle.py", line 606, in save_list | |
| self._batch_appends(iter(obj)) | |
| File "c:\python27\lib\pickle.py", line 642, in _batch_appends | |
| save(tmp[0]) | |
| File "c:\python27\lib\pickle.py", line 331, in save | |
| self.save_reduce(obj=obj, *rv) | |
| File "c:\python27\lib\pickle.py", line 425, in save_reduce | |
| save(state) | |
| File "c:\python27\lib\pickle.py", line 286, in save | |
| f(self, obj) # Call unbound method with explicit self | |
| File "c:\python27\lib\pickle.py", line 655, in save_dict | |
| self._batch_setitems(obj.iteritems()) | |
| File "c:\python27\lib\pickle.py", line 687, in _batch_setitems | |
| save(v) | |
| File "c:\python27\lib\pickle.py", line 331, in save | |
| self.save_reduce(obj=obj, *rv) | |
| File "c:\python27\lib\pickle.py", line 425, in save_reduce | |
| save(state) | |
| File "c:\python27\lib\pickle.py", line 286, in save | |
| f(self, obj) # Call unbound method with explicit self | |
| File "c:\python27\lib\pickle.py", line 655, in save_dict | |
| self._batch_setitems(obj.iteritems()) | |
| File "c:\python27\lib\pickle.py", line 687, in _batch_setitems | |
| save(v) | |
| File "c:\python27\lib\pickle.py", line 306, in save | |
| rv = reduce(self.proto) | |
| TypeError: can't pickle thread.lock objects | |
| C:\Users\Johnny\Documents\GitHub\swift-bulk-uploader>Traceback (most recent call last): | |
| File "<string>", line 1, in <module> | |
| File "c:\python27\lib\multiprocessing\forking.py", line 381, in main | |
| self = load(from_parent) | |
| File "c:\python27\lib\pickle.py", line 1384, in load | |
| return Unpickler(file).load() | |
| File "c:\python27\lib\pickle.py", line 864, in load | |
| dispatch[key](self) | |
| File "c:\python27\lib\pickle.py", line 886, in load_eof | |
| raise EOFError | |
| EOFError |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment