Created
March 9, 2010 14:07
-
-
Save lukmdo/326600 to your computer and use it in GitHub Desktop.
Gearman4Perl and in general
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Start gearman `server` on some machine (here `worker` machine): | |
$ gearmand --port 8888 | |
2. Connect a `worker` to the server | |
$ gearman -h IP -p 8888 -w -f test wc | |
$ gearman -h 127.0.0.1 -p 8888 -w -f test executable_script | |
3. Push a job to queue by calling: | |
$ gearman -h IP -p 8888 -f test "SOME DATA TO PROCESS OR ARGS" | |
Notes: | |
* Gearman does NOT spawn any `worker` processes, therefore if a number of `workers` | |
becomes exceeded the `client` has to wait to be served. | |
* Gearmand debug-info problem none of --debug=1 or -vv options of gearmand seems to work | |
* Gearmand server can go down and up without the need to re-attach workers to it | |
* If the `worker` crushes (or machine it runs on) there is the `client` that has to decide what | |
should be done | |
* If the `client` crushes then the way to manage that is: | |
- use a monitor service like launchd, or daemntools/supervise | |
http://cr.yp.to/daemontools/supervise.html | |
http://launchd.macosforge.org/ AND http://developer.apple.com/macosx/launchd.html | |
- run those `clients` periodically repeatable with check if completed in previous run | |
* `client` can't pass complex data structures to the `worker`. | |
Passed data needs to be scalar or eventually a scalar reference. In case application | |
forces passing some input data structure the serialization will make it possible: | |
- Text format: | |
+ at `client` JSON::XS::encode_json and than on `worker` JSON::XS::decode_json | |
- Bin format: | |
+ at `client` Storable::nfreeze (for portable image) and than Storable::thaw on `worker` | |
and then it can be passed as a scalar. Alternatively consider the possibility of getting the input | |
data by the `worker` by itself. | |
Additional info: | |
* http://assets.en.oreilly.com/1/event/27/Gearman_%20Bringing%20the%20Power%20of%20Map_Reduce%20to%20Everyday%20Applications%20Presentation.pdf | |
* http://www.slideshare.net/andy.sh/gearman-and-perl | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment