nextgenbears.txt

from multiprocessing import Pool

if __name__ == "__main__":
    # Initialize pool workers already
    #
    # Instantiate bears (params: the section and list of files that contains proxy objects serving the file as a string, presplitted lines and the file name, maybe other parameters)
    #
    # The file proxy objects contains lazy functions! So memory is only reserved on need
    # for specific stuff. For that purposes maybe one can create a 
    # "MultiRepresentationObject" class that needs one specific "representation" of an
    # object and can generate on demand other stuff from it.
    #
    # Collect bear tasks (using bear.get_tasks())
    # - A task is a callable with following parameters:
    #   - A queue for logging and retrieving results. The coala main thread checks
    #     instances to distinct them. So we need also a suitable Log class. There's only
    #     a single queue for all tasks.
    #   - An ID. Used to determine which thread actually exited/processed something.
    #   - !!! Shall the queue call close()? I think yes!!!
    #   - We need a "Done" signal (maybe it contains meta information like
    #     the callable processed). Can't use the pool to determine open tasks.
    # - The bear generates them using functools.partial.
    #
    # Execute them async inside the pool. -> Maybe no queue for results? Maybe one results
    # can put inside the AsyncResult. The problem is result blocks until function is done.
    # So one could deliver exceptions maybe. Yeah I think this is good. What's to be done:
    # - Create a BearException
    #   - The BearException contains the actual exception raised from the analysis body
    #   - BearException's are passed to the logger and it can recognize them as such
    #   - Other exceptions lead to a fatal error. Program resumes but prints a serious
    #     message (inform coala devs etc etc).
    # AsyncResult delivers the exception. To check for that, the Done signal is used that
    # contains the callable if possible or bear instance identifying information. This is
    # used to get the according AsyncResult and fetch the exception from it.
    # UPDATE: Use the ID passed to the called function.
    # Why using AsyncResult for this explicitly? Python should handle exception catching
    # better than me.
    #
    # Run a log message and result dispatcher.
    #
    # Finish when done
    #
    # Additional (after this implemented): Timeouts. General and bear-specific.
    #
    # This would also simplify KeyboardInterrupt's. This makes them quite bear specific,
    # but I think this is okay.