librados3

librados3

in memory of closed/reverted PRs

- pass "bufferlist&&" to setxattr(): https://github.com/ceph/ceph/pull/12206
- bump up the so version of librados: https://github.com/ceph/ceph/pull/11843
- unify the list::const_iterator and list::iterator https://github.com/ceph/ceph/pull/6215
- pass size_t and ssize_t to advance() and seek() https://github.com/ceph/ceph/pull/11993

- kill ack/commit distinction and replace with a single completion callback/state.


jcsp suggestions:
     - remove TMAP calls
     - remove rados_nobjects_list* calls (deprecated by rados_object_list_begin etc)

Adam:
    - Get rid of all the unnecessary locking!
    - Divorce IoCtx from pools
    - Divorce namespace and locator from IoCtx. Either have expanded versions of the calls or have
      calls take objects which can be implicitly constructed from strings but can be expliitly built
      to have namespaces, locators, etc.
    - Don't be as /thick/ as it is.
    - Alocate a lot less. I'm not fond of the pImpl idiom, but we could keep it and still avoid
      allocating callbacks on heap.
    - Ditch the AioCompletion model and either use a futures based interface or borrow something
      from Asio.

dillaman:
    - zero-copy read APIs where the messenger reads into the provided buffer
    - zero-copy write API for C-style strings
    - don't expose buffer::list to the public API :-p
    - eliminate duplicitous AIO interfaces (e.g. aio_remove vs aio_operate(w/ remove op))
    - expose internal dispatch threading model so that librbd can re-use for its AIO operations
    - eliminate single-threaded AIO finisher thread -- combine into new core threading model

Jesse:
    Depending on how far we want to take this...                                                                                                      
                                                                                                                                        
- in addition to unifying the iterators, make them compliant with the                                                                   
  standard iterators, and eliminate all behaviors like advance(), seek(),                                                               
  etc. and especially anything tying or exposing the iterator to its                                                                    
  buffer or allowing the internal state or representation of the buffer to                                                              
  be accessed from an iterator. This will certainly break quite a bit of                                                                
  code, but also has benefits including the ability to change the backing                                                               
  of buffer::list (ie. deque or vector).   
  
- Can the special cursors (ie. ObjectCursor) be regular iterators?                                                                                             
                                                                                                                                        
- work with standard library types, at least on the library interface's                                                                 
surface                                                                                                                                 
        - allow view types to be consumed                                                                                               
        - generally avoid copying 
        - less new stuff for users to understand                                                                                                      
                                                                                                                                        
- generally, greatly reduce coupling:                                                                                                   
        - separate configuration from initialization from connection, etc.                                                              
        - high-level convenience functions can still be provided, but now they are 
        tiered and   flexible                                                                                                                                                                                                                                                                                                                                                                                                                                                       
        - allow view types to be consumed                                                                                               
        - generally avoid copying          
        - would potentially also allow users to use their own Allocators                      
        
- review places where we use concrete types and consider if it's worthwhile to make them parametric
   - for example, where we call for a map<string, bufferlist>, what if a user wants to use an unordered_map? (Might be
   fine, just putting the consideration forth. Might be cumbersome to implement w/ regular SFINAE and enable_if<>)                                                                       
                                                                                                                                        
- generally, greatly reduce coupling:                                                                                                   
        - separate configuration from initialization from connection, etc.                                                              
        - high-level convenience functions can still be provided, but now                                                               
        they are tiered and flexible            
        - this also allows us to work on other, higher-level library designs-- something very difficult right now because
        of the coupling                                                                                        
                                                                                                                                        
- leverage the type system to guide call orders and improve safety                                                                      
        - for example, in separating configuration and making a ceph                                                                    
        handle, it's possible to ensure that if you have a handle you                                                                   
        must have a valid configuration. (ie. disable ctors and use
        friends so you can only create one from the other in the 
        correct order-- config failure throws, you never wind up in a bad state).                                                                       
                                                                                                                                        
- I feel like librados does too much in one place, that smaller                                                                     
libraries feel like they live in here.                                                                                                  
        - prefer overloading and explicit types to runtime switches
        - also for things like pool_* vs. pool_*_async, etc.
                                                                                                         
- clearly mark deprecated features                                                                                                      
        - with comments if we must                                                                                                      
        - C++14 adds [[deprecated]], it would be great to be able to                                                                    
        do this explicitly; (We're using extensions now.)                                                                                                            
        - is this an opportunity to target a current C++ standard? 
        - also take the opportunity to get rid of or simplify things added for ABI compatibility (f(), f2(), f3(), etc.)                                                                     
                                                                                                                                        
- generally, allow things to feel more like modern C++ (this is likely out of scope for what we're doing here, but I'll throw some
ideas in the mix-- if nothing else, maybe thoughts as new features are added):                                                                               
        - embrace exceptions /and/ result codes as appropriate     
        - review enumerations, make them strongly-typed unless there's reason not to do so   
        - review NULL and 0, move to nullptr_t (and see if we really need those bare pointers)                                                                  
        - end idioms like out-parameters (especially when by pointer)
        - suggest also avoiding default arguments (mostly-prefer overloads)                                                                   
        - leverage system_error, and the standard library unless entirely inappropriate (futures and promises were mentioned above,
          along with things that work like the networking TS)
        - variadic templates make a lot of function styles easier to write now
        
- wishlist:
        - making a librados client should not be embarassing in C++ vis-a-vis Python ;-)
          (I'm NOT proposing copying the Python API, just pointing out that we should make
          things simple for the user.)      
auto ceph = rados::make_ceph_handle(); // from environment, then file, etc. (as we do now)
auto ceph = rados::make_ceph_handle(config_map); // ...from an associative array of config data, via conversion to configuration_source
auto ceph = rados::make_ceph_handle(path("my_ceph.conf")); // since type is path, a file
auto ceph = rados::make_ceph_handle(rados::configuration_source(path("my_ceph.conf")); // very explicit, from filesystem
          ceph["key"] = value; // ...create or update, in kv mode, or some "no worse than this" syntax; for selecting PGs, etc. see if we can do it via types
          ...so on...
          For trickier things, we haven't lost the ability to go directly to lower level calls.