Design proposal for notary transaction api and other enablers for metering

CordaNotaryTransactionApi.png

meteringapiproposal.md

Design proposal for notary transaction api and other enablers for metering

TL;DR

This design doc contains the following:

• A proposal for new notary transaction api to extract notarised transactions in a standard format.
• A number of questions on how this api should be surfaced while maintaining security.
• Notarise method to support meta data to enable prioritisation of transactions.
• Notary and Node blacklisting / whitelisting to enable tight control over usage and connectivity for notaries and business networks.

Introduction

Metering is a method of incentivising organisations to run a notary by enabling them to charge for their use on a transaction by transaction basis.

A detailed explanation of why, what and how metering is implemented in cordite can be found here

Breaking down the metering process

In the current cordite metering implementation, the process is split into the following stages:

• Extraction of Transactions

  This is the process and extracting the history of transactions from the notary's commit log.

• Billing

  This is the process of sending an invoice to a Corda node that has used the notary for metering.

• Payment

  This is the process by which the Corda node pays the invoice for the metered transaction(s).

• Fee Distribution / Dispersal

  This is the process by which the fees paid for metering are distributed amongst the metering parties which would include the original notary and potentially other funds which are used for the maintenance of the network infrastructure and software.

• Disputes

  This is the process by which the parties involve in a metering transaction can dispute incorrect notarisation, billing or fee distribution.

Each of the above processes could be implemented in a number of different ways to suit the use cases for particular business networks or notary clusters. Therefore it is important that we are able to vary these processes independently.

There is an implementation of metering in Cordite that currently performs all of the above for single node validating or non-validating notaries. A future requirement for metering is to support notary clusters and the ability to support multiple business networks.

Currently Cordite metering directly accesses the notary commit log table : NODE_NOTARY_COMMIT_LOG Long term, this is not a sustainable approach as the underlying store and data structure for the notary transactions may vary for different notary implementations.

So, the first step is to make a consistent way of extracting transactions from the notary commit log with an Api.

The Notary Transaction Api

As discussed with the R3 notary team (Rick, Mark, Andrius) We came to consensus (sorry) on the following fields:

Api Input:

• lastTransactionTime

  This will extract all the notary transactions greater than equal to the input time which is specified in UTC time.

  Note: Instead of using transaction time, we did discuss using a sequence number but decided that this would be more difficult to implemented, particularly in a BFT scenario. However, we are open to this approach if achievable.

Api Outputs:

• transactionIdHash

  The hash of the transactionId is used here to prevent following of the transaction chain.

• transactionTime

  The UTC time of the transaction. This will be used by the consumers of the Api to track how far they have got - we would expect them to store the last transaction value in their own vaults or databases and then use this value to extract the next 'batch' of notarised transactions.

• abstractParty

  The party that instigated the transaction, so that we know where to send the metering invoice.

  There is no need to identify the party by name here, unless it is required by the 'rules' of a notary cluster.

• transactionSignature

  The instigating node will use this to verify that it did actually create the transaction in the first instance. The current metering process checks that the transactionId is in its vault which works fine, but does give away the transaction tree to the notary - hence using the hash of the transactionId above.

• Transaction metadata - this is Key/Value map of optional data that can be used by metering services to charge appropriately for the transaction such as :

  • numberOfStates

    This gives the billing 'engine' a rough size of the transaction and potentially it may charge more for large transactions

  • Transaction Priority

    We may introduce notaries that can process transactions according to priority and be rewarded with higher transaction fees.

  • transactionType (i.e. is it a notary transfer)

    Notaries transfers can potentially be expensive and so the billing engine could charge more for these

  • numberOfStatesResolved

    In the case of validating notaries the number of states pulled in a notary transfer could have significant impact on the notary and therefore be expensive.

  • retries

    It is possible that there will be duplicate transactions held in the notary as the result of transaction retries. The billing service may choose to bill for these if there are excessive to prevent DDoS style behaviour.

  Note: We have also discussed having a list of all the signers of a transaction for clustered notaries, so that fees can be selectively distributed to notaries that actually took part in the transaction. This idea has been dismissed as its quite heavyweight, and we'd have to wait for all signers slowing down notarisation.

How will the Notary Transaction api be accessed - current design questions

• If metering runs as a service inside the notary: Will notary operators be happy to do so? Note: this does have the greatest bandwidth.
• If it runs via RPC: Do we then need to expose the Notary RPC to the internet and make sure RPC users are permissioned appropriately? or set up some additional firewall rules for Notaries to allow the metering service access?
• Could we offer via P2P? i.e. like other Notary flows (primarily the one to notarise), we make it an initiated flow with a client flow for the metering service to invoke. We take advantage of visibility of the P2P port (Corda Firewall etc etc), and all other versioning and compatibility work.
• Could we do this with a non persistent artemis queue?
  • i.e. the notary just streams the transactions via a queue to the listener after it has sent the initial request to ‘get transactions from ‘this’ date'

Challenges for Metering / billing in a clustered environment

The cordite metering service currently runs the billing operation for a single node notary.

We need further design work to flesh out how the metering service will run with clustered notaries.

Current thoughts are that:

1. The metering service will collect transactions from a quorum of the notary cluster until it is happy that the transaction is valid

   Note: the ‘invoice’ for the transaction can ultimately be rejected by the original node if it does not recognise it, so the ‘quorum’ can be set to a ‘reasonable’ number.

2. The metering service can only run one instance at a time, so could operate in a round robin or leadership election manner in the cluster or it could just be an external service running in its own process.

3. Metering invoices need to be recorded in a vault and have the capability to be disputed, so we must be mindful of this in the overall design. Current thoughts on this are that disputes can go to any node in the cluster which can then verify the validity of the invoice by checking the signature and getting a quorum from the other nodes in the cluster. We also have to consider that the original notary that created the metering invoice may be temporarily offline or have left the cluster and so other member of the cluster should be able to pick it up. Lastly if a metering notary is leaving the cluster we should make sure that any outstanding invoices are either completed or delegated to another member of the cluster.

Challenges for fee distribution in a clustered environment

Once metering invoices have been paid, the fees then need to be dispersed to each notary in the metering pool. We decided that recording the signatures of all notaries in the pool on each transaction would be ‘expensive’ in terms of data and network traffic even if in theory we could distribute fees at this granular level. There a number of more coarse grained approaches to check that notaries of a cluster are contributing positively, such as periodically checking availability. As above, fee distribution is a separate concern to billing which gives business networks the flexibility to create their own mechanisms.

Notary Transaction Meta Data to improve billing opportunities

In order to provide a more flexible notarisation and metering model, giving the ability for the ‘client’ node to put meta data on the ’Notarise’ method would enable: * Transactions to be prioritised for processing - potentially for a greater metering fee. * Transaction importance can be specified - i.e. high value transactions which carry financial risk could be highlighted, so that the notary knows the risk they are undertaking Note: we are throwing this 'importance' as an idea to see if it resonates with anyone.

So we propose that we add some extra parameters to the ‘notarise’ function for these values to specified i.e. * priority (ushort) - 0 no priority - 65535 - max priority * weight (ushort) - 0 no weight - 65535 - highly valuable

These parameters could be hardwired or be part of a key/value pair input - comments welcome!

The need for Notary and Node Blacklisting / Whitelisting

Running metering as a Cordapp gives us the flexibility to decide what flavour of metering we want, if at all. The current Cordite metering implementation uses all the great stuff from corda (states, flows, transactions, services) to secure and create trustworthy billing. The downside of this of course is that the client node has to have the cordite libraries for this to work. This then means if you are in the same compatibility zone, you can use any notary you fancy and get free transactions if you refuse to run the metering libraries. Another case is that even though a node has a metering invoice, she can choose not to pay it.

In order to prevent nodes from using you as a notary, we need to have a flexible mechanism to * blacklist - i.e. refuse transactions from certain nodes/parties if they are behaving badly * whitelist - i.e. only allow notarisation for a pre-approved list of nodes/parties

Business networks may also want to control who connects to nodes in their network - so again blacklisting/whitelisting would also be a useful feature.

For the next steps I will raise 3 issues:

1. proposal for the notary transaction api
2. proposal for whitelisting / blacklisting
3. proposal for notarise meta-data

all comments and suggestions welcome!

This sounds great! One requirement that may be missing is that there may be a requirement for certain fields in a state to be available (i.e. cash owed/paid) in order to charge fees based on a % of the amount. Not sure if the notary even has this information.

The state fields required may be different for each state.

thanks @countfloyd This has been brought up. There is a pro/con to this - if we have the access to the state then metering can be more sophisticated- but it does then mean that the notary can see all of the transactions and their values - which is a privacy leak and could breach data protection laws. In the future though, when SGX is up and running, in theory we'd be able to securely do this kind of thing. I hope that makes sense.