Retrieval Market in Filecoin #

Components #

The retrieval market refers to the process of negotiating deals for a provider to serve stored data to a client.

The main components are as follows:

A payment channel actor (see Payment Channel Actor for details)
2 libp2p services -
- a protocol for making queries
- a protocol for negotiating and carrying out retrieval deals (will disappear as a seperate libp2p protocol as retrieval markets evolve)
A chain-based content routing interface
A client module to query retrieval miners and initiate deals for retrieval
A provider module to respond to queries and deal proposals

VO & V1 #

V0 of the protocol has participants send data over the retrieval protocol itself in a series of Blocks encoded in Bitswap format and verify received blocks manually. It will only support fetching the payload CID which is at the root of PieceCID’s .car File, and will only support fetching the whole DAG.

In V1, the retrieval markets will evolve to support sending arbitrary payload CID’s & selectors within a piece. Further, it will piggy back on the Data Transfer system and Graphsync to handle transfer and verification, to support arbitrary selectors, and to reduce round trips. The Data Transfer System will accordingly be augmented to support pausing/resuming and sending intermediate vouchers to facilitate this. V1 will also include additional mechanisms for timeouts and cancellations (to be specified).

Though the underlying protocols will change, the API interfaces for the client & provider will not change from V0 to V1.

Deal Flow (V0) #

Retrieval Flow - V0 #

The baseline version of proposing and accepting a deal will work as follows:

The client finds a provider of a given piece with FindProviders().
The client queries a provider to see if it meets its retrieval criteria (via Query Protocol)
The client sends a RetrievalDealProposal to the retrieval miner. (via RetrievalProtocol)
The provider validates the proposal and rejects it if it is invalid
If the request is valid, the provider responds to it with an accept message
The client creates a payment channel as neccesary and a lane, ensures there are free funds in the channel
The provider unseals the sector as neccesary
The provider sends blocks over the protocol until it requires payment
The client consumes blocks over the retrieval protocol and manually verifies them
When the provider requires payment to proceed, it sends payment request and does not send any more blocks
The client creates and stores a payment voucher off-chain
The client responds to the provider with a reference to the payment voucher
The provider redeems the payment voucher off-chain
The provider resumes sending blocks
The client consumes blocks until payment is required again
The process continues until the end of the query

Deal Flow (V1) #

Retrieval Flow - V1 #

The evolved protocol for proposing and accepting a deal will work as follows:

The client finds a provider of a given piece with FindProviders().
The client queries a provider to see if it meets its retrieval criteria (via Query Protocol)
The client schedules a Data Transfer Pull Request passing the RetrievalDealProposal as a voucher.
The provider validates the proposal and rejects it if it is invalid
If the proposal is valid, the provider responds with an accept message and begins monitoring the data transfer process
The client creates a payment channel as necessary and a lane, ensures there are free funds in the channel
The provider unseals the sector as necessary
The provider monitors data transfer as it sends blocks over the protocol, until it requires payment
When the provider requires payment, it pauses the data transfer and sends a request for payment as an intermediate voucher
The client receives the request for payment
The client creates and stores payment voucher off-chain
The client responds to provider with a reference to the payment voucher, sent as an intermediate voucher
The provider redeems the payment voucher off-chain
The provider resumes both the request and sending data
The process continues until the end of the query

Bootstrapping Trust #

Neither the client nor the provider have any specific reason to trust the other. Therefore, payment for a retrieval deal is done incrementally, sending vouchers as bytes are sent and verified.

The trust process is as follows:

When the deal is created, client & provider agree to a “payment interval” in bytes, which is the minimum amount of data the provider will send before each required increment
They also agree to a “payment interval increase” – the interval will increase by this value after each successful transfer and payment, as trust develops
Example:
- If my “payment interval” is 1000, and my “payment interval increase” is 300:
- The provider must send at least 1000 bytes before it requires any payment (it may send more cause block boundaries are uneven)
- The client must pay for all bytes sent when the provider requests payment, if the provider has sent at least 1000 bytes
- The provider now must send at least 1300 bytes before it requests payment again
- The client must pay for all bytes it hasn’t yet paid for when the provider requests payment, assuming it’s received at least 1300 bytes since last payment
- The process continues till the end of the retrieval, when the last payment will simply be for the remainder of bytes
Additional trust mechanisms in the V1 version of the protocol will include agreed upon timeouts and cancellation fees

Common Data Types #

import ipld "github.com/filecoin-project/specs/libraries/ipld"
import addr "github.com/filecoin-project/go-address"
import abi "github.com/filecoin-project/specs-actors/actors/abi"
import peer "github.com/libp2p/go-libp2p-core/peer"
import cid "github.com/ipfs/go-cid"

type PaymentVoucher struct {}

type RetrievalPeer struct {
    Address  addr.Address
    ID       peer.ID  // optional
}

type Available struct {}
type Unavailable struct {}
type Unknown struct {}

type RetrievalQueryResponseStatus union {
    Available
    Unavailable
}

type RetrievalQueryItemStatus union {
    Available
    Unavailable
    Unknown
}

type RetrievalQueryParams struct {
    PayloadCID                  cid.Cid  // optional, query if miner has this cid in this piece. some miners may not be able to respond.
    Selector                    ipld.Selector  // optional, query if miner has this cid in this piece. some miners may not be able to respond.
    MaxPricePerByte             abi.TokenAmount  // optional, tell miner uninterested if more expensive than this
    MinPaymentInterval          UInt  // optional, tell miner uninterested unless payment interval is greater than this
    MinPaymentIntervalIncrease  UInt  // optional, tell miner uninterested unless payment interval increase is greater than this
}

type RetrievalQuery struct {
    PieceCID abi.PieceCID  // V0
    RetrievalQueryParams  // V1
}

type RetrievalQueryResponse struct {
    Status                      RetrievalQueryResponseStatus
    PayloadCIDFound             RetrievalQueryItemStatus  // V1 - if a PayloadCid was requested, the result
    SelectorFound               RetrievalQueryItemStatus  // V1 - if a Selector was requested, the result

    Size                        uint64  // Total size of piece in bytes
    ExpectedPayloadSize         uint64  // V1 - optional, if PayloadCID + selector are specified and miner knows, can offer an expected size

    PaymentAddress              addr.Address  // address to send funds to -- may be different than miner addr
    MinPricePerByte             abi.TokenAmount
    MaxPaymentInterval          UInt
    MaxPaymentIntervalIncrease  UInt

    PieceRetrievalPrice()  // == MinPricePerByte * Size
    PayloadRetrievalPrice()  // V1 - optional == MinPricePerByte * ExpectedPayloadSize
}

type Accepted struct {}
type Failed struct {}
type Rejected struct {}
type Unsealing struct {}
type FundsNeeded struct {}
type Ongoing struct {}
type FundsNeededLastPayment struct {}
type Completed struct {}
type DealNotFound struct {}

type DealStatus union {
    Accepted
    Failed
    Rejected
    Unsealing
    FundsNeeded
    Ongoing
    FundsNeededLastPayment
    Completed
    DealNotFound
}

type RetrievalParams struct {
    PayloadCID               cid.Cid  // V1
    Selector                 ipld.Selector  // V1
    PricePerByte             abi.TokenAmount
    PaymentInterval          UInt
    PaymentIntervalIncrease  UInt
}

type RetrievalDealID UInt

type RetrievalDealProposal struct {
    PieceCID         abi.PieceCID
    ID               RetrievalDealID  // V1 - an identifier for the retrieval
    RetrievalParams
}

type Block struct {
    Prefix  Bytes
    Data    Bytes
}

type RetrievalDealResponse struct {
    Status       DealStatus
    ID           RetrievalDealID  // V1 - an identifier for the retrieval

    // payment required to proceed
    PaymentOwed  abi.TokenAmount

    Message      string
    Blocks       [Block]  // V0 only
}

type RetrievalDealPayment struct {
    ID              RetrievalDealID  // V1 - an identifier for the retrieval
    PaymentChannel  addr.Address
    PaymentVoucher
}