Retrieval Market in Filecoin #
Components #
The retrieval market
refers to the process of negotiating deals for a provider to serve stored data to a client.
The main components are as follows:
- A payment channel actor (see Payment Channel Actor for details)
- 2
libp2p
services -- a protocol for making queries
- a protocol for negotiating and carrying out retrieval deals (will disappear as a seperate libp2p protocol as retrieval markets evolve)
- A chain-based content routing interface
- A client module to query retrieval miners and initiate deals for retrieval
- A provider module to respond to queries and deal proposals
VO & V1 #
V0 of the protocol has participants send data over the retrieval protocol itself in a series of Blocks encoded in Bitswap format and verify received blocks manually. It will only support fetching the payload CID which is at the root of PieceCID’s .car
File, and will only support fetching the whole DAG.
In V1, the retrieval markets will evolve to support sending arbitrary payload CID’s & selectors within a piece. Further, it will piggy back on the Data Transfer system and Graphsync to handle transfer and verification, to support arbitrary selectors, and to reduce round trips. The Data Transfer System will accordingly be augmented to support pausing/resuming and sending intermediate vouchers to facilitate this. V1 will also include additional mechanisms for timeouts and cancellations (to be specified).
Though the underlying protocols will change, the API interfaces for the client & provider will not change from V0 to V1.
Deal Flow (V0) #
The baseline version of proposing and accepting a deal will work as follows:
- The client finds a provider of a given piece with
FindProviders()
. - The client queries a provider to see if it meets its retrieval criteria (via Query Protocol)
- The client sends a RetrievalDealProposal to the retrieval miner. (via RetrievalProtocol)
- The provider validates the proposal and rejects it if it is invalid
- If the request is valid, the provider responds to it with an accept message
- The client creates a payment channel as neccesary and a lane, ensures there are free funds in the channel
- The provider unseals the sector as neccesary
- The provider sends blocks over the protocol until it requires payment
- The client consumes blocks over the retrieval protocol and manually verifies them
- When the provider requires payment to proceed, it sends payment request and does not send any more blocks
- The client creates and stores a payment voucher off-chain
- The client responds to the provider with a reference to the payment voucher
- The provider redeems the payment voucher off-chain
- The provider resumes sending blocks
- The client consumes blocks until payment is required again
- The process continues until the end of the query
Deal Flow (V1) #
The evolved protocol for proposing and accepting a deal will work as follows:
- The client finds a provider of a given piece with
FindProviders()
. - The client queries a provider to see if it meets its retrieval criteria (via Query Protocol)
- The client schedules a
Data Transfer Pull Request
passing theRetrievalDealProposal
as a voucher. - The provider validates the proposal and rejects it if it is invalid
- If the proposal is valid, the provider responds with an accept message and begins monitoring the data transfer process
- The client creates a payment channel as necessary and a lane, ensures there are free funds in the channel
- The provider unseals the sector as necessary
- The provider monitors data transfer as it sends blocks over the protocol, until it requires payment
- When the provider requires payment, it pauses the data transfer and sends a request for payment as an intermediate voucher
- The client receives the request for payment
- The client creates and stores payment voucher off-chain
- The client responds to provider with a reference to the payment voucher, sent as an intermediate voucher
- The provider redeems the payment voucher off-chain
- The provider resumes both the request and sending data
- The process continues until the end of the query
Bootstrapping Trust #
Neither the client nor the provider have any specific reason to trust the other. Therefore, payment for a retrieval deal is done incrementally, sending vouchers as bytes are sent and verified.
The trust process is as follows:
- When the deal is created, client & provider agree to a “payment interval” in bytes, which is the minimum amount of data the provider will send before each required increment
- They also agree to a “payment interval increase” – the interval will increase by this value after each successful transfer and payment, as trust develops
- Example:
- If my “payment interval” is 1000, and my “payment interval increase” is 300:
- The provider must send at least 1000 bytes before it requires any payment (it may send more cause block boundaries are uneven)
- The client must pay for all bytes sent when the provider requests payment, if the provider has sent at least 1000 bytes
- The provider now must send at least 1300 bytes before it requests payment again
- The client must pay for all bytes it hasn’t yet paid for when the provider requests payment, assuming it’s received at least 1300 bytes since last payment
- The process continues till the end of the retrieval, when the last payment will simply be for the remainder of bytes
- Additional trust mechanisms in the V1 version of the protocol will include agreed upon timeouts and cancellation fees
Common Data Types #
import ipld "github.com/filecoin-project/specs/libraries/ipld"
import addr "github.com/filecoin-project/go-address"
import abi "github.com/filecoin-project/specs-actors/actors/abi"
import peer "github.com/libp2p/go-libp2p-core/peer"
import cid "github.com/ipfs/go-cid"
type PaymentVoucher struct {}
type RetrievalPeer struct {
Address addr.Address
ID peer.ID // optional
}
type Available struct {}
type Unavailable struct {}
type Unknown struct {}
type RetrievalQueryResponseStatus union {
Available
Unavailable
}
type RetrievalQueryItemStatus union {
Available
Unavailable
Unknown
}
type RetrievalQueryParams struct {
PayloadCID cid.Cid // optional, query if miner has this cid in this piece. some miners may not be able to respond.
Selector ipld.Selector // optional, query if miner has this cid in this piece. some miners may not be able to respond.
MaxPricePerByte abi.TokenAmount // optional, tell miner uninterested if more expensive than this
MinPaymentInterval UInt // optional, tell miner uninterested unless payment interval is greater than this
MinPaymentIntervalIncrease UInt // optional, tell miner uninterested unless payment interval increase is greater than this
}
type RetrievalQuery struct {
PieceCID abi.PieceCID // V0
RetrievalQueryParams // V1
}
type RetrievalQueryResponse struct {
Status RetrievalQueryResponseStatus
PayloadCIDFound RetrievalQueryItemStatus // V1 - if a PayloadCid was requested, the result
SelectorFound RetrievalQueryItemStatus // V1 - if a Selector was requested, the result
Size uint64 // Total size of piece in bytes
ExpectedPayloadSize uint64 // V1 - optional, if PayloadCID + selector are specified and miner knows, can offer an expected size
PaymentAddress addr.Address // address to send funds to -- may be different than miner addr
MinPricePerByte abi.TokenAmount
MaxPaymentInterval UInt
MaxPaymentIntervalIncrease UInt
PieceRetrievalPrice() // == MinPricePerByte * Size
PayloadRetrievalPrice() // V1 - optional == MinPricePerByte * ExpectedPayloadSize
}
type Accepted struct {}
type Failed struct {}
type Rejected struct {}
type Unsealing struct {}
type FundsNeeded struct {}
type Ongoing struct {}
type FundsNeededLastPayment struct {}
type Completed struct {}
type DealNotFound struct {}
type DealStatus union {
Accepted
Failed
Rejected
Unsealing
FundsNeeded
Ongoing
FundsNeededLastPayment
Completed
DealNotFound
}
type RetrievalParams struct {
PayloadCID cid.Cid // V1
Selector ipld.Selector // V1
PricePerByte abi.TokenAmount
PaymentInterval UInt
PaymentIntervalIncrease UInt
}
type RetrievalDealID UInt
type RetrievalDealProposal struct {
PieceCID abi.PieceCID
ID RetrievalDealID // V1 - an identifier for the retrieval
RetrievalParams
}
type Block struct {
Prefix Bytes
Data Bytes
}
type RetrievalDealResponse struct {
Status DealStatus
ID RetrievalDealID // V1 - an identifier for the retrieval
// payment required to proceed
PaymentOwed abi.TokenAmount
Message string
Blocks [Block] // V0 only
}
type RetrievalDealPayment struct {
ID RetrievalDealID // V1 - an identifier for the retrieval
PaymentChannel addr.Address
PaymentVoucher
}