Introduction

Welcome to the Polka Storage project!

This project aims to deliver a Polkadot-native system parachain for data storage.

Since the Referendum approval, we've been busy developing the parachain, and this is our deliverable for Phase 1, composed of:

• Filecoin actor ports:
  • Storage Provider — excluding proving mechanisms
  • Market
• CAR file conversion server
• Dedicated CLIs
  • storage-provider-cli to generate keys and test our CARv2 Rust implementation!
  • storagext-cli (shown below) to take the parachain for a spin!

The following on-chain logic has been implemented:

• Keeping track of Storage Providers,
• Publishing Market Deals on-chain,
• Investing tokens into the Storage Market,
• Receiving funds after completing a deal,
• Commiting to the Storage and Proving the storage,
• Declaring failures to deliver committed storage and Recovering from it,
• Continuously proving that the promise of storage has been kept up PoSt proof,
• Punishing for failing to provide storage.

We present a complete real-world scenario in which a Storage Provider and a Storage User negotiate a deal, perform all the steps necessary to start the storage and then receive rewards (or punishments) for making it happen.

More information available about the project's genesis in:

• OpenGov Referendum — https://polkadot.polkassembly.io/referenda/494
• Research Report — https://github.com/eigerco/polkadot-native-storage/blob/main/doc/report/polkadot-native-storage-v1.0.0.pdf
• Polkadot Forum News Post — https://forum.polkadot.network/t/polkadot-native-storage/4551

Getting Started

This chapter goes through the process of setting up, running, and trying out the components implemented so far.

System Requirements

Before proceeding with the setup, please ensure the host system meets the following requirements:

• OS: Linux x86_64/MacOS ARM x64
• RAM: Minimum 8GB, recommended 16GB or more

Guides

• Local Testnet - Polka Storage Parachain — Covers how to setup a local testnet for the Polka Storage parachain, using Zombienet.
• CAR server — Covers how to setup and run the CARv2 server, which transforms files into .car archives.
• Real-world use case demo — Covers a real-world use case, adding balances, submitting sectors, and more!

Local Testnet - Polka Storage Parachain

This guide helps to set up a local parachain network using zombienet. At the end, we will have three nodes: Alice, Bob and Charlie. Alice and Bob will be running Polkadot relay chain nodes as validators, and Charlie will run a relay chain and parachain node. Charlie will be our contact point to the parachain network.

Native Binaries

Our latest releases's binaries are available to download and can be run without additional dependencies. We support Linux x86_64 and MacOS ARM x64. The commands below will download:

• Relay Chain binaries (polkadot, polkadot-prepare-worker, polkadot-execute-worker),
• Polka Storage Parachain binary (polka-storage-node),
• Polka Storage Provider internal node (polka-storage-provider),
• CLI for interacting with the parachain (storagext-cli),
• zombienet to spawn local testnets and orchestrate them (zombienet),
• Polka Storage Parachain out-of-the-box zombienet's configuration (polka-storage-testnet.toml).

Linux x86_64

1. Download the binaries:

wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot
wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot-prepare-worker
wget https://github.com/paritytech/polkadot-sdk/releases/download/polkadot-v1.13.0/polkadot-execute-worker
wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/polka-storage-node
wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/polka-storage-provider
wget https://s3.eu-central-1.amazonaws.com/polka-storage/linux_x86-64/storagext-cli
wget https://github.com/paritytech/zombienet/releases/download/v1.3.106/zombienet-linux-x64 -O zombienet

2. Setup permissions:

chmod +x zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker

3. Run zombienet:

export PATH=$(pwd):$PATH

wget https://s3.eu-central-1.amazonaws.com/polka-storage/polka-storage-testnet.toml
zombienet -p native spawn polka-storage-testnet.toml

MacOS ARM

1. Download the binaries:

wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot
wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot-prepare-worker
wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polkadot-execute-worker
wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polka-storage-node
wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/polka-storage-provider
wget https://s3.eu-central-1.amazonaws.com/polka-storage/macos_arm/storagext-cli
wget https://github.com/paritytech/zombienet/releases/download/v1.3.106/zombienet-macos-arm64 -O zombienet

2. Setup permissions & de-quarantine:

chmod +x zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker
xattr -d com.apple.quarantine zombienet polka-storage-node polka-storage-provider storagext-cli polkadot polkadot-prepare-worker polkadot-execute-worker

3. Run zombienet:

export PATH=$(pwd):$PATH

wget https://s3.eu-central-1.amazonaws.com/polka-storage/polka-storage-testnet.toml
zombienet -p native spawn polka-storage-testnet.toml

The parachain is also accessible using the Polkadot.js Apps interface by clicking on this link: https://polkadot.js.org/apps/?rpc=ws://127.0.0.1:42069#/explorer

Or interact with the chain via storagext-cli, for example:

storagext-cli --sr25519-key "//Alice" storage-provider register Alice

Kubernetes

Prerequisites

• zombienet v1.3.106 - cli tool to easily spawn ephemeral Polkadot/Substrate networks and perform tests against them.
• minikube — to run the parachain nodes
• a configured kubectl — is used to set up the required kubernetes resources by zombienet
  • https://minikube.sigs.k8s.io/docs/handbook/kubectl/
  • https://kubernetes.io/docs/tasks/tools/#kubectl

Start up the Kubernetes cluster

Using minikube, start the cluster with the following command:

minikube start

More information about minikube is available on its Getting Started page.

Running the Parachain

1. Create a local-kube-testnet.toml file with the following content:

[settings]
image_pull_policy = "IfNotPresent"

[relaychain]
chain = "rococo-local"
default_args = ["--detailed-log-output", "-lparachain=debug,xcm=trace,runtime=trace"]
default_image = "docker.io/parity/polkadot:v1.13.0"

[[relaychain.nodes]]
name = "alice"
validator = true

[[relaychain.nodes]]
name = "bob"
validator = true

[[parachains]]
cumulus_based = true

# We need to use a Parachain of an existing System Chain (https://github.com/paritytech/polkadot-sdk/blob/master/polkadot/runtime/rococo/src/xcm_config.rs).
# The reason: being able to get native DOTs from Relay Chain to Parachain via XCM Teleport.
# We'll have a proper Parachain ID in the *future*, but for now, let's stick to 1000 (which is AssetHub and trusted).
id = 1000

# Run Charlie as parachain collator
[[parachains.collators]]
args = ["--detailed-log-output", "-lparachain=debug,xcm=trace,runtime=trace"]
command = "polka-storage-node"
image = "polkadotstorage.azurecr.io/parachain-node:0.1.0"
name = "charlie"
rpc_port = 42069
validator = true

2. Run the Parachain, and spawn the zombienet testnet in the Kubernetes cluster:

zombienet -p kubernetes spawn local-kube-testnet.toml

│ /ip4/10.1.0.16/tcp/30333/ws/p2p/12D3KooWPKzmmE2uYgF3z13xjpbFTp63g9dZFag8pG6MgnpSLF4S                                   │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

         Warn: Tracing collator service doesn't exist
┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                                       Network launched 🚀🚀                                                       │
├──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Namespace                    │ zombie-1cecb9b5e0f9a14208f2fbefd9384490                                                            │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Provider                     │ kubernetes                                                                                         │
├──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                         Node Information                                                          │
├──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Name                         │ alice                                                                                              │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Direct Link                  │ https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A34341#/explorer                           │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Prometheus Link              │ http://127.0.0.1:35537/metrics                                                                     │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Log Cmd                      │ kubectl logs -f alice -c alice -n zombie-1cecb9b5e0f9a14208f2fbefd9384490                          │
├──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                         Node Information                                                          │
├──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Name                         │ bob                                                                                                │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Direct Link                  │ https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A44459#/explorer                           │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Prometheus Link              │ http://127.0.0.1:43841/metrics                                                                     │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Log Cmd                      │ kubectl logs -f bob -c bob -n zombie-1cecb9b5e0f9a14208f2fbefd9384490                              │
├──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                         Node Information                                                          │
├──────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Name                         │ charlie                                                                                            │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Direct Link                  │ https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A42069#/explorer                           │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Prometheus Link              │ http://127.0.0.1:42675/metrics                                                                     │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Log Cmd                      │ kubectl logs -f charlie -c charlie -n zombie-1cecb9b5e0f9a14208f2fbefd9384490                      │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Parachain ID                 │ 1000                                                                                               │
├──────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ChainSpec Path               │ /tmp/zombie-1cecb9b5e0f9a14208f2fbefd9384490_-29755-WOCdKtq9zPGA/1000-rococo-local.json            │
└──────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘

Verifying the Setup

Check if all zombienet pods were started successfully:

kubectl get pods --all-namespaces

...
zombie-01b7920d650c18d3d78f75fd8b0978af   alice                              1/1     Running     0               77s
zombie-01b7920d650c18d3d78f75fd8b0978af   bob                                1/1     Running     0               62s
zombie-01b7920d650c18d3d78f75fd8b0978af   charlie                            1/1     Running     0               49s
zombie-01b7920d650c18d3d78f75fd8b0978af   fileserver                         1/1     Running     0               2m28s
zombie-01b7920d650c18d3d78f75fd8b0978af   temp                               0/1     Completed   0               2m25s
zombie-01b7920d650c18d3d78f75fd8b0978af   temp-1                             0/1     Completed   0               2m25s
zombie-01b7920d650c18d3d78f75fd8b0978af   temp-2                             0/1     Completed   0               2m15s
zombie-01b7920d650c18d3d78f75fd8b0978af   temp-3                             0/1     Completed   0               2m1s
zombie-01b7920d650c18d3d78f75fd8b0978af   temp-4                             0/1     Completed   0               114s
zombie-01b7920d650c18d3d78f75fd8b0978af   temp-5                             0/1     Completed   0               91s
zombie-01b7920d650c18d3d78f75fd8b0978af   temp-collator                      0/1     Completed   0               104s

Accessing the Parachain

The parachain is available through the Polkadot.js Apps interface by clicking on this link:

https://polkadot.js.org/apps/?rpc=ws%3A%2F%2F127.0.0.1%3A42069#/explorer

This link will automatically connect to Charlie's node running on a local machine at port 42069. The port is configured in local-kube-testnet.toml under rpc_port for Charlie's node.

Zombienet Configuration Breakdown

During the setup either download a file in the third step of Linux/MacOS — polka-storage-testnet.toml — or copy it from the first step of Running the parachain.

Similarities

The two files share most of the contents, so we'll start by covering their similarities. For more details refer to the zombienet documentation:

relaychain

nodes

parachains

A list of tables defining multiple parachains, in our case, we only care for our own parachain.

collators

Differences

The difference between them lies in the usage of container configurations:

CAR server

It is an HTTP server that converts arbitrary content into a CARv2 file and serves it over HTTP - supporting the latest CARv2 format, which is not yet entirely supported by other crates in the Rust ecosystem. The next steps describe how to run the server locally and use it to upload and download files.

Start the server

1. Create a Docker volume to store uploaded files:

docker volume create storage_provider

2. Start the server:

docker run \
    -p 127.0.0.1:9000:9000 \
    --mount source=storage_provider,destination=/app/uploads \
    polkadotstorage.azurecr.io/polka-storage-provider:0.1.0 storage \
        --listen-addr 0.0.0.0:9000

• -p 127.0.0.1:9000:9000: Maps port 9000 on the localhost to port 9000 on the container.
• --mount source=storage_provider,destination=/app/uploads: Mounts the storage_provider volume to /app/uploads inside the container.
• polkadotstorage.azurecr.io/polka-storage-provider:0.1.0 storage: Runs the polkadotstorage.azurecr.io/polka-storage-provider:0.1.0 image with the storage command.
• --listen-addr 0.0.0.0:9000: Configures the server to listen on all available network interfaces.

Verifying the Setup

After setting up and starting the CAR server, it's essential to verify that everything works correctly. Follow these steps to ensure your setup works as expected:

1. Upload a test file using the instructions in the Upload a file section. Make sure to note the CID returned by the server.

2. Download the CAR file using the retrieved CID, following the steps in the Download the CAR File section.

3. [Optional] Verify the contents of the downloaded CAR file. Using, for example, go-car's inspect command:

   car inspect <target_file>
   

   The user can use debug for more detailed output:

   car debug <target_file>
   

   If the user desires, they can extract the contents of the file:

   car extract <target_file>
   

If a file can be successfully uploaded, the server produces a CID, allows download the corresponding CAR file, and verify its contents, the CAR server setup is working correctly.

Real-world use case demo

For convenience's sake, we have a script that automates the actions in this guide. The script is available at the following link: https://polka-storage.s3.eu-central-1.amazonaws.com/demo.sh

Alternatively, download and run in a single step:

wget https://polka-storage.s3.eu-central-1.amazonaws.com/demo.sh
chmod +x demo.sh
./demo.sh

The script has some pre-requisites:

• The storagext-cli binary must exist and be present in the $PATH — instructions on how to achieve that are available in the Local Testnet - Polka Storage Parachain chapter.
• You need to launch a fresh parachain, as the script needs to be run while the Charlie node is booting up; the script will wait for the first block before starting. This process may need more than one attempt.

A high-level overview with diagrams of the process described below can be found in Pallets section.

1. Publishing a deal

Charlie heard he could provide storage to people worldwide and earn some tokens, so he decided to register as a Storage Provider.

$ storagext-cli --sr25519-key "//Charlie" storage-provider register Charlie
[0xd14d…4b6a] Storage Provider Registered: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, info: Storage Provider Info: { peer_id: 3ZAB4sc5BS, window_post_proof_type: StackedDRGWindow2KiBV1P1, sector_size: _2KiB, window_post_partition_sectors: 2 }, proving_period_start: 21 }

Alice is a Storage User and wants to store an image of her lovely Husky (husky.jpg) in the Polka Storage parachain.

Alice knows¹ that she needs to get a CID of the image, so she uploaded it to the CAR server and received the CID: bafybeihxgc67fwhdoxo2klvmsetswdmwwz3brpwwl76qizbsl6ypro6vxq.

Alice heard somewhere² in the hallways of her favourite gym that Charlie is a Storage Provider. She calls him (off-chain), and they negotiate a deal:

husky-deal.json

[
  {
    "piece_cid": "bafybeihxgc67fwhdoxo2klvmsetswdmwwz3brpwwl76qizbsl6ypro6vxq",
    "piece_size": 1278,
    "client": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    "provider": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
    "label": "My lovely Husky (husky.jpg)",
    "start_block": 25,
    "end_block": 50,
    "storage_price_per_block": 1000000000,
    "provider_collateral": 12500000000,
    "state": "Published"
  }
]

After the negotiation, they need to add funds to the Pallet and then publish their intent so it can be checked by the parachain. So here they go:

# Adding balance to Alice's account
$ storagext-cli --sr25519-key "//Alice" market add-balance 25000000000
[0xe0bd…06f9] Balance Added: { account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, amount: 25000000000 }
# Adding balance to Charlie's account
$ storagext-cli --sr25519-key "//Charlie" market add-balance 12500000000
[0x25aa…edd3] Balance Added: { account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, amount: 12500000000 }
# Publishing their deal
$ storagext-cli --sr25519-key  "//Charlie" market publish-storage-deals --client-sr25519-key  "//Alice" "@husky-deal.json"
[0xd50b…dee6] Deal Published: { deal_id: 0, provider_account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, client_account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY }

2. Committing a deal

After the deals have been published, the rest is up to Charlie. If Charlie does not behave appropriately, pre-commit and prove the deal by block 25 (start_block), he will be slashed, and all his funds (provider_collateral) will be gone.³ So he should do his part!

pre-commit-husky.json

{
  "sector_number": 1,
  "sealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu",
  "deal_ids": [0],
  "expiration": 75,
  "unsealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu",
  "seal_proof": "StackedDRG2KiBV1P1"
}

prove-commit-husky.json

{
  "sector_number": 1,
  "proof": "1230deadbeef"
}

• proof: hex string of bytes of the proof, it's WIP, so any non-zero value works.

$ storagext-cli --sr25519-key "//Charlie" storage-provider pre-commit "@pre-commit-husky.json"
[0xf2ad…dc3d] Sector Pre-Committed: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, sector_number: Sector Pre-Commit Info: { sector_number: 1, expiration: 75, seal_proof: RegisteredSealProof::StackedDRG2KiBV1P1, unsealed_cid: bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu, sealed_cid: bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu } }
$ storagext-cli --sr25519-key "//Charlie" storage-provider prove-commit "@prove-commit-husky.json"
[0x0743…6fd3] Sector Proven: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, sector_number: 1, partition_number: 0, deadline_idx: 0 }

3. Proofs and faults

Aside on Deadlines

There is a little something that Charlie needs to know about: deadlines (don't we all...).

Each Storage Provider has a Proving Period, a time divided into segments (deadlines). To simplify, let's say a proving period lasts a day (24 hours), and between the start and end of each hour, there is a segment, just like on a clock.

Now, when Charlie gathers enough data and stores it in a sector, he must keep proving that he has some data. Charlie can have multiple sectors and he doesn't want to overload the system by proving all of them simultaneously. So what if each sector got assigned a segment (1 hour) during the day, and Charlie would need to submit proof that he has data roughly on the same hour each day? That'd work, right?

So this is what a Proving Period and Deadlines are. We divide a proving period into deadlines and when we prove commit, we assign a sector to a deadline. From now on, the sector must be proven periodically and daily during this lifetime.

windowed-post.json

{
  "deadline": 0,
  "partitions": [0],
  "proof": {
    "post_proof": "2KiB",
    "proof_bytes": "1230deadbeef"
  }
}

Now that the sector has been proven, Charlie must keep confirming that he stores the data. Charlie's proving period starts at block 21 (as register-storage-provider tells us), so the first deadline is between blocks [21, 31), second [31, 41). That's because there are 2 deadlines within a proving period, and a deadline has a window of 10 blocks. Charlie got assigned the first deadline, so he waits until block 21 to send the proof that he still stores the data.

$ storagext-cli --sr25519-key "//Charlie" storage-provider submit-windowed-post "@windowed-post.json"
[0x3aa1…a12a] Valid PoSt Submitted: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y }

The next proving period starts at block 41, with the deadline [41, 51), so Charlie must submit his proof on this block. He knows he won't be able to create a proof, as his hard drives went down, so he reports it. If he didn't report, he'd get slashed.

fault-declaration.json

[
  {
    "deadline": 0,
    "partition": 0,
    "sectors": [1]
  }
]

$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults "@fault-declaration.json"
[0x49cd…9a88] Faults Declared: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, faults: [Fault Declaration: { deadline: 0, partition: 0, sectors: [1] }] }

Charlie fixed his issues with storage and now wants to declare that he can still provide data in this sector. If he does this too late (1 minute before the next deadline starts), he won't be able to.

$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults-recovered "@fault-declaration.json"
Error: Runtime error: Pallet error: StorageProvider::FaultRecoveryTooLate

Caused by:
    Pallet error: StorageProvider::FaultRecoveryTooLate

If he does it at least a minute before, it succeeds:

$ storagext-cli --sr25519-key "//Charlie" storage-provider declare-faults-recovered "@fault-declaration.json"
[0xca6b…875c] Faults Recovered: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, recoveries: [Recovery Declaration: { deadline: 0, partition: 0, sectors: [1] }] }

And then, at the next deadline, in the next proving period [41, 51) he needs to remember to submit windowed PoSt again.

$ storagext-cli --sr25519-key "//Charlie" storage-provider submit-windowed-post "@windowed-post.json"
[0x3e46…5636] Valid PoSt Submitted: { owner: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y }

4. Reaping the rewards

After the deal ends (after block 50), Charlie goes to get his rewards! First, he settles all of the locked funds, so his balance gets unlocked, and then he withdraws the balance from his Market account to use his DOTs for a new shiny dumbbell.

$ storagext-cli --sr25519-key "//Charlie" market settle-deal-payments 0
[0x1633…d17b] Deals settled: { successful: [Settled Deal { deal_id: 0, provider_account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, client_account: 5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY, amount: 25000000000 }], unsuccessful: [] }
$ storagext-cli --sr25519-key "//Charlie" market withdraw-balance 37500000000
[0x0197…bdd9] Balance Withdrawn: { account: 5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y, amount: 37500000000 }

Polka Storage Provider

The Polka Storage Provider CLI provides two commands — wallet and storage.

storage

The storage command launches a server that converts arbitrary files into CARv2 files — it does so using our CARv2 Rust library (the first Rust CARv2 implementation).

More information available on the storage page.

wallet

The wallet command is a thin wrapper over the subkey utility provided by Polkadot.

More information available on the wallet page.

The storage command

The storage command launches a server that can convert files into CAR files. It has two customizable parameters:

Start the Storage Server

Next, start the storage server using the created volume:

polka-storage-provider storage --listen-addr 0.0.0.0:9000

• --listen-addr 0.0.0.0:9000: Configures the server to listen on all available network interfaces.

Upload a file

To upload a file to the provider's server, use the following curl command.

curl \
    -X POST \
    --data-binary "@image.jpg" \
    http://localhost:9000/upload

This command uploads the file image.jpg to the server running at http://localhost:9000/upload. The server converts the uploaded content to a CAR file and saves it to the mounted volume. The returned CID can later be used to fetch a CAR file from the server.

Download the CAR File

After uploading, a CID (Content Identifier) for the file will be produced. Use this CID to download the corresponding CAR file. Replace :cid with the actual CID provided:

curl \
    -X GET \
    --output ./content.car \
    http://localhost:9000/download/<cid>

• -X GET: Specifies the GET request method.
• http://localhost:9000/download/:cid: The URL to download the CAR file, with <cid> being the placeholder for the actual CID.
• --output ./content.car: Saves the downloaded CAR file as content.car in the current directory.

Here's a quick example:

curl \
    -X GET \
    --output ./content.car \
    http://localhost:9000/download/bafkreicqsawmxavfxpqncjy545bgulr54b5xliriexxjiaof6uue5ovduu

The wallet command

The wallet command is a re-export of the Substrate CLI. The detailed documentation is available under:

• sc_cli
• subkey subcommands

Subcommands

The following commands are available with the wallet subcommand:

Examples

Generate a new random key to interact with the Polka Storage parachain:

> polka-storage-provider wallet generate
Secret phrase:       offer payment boost boy manage car asset lock cousin mountain vehicle setup
  Network ID:        substrate
  Secret seed:       0xfe36ee692552b0ce54de06ce4f5cc152fe2fa808cb40f58c81168bc1237208bb
  Public key (hex):  0x3ae6bdc05a6657cea011084d32b9970891be5d02b2101bbad0ca95d287f0226e
  Account ID:        0x3ae6bdc05a6657cea011084d32b9970891be5d02b2101bbad0ca95d287f0226e
  Public key (SS58): 5DPwBLBRGunws9T2aF59cht37HeBg9aSTAc6Fh2aFBJPSsr6
  SS58 Address:      5DPwBLBRGunws9T2aF59cht37HeBg9aSTAc6Fh2aFBJPSsr6

The password may be added interactively, using --password-interactive flag:

> polka-storage-provider wallet generate --password-interactive
Key password: <top secret hidden password>
Secret phrase:       comfort distance rack number assist nasty young universe lamp advice neglect ladder
  Network ID:        substrate
  Secret seed:       0x4243f3f1d78beb5c0408bbaeae58845881b638060380437967482be2d4d42bce
  Public key (hex):  0x3acb66c0313d0e8ef896bc2317545582c1f0a928f402bcbe4cdf6f37489ddb16
  Account ID:        0x3acb66c0313d0e8ef896bc2317545582c1f0a928f402bcbe4cdf6f37489ddb16
  Public key (SS58): 5DPo4H1oPAQwReNVMi9XckSkvW4me1kJoageggJSMDF2EzjZ
  SS58 Address:      5DPo4H1oPAQwReNVMi9XckSkvW4me1kJoageggJSMDF2EzjZ

Or it can be passed directly, beforehand:

> polka-storage-provider wallet generate --password <top secret password>
Secret phrase:       cactus art crime burden hope also thought asset lake only cheese obtain
  Network ID:        substrate
  Secret seed:       0xb69c2d238fa7641f0d69911ca8f107f1b97a51cfc71e8a06e0ec9c7329d69ff7
  Public key (hex):  0xb60a716e488bcb2a54ef1b1cf8874569d2d927cc830ae0ae1cc2612fac27f55d
  Account ID:        0xb60a716e488bcb2a54ef1b1cf8874569d2d927cc830ae0ae1cc2612fac27f55d
  Public key (SS58): 5GBPg51VZG8PobmkLNSn9vDkNvoBXV5vCGhbetifgxwjPKAg
  SS58 Address:      5GBPg51VZG8PobmkLNSn9vDkNvoBXV5vCGhbetifgxwjPKAg

Storagext CLI

Alongside the pallets, we've also developed a CLI to enable calling extrinsics without Polkadot.js.

The CLI's goal is to ease development and testing and to sidestep some limitations of the Polkadot.js visual interface.

This chapter covers how to use the storagext-cli, along with that, there are several usage examples available throughout the book.

Getting started

The storagext-cli takes two main flags — the node's RPC address and a key¹, the latter is split into three kinds, and one is required for most operations (for example, if the operation being called is a signed extrinsic²):

• Sr25519 — --sr25519-key or the SR25519_KEY environment variable
• ECDSA — --ecdsa-key or the ECDSA_KEY environment variable
• Ed25519 — --ed25519-key or the ED25519_KEY environment variable

For example, to connect to a node at supercooldomain.com:1337 using Charlie's Sr25519 key:

storagext-cli --node-rpc "supercooldomain.com:1337" --sr25519-key "//Charlie" <commands>

Or, retrieving the same key but using the environment variable form:

SR25519_KEY="//Charlie" storagext-cli --node-rpc "supercooldomain.com:1337" <commands>

Flags

Sub-chapters

• Subcommand market
• Subcommand storage-provider
• Subcommand system

The market command

Under the market subcommand Market related extrinsics are available. This chapter covers the provided commands and how to use them.

add-balance

The add-balance adds balance to the market account of the extrinsic signer. It takes a single AMOUNT argument, the balance to add to the market account, the balance will be added to the free balance.

Parameters

Example

Adding 1000000000 Plancks to Alice's account.

storagext-cli --sr25519-key "//Alice" market add-balance 1000000000

Error: Runtime error: Token error: Account cannot exist with the funds that would be given.

More information about the add_balance extrinsic is available in Pallets/Market Pallet/Add Balance.

withdraw-balance

The withdraw-balance withdraws balance from the market account of the extrinsic signer. Like add-balance, withdraw-balance takes a single AMOUNT argument; note that only free balance can be withdrawn. Likewise, withdrawal of a balance amount lesser than or equal to the free amount and greater than 0 (\({free} \ge {amount} \gt 0\)).

Parameters

Example

Withdrawing 10000 Plancks from Alice's account.

storagext-cli --sr25519-key "//Alice" market withdraw-balance 10000

More about the withdraw_balance extrinsic is available in Pallets/Market Pallet/Withdraw Balance.

publish-storage-deals

The publish-storage-deals publishes storage deals that have been agreed upon off-chain. The deals are to be submitted by the storage provider, having been previously signed by the client.

Parameters

The client keypair can be passed using --client-<key kind>, where <key kind> is one of the three supported keys, like the global keys, one is required.

The DEALS JSON array is composed of objects:

Example

Publishing deals between Alice (the Storage Provider) and Charlie (the client).

storagext-cli --sr25519-key "//Alice" market publish-storage-deals \
  --client-sr25519-key "//Charlie" \
  "@deals.json"

Where deals.json is a file with contents similar to:

[
  {
    "piece_cid": "bafk2bzacecg3xxc4f2ql2hreiuy767u6r72ekdz54k7luieknboaakhft5rgk",
    "piece_size": 1337,
    "client": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
    "provider": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    "label": "Super Cool (but secret) Plans for a new Polkadot Storage Solution",
    "start_block": 69,
    "end_block": 420,
    "storage_price_per_block": 15,
    "provider_collateral": 2000,
    "state": "Published"
  },
  {
    "piece_cid": "bafybeih5zgcgqor3dv6kfdtv3lshv3yfkfewtx73lhedgihlmvpcmywmua",
    "piece_size": 1143,
    "client": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
    "provider": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    "label": "List of problematic (but flying) Boeing planes",
    "start_block": 1010,
    "end_block": 1997,
    "storage_price_per_block": 1,
    "provider_collateral": 3900,
    "state": "Published"
  }
]

More information about the publish_storage_deals extrinsic is available in Pallets/Market Pallet/Publish Storage Deals.

settle-deal-payments

The settle-deal-payments command makes the storage provider receive the owed funds from storing data for their clients. Non-existing deal IDs will be ignored.

Anyone can settle anyone's deals, though there's little incentive to do so as it costs gas, so the Storage Provider will end up being the caller most of the time.

Parameters

Example

Settling deals with the IDs 97, 1010, 1337, 42069:

storagext-cli --sr25519-key "//Alice" market settle-deal-payments 97 1010 1337 42069

More information about the publish_storage_deals extrinsic is available in Pallets/Market Pallet/Settle Deal Payments.

retrieve-balance

The retrieve-balance command checks the balance of a given market account.

Parameters

Example

storagext-cli market retrieve-balance "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY" # Alice's account

This extrinsic is not signed, and does not need to be called using any of the --X-key flags.

The storage-provider command

Under the storage-provider subcommand Storage Provider related extrinsics are available. This chapter covers the provided commands and how to use them.

register

The register command registers as a storage provider. Before a user can start providing storage, they need to register to be able to deal with the clients and perform any storage provider duties.

Parameters

Example

Registering the provider with the specific peer_id

storagext-cli --sr25519-key <key> storage-provider register <peer_id>

More information about the register extrinsic is available in Pallets/Storage Provider/Register.

pre-commit

The pre-commit command pre-commits a sector with deals that have been published by market publish-storage-deals. The pre-committed sector has to be proven or the deals will not activate and will be slashed.

Parameters

The PRE_COMMIT_SECTOR JSON object has the following structure:

Example

Pre-commits a sector with specified deals.

storagext-cli --sr25519-key <key> storage-provider pre-commit \
    "@pre-commit-sector.json"

Where pre-commit-sector.json is a file with contents similar to:

{
  "sector_number": 0,
  "sealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu",
  "deal_ids": [0],
  "expiration": 100,
  "unsealed_cid": "bafkreibme22gw2h7y2h7tg2fhqotaqjucnbc24deqo72b6mkl2egezxhvy",
  "seal_proof": "StackedDRG2KiBV1P1"
}

More information about the pre_commit extrinsic is available in Pallets/Storage Provider/Pre-commit sector.

prove-commit

The prove-commit command proves a sector commitment. Currently, any proof that is a hex encoded string of length >= 1 is accepted. After the sector is proven, the deals will become Active.

Parameters

The PROVE_COMMIT_SECTOR JSON object has the following structure:

Example

Proves a sector commitment.

storagext-cli --sr25519-key <key> storage-provider prove-commit \
    "@prove-commit-sector.json"

Where prove-commit-sector.json is a file with contents similar to:

{
  "sector_number": 0,
  "proof": "beef"
}

More information about prove_commit extrinsic is available in Pallets/Storage Provider/Prove-commit sector.

submit-windowed-post

The submit-windowed-post command submits a windowed PoSt proof. The post proof needs to be periodically submitted to prove that some sector is still stored. Sectors are proven in batches called partitions.

Parameters

The WINDOWED_POST JSON object has the following structure:

The PROOF JSON object has the following structure:

Example

Proves a partition in a specific deadline.

storagext-cli --sr25519-key <key> storage-provider submit-windowed-post \
    "@window-proof.json"

Where window-proof.json is a file with contents similar to:

{
  "deadline": 0,
  "partition": 0,
  "proof": {
    "post_proof": "2KiB",
    "proof_bytes": "07482439"
  }
}

More information about the submit_windowed_post extrinsic is available in Pallets/Storage Provider/Submit Windowed Post.

declare-faults

The declare-faults command declares faulty sectors. This is required to avoid penalties for not submitting Window PoSt at the required time.

Parameters

The FAULTS JSON object has the following structure:

Example

Declares a list of faulty sectors in a specific deadline and partition.

storagext-cli --sr25519-key <key> storage-provider declare-faults \
    "@faults.json"

Where faults.json is a file with contents similar to:

[
  {
    "deadline": 0,
    "partition": 0,
    "sectors": [0]
  }
]

More information about the declare_faults extrinsic is available in Pallets/Storage Provider/Declare Faults.

declare-faults-recovered

The declare-faults-recovered command declares recovered faulty sectors.

Parameters

The RECOVERIES JSON object has the following structure:

Example

Declares a list of sectors as recovered in a specific deadline and partition.

storagext-cli --sr25519-key <key> storage-provider declare-faults-recovered \
    "@recoveries.json"

Where recoveries.json is a file with contents similar to:

[
  {
    "deadline": 0,
    "partition": 0,
    "sectors": [0]
  }
]

More information about the declare_faults_recovered extrinsic is available in Pallets/Storage Provider/Declare Faults Recovered.

The system command

The command provides various utilities for interacting with the blockchain. It retrieves information about the current state of the chain.

get-height

The command gets the current block height of the chain.

Example

Getting the current block height of the chain.

storagext-cli system get-height

wait-for-height

The command awaits for the chain to reach a specific block height. It will exit once the chain has reached the specified height.

Parameters

Example

Waiting for the chain to reach block height 100.

storagext-cli system wait-for-height 100

Polka Storage pallets

• storage-provider - A pallet that manages storage providers and their associated data.
• market - A pallet that handles the storage market operations.

Overview

The Polka Storage parachain is all about making storage deals. Let us go over how a deal is done!

Before anything else, Storage Providers need to register themselves with the Storage Provider Pallet — they can do so using the register_storage_provider extrinsic.

Now that storage providers can be registered in the storage provider pallet, we need to add some balance to both the Storage User's and the Provider's accounts, which is done using the Market's add_balance extrinsic.

Afterwards, storage users and providers negotiate data storage deals off-chain. Once a deal between the two parties is reached, the client can sign the deal and send it to the storage provider for publishing — the storage provider will then publish the signed deal using the publish_storage_deals extrinsic.

After publishing, the funds allocated for the deal will be moved from free to locked, and they can no longer be withdrawn until the deal has ended.

At this point, the remaining responsibility is shifted to the storage provider, which needs to activate the deal; to do so, the storage provider first needs to pre-commit the deal's sectors, and then prove they stored the sectors — these two steps are done using the pre_commit_sector and prove_commit_sector extrinsics.

Verification is done and reported to the Market pallet to terminate the deal and apply penalties to the storage provider (remove and burn its collateral — i.e. locked funds) if they fail to activate the deal on time and return the funds to the client.

Suppose the deal has been completed successfully or is Active. In that case, the storage provider is now required to periodically submit proofs that they're still storing the user's data — the storage provider does this by calculating a proof and submitting it using submit_windowed_post.

Finally, storage providers can then settle deal payments to receive their fair share for keeping the user's data safe — using the settle_deal_payments extrinsic.

Putting it all together, we get the following:

Market Pallet

Table of Contents

• Overview
• Extrinsics
  • add_balance
  • withdraw_balance
  • settle_deal_payments
  • publish_storage_deals
• Events
• Errors
• Constants

Overview

The purpose of the pallet is to manage storage deals between storage market participants and to track their funds. Market Pallet is tightly coupled with Storage Provider Pallet because it's a source of truth for deals. Storage Provider Pallet cannot exist without deal information from Market Pallet.

Extrinsics^*

add_balance

Reserves a given amount of currency for usage in the Storage Market.

The reserved amount will be considered free until it is used in a deal when moved to locked and used to pay for the deal.

Example

Using the storagext-cli to add 1000000000¹ Plancks to Alice's account with the following command²:

storagext-cli --sr25519-key "//Alice" market add-balance 1000000000

This value is the minimum amount due to Polkadot's existential deposit.
More information available in: <https://support.polkadot.network/support/solutions/articles/65000168651-what-is-the-existential-deposit->.

Withdraws funds from the Storage Market.

The funds will be withdrawn from the free balance, meaning that the amount must be lesser than or equal to free and greater than 0 (\({free} \ge {amount} \gt 0\)).

Example

Using the storagext-cli to withdraw 10000 Plancks from Alice's free balance using the following command³:

storagext-cli --sr25519-key "//Alice" market withdraw-balance 10000

publish_storage_deals

Publishes list of deals to the chain.

This extrinsic must be called by a storage provider.

The client_signature, as the name indicates, is generated by the client by signing the deal proposal with their private key — the storagext-cli does this for the user automatically⁴. This signature ensures that the storage provider cannot forge a deal with an arbitrary client. The type of signature is dependent on the key the signer has, currently supported key types are Sr25519, ECDSA and Ed25519.

This step corresponds to the "sign & send proposal" step in the deal overview.

Deal Proposal Components

See the original Filecoin specification for details.

Example

Using the storagext-cli to publish deals with //Alice as the storage provider and //Charlie as the client by running the following command⁶:

storagext-cli --sr25519-key "//Alice" market publish-storage-deals \
  --client-sr25519-key "//Charlie" \
  "@deals.json"

Where deals.json is a file with contents similar to:

[
  {
    "piece_cid": "bafk2bzacecg3xxc4f2ql2hreiuy767u6r72ekdz54k7luieknboaakhft5rgk",
    "piece_size": 1337,
    "client": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
    "provider": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    "label": "Super Cool (but secret) Plans for a new Polkadot Storage Solution",
    "start_block": 69,
    "end_block": 420,
    "storage_price_per_block": 15,
    "provider_collateral": 2000,
    "state": "Published"
  },
  {
    "piece_cid": "bafybeih5zgcgqor3dv6kfdtv3lshv3yfkfewtx73lhedgihlmvpcmywmua",
    "piece_size": 1143,
    "client": "5FLSigC9HGRKVhB9FiEo4Y3koPsNmBmLJbpXg2mp1hXcS59Y",
    "provider": "5GrwvaEF5zXb26Fz9rcQpDWS57CtERHpNehXCPcNoHGKutQY",
    "label": "List of problematic (but flying) Boeing planes",
    "start_block": 1010,
    "end_block": 1997,
    "storage_price_per_block": 1,
    "provider_collateral": 3900,
    "state": "Published"
  }
]

settle_deal_payments

Settle specified deals between providers and clients.

Both clients and providers can call this extrinsic. However, since the settlement is the mechanism through which the provider gets paid, a client has no reason to call this extrinsic. Non-existing deal IDs will be ignored.

Example

Using the storagext-cli to settle deal payments for IDs 97, 1010, 1337 and 42069 using the following command⁷:

storagext-cli --sr25519-key "//Alice" market settle-deal-payments 97 1010 1337 42069

Events

The Market Pallet emits the following events:

• BalanceAdded - Indicates that some balance was added as free to the Market Pallet account for usage in the storage market.
  • who - SS58 address of then account which added balance
  • amount - Amount added
• BalanceWithdrawn - Some balance was transferred (free) from the Market Account to the Participant's account.
  • who - SS58 address of the account which had withdrawn the balance
  • amount - Amount withdrawn
• DealPublished - Indicates that a deal was successfully published with publish_storage_deals.
  • deal_id - Unique deal ID
  • client - SS58 address of the storage client
  • provider - SS58 address of the storage provider
• DealActivated - Deal's state has changed to Active.
  • deal_id - Unique deal ID
  • client - SS58 address of the storage client
  • provider - SS58 address of the storage provider
• DealsSettled - Published after the settle_deal_payments extrinsic is called. Indicates which deals were successfully and unsuccessfully settled.
  • successful - List of deal IDs that were settled
  • unsuccessful - List of deal IDs with the corresponding errors
• DealSlashed - Is emitted when some deal expired
  • deal_id - Deal ID that was slashed
• DealTerminated - Is emitted it indicates that the deal was voluntarily or involuntarily terminated.
  • deal_id - Terminated deal ID
  • client - SS58 address of the storage client
  • provider - SS58 address of the storage provider

Errors

The Market Pallet actions can fail with following errors:

• InsufficientFreeFunds - Market participants do not have enough free funds.
• NoProposalsToBePublished - publish_storage_deals was called with an empty list of deals.
• ProposalsNotPublishedByStorageProvider - Is returned when calling publish_storage_deals and the deals in a list are not published by the same storage provider.
• AllProposalsInvalid - publish_storage_deals call was supplied with a list of deals which are all invalid.
• DuplicateDeal - There is more than one deal with this ID in the Sector.
• DealNotFound - Tried to activate a deal that is not in the system.
• DealActivationError - Tried to activate a deal, but data was malformed.
  • Invalid specified provider.
  • The deal already expired.
  • Sector containing the deal expires before the deal.
  • Invalid deal state.
  • Deal is not pending.
• DealsTooLargeToFitIntoSector - Sum of all deals piece sizes for a sector exceeds sector size. The sector size is based on the registered proof type. We currently only support registered StackedDRG2KiBV1P1 proofs, which have 2KiB sector sizes.
• TooManyDealsPerBlock - Tried to activate too many deals at a given start_block.
• UnexpectedValidationError - publish_storage_deals's core logic was invoked with a broken invariant that should be called by validate_deals. Please report an issue to the developers.
• DealPreconditionFailed - Due to a programmer bug. Please report an issue to the developers.

Constants

Storage Provider Pallet

Table of Contents

• Overview
• Usage
  • Declaring storage faults and recoveries
• Extrinsics
  • register_storage_provider
  • pre_commit_sector
  • prove_commit_sector
  • submit_windowed_post
  • declare_faults
  • declare_faults_recovered
• Events
• Errors
• Pallet constants

Overview

The Storage Provider Pallet handles the creation of storage providers and facilitates storage providers and clients in creating storage deals. Storage providers must provide the Proof of Space-time (PoSt) and the Proof of Replication (PoRep) to the Storage Provider Pallet to prevent the pallet from imposing penalties on storage providers through slashing.

Usage

Declaring storage faults and recoveries

Faulty sectors are subject to penalties. To minimize said penalties, the storage provider should declare any sector for which they cannot generate a PoSt as faulty, this will mask said sectors in future deadlines, minimizing the suffered penalties. A storage provider must declare the sector as faulty before the challenge window.

Through the declare_faults and declare_faults_recovered extrinsics the storage provider can declare sectors as faulty or recovered¹.

Substrate pallet hooks execute actions when certain conditions are met.

Substrate pallet hooks execute some actions when certain conditions are met. We use these hooks — when a block finalizes — to check if storage providers are up to date with their proofs. If a storage provider fails to submit proof on time, the Storage Provider pallet will signal the Market pallet to penalize the storage provider. Accordingly, removing and burning the collateral locked up during the pre-commit.

Extrinsics

register_storage_provider

Storage Provider registration is the first extrinsic any storage provider must call. Without being registered, other extrinsics will return an error.

Before a storage provider can register, they must set up a PeerId. This PeerId is used in the p2p network to connect to the storage provider.

Example

Registering a storage provider with keypair //Alice and peer ID alice with the following command²:

storagext-cli --sr25519-key "//Alice" storage-provider register alice

pre_commit_sector

After publishing a deal, the storage provider needs to pre-commit the sector information to the chain. Sectors are not valid after pre-commit. The sectors need to be proven first.

Example

Storage provider //Alice pre-committing⁴ a sector number 1, with a single deal ID 0.

storagext-cli --sr25519-key "//Alice" storage-provider pre-commit @pre-commit-sector.json

Where pre-commit-sector.json is a file with contents similar to:

{
  "sector_number": 1,
  "sealed_cid": "bafk2bzaceajreoxfdcpdvitpvxm7vkpvcimlob5ejebqgqidjkz4qoug4q6zu",
  "deal_ids": [0],
  "expiration": 100,
  "unsealed_cid": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi",
  "seal_proof": "StackedDRG2KiBV1P1"
}

prove_commit_sector

After pre-committing some new sectors the storage provider needs to supply a Proof-of-Replication for these sectors ³.

Example

This example follows up on the pre-commit example. Storage provider //Alice is proven committing⁵ sector number 1.

storagext-cli --sr25519-key "//Alice" storage-provider prove-commit @prove-commit-sector.json

Where prove-commit-sector.json is a file with contents similar to:

{
  "sector_number": 1,
  "proof": "1230deadbeef"
}

submit_windowed_post

A storage provider needs to periodically submit a Proof-of-Spacetime to prove that they are still storing the data they promised. Multiple proofs can be submitted at once.

Example

Storage provider //Alice submitting⁶ proof for deadline 0, partition 0.

storagext-cli --sr25519-key "//Alice" storage-provider submit-windowed-post @submit-windowed-post.json

Where submit-windowed-post.json is a file with contents similar to:

{
  "deadline": 0,
  "partition": [0],
  "proof": {
    "post_proof": "2KiB",
    "proof_bytes": "1230deadbeef"
  }
}

declare_faults

A storage provider can declare faults when they know they cannot submit PoSt on time to prevent getting penalized. Faults have an expiry of 42 days. The sectors will be terminated if the faults have not been recovered before this time. Multiple faults can be declared at once.

declare_faults can take in multiple fault declarations:

Where the fault declarations contain:

Example

Storage provider //Alice declaring faults⁷ on deadline 0, partition 0, sector 1.

storagext-cli --sr25519-key "//Alice" storage-provider declare-faults @fault-declaration.json

Where fault-declaration.json is a file with contents similar to:

[
  {
    "deadline": 0,
    "partition": 0,
    "sectors": [1]
  }
]

declare_faults_recovered

After declaring sectors as faulty, a storage provider can recover them. The storage provider must recover the faults if the system has marked some sectors as faulty due to a missing PoSt. Faults are not fully recovered until the storage provider submits a valid PoSt after the declare_faults_recovered extrinsic.

declare_faults_recovered can take in multiple fault recoveries:

Where the fault recoveries contain:

Example

Storage provider //Alice declaring recoveries⁸ on deadline 0, partition 0, sector 1.

storagext-cli --sr25519-key "//Alice" storage-provider declare-faults-recovered @fault-declaration.json

Where fault-declaration.json is a file with contents similar to:

[
  {
    "deadline": 0,
    "partition": 0,
    "sectors": [1]
  }
]

Events

The Storage Provider Pallet emits the following events:

• StorageProviderRegistered - Indicates that a new storage provider has been registered.
  • owner - SS58 address of the storage provider.
  • info - The static information about the new storage provider. This information includes:
    • peer_id - Libp2p identity that should be used when connecting to the storage provider.
    • window_post_proof_type - The proof type used by the storage provider for sealing sectors.
    • sector_size - Amount of space in each sector committed to the network by the storage provider.
    • window_post_partition_sectors - The number of sectors in each Window PoSt partition (proof).
• SectorPreCommitted - A storage provider has pre-committed some new sector after publishing some new deal.
  • owner - SS58 address of the storage provider.
  • sector - The sector number being pre-committed.
• SectorProven - A storage provider has proven a sector that they previously pre-committed.
  • owner - SS58 address of the storage provider.
  • sector_number - The sector number that was proven.
• SectorSlashed - A previously pre-committed sector, but not proven, has been slashed by the system because it has expired.
  • owner - SS58 address of the storage provider.
  • sector_number - The sector number that has been slashed because of expiry.
• ValidPoStSubmitted - A valid PoSt has been submitted by a storage provider.
  • owner - SS58 address of the storage provider.
• FaultsDeclared - A storage provider has declared some sectors as faulty.
  • owner - SS58 address of the storage provider.
  • faults - An array with information about the fault declarations. This information includes:
    • deadline - The deadline to which the faulty sectors are assigned.
    • partition - Partition number within the deadline containing the faulty sectors.
    • sectors - Sectors in the partition being declared as faulty.
• FaultsRecovered - A storage provider has recovered some sectors previously declared as faulty.
  • owner - SS58 address of the storage provider.
  • recoveries - An array with information about the fault recoveries. This information includes:
    • deadline - The deadline to which the recovered sectors are assigned.
    • partition - Partition number within the deadline containing the recovered sectors.
    • sectors - Sectors in the partition being declared as recovered.
• PartitionFaulty - It was detected that a storage provider has not submitted their PoSt on time and has marked some sectors as faulty.
  • owner - SS58 address of the storage provider.
  • partition - Partition number for which the PoSt was missed.
  • sectors - The sectors in the partition declared faulty by the system.

Errors

The Storage Provider Pallet actions can fail with the following errors:

• StorageProviderExists - A storage provider is already registered and tries to register again.
• StorageProviderNotFound - This error is emitted by all extrinsics except registration in the storage provider pallet when a storage provider tries to call an extrinsic without registering first.
• InvalidSector - This error can be emitted when:
  • A storage provider supplies a sector number during pre-commit exceeding the maximum number of sectors.
  • A storage provider supplies a sector number during proof commit that exceeds the maximum amount of sectors.
• InvalidProofType - This error can be emitted when:
  • A storage provider submits a seal-proof type during pre-commit that is different than the one configured during registration.
  • During a prove commit extrinsic, the proof type supplied by the storage provider is invalid.
  • A storage provider submits a windowed PoSt proof type that is different from the one configured during registration.
• NotEnoughFunds - Emitted when a storage provider does not have enough funds for the pre-commit deposit.
• SectorNumberAlreadyUsed - A storage provider tries to pre-commit a sector number that has already been used.
• ExpirationBeforeActivation - A storage provider tries to pre-commit a sector where that sector expires before activation.
• ExpirationTooSoon - A storage provider tries to pre-commit a sector with a total lifetime less than MinSectorExpiration.
• ExpirationTooLong - A storage provider tries to pre-commit a sector with an expiration that exceeds MaxSectorExpirationExtension.
• MaxSectorLifetimeExceeded - A storage provider tries to pre-commit a sector with a total lifetime that exceeds SectorMaximumLifetime.
• InvalidCid - Emitted when a storage provider submits an invalid unsealed CID when trying to pre-commit a sector.
• ProveCommitAfterDeadline - A storage provider has tried to prove a previously pre-committed sector after the proving deadline.
• PoStProofInvalid - A proof that the storage provider submitted is invalid. Currently, this error is emitted when the proof length is 0.
• InvalidUnsealedCidForSector - This error is emitted when the declared unsealed_cid for pre_commit is different from the one calculated by the system.
• FaultDeclarationTooLate - A fault declaration was submitted after the fault declaration cutoff. The fault declaration can be submitted after the upcoming deadline is closed.
• FaultRecoveryTooLate - A fault recovery was submitted after the fault recovery cutoff. The fault recovery can be submitted after the upcoming deadline is closed.
• DeadlineError - An error was encountered in the deadline module. Please report an issue to the developers.
• PartitionError - An error was encountered in the partition module. Please report an issue to the developers.
• StorageProviderError - An error was encountered in the storage provider module. Please report an issue to the developers.
• SectorMapError - An error was encountered in the sector map module. These errors can be:
  • FailedToInsertSector - Internal bounds violation with Sectors. Please report an issue to the developers.
  • FailedToInsertPartition - Internal bounds violation with partitions. Please report an issue to the developers.
• CouldNotActivateSector - Failure during prove commit when trying to convert a previously pre-committed sector due to a programming error. Please report an issue to the developers.
• CouldNotVerifySectorForPreCommit - Failure during pre-commit due to the commd calculation failing due to a programming error. Please report an issue to the developers.
• SlashingFailed - Slashing of funds fails due to a programmer error. Please report an issue to the developers.
• ConversionError - Due to a programmer error. Please report an issue to the developers.

Pallet constants

The Storage Provider Pallet has the following constants:

Glossary and Anti-Glossary

This document provides definitions and explanations for terms used throughout the project and a list of terms that should not be used.

Table of Contents

• Glossary and Anti-Glossary
  • Table of Contents
  • Glossary
    • Actor
    • Bond
    • Collateral
    • Collator
    • Committed Capacity
    • Commitment of Data
    • Commitment of Replication
    • Crowdloan
    • Deadline
    • Extrinsics
    • Fault
    • Full Node
    • Invulnerable
    • Node
    • Parachain
    • Partition
    • Planck
    • Polkadot
  • Proofs
    • Proving Period
    • Relay Chain
    • Sector
    • Session
    • Slashing
    • Slot Auction
    • Staking
    • Storage Provider
    • Storage User
    • System Parachain
  • Anti-Glossary
    • Term to Avoid: Miner
    • Term to Avoid: Pledge

Glossary

This section lists terms used throughout the project.

Actor

In Filecoin, an actor is an on-chain object with its state and set of methods. Actors define how the Filecoin network manages and updates its global state.

Bond

This term is used in:

• Parachain Slot Auction. To bid in an auction, parachain teams agree to lock up (or bond) a portion of DOT tokens for the duration of the lease. While bonded for a lease, the DOT cannot be used for other activities like staking or transfers.

• Collator slot auction (selection mechanism). It is used as a deposit to become a collator. Candidates can register by placing the minimum bond. Then, if an account wants to participate in the collator slot auction, they have to replace an existing candidate by placing a more significant deposit (bond).

Collateral

Collaterals are assets locked up or deposited as a form of security to mitigate risks and ensure the performance of specific actions. Collateral acts as a guarantee that an individual will fulfil their obligations. Failing to meet obligations or behaving maliciously can result in the loss of staked assets or collateral as a penalty for non-compliance or misconduct by slashing.

Collator

Collators maintain parachains by collecting parachain transactions from users and producing state transition proofs for Relay Chain validators. In other words, collators maintain parachains by aggregating parachain transactions into parachain block candidates and producing state transition proofs (Proof-of-Validity, PoV) for validators. They must provide a financial commitment (collateral) to ensure they are incentivized to perform their duties correctly and to dissuade malicious behaviour.

Committed Capacity

The Committed Capacity (CC) is one of three types of deals in which there is effectively no deal, and the Storage Provider stores random data inside the sector instead of customer data.

If a storage provider doesn't find any available deal proposals appealing, they can alternatively make a capacity commitment, filling a sector with arbitrary data, rather than with client data. Maintaining this sector allows the storage provider to provably demonstrate that they are reserving space on behalf of the network.

Commitment of Data

This value is also known as commD or unsealed_cid. As the storage miner receives each piece of client data, they place it into a sector. Sectors are the fundamental units of storage in Filecoin, and can contain pieces from multiple deals and clients.

Once a sector is full, a CommD (Commitment of Data, aka UnsealedSectorCID) is produced, representing the root node of all the piece CIDs contained in the sector.

Commitment of Replication

The terms commR, sealed_cid, commitment of replication are interchangeable. During sealing, the sector data (identified by the CommD) is encoded through a sequence of graph and hashing processes to create a unique replica. The root hash of the merkle tree of the resulting replica is the CommRLast.

The CommRLast is then hashed together with the CommC (another merkle root output from Proof of Replication). This generates the CommR (Commitment of Replication, aka SealedSectorCID), which is recorded to the public blockchain. The CommRLast is saved privately by the miner for future use in Proof of Spacetime, but is not saved to the chain.

Crowdloan

Projects can raise DOT tokens from the community through crowdloans. Participants pledge their DOT tokens to help the project win a parachain slot auction. If successful, the tokens are locked up for the duration of the parachain lease, and participants might receive rewards or tokens from the project in return.

Deadline

A deadline is one of the multiple points during a proving period when proofs for some partitions are due.

For more information on deadlines, read the original Filecoin specification: https://spec.filecoin.io/#section-algorithms.pos.post.design

Extrinsics

From the Polkadot Wiki:

Within each functional pallet on the blockchain, one can call its functions and execute them successfully, provided they have the permission to do so. Because these calls originate outside of the blockchain runtime, such transactions are referred to as extrinsics.

Fault

A fault happens when a proof is not submitted within the proving period. For a sector to stop being considered in proving periods, it needs to be declared as faulty — indicating the storage provider is aware of the faulty sector and will be working to restore it. If a sector is faulty for too long, it will be terminated and the deal will be slashed.

For more information on faults, read the original Filecoin specification: https://spec.filecoin.io/#section-glossary.fault

Full Node

A device (computer) that fully downloads and stores the entire blockchain of the parachain, validating and relaying transactions and blocks within the network. It is one of the node types.

Invulnerable

A status assigned to certain collators that makes them exempt from being removed from the active set of collators.

Node

A device (computer) that participates in running the protocol software of a decentralized network; in other words, a participant of the blockchain network who runs it locally.

Parachain

A parachain is a specialized blockchain that runs in parallel to other parachains within a larger network, benefiting from shared security and interoperability, and can be validated by the validators of the Relay Chain.

Partition

Partitions are logical groups¹ of sectors to be proven together.

The number of sectors to be proven at once is 2349², as defined by Filecoin.

For more information on partitions, read the original Filecoin specification: https://spec.filecoin.io/#section-algorithms.pos.post.constants--terminology

Planck

From the Polkadot Wiki:

The smallest unit for the account balance on Substrate based blockchains (Polkadot, Kusama, etc.) is Planck (a reference to Planck Length, the smallest possible distance in the physical Universe). DOT's Planck is like BTC's Satoshi or ETH's Wei. Polkadot's native token DOT equals to \(10^{10}\) Planck and Kusama's native token KSM equals to \(10^{12}\) Planck.

Polkadot

“Layer-0” blockchain platform designed to facilitate interoperability, scalability and security among different “Layer-1” blockchains, called parachains.

Proofs

Cryptographic evidence used to verify that storage providers have received, are storing, and are continuously maintaining data as promised.

There are two main types of proofs:

• Proof-of-Replication (PoRep): In order to register a sector with the network, the sector has to be sealed. Sealing is a computation-heavy process that produces a unique representation of the data in the form of a proof, called Proof-of-Replication or PoRep.

• Proof-of-Spacetime (PoSt): Used to verify that the storage provider continues to store the data over time. Storage providers must periodically generate and submit proofs to show that they are still maintaining the stored data as promised.

Proving Period

A proving period is when storage providers' commitments are audited, and they must prove they are still storing the data from the deals they signed

• the average period for proving all sectors maintained by a provider (default set to 24 hours).

For more information on proving periods, read the original Filecoin specification:

• Proving periods in the context of Window Proof of Spacetime — https://spec.filecoin.io/#section-algorithms.pos.post.windowpost
• Proving periods in the context of Filecoin's system design — https://spec.filecoin.io/#section-algorithms.pos.post.design

Relay Chain

The Relay Chain in Polkadot is the central chain (blockchain) responsible for the network's shared security, consensus, and cross-chain interoperability.

Sector

The sector is the default unit of storage that providers put in the network. A sector is a contiguous array of bytes on which a storage provider puts together, seals,and performs Proofs of Spacetime on. Storage providers store data on the network in fixed-size sectors.

For more information on sectors, read the original Filecoin specification: https://spec.filecoin.io/#section-glossary.sector

Session

A predefined period during which a set of collators remains constant.

Slashing

The process of penalizing network participants, including validators, nominators, and collators, for various protocol violations. These violations could include producing invalid blocks, equivocation (double signing), inability of the Storage Provider to prove that the data is stored and maintained as promised, or other malicious activities. As a result of slashing, participants may face a reduction in their staked funds or other penalties depending on the severity of the violation.

Slot Auction

To secure a parachain slot, a project must win an auction by pledging (locking up) a significant amount of DOT tokens. These tokens are used as collateral to secure the slot for a specified period. Once the slot is secured, the project can launch and operate its parachain.

Staking

Staking is when DOT holders lock up their tokens to support the network's security and operations. In return, they can earn rewards. There are two main roles involved in staking:

• Validators: Validators produce new blocks, validate transactions, and secure the network. They are selected based on their stake and performance. Validators must run a node and have the technical capability to maintain it.

• Nominators: Nominators support the network by backing (nominating) validators they trust with their DOT tokens. Nominators share in the rewards earned by the validators they support. This allows DOT holders who don't want to run a validator node to still participate in the network's security and earn rewards.

Our parachain will use staking to back up the collators similarly to "Nominators". In this regard, "Nominators" will fall to Storage Providers, while " Validators" will be assigned to Collators accordingly.

Storage Provider

The user who offers storage space on their devices to store data for others.

Storage User

Aka Client: The user who initiates storage deals by providing data to be stored on the network by the Storage Provider.

System Parachain

System-level chains move functionality from the Relay Chain into parachains, minimizing the administrative use of the Relay Chain. For example, a governance parachain could move all the Polkadot governance processes from the Relay Chain into a parachain.

Anti-Glossary

This section lists terms that should not be used within the project, along with preferred alternatives.

Term to Avoid: Miner

In Filecoin, a "Lotus Miner" is responsible for storage-related operations, such as sealing sectors (PoRep (Proof-of-Replication)), proving storage (PoSt (Proof-of-Spacetime)), and participating in the Filecoin network as a storage miner.

Reason: In the Filecoin network, the miner simultaneously plays the roles of storage provider and block producer. However, this term cannot be used in the Polkadot ecosystem because there are no block producers in parachains; the Relay Chain is responsible for block production. Parachains can only prepare block candidates via the Collator node and pass them to the Relay Chain.

Term to Avoid: Pledge

It's better to apply this term within its proper context rather than avoiding it altogether. It's easy to confuse it with staking, but they have distinct meanings.

Reason: Pledging generally refers to locking up tokens as collateral to participate in certain network activities or services like: Parachain Slot Auctions and Crowdloans.