Singularity logo

Singularity

An end-to-end solution for data onboarding to Filecoin.

Facilitating petabyte-scale data ingestion for storage clients and providers.

Features

Simple data processing

Singularity automates data preparation by automatically chunking your data and generating CAR files.

Configurable and modular

Pick-and-choose the components you want. Process local data or point to a public S3 bucket.

Robust deal execution

Customize the deal execution strategy: deal with your preferred storage providers with a schedule that works.

Indexing and retrieval

Data is automatically indexed on IPFS. Retrievals are done with a simple command.

Singularity is an open-source tool designed by @Xinan Xu and @Fei Yan and maintained by the Data Onboarding team at Protocol Labs. Our goal is to develop tools that help data owners and storage providers ingest massive amounts of data to the Filecoin network.

It is a easy-to-use NPM package that works with Lotus and Boost to tie together every step of data onboarding. 

How it works

1. Prepare the data

The data preparation module will scan and convert a local folder recursively into CAR files ready to be onboarded to Filecoin Network. Singularity also indexes your files on IPFS for easy retrieval.

2. Distribute the CAR files

Easily host an HTTP server with the CAR files to be downloaded by the storage provider. You can also send the CAR files offline if the files are too large.

3. Pick your storage providers

Use a solution for finding storage providers or reach out on Slack. Most storage providers today are interested in taking verified deals, so you need to familiarize yourself with Filecoin Plus.

4. Make storage deals

Leverage Lotus and Boost to automatically make deals with the storage providers of your choice.