-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: a new ChainIndexer to index tipsets, messages and events #12421
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions
Co-authored-by: Rod Vagg <rod@vagg.org>
…issing and not backfill parents (#12619) * fix backfilling UX * Update chain/index/api.go Co-authored-by: Rod Vagg <rod@vagg.org> * address review --------- Co-authored-by: Rod Vagg <rod@vagg.org>
@@ -1,6 +1,13 @@ | |||
# Lotus changelog | |||
|
|||
# UNRELEASED | |||
(`ChainIndexer`) to index Filecoin chain state such as tipsets, messages, events and ETH transactions for accurate and faster RPC responses. The `ChainIndexer` replaces the existing `MsgIndex`, `EthTxIndex` and `EventIndex` implementations in Lotus, which [suffer from a multitude of known problems](https://github.com/filecoin-project/lotus/issues/12293). If you are an RPC provider/node operator, please refer to the [ChainIndexer documentation for RPC providers](TODO: URL) for information on how to enable, configure and use the new Indexer. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
blocking merge with this comment because this needs to be updated by the docs PR
For #12453.
Subsumes #12388 to implement the
ChainIndexer
.TODO
ChainIndexer
(tests are at feat: tests for the chain indexer #12521)Migration
Index "migration" will effectively involve re-indexing the chain state here. This is because the data in the old Indices isn't reliable (we're already aware of multiple bugs in the old Indices which motivated this PR in the first place) and the old Indices have never been gc'd/kept in sync with the chain state on nodes with splitstore enabled.
For Snapshot synced nodes with Splitstore enabled
ReConcileEmptyIndex
to true and setting theMaxReconcileTipsets
as needed.For archival nodes
If archival nodes want to backfill the index as part of daemon startup, they can always bootup the node with
ReConcileEmptyIndex
set to true andMaxReconcileTipsets
set to infinity. This will drastically slow down the node startup but will ensure that the node starts up with it's entire history indexed.The other option for users is to use the
Backfill
RPC API with a range of height they want backfilled. However, we need to handle race conditions here where backfilling races with new tipsets coming in and need to ensure that backfilling does not end up overwriting updates caused by new tipsets being indexed.Sqllite Query Plans
For looking up
MsgInfo
reverted != 0
and only one withreverted=0
for a givenmessage_cid
so no need to index byreverted
as well here.For looking up an eth tx hash
For checking if a tipset exists
For getting the minimum no-reverted height for reconciliation
reverted != 0
and only one entry withreverted=0
, this should be allright.Looking up a
message_id
to insert an eventLooking up events at a specific height
Looking up events in a height range
Looking up events for a specific tipset