Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: a new ChainIndexer to index tipsets, messages and events #12421

Open
wants to merge 118 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
118 commits
Select commit Hold shift + click to select a range
fb19d79
chain index complete for msgs and txns
aarshkshah1992 Aug 29, 2024
58569d6
dont need observer changes for now
aarshkshah1992 Aug 29, 2024
5a3f76f
changes
aarshkshah1992 Aug 29, 2024
9ea48f3
fix tests
aarshkshah1992 Aug 29, 2024
1e3a9d5
fix tests
aarshkshah1992 Aug 29, 2024
4c34bc7
use th right context
aarshkshah1992 Aug 29, 2024
285ce26
index empty tipsets correctly
aarshkshah1992 Aug 30, 2024
12e67fe
implement automated backfilling
aarshkshah1992 Aug 30, 2024
3377987
add event indexing and remove all old indices
aarshkshah1992 Sep 4, 2024
e6331da
fix test
aarshkshah1992 Sep 4, 2024
f1f24c8
revert deployment test changes
aarshkshah1992 Sep 5, 2024
3f09e1e
revert test changes and better error handling for eth tx index lookups
aarshkshah1992 Sep 5, 2024
c3865b7
fix sql statments naming convention
aarshkshah1992 Sep 5, 2024
8ec7cd2
address review for Index GC
aarshkshah1992 Sep 5, 2024
a1c5201
more changes as per review
aarshkshah1992 Sep 5, 2024
e0831ee
changes as per review
aarshkshah1992 Sep 5, 2024
06ca87a
fix config
aarshkshah1992 Sep 5, 2024
c30079e
mark events as reverted during reconciliation
aarshkshah1992 Sep 5, 2024
01d78e3
better reconciliation; pens down and code complete; also reconcile ev…
aarshkshah1992 Sep 6, 2024
994a717
fix tests
aarshkshah1992 Sep 6, 2024
ad9bcb1
improve config and docs
aarshkshah1992 Sep 6, 2024
0c9a0ca
improve docs and error handling
aarshkshah1992 Sep 6, 2024
89cedb2
improve read logic
aarshkshah1992 Sep 6, 2024
a2a2b76
improve docs
aarshkshah1992 Sep 6, 2024
44af9f8
better logging and handle ennable event storage
aarshkshah1992 Sep 9, 2024
6608b80
improve logs and index init proc
aarshkshah1992 Sep 9, 2024
625d8c8
better logging
aarshkshah1992 Sep 9, 2024
fed08b0
fix bugs based on calibnet testing
aarshkshah1992 Sep 9, 2024
a5c56c1
create sqliite Indices
aarshkshah1992 Sep 10, 2024
7acd481
gc should be based on epochs
aarshkshah1992 Sep 10, 2024
821dcd4
fix event query
aarshkshah1992 Sep 10, 2024
cde46cb
foreign keys should be enabled on the DB
aarshkshah1992 Sep 10, 2024
727dae3
reverted tipsets should be removed as part of GC
aarshkshah1992 Sep 10, 2024
c07784d
release read lock
aarshkshah1992 Sep 10, 2024
896048a
make it easy to backfill an empty index using reconciliation
aarshkshah1992 Sep 10, 2024
602f660
better docs for reconciliation
aarshkshah1992 Sep 10, 2024
9750571
Merge remote-tracking branch 'origin/master' into feat/msg-eth-tx-index
aarshkshah1992 Sep 12, 2024
13c2824
fix conflicts with master
aarshkshah1992 Sep 12, 2024
37d6746
Apply suggestions from code review
aarshkshah1992 Sep 13, 2024
c4490bb
fix go mod
aarshkshah1992 Sep 13, 2024
93a8b76
fix formatting
aarshkshah1992 Sep 13, 2024
6f8530e
revert config changes
aarshkshah1992 Sep 13, 2024
627aff2
address changes in observer
aarshkshah1992 Sep 13, 2024
7244b66
remove top level chainindex package
aarshkshah1992 Sep 13, 2024
531cd38
changes as per review
aarshkshah1992 Sep 13, 2024
77fc462
changes as per review
aarshkshah1992 Sep 13, 2024
c2e5f68
changes as per review
aarshkshah1992 Sep 13, 2024
286af22
handle index with reverted tipsets during reconciliation
aarshkshah1992 Sep 13, 2024
d67a30a
changes as per review
aarshkshah1992 Sep 13, 2024
5f5ef3a
fix type of max reconcile epoch
aarshkshah1992 Sep 13, 2024
f5a5c61
changes to reconciliation as per review
aarshkshah1992 Sep 14, 2024
730d00a
log ipld error
aarshkshah1992 Sep 14, 2024
c099abf
better logging of progress
aarshkshah1992 Sep 14, 2024
951ce77
disable chain indexer hydrate from snapshot based on config
aarshkshah1992 Sep 14, 2024
ad6c086
always populate index
aarshkshah1992 Sep 14, 2024
52e104d
make config easy to reason about
aarshkshah1992 Sep 14, 2024
9c6c728
fix config
aarshkshah1992 Sep 14, 2024
1da1e07
fix messaging
aarshkshah1992 Sep 14, 2024
efe90f8
revert config changes
aarshkshah1992 Sep 14, 2024
c121321
Apply suggestions from code review
aarshkshah1992 Sep 16, 2024
2ff1d42
changes as per review
aarshkshah1992 Sep 16, 2024
c945bb5
make error messages homogenous
aarshkshah1992 Sep 16, 2024
432e09a
fix indentation
aarshkshah1992 Sep 16, 2024
af9bc23
changes as per review
aarshkshah1992 Sep 16, 2024
6d84b03
feat: recompute tipset to generate missing events if event indexing i…
aarshkshah1992 Sep 16, 2024
b9f1583
better docs for gc retention epoch
aarshkshah1992 Sep 16, 2024
1921abd
imrpove DB handling (#12485)
aarshkshah1992 Sep 19, 2024
c03c4c7
Merge branch 'master' into feat/msg-eth-tx-index
aarshkshah1992 Sep 30, 2024
b7966e3
fix conflict
aarshkshah1992 Sep 30, 2024
da76042
fix lite node config for indexer
aarshkshah1992 Oct 1, 2024
1e8f8ce
exclude reverted events from eth get logs if client queries by epoch
aarshkshah1992 Oct 11, 2024
7b304a4
Simply addressing for event lookups in the index.
aarshkshah1992 Oct 11, 2024
5e2dba4
Merge remote-tracking branch 'origin/master' into feat/msg-eth-tx-index
aarshkshah1992 Oct 14, 2024
593e724
Apply suggestions from code review
aarshkshah1992 Oct 14, 2024
aa2aa4c
Apply suggestions from code review
aarshkshah1992 Oct 14, 2024
0339c46
fix tests
aarshkshah1992 Oct 14, 2024
105d85b
Apply suggestions from code review
aarshkshah1992 Oct 14, 2024
a9788f5
feat: migration("re-indexing"), backfilling and diasgnostics tooling …
aarshkshah1992 Oct 14, 2024
fcb6012
fix lint
aarshkshah1992 Oct 14, 2024
e3be131
Apply suggestions from code review
aarshkshah1992 Oct 15, 2024
8f0ee3c
Apply suggestions from code review
aarshkshah1992 Oct 15, 2024
04ac694
remove reverted flag from RPC
aarshkshah1992 Oct 15, 2024
f92b03c
Apply suggestions from code review
aarshkshah1992 Oct 15, 2024
5b58fe5
fix testing of events and dummy chain store
aarshkshah1992 Oct 15, 2024
86a8d26
remove lotus shed commands for old Indices
aarshkshah1992 Oct 15, 2024
aff1a6c
change type of event counts to uint64
aarshkshah1992 Oct 15, 2024
fe76994
only recompute events if theyre not found
aarshkshah1992 Oct 15, 2024
361d7bf
short-circuit empty events path for older tipsets
aarshkshah1992 Oct 15, 2024
f628cb2
chain indexer must be enabled if ETH RPC is enabled
aarshkshah1992 Oct 15, 2024
c96f4ed
change name of message_id column to id in tipset_message table
aarshkshah1992 Oct 15, 2024
cb7150c
only expose SetRecomputeTipSetStateFunc
aarshkshah1992 Oct 16, 2024
5575a1d
dont block on head indexing for reading messages
aarshkshah1992 Oct 16, 2024
e2013a0
document why we're only checking for missing events for a single tipset
aarshkshah1992 Oct 16, 2024
fcda11b
document when we query for reverted events
aarshkshah1992 Oct 16, 2024
ee6e589
simplify event collection
aarshkshah1992 Oct 16, 2024
8aad8e4
Apply suggestions from code review
aarshkshah1992 Oct 16, 2024
ab4de90
fix test
aarshkshah1992 Oct 16, 2024
bc97ebe
change event_id to id in the event table
aarshkshah1992 Oct 16, 2024
06e139f
change head indexed timeout
aarshkshah1992 Oct 16, 2024
947e6d1
remove deprecated config options
aarshkshah1992 Oct 16, 2024
5887cdf
fail ETH RPC calls if ChainIndexer is disabled
aarshkshah1992 Oct 16, 2024
03627aa
fix docs
aarshkshah1992 Oct 16, 2024
b4939ae
remove the tipset key cid func from lotus shed
aarshkshah1992 Oct 16, 2024
091574c
Merge remote-tracking branch 'origin/master' into feat/msg-eth-tx-index
aarshkshah1992 Oct 16, 2024
54d6632
address review comments
aarshkshah1992 Oct 16, 2024
1135da7
Apply suggestions from code review
aarshkshah1992 Oct 17, 2024
6dd3fcc
chore(events): remove unnecessary DisableRealTimeFilterAPI (#12610)
rvagg Oct 17, 2024
16123bc
feat(cli): add --quiet to chainindex validate-backfill + cleanups (#1…
rvagg Oct 17, 2024
488202b
fix tests
aarshkshah1992 Oct 17, 2024
8e6229e
Apply suggestions from code review
aarshkshah1992 Oct 17, 2024
5836ccb
error type for disabled chainindexer
aarshkshah1992 Oct 17, 2024
d0fb6e2
Merge remote-tracking branch 'origin/master' into feat/msg-eth-tx-index
aarshkshah1992 Oct 17, 2024
70aaef4
fix(chainindex): recompute tipset when we find no receipts
rvagg Oct 18, 2024
702f1f5
fix(chainindexer): backfilling should halt when chain state data is m…
aarshkshah1992 Oct 21, 2024
bd72eec
reduce log noise
aarshkshah1992 Oct 21, 2024
c3937d7
Merge remote-tracking branch 'origin/master' into feat/msg-eth-tx-index
aarshkshah1992 Oct 21, 2024
0311749
make jen
aarshkshah1992 Oct 21, 2024
fe16f30
make jen
aarshkshah1992 Oct 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
# Lotus changelog

# UNRELEASED
(`ChainIndexer`) to index Filecoin chain state such as tipsets, messages, events and ETH transactions for accurate and faster RPC responses. The `ChainIndexer` replaces the existing `MsgIndex`, `EthTxIndex` and `EventIndex` implementations in Lotus, which [suffer from a multitude of known problems](https://github.com/filecoin-project/lotus/issues/12293). If you are an RPC provider/node operator, please refer to the [ChainIndexer documentation for RPC providers](TODO: URL) for information on how to enable, configure and use the new Indexer.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocking merge with this comment because this needs to be updated by the docs PR

Add `EthGetBlockReceipts` RPC method to retrieve transaction receipts for a specified block. This method allows users to obtain Ethereum format receipts of all transactions included in a given tipset as specified by its Ethereum block equivalent. ([filecoin-project/lotus#12478](https://github.com/filecoin-project/lotus/pull/12478))


## ☢️ Upgrade Warnings ☢️

- Minimum go-version been updated to v1.22.7 ([filecoin-project/lotus#12459](https://github.com/filecoin-project/lotus/pull/12459))

## New features
- Update `EthGetBlockByNumber` to return a pointer to ethtypes.EthBlock or nil for null rounds. ([filecoin-project/lotus#12529](https://github.com/filecoin-project/lotus/pull/12529))
Expand Down
31 changes: 31 additions & 0 deletions api/api_full.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,37 @@ type FullNode interface {
Common
Net

// MethodGroup: ChainIndexer
// The ChainIndexer method group contains methods for interacting with the chain indexer.

// ChainValidateIndex validates the integrity of and optionally backfills
// the chain index at a specific epoch.
//
// It can be used to:
//
// 1. Validate the chain index at a specific epoch:
// - Ensures consistency between indexed data and actual chain state
// - Reports any errors found during validation (i.e. the indexed data does not match the actual chain state, missing data, etc.)
//
// 2. Optionally backfill missing data:
// - Backfills data if the index is missing information for the specified epoch
// - Backfilling only occurs when the `backfill` parameter is set to `true`
//
// 3. Detect "holes" in the index:
// - If `backfill` is `false` and the index lacks data for the specified epoch, the API returns an error indicating missing data
//
// Parameters:
// - epoch: The specific chain epoch for which to validate/backfill the index.
// - backfill: A boolean flag indicating whether to attempt backfilling of missing data if the index does not have data for the
// specified epoch.
//
// Returns:
// - *types.IndexValidation: A pointer to an IndexValidation struct containing the results of the validation/backfill.
// - error: An error object if the validation/backfill fails. The error message will contain details about the index
// corruption if the call fails because of an incosistency between indexed data and the actual chain state.
// Note: The API returns an error if the index does not have data for the specified epoch and backfill is set to false.
ChainValidateIndex(ctx context.Context, epoch abi.ChainEpoch, backfill bool) (*types.IndexValidation, error) //perm:write

// MethodGroup: Chain
// The Chain method group contains methods for interacting with the
// blockchain, but that do not require any form of state computation.
Expand Down
15 changes: 15 additions & 0 deletions api/mocks/mock_full.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions api/proxy_gen.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading