4.1.0
Migration from 4.0 (optional)
- Run
dipdup schema init
on the existing database to enabledipdup_head_status
view and REST endpoint.
What's New
Index only the current state of big maps
big_map
indexes allow achieving faster processing times than operation
ones when storage updates are the only on-chain data your dapp needs to function. With this DipDup release, you can go even further and index only the current storage state, ignoring historical changes.
indexes:
foo:
kind: big_map
...
skip_history: never|once|always
When this option is set to once
, DipDup will skip historical changes only on initial sync and switch to regular indexing afterward. When the value is always
, DipDup will fetch all big map keys on every restart. Preferrable mode depends on your workload.
All big map diffs DipDup pass to handlers during fast sync have action
field set to BigMapAction.ADD_KEY
. Keep in mind that DipDup fetches all keys in this mode, including ones removed from the big map. If needed, you can filter out the latter by BigMapDiff.data.active
field.
New datasource for contract and token metadata
Since the first version DipDup allows to fetch token metadata from Better Call Dev API with bcd
datasource. Now it's time for a better solution. Firstly, BCD is far from being reliable in terms of metadata indexing. Secondly, spinning up your own instance of BCD requires significant effort and computing power. Lastly, we plan to deprecate Better Call Dev API soon (but do not worry - it won't affect the explorer frontend).
Luckily, we have dipdup-metadata, a standalone companion indexer for DipDup written in Go. Configure a new datasource in the following way:
datasources:
metadata:
kind: metadata
url: https://metadata.dipdup.net
network: mainnet|ghostnet|limanet
Now you can use it anywhere in your callbacks:
datasource = ctx.datasources['metadata']
token_metadata = await datasource.get_token_metadata(address, token_id)
bcd
datasource will remain available for a while, but we discourage using it for metadata processing.
Nested packages for hooks and handlers
Callback modules are no longer have to be in top-level hooks
/handlers
directories. Add one or multiple dots to the callback name to define nested packages:
package: indexer
hooks:
foo.bar:
callback: foo.bar
After running init
command, you'll get the following directory tree (shortened for readability):
indexer
├── hooks
│ ├── foo
│ │ ├── bar.py
│ │ └── __init__.py
│ └── __init__.py
└── sql
└── foo
└── bar
└── .keep
The same rules apply to handler callbacks. Note that callback
field must be a valid Python package name - lowercase letters, underscores, and dots.
New CLI commands and flags
-
schema init
is a new command to prepare a database for running DipDip. It will create tables based on your models, then callon_reindex
SQL hook to finish preparation - the same things DipDup does when run on a clean database. -
hasura configure --force
flag allows to configure Hasura even if metadata hash matches one saved in database. It may come in handy during development. -
init --keep-schemas
flag makes DipDup preserve contract JSONSchemas. Usually, they are removed after generating typeclasses withdatamodel-codegen
, but you can keep them to convert to other formats or troubleshoot codegen issues.
Built-in dipdup_head_status
view and REST endpoint
DipDup maintains several internal models to keep its state. As Hasura generates GraphQL queries and REST endpoints for those models, you can use them for monitoring. However, some SaaS monitoring solutions can only check whether an HTTP response contains a specific word or not. For such cases dipdup_head_status
view was added - a simplified representation of dipdup_head
table. It returns OK
when datasource received head less than two minutes ago and OUTDATED
otherwise. Latter means that something's stuck, either DipDup (e.g., because of database deadlock) or TzKT instance. Or maybe the whole Tezos blockchain, but in that case, you have problems bigger than indexing.
$ curl "http://127.0.0.1:41000/api/rest/dipdupHeadStatus?name=https%3A%2F%2Fapi.tzkt.io"
{"dipdupHeadStatus":[{"status":"OUTDATED"}]}%
Note that dipdup_head
update may be delayed during sync even if the --early-realtime
flag is enabled, so don't rely exclusively on this endpoint.
Changelog
Added
- cli: Added
schema init
command to initialize database schema. - cli: Added
--force
flag tohasura configure
command. - codegen: Added support for subpackages inside callback directories.
- hasura: Added
dipdup_head_status
view and REST endpoint. - index: Added an ability to skip historical data while synchronizing
big_map
indexes. - metadata: Added
metadata
datasource. - tzkt: Added
get_big_map
andget_contract_big_maps
datasource methods.