4.0.0
⚠ Breaking Changes
run --oneshot
option is removed. The oneshot mode (DipDup stops after the sync is finished) applies automatically whenlast_level
field is set in the index config.clear-cache
command is removed. Usecache clear
instead.
Migration from 3.x
- Run
dipdup init
command to generateon_synchronized
hook stubs. - Run
dipdup schema approve
command on every database you want to use with 4.0.0. Runningdipdup migrate
is not necessary sincespec_version
hasn't changed in this release.
What's New
Performance optimizations
Overall indexing performance has been significantly improved. Key highlights:
- Configuration files are loaded 10x times faster. The more indexes in the project, the more noticeable difference is.
- Significantly reduced CPU usage in realtime mode.
- Datasource default HTTP connection options optimized for a reasonable balance between resource consumption and indexing speed.
Also, two new flags were added to improve DipDup performance in several scenarios: merge_subscriptions
and early_relatime
. See this paragraph for details.
Configurable action on reindex
There are several reasons that trigger reindexing:
reason | description |
---|---|
manual | Reindexing triggered manually from callback with ctx.reindex . |
migration | Applied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared. |
rollback | Reorg message received from TzKT, and can not be processed. |
config_modified | One of the index configs has been modified. |
schema_modified | Database schema has been modified. Try to avoid manual schema modifications in favor of SQL hooks. |
Now it is possible to configure desirable action on reindexing triggered by the specific reason.
action | description |
---|---|
exception (default) | Raise ReindexingRequiredError and quit with error code. The safest option since you can trigger reindexing accidentally, e.g., by a typo in config. Don't forget to set up the correct restart policy when using it with containers. |
wipe | Drop the whole database and start indexing from scratch. Be careful with this option! |
ignore | Ignore the event and continue indexing as usual. It can lead to unexpected side-effects up to data corruption; make sure you know what you are doing. |
To configure actions for each reason, add the following section to DipDup config:
advanced:
...
reindex:
manual: wipe
migration: exception
rollback: ignore
config_modified: exception
schema_modified: exception
New CLI commands and flags
command or flag | description |
---|---|
cache show | Get information about file caches used by DipDup. |
config export | Print config after resolving all links and variables. Add --unsafe option to substitute environment variables; default values from config will be used otherwise. |
run --early-realtime | Establish a realtime connection before all indexes are synchronized. |
run --merge-subscriptions | Subscribe to all operations/big map diffs during realtime indexing. This flag helps to avoid reaching TzKT subscriptions limit (currently 10000 channels). Keep in mind that this option could significantly improve RAM consumption depending on the time required to perform a sync. |
status | Print the current status of indexes from the database. |
advanced
top-level config section
This config section allows users to tune system-wide options, either experimental or unsuitable for generic configurations.
field | description |
---|---|
early_realtime merge_subscriptions postpone_jobs | Another way to set run command flags. Useful for maintaining per-deployment configurations. |
reindex | Configure action on reindexing triggered. See this paragraph for details. |
CLI flags have priority over self-titled AdvancedConfig
fields.
aiosignalrcore
replaced with pysignalr
It may not be the most noticeable improvement for end-user, but it still deserves a separate paragraph in this article.
Historically, DipDup used our own fork of signalrcore
library named aiosignalrcore
. This project aimed to replace the synchronous websocket-client
library with asyncio-ready websockets
. Later we discovered that required changes make it hard to maintain backward compatibility, so we have decided to rewrite this library from scratch. So now you have both a modern and reliable library for SignalR protocol and a much more stable DipDup. Ain't it nice?
Changes since 3.1.3
This is a combined changelog of -rc versions released since the last stable release until this one.
Added
- cli: Added
run --early-realtime
flag to establish a realtime connection before all indexes are synchronized. - cli: Added'run --merge-subscriptions` flag to subscribe to all operations/big map diffs during realtime indexing.
- cli: Added
status
command to print the current status of indexes from the database. - cli: Added
config export [--unsafe]
command to print config after resolving all links and variables. - cli: Added
cache show
command to get information about file caches used by DipDup. - config: Added
first_level
andlast_level
optional fields toTemplateIndexConfig
. These limits are applied after ones from the template itself. - config: Added
daemon
boolean field toJobConfig
to run a single callback indefinitely. Conflicts withcrontab
andinterval
fields. - config: Added
advanced
top-level section. - hooks: Added
on_synchronized
hook, which fires each time all indexes reach realtime state.
Fixed
- cli: Fixed config not being verified when invoking some commands.
- cli: Fixed crashes and output inconsistency when piping DipDup commands.
- cli: Fixed missing
schema approve --hashes
argument. - cli: Fixed
schema wipe --immune
flag being ignored. - codegen: Fixed contract address used instead of an alias when typename is not set.
- codegen: Fixed generating callback arguments for untyped operations.
- codegen: Fixed missing imports in handlers generated during init.
- coinbase: Fixed possible data inconsistency caused by caching enabled for method
get_candles
. - hasura: Fixed unnecessary reconfiguration in restart.
- http: Fixed increasing sleep time between failed request attempts.
- index: Fixed
CallbackError
raised instead ofReindexingRequiredError
in some cases. - index: Fixed crash while processing storage of some contracts.
- index: Fixed incorrect log messages, remove duplicate ones.
- index: Fixed invocation of head index callback.
- index: Fixed matching of untyped operations filtered by
source
field (@pravin-d). - tzkt: Fixed filtering of big map diffs by the path.
- tzkt: Fixed
get_originated_contracts
andget_similar_contracts
methods whose output was limited toHTTPConfig.batch_size
field. - tzkt: Fixed lots of SignalR bugs by replacing
aiosignalrcore
library withpysignalr
. - tzkt: Fixed processing operations with entrypoint
default
. - tzkt: Fixed regression in processing migration originations.
- tzkt: Fixed resubscribing when realtime connectivity is lost for a long time.
- tzkt: Fixed sending useless subscription requests when adding indexes in runtime.
Changed
- cli:
schema wipe
command now requires confirmation when invoked in the interactive shell. - cli:
schema approve
command now also causes a recalculation of schema and index config hashes. - index: DipDup will recalculate respective hashes if reindexing is triggered with
config_modified: ignore
orschema_modified: ignore
in advanced config.
Removed
- cli: Removed deprecated
run --oneshot
argument andclear-cache
command.
Performance
- config: Configuration files are loaded 10x times faster.
- index: Checks performed on each iteration of the main DipDup loop are slightly faster now.
- index: Number of operations processed by matcher reduced by 40%-95% depending on the number of addresses and entrypoints used.
- tzkt: Improved performance of response deserialization.
- tzkt: Rate limit was increased. Try to set
connection_timeout
to a higher value if requests fail withConnectionTimeout
exception.