DragonX/dragonx
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

IBD/sync speedups: parallel RandomX pre-verify, adaptive dbcache, P2P download fixes

Browse Source 
- Parallel RandomX PoW pre-verification pool (CCheckQueue) run ahead of the serial
  connect; consensus-neutral (inline CheckRandomXSolution fallback still verifies
  anything not pre-verified). New -randomxverifythreads (default = -par).
- Adaptive dbcache: default sizes the UTXO/coins cache to most of RAM and shrinks
  under memory pressure, always leaving a reserve free; -dbcache pins a fixed value.
- P2P block download: bounded socket recv-drain loop (tlsmanager); frontier-block
  reassignment to break head-of-line stalls (-blockreassigntimeout); ProcessGetData
  serves a bounded batch of blocks per pass instead of one (fixes the serve-side
  one-block-per-tick throttle that caps download network-wide).
- assumeutxo: dumptxoutset RPC + LoadSnapshot machinery + AssumeutxoData chainparams.
- Signed bootstrap verification (util/bootstrap-dragonx.sh, util/sign-bootstrap.md).
- gtest: RandomX pre-verify consensus-equivalence test + UTXO-snapshot round-trip;
  revived the gtest harness (Makefile.am include fix, Makefile.gtest.include).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
DanS
2026-06-19 12:30:10 -05:00
parent 2b011d6ee2
commit 1673cfb6dc
18 changed files with 1599 additions and 154 deletions
Download Patch File Download Diff File Expand all files Collapse all files

217
src/pow.cpp
View File

@@ -18,6 +18,7 @@
* *
******************************************************************************/
#include "pow.h"
#include "checkpoints.h"
#include "consensus/upgrades.h"
#include "arith_uint256.h"
#include "chain.h"
@@ -30,6 +31,8 @@
#include "sodium.h"
#include "RandomX/src/randomx.h"
#include <mutex>
#include <boost/thread/shared_mutex.hpp>
#include <boost/thread/locks.hpp>
#ifdef ENABLE_RUST
#include "librustzcash.h"
@@ -704,6 +707,7 @@ static std::mutex cs_randomx_validator;
static randomx_cache *s_rxCache = nullptr;
static randomx_vm *s_rxVM = nullptr;
static std::string s_rxCurrentKey; // tracks current key to avoid re-init
static int64_t nTimeRandomX = 0; // cumulative RandomX validation time (us), reported under -debug=bench
// Thread-local flag: skip CheckRandomXSolution when the miner is validating its own block
// The miner already computed the correct RandomX hash — re-verifying with a separate
@@ -714,26 +718,70 @@ void SetSkipRandomXValidation(bool skip) { fSkipRandomXValidation = skip; }
CBlockIndex *hush_chainactive(int32_t height);
bool CheckRandomXSolution(const CBlockHeader *pblock, int32_t height)
// Centralized predicate: does a block at this height actually require a RandomX hash check?
// Shared by CheckRandomXSolution (inline path) and the parallel pre-verify pool so the two can
// never drift. Returns false when the recompute is unnecessary:
// - non-RandomX chain, or RandomX validation disabled (activation height < 0)
// - below the RandomX activation height (those blocks used Equihash, validated elsewhere)
// - during initial on-disk block loading / reindex (HUSH_LOADINGBLOCKS)
// - below the last hardcoded checkpoint (chain pinned by checkpoint hash + linkage + work)
// Deliberately does NOT consider the thread-local fSkipRandomXValidation (miner self-check) — that
// is a property of the calling thread, handled only in the inline CheckRandomXSolution below.
bool RandomXValidationRequired(int32_t height)
{
// Only applies to RandomX chains
if (ASSETCHAINS_ALGO != ASSETCHAINS_RANDOMX)
return true;
// Disabled if activation height is negative
return false;
if (ASSETCHAINS_RANDOMX_VALIDATION < 0)
return true;
// Not yet at activation height
return false;
if (height < ASSETCHAINS_RANDOMX_VALIDATION)
return true;
// Do not affect initial block loading
return false;
extern int32_t HUSH_LOADINGBLOCKS;
if (HUSH_LOADINGBLOCKS != 0)
return false;
extern bool fCheckpointsEnabled;
if (fCheckpointsEnabled && height < Checkpoints::GetTotalBlocksEstimate(Params().Checkpoints()))
return false;
return true;
}
// Serialize the RandomX hash input: the block header without nSolution (but with nNonce). Used by
// both the inline CheckRandomXSolution and the parallel pre-verify pool, so the bytes are identical.
std::vector<unsigned char> GetRandomXInput(const CBlockHeader& block)
{
CRandomXInput rxInput(block);
CDataStream ss(SER_NETWORK, PROTOCOL_VERSION);
ss << rxInput;
return std::vector<unsigned char>(ss.begin(), ss.end());
}
// Derive the RandomX key string for a block at `height`. Below interval+lag it is the chain-params
// initial key; otherwise the block hash at the key-rotation height. MUST be called under cs_main
// (reads chainActive via hush_chainactive). Returns empty if the key-height block is unavailable.
std::string GetRandomXKey(int32_t height)
{
static int randomxInterval = GetRandomXInterval();
static int randomxBlockLag = GetRandomXBlockLag();
if (height < randomxInterval + randomxBlockLag) {
char initialKey[82];
snprintf(initialKey, 81, "%08x%s%08x", ASSETCHAINS_MAGIC, SMART_CHAIN_SYMBOL, ASSETCHAINS_RPCPORT);
return std::string(initialKey, strlen(initialKey));
}
int keyHeight = ((height - randomxBlockLag) / randomxInterval) * randomxInterval;
CBlockIndex *pKeyIndex = hush_chainactive(keyHeight);
if (pKeyIndex == nullptr)
return std::string();
uint256 blockKey = pKeyIndex->GetBlockHash();
return std::string((const char*)&blockKey, sizeof(blockKey));
}
bool CheckRandomXSolution(const CBlockHeader *pblock, int32_t height)
{
// Centralized height gate (shared with the parallel pre-verify pool, Stage 0).
if (!RandomXValidationRequired(height))
return true;
// Skip when miner is validating its own block via TestBlockValidity
// Skip when the miner is validating its own freshly-mined block via TestBlockValidity
// (thread-local; never set on the connect thread or the pre-verify worker threads).
if (fSkipRandomXValidation)
return true;
@@ -743,47 +791,44 @@ bool CheckRandomXSolution(const CBlockHeader *pblock, int32_t height)
pblock->nSolution.size(), RANDOMX_HASH_SIZE, height);
}
static int randomxInterval = GetRandomXInterval();
static int randomxBlockLag = GetRandomXBlockLag();
// Determine the correct RandomX key for this height
char initialKey[82];
snprintf(initialKey, 81, "%08x%s%08x", ASSETCHAINS_MAGIC, SMART_CHAIN_SYMBOL, ASSETCHAINS_RPCPORT);
std::string rxKey;
if (height < randomxInterval + randomxBlockLag) {
// Use initial key derived from chain params
rxKey = std::string(initialKey, strlen(initialKey));
} else {
// Use block hash at the key height
int keyHeight = ((height - randomxBlockLag) / randomxInterval) * randomxInterval;
CBlockIndex *pKeyIndex = hush_chainactive(keyHeight);
if (pKeyIndex == nullptr) {
return error("CheckRandomXSolution(): cannot get block index at key height %d for block %d", keyHeight, height);
}
uint256 blockKey = pKeyIndex->GetBlockHash();
rxKey = std::string((const char*)&blockKey, sizeof(blockKey));
}
// Serialize the block header without nSolution (but with nNonce) as RandomX input
CRandomXInput rxInput(*pblock);
CDataStream ss(SER_NETWORK, PROTOCOL_VERSION);
ss << rxInput;
// Derive the key (shared helper) and serialize the input (identical bytes to the pool path).
std::string rxKey = GetRandomXKey(height);
if (rxKey.empty())
return error("CheckRandomXSolution(): cannot derive RandomX key for height %d", height);
std::vector<unsigned char> ssInput = GetRandomXInput(*pblock);
char computedHash[RANDOMX_HASH_SIZE];
// Measurement (Track 1): isolate RandomX verification cost during IBD. The
// expensive parts are the per-key cache (re)init (~every GetRandomXInterval()
// blocks) and the hash computation itself; both happen under the lock below.
int64_t nTimeRxStart = GetTimeMicros();
bool fKeyInit = false;
{
std::lock_guard<std::mutex> lock(cs_randomx_validator);
// Initialize cache + VM if needed, or re-init if key changed
if (s_rxCache == nullptr) {
randomx_flags flags = randomx_get_flags();
s_rxCache = randomx_alloc_cache(flags);
// Try large pages for the 256MB validator cache: fewer TLB misses → ~15-30% faster
// light-mode validation where the OS has hugepages configured. Falls back transparently
// when unavailable, exactly as the miner does (miner.cpp:1097). Page size does not affect
// the computed hash, so this is consensus-neutral.
bool fLargePages = true;
s_rxCache = randomx_alloc_cache(flags | RANDOMX_FLAG_LARGE_PAGES);
if (s_rxCache == nullptr) {
fLargePages = false;
s_rxCache = randomx_alloc_cache(flags);
}
if (s_rxCache == nullptr) {
return error("CheckRandomXSolution(): failed to allocate RandomX cache");
}
// Confirm the fast paths are active (JIT off would be ~9x slower; see randomx-benchmark).
LogPrint("bench", "CheckRandomXSolution: RandomX flags=0x%x JIT=%d HARD_AES=%d largePages=%d\n",
(unsigned int)flags, !!(flags & RANDOMX_FLAG_JIT), !!(flags & RANDOMX_FLAG_HARD_AES), (int)fLargePages);
randomx_init_cache(s_rxCache, rxKey.data(), rxKey.size());
s_rxCurrentKey = rxKey;
fKeyInit = true;
s_rxVM = randomx_create_vm(flags, s_rxCache, nullptr);
if (s_rxVM == nullptr) {
randomx_release_cache(s_rxCache);
@@ -793,11 +838,17 @@ bool CheckRandomXSolution(const CBlockHeader *pblock, int32_t height)
} else if (s_rxCurrentKey != rxKey) {
randomx_init_cache(s_rxCache, rxKey.data(), rxKey.size());
s_rxCurrentKey = rxKey;
fKeyInit = true;
randomx_vm_set_cache(s_rxVM, s_rxCache);
}
randomx_calculate_hash(s_rxVM, &ss[0], ss.size(), computedHash);
randomx_calculate_hash(s_rxVM, ssInput.data(), ssInput.size(), computedHash);
}
int64_t nTimeRxEnd = GetTimeMicros();
nTimeRandomX += nTimeRxEnd - nTimeRxStart;
LogPrint("bench", " - RandomX verify ht=%d: %.2fms%s [%.2fs]\n",
height, (nTimeRxEnd - nTimeRxStart) * 0.001,
fKeyInit ? " (key-init)" : "", nTimeRandomX * 0.000001);
// Compare computed hash against nSolution
if (memcmp(computedHash, pblock->nSolution.data(), RANDOMX_HASH_SIZE) != 0) {
@@ -814,7 +865,7 @@ bool CheckRandomXSolution(const CBlockHeader *pblock, int32_t height)
fprintf(stderr, " computed : %s\n", computedHex.c_str());
fprintf(stderr, " nSolution: %s\n", solutionHex.c_str());
fprintf(stderr, " rxKey size=%lu, input size=%lu, nNonce=%s\n",
rxKey.size(), ss.size(), pblock->nNonce.ToString().c_str());
rxKey.size(), ssInput.size(), pblock->nNonce.ToString().c_str());
fprintf(stderr, " nSolution.size()=%lu, RANDOMX_HASH_SIZE=%d\n",
pblock->nSolution.size(), RANDOMX_HASH_SIZE);
// Also log to debug.log
@@ -822,7 +873,7 @@ bool CheckRandomXSolution(const CBlockHeader *pblock, int32_t height)
LogPrintf(" computed : %s\n", computedHex);
LogPrintf(" nSolution: %s\n", solutionHex);
LogPrintf(" rxKey size=%lu, input size=%lu, nNonce=%s\n",
rxKey.size(), ss.size(), pblock->nNonce.ToString());
rxKey.size(), ssInput.size(), pblock->nNonce.ToString());
return false;
}
@@ -830,6 +881,88 @@ bool CheckRandomXSolution(const CBlockHeader *pblock, int32_t height)
return true;
}
// ============================================================================================
// Parallel RandomX pre-verification pool (Stage 2).
// One shared light-mode cache (holding a single key at a time) + per-thread VMs, mirroring the
// miner's RandomXDatasetManager pattern (miner.cpp). The connect thread (ActivateBestChainStep)
// loads the cache key for a same-key group of about-to-be-connected blocks, dispatches them to
// this pool, and barrier-waits; each worker hashes on its own VM (sharing the read-only cache)
// and, on a match, sets the block's transient fRandomXVerified flag so the inline check in
// CheckBlockHeader can be skipped. The inline path remains the consensus authority for anything
// not pre-verified, so the pool can only ever flip false->true on a real hash match.
static boost::shared_mutex g_rxvMutex; // shared = hashing; exclusive = cache (re)init
static randomx_cache* g_rxvCache = nullptr; // shared, read-only during hashing
static std::string g_rxvKey; // key currently loaded into g_rxvCache
static randomx_flags g_rxvFlags;
static thread_local randomx_vm* tls_rxvVM = nullptr;
static thread_local std::string tls_rxvVMKey;
CCheckQueue<CRandomXCheck> rxCheckQueue(1); // batch size 1: each item is ~tens of ms
bool RandomXValidatorPrepareKey(const std::string& rxKey)
{
boost::unique_lock<boost::shared_mutex> lock(g_rxvMutex);
if (g_rxvCache == nullptr) {
g_rxvFlags = randomx_get_flags();
g_rxvCache = randomx_alloc_cache(g_rxvFlags | RANDOMX_FLAG_LARGE_PAGES);
if (g_rxvCache == nullptr)
g_rxvCache = randomx_alloc_cache(g_rxvFlags);
if (g_rxvCache == nullptr) {
LogPrintf("RandomXValidatorPrepareKey: cache alloc failed; parallel pre-verify disabled\n");
return false;
}
randomx_init_cache(g_rxvCache, rxKey.data(), rxKey.size());
g_rxvKey = rxKey;
return true;
}
if (g_rxvKey != rxKey) {
randomx_init_cache(g_rxvCache, rxKey.data(), rxKey.size());
g_rxvKey = rxKey;
}
return true;
}
bool CRandomXCheck::operator()()
{
boost::shared_lock<boost::shared_mutex> lock(g_rxvMutex);
// The connect thread set the shared cache to one key before dispatching this group. If this
// item's key doesn't match (e.g. a key-rotation straggler) or the cache is unavailable, skip it
// and leave *presult false — the inline CheckRandomXSolution will verify it.
if (g_rxvCache == nullptr || g_rxvKey != rxKey)
return true;
if (tls_rxvVM == nullptr) {
tls_rxvVM = randomx_create_vm(g_rxvFlags, g_rxvCache, nullptr);
if (tls_rxvVM == nullptr)
return true; // cannot verify here -> inline fallback
tls_rxvVMKey = g_rxvKey;
} else if (tls_rxvVMKey != g_rxvKey) {
// Cache was re-initialized to a new key since this VM last ran; rebind.
randomx_vm_set_cache(tls_rxvVM, g_rxvCache);
tls_rxvVMKey = g_rxvKey;
}
unsigned char h[RANDOMX_HASH_SIZE];
randomx_calculate_hash(tls_rxvVM, input.data(), input.size(), h);
if (memcmp(h, expected, RANDOMX_HASH_SIZE) == 0 && presult != nullptr)
*presult = true;
return true; // ALWAYS true: never short-circuit the queue; per-block result is in *presult
}
void ThreadRandomXVerify()
{
RenameThread("hush-rxverify");
rxCheckQueue.Thread();
}
void RandomXValidatorShutdown()
{
boost::unique_lock<boost::shared_mutex> lock(g_rxvMutex);
// Per-thread VMs are intentionally leaked (process exiting); release the shared cache.
if (g_rxvCache != nullptr) {
randomx_release_cache(g_rxvCache);
g_rxvCache = nullptr;
}
}
int32_t hush_chosennotary(int32_t *notaryidp,int32_t height,uint8_t *pubkey33,uint32_t timestamp);
int32_t hush_currentheight();
void hush_index2pubkey33(uint8_t *pubkey33,CBlockIndex *pindex,int32_t height);