wallet/rpc: gather Sapling keys once per history RPC, not per output per address

zsWalletTxJSON (z_listsentbyaddress, z_listreceivedaddress, getalldata)
runs once per wallet tx under LOCK2(cs_main, cs_wallet). For every
shielded output it copied the wallet's address set, and for each address
fetched the spending key, re-derived its full viewing key (an
elliptic-curve operation), and trial-decrypted the output; the sends side
did the same with the ovk. Every received or sent note in a tx with
spends walked the whole wallet in GetNullifiersForAddresses for the
change check. The spends side decrypted every spent note again and
deep-copied its SaplingNoteData (witnesses included). The GUI runs
z_listsentbyaddress over the whole wallet on every launch.

SaplingHistoryKeys, built once per RPC call, holds each address with a
usable spending key (in GetSaplingPaymentAddresses order, with its ivk
and ovk as the spending key gives them, and its ivk as
GetSaplingIncomingViewingKey gives it). It also gives the change check
as a lookup into the wallet's (ivk, nullifier) pairs, collected on first
use. The helpers keep their exact structure and quirks. Each output's
JSON object and change flag still accumulate across the addresses that
decrypt it. Diversified addresses sharing a key still produce one entry
each. An unknown address still checks change under a zero ivk. A locked
wallet still reports no shielded received or sent entries. Only the
repeated work is gone:

- each output is decrypted at most once per distinct key. Once it is
  known which ivk decrypts an output, no other ivk is tried: the cached
  plaintext, or a decryption under the ivk the wallet's note data
  records. Two canonical ivks can't both pass the note commitment check,
  which pins pk_d = [ivk]g_d;
- outCiphertexts are decrypted once per distinct ovk; the t->z shielding
  ovk is derived from the seed once per tx;
- spends use the cached plaintext of the spent note.

Two deliberate behaviour changes. The spends helper looked notes up with
operator[]: it inserted a null-outpoint entry into
mapSaplingNullifiersToNotes for every other wallet's nullifier it met.
After getalldata or z_listsentbyaddress had run, z_viewtransaction of
any tx spending someone else's note (every incoming z->z payment) failed
with map::at. And with an inconsistent nullifier map, it inserted an
empty note into a wallet tx. The helpers now only read the wallet. Also,
if the walletpassphrase timer relocks the wallet during the call, the
whole call reports the keys as of its start; the old per-output lookups
switched over partway.

test_wallet_history.cpp holds 69bb035e7's helpers verbatim (renamed)
and a wallet exercising every branch:
- default and diversified addresses, a viewing-key-only address, and a
  key imported after its note arrived;
- z->z sends with change, and with text, empty and binary memos;
- a send nobody's ovk reads, and a t->z shielding send;
- our spends, someone else's, and a mixed tx.
Every tx x every return type x nine address filters gives byte-identical
JSON: with the plaintext cache warm, cold, and on an encrypted wallet
both locked and unlocked. Mutation-checked: dropping the ivk check from
the uniqueness shortcut fails the test. DoesNotModifyTheWallet pins the
read-only change (and shows that the old helpers did insert the foreign
nullifier).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

src/Makefile.gtest.include

@@ -19,7 +19,8 @@ hush_gtest_SOURCES = \
 	gtest/test_net_closesocket.cpp \
 	gtest/test_fd_defense.cpp \
 	gtest/test_wallet_dbenv.cpp \
-	gtest/test_wallet_notecache.cpp
+	gtest/test_wallet_notecache.cpp \
+	gtest/test_wallet_history.cpp
 
 hush_gtest_CPPFLAGS = $(AM_CPPFLAGS) -DMULTICORE -fopenmp -DBINARY_OUTPUT -DCURVE_ALT_BN128 -DSTATIC $(BITCOIN_INCLUDES)
 hush_gtest_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)

src/gtest/test_wallet_history.cpp

@@ -0,0 +1,816 @@
+// Copyright (c) 2016-2024 The Hush developers
+// Distributed under the GPLv3 software license, see the accompanying
+// file COPYING or https://www.gnu.org/licenses/gpl-3.0.en.html
+
+#include <gtest/gtest.h>
+
+#include <deque>
+#include <utf8.h>
+
+#include "chain.h"
+#include "chainparams.h"
+#include "coins.h"
+#include "init.h"
+#include "key_io.h"
+#include "main.h"
+#include "primitives/block.h"
+#include "primitives/transaction.h"
+#include "random.h"
+#include "rpc/server.h"
+#include "script/standard.h"
+#include "utilmoneystr.h"
+#include "wallet/wallet.h"
+#include "wallet/rpchushwallet.h"
+#include "zcash/Address.hpp"
+#include "zcash/Note.hpp"
+#include "zcash/NoteEncryption.hpp"
+#include "zcash/zip32.h"
+
+using namespace std;
+using namespace libzcash;
+
+// The history-RPC helpers (src/wallet/rpchushwallet.cpp) as of 69bb035e7, verbatim apart from the Legacy_
+// prefix and static linkage: the reference the optimized helpers must reproduce byte for byte.
+static void Legacy_zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
+
+  LOCK2(cs_main, pwalletMain->cs_wallet);
+
+  //Used to identify incomplete key sets
+  int vinCount = 0;
+  int vinSpendCount = 0;
+  int shieldedSpendCount = 0;
+  int shieldedSpendsSpentCount = 0;
+
+  //Check address
+  bool isTAddress = false;
+  bool isZsAddress = false;
+
+  CTxDestination tAddress = DecodeDestination(strAddress);
+  auto zAddress = DecodePaymentAddress(strAddress);
+  SaplingPaymentAddress zsAddress;
+
+  if (filterByAddress) {
+
+    if (IsValidDestination(tAddress))
+      isTAddress = true;
+
+    if (IsValidPaymentAddress(zAddress)) {
+      if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
+          zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
+          isZsAddress = true;
+      }
+    }
+  }
+
+
+  // Transparent Inputs belonging to the wallet
+  UniValue tSpends(UniValue::VARR);
+  if (isTAddress || !filterByAddress) {
+    for (int i = 0; i < wtx.vin.size(); i++) {
+      vinCount++;
+      const CTxIn& txin = wtx.vin[i];
+      UniValue obj(UniValue::VOBJ);
+      CTxDestination address;
+      const CWalletTx* parent = pwalletMain->GetWalletTx(txin.prevout.hash);
+      if (parent != NULL) {
+        const CTxOut& parentOut = parent->vout[txin.prevout.n];
+        ExtractDestination(parentOut.scriptPubKey, address);
+        if(IsMine(*pwalletMain, address)){
+          vinSpendCount++;
+          totalSpends += CAmount(-parentOut.nValue);
+          obj.push_back(Pair("address",EncodeDestination(address)));
+          obj.push_back(Pair("scriptPubKey",HexStr(parentOut.scriptPubKey.begin(), parentOut.scriptPubKey.end())));
+          obj.push_back(Pair("amount",ValueFromAmount(-parentOut.nValue)));
+          obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
+          obj.push_back(Pair("spendVout",(int)txin.prevout.n));
+          CTxDestination filterAddress = DecodeDestination(strAddress);
+          if (address == tAddress || !filterByAddress) {
+            filteredSpends += CAmount(-parentOut.nValue);
+            tSpends.push_back(obj);
+          }
+        }
+      }
+    }
+  }
+  spends.push_back(Pair("transparentSpends",tSpends));
+
+  // Sapling Inputs belonging to the wallet
+  UniValue zsSpends(UniValue::VARR);
+  if (isZsAddress || !filterByAddress) {
+    for (int i = 0; i < wtx.vShieldedSpend.size(); i++) {
+      shieldedSpendCount++;
+      const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
+      UniValue obj(UniValue::VOBJ);
+
+      SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier];
+
+      if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
+        const CWalletTx* parent = pwalletMain->GetWalletTx(pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash);
+        const OutputDescription& output = parent->vShieldedOutput[op.n];
+        auto nd = pwalletMain->mapWallet[pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash].mapSaplingNoteData[op];
+        auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
+
+        if (pt) {
+          auto note = pt.get();
+          auto pa = nd.ivk.address(note.d);
+          auto address = pa.get();
+          shieldedSpendsSpentCount++;
+          totalSpends += CAmount(-note.value());
+          obj.push_back(Pair("address",EncodePaymentAddress(address)));
+          obj.push_back(Pair("amount", ValueFromAmount(CAmount(-note.value()))));
+          obj.push_back(Pair("spendTxid",parent->GetHash().ToString()));
+          obj.push_back(Pair("spendshieldedOutputIndex",(int)op.n));
+          if (address == zsAddress || !filterByAddress) {
+            filteredSpends += CAmount(-note.value());
+            zsSpends.push_back(obj);
+          }
+        }
+      }
+    }
+  }
+  spends.push_back(Pair("saplingSpends",zsSpends));
+
+ 
+  spends.push_back(Pair("totalSpends",ValueFromAmount(filteredSpends)));
+
+  if (!filterByAddress) {
+    if (filteredSpends != 0 && (vinCount != vinSpendCount || shieldedSpendCount != shieldedSpendsSpentCount)) {
+      spends.push_back(Pair("missingSpendingKeys", true));
+      spends.push_back(Pair("vinCount", vinCount));
+      spends.push_back(Pair("vinSpendCount", vinSpendCount));
+      spends.push_back(Pair("shieldedSpendCount", shieldedSpendCount));
+      spends.push_back(Pair("shieldedSpendsSpentCount", shieldedSpendsSpentCount));
+    } else {
+        spends.push_back(Pair("missingSpendingKeys", false));
+    }
+  }
+}
+
+
+static void Legacy_zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress) {
+
+  if(fZdebug)
+    fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
+
+  LOCK2(cs_main, pwalletMain->cs_wallet);
+
+  //Check address
+  bool isTAddress = false;
+  bool isZsAddress = false;
+
+  CTxDestination tAddress = DecodeDestination(strAddress);
+  auto zAddress = DecodePaymentAddress(strAddress);
+  SaplingPaymentAddress zsAddress;
+
+  if (filterByAddress) {
+
+    if (IsValidDestination(tAddress))
+      isTAddress = true;
+
+    if (IsValidPaymentAddress(zAddress)) {
+      if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
+          zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
+          isZsAddress = true;
+      }
+    }
+  }
+
+
+  //Transparent Received txos belonging to the wallet
+  UniValue tReceived(UniValue::VARR);
+  if (isTAddress || !filterByAddress) {
+    for (int i = 0; i < wtx.vout.size(); i++) {
+      const CTxOut& txout = wtx.vout[i];
+      UniValue obj(UniValue::VOBJ);
+      CTxDestination address;
+      ExtractDestination(txout.scriptPubKey, address);
+      if(IsMine(*pwalletMain, address)){
+        obj.push_back(Pair("address",EncodeDestination(address)));
+        obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
+        obj.push_back(Pair("amount",ValueFromAmount(txout.nValue)));
+        obj.push_back(Pair("vout", i));
+        if (address == tAddress || !filterByAddress) {
+          totalReceived += CAmount(txout.nValue);
+          tReceived.push_back(obj);
+        }
+      }
+    }
+  }
+  received.push_back(Pair("transparentReceived",tReceived));
+
+
+  //Sapling Sends belonging to the wallet
+  UniValue zsReceived(UniValue::VARR);
+  if (isZsAddress || !filterByAddress) {
+    for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
+      const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
+      UniValue obj(UniValue::VOBJ);
+      bool changeTx = false;
+      //Decrypt sapling incoming commitments using IVK
+      std::set<libzcash::SaplingPaymentAddress> addresses;
+      pwalletMain->GetSaplingPaymentAddresses(addresses);
+      for (auto addr : addresses) {
+        libzcash::SaplingExtendedSpendingKey extsk;
+        if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
+          auto pt = libzcash::SaplingNotePlaintext::decrypt(
+                  outputDesc.encCiphertext, extsk.expsk.full_viewing_key().in_viewing_key(), outputDesc.ephemeralKey, outputDesc.cm);
+
+          if (pt) {
+            auto note = pt.get();
+            auto pt_unwrapped = pt.get();
+            auto memo = pt_unwrapped.memo();
+            obj.push_back(Pair("address",EncodePaymentAddress(addr)));
+            obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
+            obj.push_back(Pair("memo", HexStr(memo)));
+
+              if (memo[0] <= 0xf4) {
+                  auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
+                  std::string memoStr(memo.begin(), end.base());
+                if (utf8::is_valid(memoStr)) {
+                obj.push_back(Pair("memoStr", memoStr));
+            }
+        }
+            obj.push_back(Pair("shieldedOutputIndex",i));
+
+            //Check Change Status
+            if (wtx.vShieldedSpend.size()!=0) {
+              std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
+              nullifierSet = pwalletMain->GetNullifiersForAddresses({addr});
+              BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
+                if (nullifierSet.count(std::make_pair(addr, spendDesc.nullifier))) {
+                    changeTx = true;
+                }
+              }
+            }
+            obj.push_back(Pair("change",changeTx));
+            if (addr == zsAddress || !filterByAddress) {
+              totalReceived += CAmount(note.value());
+              zsReceived.push_back(obj);
+            }
+          }
+        }
+      }
+    }
+  }
+  received.push_back(Pair("saplingReceived",zsReceived));
+
+  received.push_back(Pair("totalReceived",ValueFromAmount(totalReceived)));
+}
+
+
+static void Legacy_zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress) {
+
+  LOCK2(cs_main, pwalletMain->cs_wallet);
+
+  //Used to identify incomplete key sets
+  int shieldedOutputCount = 0;
+  int shieldedOutputDecryptedCount = 0;
+
+  //Check address
+  bool isTAddress = false;
+  bool isZsAddress = false;
+
+  CTxDestination tAddress = DecodeDestination(strAddress);
+  auto zAddress = DecodePaymentAddress(strAddress);
+  SaplingPaymentAddress zsAddress;
+
+  if (filterByAddress) {
+
+    if (IsValidDestination(tAddress))
+      isTAddress = true;
+
+    if (IsValidPaymentAddress(zAddress)) {
+      if (boost::get<libzcash::SaplingPaymentAddress>(&zAddress) != nullptr) {
+          zsAddress = boost::get<libzcash::SaplingPaymentAddress>(zAddress);
+          isZsAddress = true;
+      }
+    }
+  }
+
+  //All Transparent Sends in the transaction
+
+  UniValue tSends(UniValue::VARR);
+  if (isTAddress || !filterByAddress) {
+    for (int i = 0; i < wtx.vout.size(); i++) {
+      const CTxOut& txout = wtx.vout[i];
+      UniValue obj(UniValue::VOBJ);
+      CTxDestination address;
+      ExtractDestination(txout.scriptPubKey, address);
+      obj.push_back(Pair("address",EncodeDestination(address)));
+      obj.push_back(Pair("scriptPubKey",HexStr(txout.scriptPubKey.begin(), txout.scriptPubKey.end())));
+      obj.push_back(Pair("amount",ValueFromAmount(-txout.nValue)));
+      obj.push_back(Pair("vout", i));
+      if (address == tAddress || !filterByAddress) {
+        totalSends += CAmount(-txout.nValue);
+        tSends.push_back(obj);
+      }
+    }
+  }
+  sends.push_back(Pair("transparentSends",tSends));
+
+  //All Shielded Sends in the transaction
+  UniValue zsSends(UniValue::VARR);
+  if (isZsAddress || !filterByAddress) {
+    for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
+      const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
+      shieldedOutputCount++;
+      UniValue obj(UniValue::VOBJ);
+      bool changeTx = false;
+
+      //Decrypt sapling outgoing t to z transaction using HDseed
+      if (wtx.vShieldedSpend.size()==0) {
+        HDSeed seed;
+        if (pwalletMain->GetHDSeedForDerivation(seed)) {
+          auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
+                  outputDesc.outCiphertext,ovkForShieldingFromTaddr(seed),outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
+
+          if (opt) {
+            auto opt_unwrapped = opt.get();
+            auto pt = libzcash::SaplingNotePlaintext::decrypt(
+                    outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
+
+            if (pt) {
+              shieldedOutputDecryptedCount++;
+              auto pt_unwrapped = pt.get();
+              libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
+              obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
+              obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
+              obj.push_back(Pair("shieldedOutputIndex",i));
+              obj.push_back(Pair("change",false));
+              if (sentAddr == zsAddress || !filterByAddress) {
+                totalSends += CAmount(-pt_unwrapped.value());
+                zsSends.push_back(obj);
+              }
+            }
+          }
+        }
+
+      //attempt Decryption of Outgoing Sapling using wallet extended spending keys
+      } else {
+        std::set<libzcash::SaplingPaymentAddress> addresses;
+        pwalletMain->GetSaplingPaymentAddresses(addresses);
+        for (auto addr : addresses) {
+          libzcash::SaplingExtendedSpendingKey extsk;
+          if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
+            auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
+                    outputDesc.outCiphertext,extsk.expsk.full_viewing_key().ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
+
+            if (opt) {
+              auto opt_unwrapped = opt.get();
+              auto pt = libzcash::SaplingNotePlaintext::decrypt(
+                      outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
+
+              if (pt) {
+                auto pt_unwrapped = pt.get();
+                auto memo = pt_unwrapped.memo();
+                shieldedOutputDecryptedCount++;
+                libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
+                obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
+                obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
+                obj.push_back(Pair("memo", HexStr(memo)));
+
+              if (memo[0] <= 0xf4) {
+                  auto end = std::find_if(memo.rbegin(), memo.rend(), [](unsigned char v) { return v != 0; });
+                  std::string memoStr(memo.begin(), end.base());
+                if (utf8::is_valid(memoStr)) {
+                obj.push_back(Pair("memoStr", memoStr));
+            }
+        }
+                obj.push_back(Pair("shieldedOutputIndex",i));
+
+                //Check Change Status
+                if (wtx.vShieldedSpend.size()!=0) {
+                  std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
+                  nullifierSet = pwalletMain->GetNullifiersForAddresses({sentAddr});
+                  BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
+                    if (nullifierSet.count(std::make_pair(sentAddr, spendDesc.nullifier))) {
+                        changeTx = true;
+                    }
+                  }
+                }
+                obj.push_back(Pair("change",changeTx));
+                if (sentAddr == zsAddress || !filterByAddress) {
+                  totalSends += CAmount(-pt_unwrapped.value());
+                  zsSends.push_back(obj);
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+  sends.push_back(Pair("saplingSends",zsSends));
+
+  if (shieldedOutputCount != shieldedOutputDecryptedCount) {
+    sends.push_back(Pair("missingSaplingOVK", true));
+  } else {
+    sends.push_back(Pair("missingSaplingOVK", false));
+  }
+
+  sends.push_back(Pair("totalSends",ValueFromAmount(totalSends)));
+}
+
+
+static void Legacy_zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType) {
+
+  LOCK2(cs_main, pwalletMain->cs_wallet);
+
+  //Track total wallet spend and received
+  CAmount totalSpends = 0;
+  CAmount filteredSpends = 0;
+  CAmount totalReceived = 0;
+  CAmount totalSends = 0;
+
+  //Various Univalue to be added to the final transaction
+  UniValue spends(UniValue::VOBJ);
+  UniValue received(UniValue::VOBJ);
+  UniValue sends(UniValue::VOBJ);
+  UniValue tx(UniValue::VOBJ);
+
+  //Begin Compiling the Decrypted Transaction
+  tx.push_back(Pair("txid", wtx.GetHash().ToString()));
+
+  if(fZdebug)
+    fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
+
+  if (wtx.IsCoinBase())
+  {
+      tx.push_back(Pair("coinbase", true));
+      if (wtx.GetDepthInMainChain() < 1)
+          tx.push_back(Pair("category", "orphan"));
+      else if (wtx.GetBlocksToMaturity() > 0)
+          tx.push_back(Pair("category", "immature"));
+      else
+          tx.push_back(Pair("category", "generate"));
+  } else {
+    tx.push_back(Pair("coinbase", false));
+    tx.push_back(Pair("category", "standard"));
+  }
+
+  tx.push_back(Pair("blockhash", wtx.hashBlock.GetHex()));
+  tx.push_back(Pair("blockindex", wtx.nIndex));
+  int confirms = wtx.GetDepthInMainChain();
+  if(confirms > 0)
+  {
+      tx.push_back(Pair("blocktime", mapBlockIndex[wtx.hashBlock]->GetBlockTime()));
+  } else {
+      tx.push_back(Pair("blocktime", 0));
+  }
+  tx.push_back(Pair("expiryheight", (int64_t)wtx.nExpiryHeight));
+  tx.push_back(Pair("confirmations", confirms));
+  tx.push_back(Pair("time", wtx.GetTxTime()));
+  tx.push_back(Pair("size", static_cast<uint64_t>(GetSerializeSize(static_cast<CTransaction>(wtx), SER_NETWORK, PROTOCOL_VERSION))));
+
+  //Wallet Conflicts
+  UniValue conflicts(UniValue::VARR);
+  BOOST_FOREACH(const uint256& conflict, wtx.GetConflicts())
+      conflicts.push_back(conflict.GetHex());
+  tx.push_back(Pair("walletconflicts", conflicts));
+
+  // Return Type used to determine what is included in the transaction
+  // 0 Spends, Received and spends
+  // 1 Sends
+  // 2 Received
+  // 3 Spends
+  // TODO - Setup Enum, maybe...
+
+  // Add Spends retrieved earlier
+  if (returnType != 2) {
+    Legacy_zsTxSpendsToJSON(wtx, spends, totalSpends, filteredSpends, strAddress, fBool);
+    if ((!fBool || filteredSpends != 0) && (returnType == 0 || returnType == 1)) {
+      tx.push_back(Pair("spends",spends));
+    }
+  }
+  // Get Received
+  if (returnType == 0 || returnType == 2) {
+    Legacy_zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool);
+    if (!fBool || totalReceived != 0) {
+      tx.push_back(Pair("received",received));
+    }
+  }
+
+  // Get Sends
+  if (returnType == 0 || returnType == 3) {
+    //Only include sends if there are spends that belong to the wallet.
+    if (totalSpends != 0 || fBool) {
+      Legacy_zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool);
+    }
+    if (!fBool || totalSends != 0) {
+      tx.push_back(Pair("sends",sends));
+    }
+  }
+
+  if ((returnType == 0 && (!fBool || filteredSpends != 0 || totalReceived != 0 || totalSends != 0))
+   || (returnType == 1 && (!fBool || filteredSpends != 0))
+   || (returnType == 2 && (!fBool || totalReceived != 0))
+   || (returnType == 3 && (!fBool || totalSends != 0))) {
+    ret.push_back(tx);
+  }
+
+}
+
+namespace {
+
+std::array<unsigned char, HUSH_MEMO_SIZE> Memo(const std::string& text)
+{
+    std::array<unsigned char, HUSH_MEMO_SIZE> memo;
+    memo.fill(0);
+    if (text == "empty")            // the "no memo" encoding: 0xF6 then zeros
+        memo[0] = 0xF6;
+    else if (text == "binary")      // starts below 0xF5 but isn't UTF-8
+        memo[0] = 0xC3, memo[1] = 0x28;
+    else
+        memcpy(memo.data(), text.data(), text.size());
+    return memo;
+}
+
+// A real encrypted Sapling output, and when ovk is given, an outCiphertext the sender can decrypt with it
+// (the wallet never checks proofs or value commitments).
+OutputDescription Output(const SaplingPaymentAddress& to, uint64_t value, const std::string& memo,
+                         const boost::optional<uint256>& ovk = boost::none)
+{
+    SaplingNote note(to, value);
+    auto enc = SaplingNotePlaintext(note, Memo(memo)).encrypt(note.pk_d);
+    assert(static_cast<bool>(enc));
+    OutputDescription od;
+    od.cv = GetRandHash();
+    od.cm = note.cm().get();
+    od.ephemeralKey = enc->second.get_epk();
+    od.encCiphertext = enc->first;
+    if (ovk) {
+        SaplingOutgoingPlaintext out(note.pk_d, enc->second.get_esk());
+        od.outCiphertext = out.encrypt(*ovk, od.cv, od.cm, enc->second);
+    } else {
+        GetRandBytes(od.outCiphertext.begin(), od.outCiphertext.size());
+    }
+    return od;
+}
+
+CTransaction SaplingTx(const std::vector<OutputDescription>& outputs, const std::vector<uint256>& spentNullifiers,
+                       const std::vector<COutPoint>& vin = {})
+{
+    CMutableTransaction mtx;
+    mtx.fOverwintered = true;
+    mtx.nVersion = SAPLING_TX_VERSION;
+    mtx.nVersionGroupId = SAPLING_VERSION_GROUP_ID;
+    mtx.vShieldedOutput = outputs;
+    for (const uint256& nf : spentNullifiers) {
+        SpendDescription sd;
+        sd.nullifier = nf;
+        mtx.vShieldedSpend.push_back(sd);
+    }
+    for (const COutPoint& prev : vin)
+        mtx.vin.push_back(CTxIn(prev));
+    return CTransaction(mtx);
+}
+
+SaplingExtendedSpendingKey Key(uint32_t account)
+{
+    std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x6b);
+    HDSeed seed(raw);
+    return SaplingExtendedSpendingKey::Master(seed).Derive(account | ZIP32_HARDENED_KEY_LIMIT);
+}
+
+}
+
+// A wallet whose txs exercise every branch of the history helpers: notes received on default and diversified
+// addresses, by a viewing-key-only address and by a key imported afterwards; z->z sends with change and memos;
+// a t->z shielding send; spends of our notes and of someone else's. pwalletMain points at it during the test.
+class HistoryWallet : public CWallet {
+public:
+    bool EncryptKeys(CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::EncryptKeys(vMasterKeyIn); }
+    bool Unlock(const CKeyingMaterial& vMasterKeyIn) { return CCryptoKeyStore::Unlock(vMasterKeyIn); }
+};
+
+class WalletHistory : public ::testing::Test {
+protected:
+    HistoryWallet wallet;
+    CWallet* savedWallet = nullptr;
+    SaplingPaymentAddress paA, paA2, paB, paC, paD, paF;
+    CTxDestination taddr;
+    std::map<uint256, int> depthOf;
+    std::vector<uint256> txids;
+    CCoinsView coinsBase;
+    CCoinsViewCache coins{&coinsBase};
+    CCoinsViewCache* savedCoinsTip = nullptr;
+    CBlockIndex* savedTip = nullptr;
+    std::deque<CBlock> blocks;
+    std::deque<CBlockIndex> index;
+    std::vector<uint256> insertedBlocks;
+
+    void SetUp() override {
+        SelectParams(CBaseChainParams::MAIN);
+        savedWallet = pwalletMain;
+        pwalletMain = &wallet;
+        savedCoinsTip = pcoinsTip;
+        pcoinsTip = &coins;
+        LOCK2(cs_main, wallet.cs_wallet);
+        savedTip = chainActive.Tip();
+
+        std::vector<unsigned char, secure_allocator<unsigned char>> raw(32, 0x2d);
+        ASSERT_TRUE(wallet.SetHDSeed(HDSeed(raw)));
+        HDSeed seed;
+        ASSERT_TRUE(wallet.GetHDSeedForDerivation(seed));
+        const uint256 seedOvk = ovkForShieldingFromTaddr(seed);
+
+        auto skA = Key(0), skB = Key(1), skC = Key(2), skD = Key(3), skF = Key(9);
+        paA = skA.DefaultAddress();
+        diversifier_index_t j;
+        j.begin()[0] = 50;
+        paA2 = skA.ToXFVK().Address(j)->second;
+        paB = skB.DefaultAddress();
+        paC = skC.DefaultAddress();
+        paD = skD.DefaultAddress();
+        paF = skF.DefaultAddress();
+        const uint256 ovkA = skA.expsk.full_viewing_key().ovk, ovkC = skC.expsk.full_viewing_key().ovk;
+        ASSERT_TRUE(wallet.AddSaplingZKey(skA, paA));
+        ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skA.expsk.full_viewing_key().in_viewing_key(), paA2));
+        ASSERT_TRUE(wallet.AddSaplingIncomingViewingKey(skB.expsk.full_viewing_key().in_viewing_key(), paB));
+        ASSERT_TRUE(wallet.AddSaplingZKey(skC, paC));
+
+        CKey tkey;
+        tkey.MakeNewKey(true);
+        ASSERT_TRUE(wallet.AddKeyPubKey(tkey, tkey.GetPubKey()));
+        taddr = tkey.GetPubKey().GetID();
+
+        const uint256 nfA = uint256S("a1"), nfA2 = uint256S("a2"), nfC = uint256S("c1"), nfForeign = uint256S("ff");
+
+        // T0: transparent funds to our taddr.
+        CMutableTransaction t0;
+        t0.vin.push_back(CTxIn(COutPoint(GetRandHash(), 0)));
+        t0.vout.push_back(CTxOut(10 * COIN, GetScriptForDestination(taddr)));
+        AddTx(CTransaction(t0), 30);
+        // T1: received by A, A2, B (viewing key only), C and someone else; no spends.
+        uint256 t1 = AddTx(SaplingTx({Output(paA, 1000, "for A"), Output(paA2, 2000, "empty"), Output(paB, 3000, "for B"),
+                                      Output(paC, 4000, "binary"), Output(paF, 5000, "not ours")}, {}),
+                           25, {{0, nfA}, {1, nfA2}, {3, nfC}});
+        (void)t1;
+        // T2: A sends to F with change back to A, spending A's note: outCiphertexts under A's ovk.
+        AddTx(SaplingTx({Output(paF, 600, "payment", ovkA), Output(paA, 399, "change", ovkA)}, {nfA}), 20);
+        // T3: t->z shielding of T0's output to A, outCiphertext under the seed's shielding ovk.
+        AddTx(SaplingTx({Output(paA, 9 * COIN, "empty", seedOvk)}, {}, {COutPoint(txids[0], 0)}), 15);
+        // T4: C spends its note: to A2 (another of our keys) and change to C.
+        AddTx(SaplingTx({Output(paA2, 1500, "to a diversified A", ovkC), Output(paC, 2499, "binary", ovkC)}, {nfC}), 10);
+        // T5: someone else's spend paying A.
+        AddTx(SaplingTx({Output(paA, 777, "from a stranger")}, {nfForeign}), 8);
+        // T6: received by D before D's key is in the wallet (an import without rescan), then mined.
+        AddTx(SaplingTx({Output(paD, 4242, "for D, found later")}, {}), 5);
+        // T8: spends one of our notes and one of someone else's (a coinjoin-like tx): missingSpendingKeys.
+        AddTx(SaplingTx({Output(paF, 100, "mixed", ovkA)}, {nfA, uint256S("fe")}), 3);
+        // T9: spends A2's note, but its one output's outCiphertext is under nobody's ovk we know: missingSaplingOVK.
+        AddTx(SaplingTx({Output(paF, 50, "unreadable send")}, {nfA2}), 2);
+        // T7: spends A2's note (A and A2 share A's key); never mined and not in the mempool (depth -1).
+        AddTx(SaplingTx({Output(paF, 2900, "", ovkA)}, {nfA2}), 0);
+        ASSERT_TRUE(wallet.AddSaplingZKey(skD, paD));
+
+        Mine();
+    }
+
+    void TearDown() override {
+        LOCK(cs_main);
+        chainActive.SetTip(savedTip);
+        for (const uint256& h : insertedBlocks)
+            mapBlockIndex.erase(h);
+        pcoinsTip = savedCoinsTip;
+        pwalletMain = savedWallet;
+    }
+
+    uint256 AddTx(const CTransaction& tx, int depth, const std::map<uint32_t, uint256>& nullifiers = {}) {
+        CWalletTx wtx(&wallet, tx);
+        mapSaplingNoteData_t found = wallet.FindMySaplingNotes(tx).first;
+        for (const auto& nf : nullifiers)
+            found.at(SaplingOutPoint(tx.GetHash(), nf.first)).nullifier = nf.second;
+        wtx.SetSaplingNoteData(found);
+        EXPECT_TRUE(wallet.AddToWallet(wtx, true, nullptr));
+        if (depth > 0)
+            depthOf[tx.GetHash()] = depth;
+        txids.push_back(tx.GetHash());
+        return tx.GetHash();
+    }
+
+    void Mine() {
+        const int tip = 40;
+        for (int h = 0; h <= tip; h++) {
+            blocks.emplace_back();
+            CBlock& block = blocks.back();
+            block.nTime = 1700000000 + 36 * h;
+            block.nNonce = uint256S(strprintf("%x", 0x900 + h));
+            if (h > 0)
+                block.hashPrevBlock = blocks[h - 1].GetHash();
+            for (const auto& d : depthOf)
+                if (tip - d.second + 1 == h)
+                    block.vtx.push_back(wallet.mapWallet.at(d.first));
+            block.hashMerkleRoot = block.BuildMerkleTree();
+            index.emplace_back(block);
+            CBlockIndex& bi = index.back();
+            bi.SetHeight(h);
+            bi.pprev = h > 0 ? &index[h - 1] : nullptr;
+            auto inserted = mapBlockIndex.insert(std::make_pair(block.GetHash(), &bi));
+            ASSERT_TRUE(inserted.second);
+            insertedBlocks.push_back(block.GetHash());
+            bi.phashBlock = &inserted.first->first;
+        }
+        chainActive.SetTip(&index[tip]);
+        for (const auto& d : depthOf)
+            wallet.mapWallet.at(d.first).SetMerkleBranch(blocks[tip - d.second + 1]);
+    }
+
+    std::string NewJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
+        UniValue ret(UniValue::VARR);
+        zsWalletTxJSON(wtx, ret, addr, fBool, returnType, GetSaplingHistoryKeys());
+        return ret.write();
+    }
+    std::string LegacyJSON(const CWalletTx& wtx, const std::string& addr, bool fBool, int returnType) {
+        UniValue ret(UniValue::VARR);
+        Legacy_zsWalletTxJSON(wtx, ret, addr, fBool, returnType);
+        return ret.write();
+    }
+    void ExpectMatchesLegacy(bool checkCoverage = true);
+};
+
+// Every tx, every return type, unfiltered and filtered by each kind of address: byte-identical JSON. Run with
+// the wallet's plaintext cache warm (as after the startup warm-up) and cold (every output trial-decrypted).
+class WalletHistoryCold : public WalletHistory {};
+
+TEST_F(WalletHistoryCold, MatchesLegacy) {
+    LOCK2(cs_main, wallet.cs_wallet);
+    for (auto& p : wallet.mapWallet)
+        for (auto& item : p.second.mapSaplingNoteData)
+            item.second.cachedPlaintext = boost::none;
+    ExpectMatchesLegacy();
+}
+
+TEST_F(WalletHistory, MatchesLegacy) {
+    LOCK2(cs_main, wallet.cs_wallet);
+    ExpectMatchesLegacy();
+}
+
+void WalletHistory::ExpectMatchesLegacy(bool checkCoverage) {
+    const std::vector<std::pair<std::string, bool>> filters = {
+        {"*", false}, {"*", true}, {EncodePaymentAddress(paA), true}, {EncodePaymentAddress(paA2), true},
+        {EncodePaymentAddress(paB), true}, {EncodePaymentAddress(paC), true}, {EncodePaymentAddress(paD), true},
+        {EncodePaymentAddress(paF), true}, {EncodeDestination(taddr), true},
+    };
+    std::string all;
+    for (const uint256& txid : txids) {
+        const CWalletTx& wtx = wallet.mapWallet.at(txid);
+        for (int returnType = 0; returnType <= 3; returnType++) {
+            for (const auto& f : filters) {
+                // New first: the legacy spends helper default-inserts into wallet maps.
+                std::string got = NewJSON(wtx, f.first, f.second, returnType);
+                std::string want = LegacyJSON(wtx, f.first, f.second, returnType);
+                EXPECT_EQ(want, got) << "tx " << txid.ToString() << " returnType " << returnType << " filter " << f.first << "/" << f.second;
+                all += got;
+            }
+        }
+    }
+    // The fixture really reaches the interesting branches.
+    if (!checkCoverage)
+        return;
+    for (const char* needle : {"\"change\":true", "\"memoStr\"", "\"saplingSpends\":[{", "\"transparentSpends\":[{",
+                               "\"saplingSends\":[{", "\"saplingReceived\":[{", "\"missingSaplingOVK\":true",
+                               "\"missingSpendingKeys\":true", "for D, found later"}) {
+        EXPECT_NE(std::string::npos, all.find(needle)) << needle;
+    }
+}
+
+// The same comparison on an encrypted wallet, locked (no spending keys or HD seed available: no shielded
+// received or sent entries, as before) and then unlocked.
+class WalletHistoryEncrypted : public WalletHistory {};
+
+TEST_F(WalletHistoryEncrypted, MatchesLegacyLockedAndUnlocked) {
+    LOCK2(cs_main, wallet.cs_wallet);
+    CKeyingMaterial masterKey(32, 0x5e);
+    ASSERT_TRUE(wallet.EncryptKeys(masterKey));
+    ASSERT_TRUE(wallet.IsLocked());
+    ASSERT_TRUE(GetSaplingHistoryKeys().spendable.empty());
+    ExpectMatchesLegacy(false);
+    ASSERT_TRUE(wallet.Unlock(masterKey));
+    ASSERT_FALSE(GetSaplingHistoryKeys().spendable.empty());
+    ExpectMatchesLegacy();
+}
+
+// The history helpers only read the wallet. The old spends helper's operator[] lookups inserted an entry into
+// mapSaplingNullifiersToNotes for every other wallet's nullifier they met, which made a later
+// z_viewtransaction of that tx fail (map::at on a null outpoint).
+TEST_F(WalletHistory, DoesNotModifyTheWallet) {
+    LOCK2(cs_main, wallet.cs_wallet);
+    const size_t nullifierEntries = wallet.mapSaplingNullifiersToNotes.size();
+    std::map<uint256, size_t> noteEntries;
+    for (const auto& p : wallet.mapWallet)
+        noteEntries[p.first] = p.second.mapSaplingNoteData.size();
+
+    for (const uint256& txid : txids)
+        for (int returnType = 0; returnType <= 3; returnType++)
+            NewJSON(wallet.mapWallet.at(txid), "*", false, returnType);
+    EXPECT_EQ(nullifierEntries, wallet.mapSaplingNullifiersToNotes.size());
+    EXPECT_EQ(0u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff"))) << "someone else's spend";
+    for (const auto& p : wallet.mapWallet)
+        EXPECT_EQ(noteEntries.at(p.first), p.second.mapSaplingNoteData.size());
+
+    // What the old helpers did, for the record.
+    for (const uint256& txid : txids)
+        LegacyJSON(wallet.mapWallet.at(txid), "*", false, 0);
+    EXPECT_EQ(1u, wallet.mapSaplingNullifiersToNotes.count(uint256S("ff")));
+}

src/wallet/rpchushwallet.cpp

@@ -16,6 +16,109 @@ using namespace libzcash;
 
 bool EnsureWalletIsAvailable(bool avoidException);
 
+SaplingHistoryKeys GetSaplingHistoryKeys()
+{
+  AssertLockHeld(pwalletMain->cs_wallet);
+  SaplingHistoryKeys keys;
+  std::set<libzcash::SaplingPaymentAddress> addresses;
+  pwalletMain->GetSaplingPaymentAddresses(addresses);
+  for (const auto& addr : addresses) {
+    libzcash::SaplingExtendedSpendingKey extsk;
+    if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
+      const libzcash::SaplingFullViewingKey fvk = extsk.expsk.full_viewing_key();
+      SaplingHistoryKeys::SpendableAddress a;
+      a.addr = addr;
+      a.ivk = fvk.in_viewing_key();
+      a.ovk = fvk.ovk;
+      // Left zero when the wallet doesn't know addr, exactly as GetNullifiersForAddresses does.
+      pwalletMain->GetSaplingIncomingViewingKey(addr, a.walletIvk);
+      keys.spendable.push_back(a);
+    }
+  }
+  return keys;
+}
+
+bool SaplingHistoryKeys::IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const
+{
+  AssertLockHeld(pwalletMain->cs_wallet);
+  if (!noteNullifiers) {
+    noteNullifiers = std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>();
+    for (const auto& p : pwalletMain->mapWallet) {
+      for (const auto& item : p.second.mapSaplingNoteData) {
+        if (item.second.nullifier)
+          noteNullifiers->insert(std::make_pair(item.second.ivk, *item.second.nullifier));
+      }
+    }
+  }
+  return noteNullifiers->count(std::make_pair(addrIvk, nullifier)) > 0;
+}
+
+namespace {
+
+// One output's decryptions under incoming viewing keys, each key tried at most once. Two distinct ivks can't
+// both decrypt an output: the cm check recomputes pk_d = [ivk]g_d, and ivks are canonical scalars below the
+// order of g_d (ka_agree and ivk_to_pkd reject the rest), so a second one would need a NoteCommit collision.
+// So once it is known which ivk decrypts the output (the wallet's note data records the one FindMySaplingNotes
+// found, with its plaintext normally cached), no other ivk is tried.
+class OutputDecryptions {
+  const OutputDescription& output;
+  const SaplingNoteData* nd;
+  std::map<libzcash::SaplingIncomingViewingKey, boost::optional<SaplingNoteCache>> tried;
+
+  boost::optional<SaplingNoteCache> Decrypt(const libzcash::SaplingIncomingViewingKey& ivk) const {
+    auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext, ivk, output.ephemeralKey, output.cm);
+    if (!both)
+      return boost::none;
+    return SaplingNoteCache{ both->second, both->first.memo() };
+  }
+  // The plaintext under the ivk the wallet recorded for this output, decrypting it if it isn't cached.
+  const boost::optional<SaplingNoteCache>& Recorded() {
+    auto it = tried.find(nd->ivk);
+    if (it == tried.end())
+      it = tried.insert(std::make_pair(nd->ivk, nd->cachedPlaintext ? nd->cachedPlaintext : Decrypt(nd->ivk))).first;
+    return it->second;
+  }
+public:
+  OutputDecryptions(const CWalletTx& wtx, uint32_t n) : output(wtx.vShieldedOutput[n]), nd(nullptr) {
+    auto it = wtx.mapSaplingNoteData.find(SaplingOutPoint(wtx.GetHash(), n));
+    if (it != wtx.mapSaplingNoteData.end())
+      nd = &it->second;
+  }
+  const SaplingNoteCache* Get(const libzcash::SaplingIncomingViewingKey& ivk) {
+    auto it = tried.find(ivk);
+    if (it == tried.end()) {
+      boost::optional<SaplingNoteCache> result;
+      if (nd && nd->ivk == ivk) {
+        result = Recorded();
+      } else if (!(nd && Recorded())) {
+        // Not a wallet note, or one whose recorded ivk doesn't decrypt it: try this key.
+        result = Decrypt(ivk);
+      }
+      it = tried.insert(std::make_pair(ivk, result)).first;
+    }
+    return it->second ? &*it->second : nullptr;
+  }
+};
+
+// What an outCiphertext tells the sender holding ovk: the note's plaintext and the address it was sent to.
+boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>> DecryptSent(
+    const OutputDescription& outputDesc, const uint256& ovk)
+{
+  auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
+          outputDesc.outCiphertext,ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
+  if (!opt)
+    return boost::none;
+  auto opt_unwrapped = opt.get();
+  auto pt = libzcash::SaplingNotePlaintext::decrypt(
+          outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
+  if (!pt)
+    return boost::none;
+  auto pt_unwrapped = pt.get();
+  return std::make_pair(pt_unwrapped, libzcash::SaplingPaymentAddress(pt_unwrapped.d, opt_unwrapped.pk_d));
+}
+
+}
+
 void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress) {
 
   LOCK2(cs_main, pwalletMain->cs_wallet);
@@ -87,18 +190,29 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
       const SpendDescription& spendDesc = wtx.vShieldedSpend[i];
       UniValue obj(UniValue::VOBJ);
 
-      SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier];
-
+      // Lookups by find(), not operator[]: that inserted an entry for every other wallet's nullifier it met
+      // (and could insert an empty note into a wallet tx).
       if (pwalletMain->IsSaplingNullifierFromMe(spendDesc.nullifier)) {
-        const CWalletTx* parent = pwalletMain->GetWalletTx(pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash);
-        const OutputDescription& output = parent->vShieldedOutput[op.n];
-        auto nd = pwalletMain->mapWallet[pwalletMain->mapSaplingNullifiersToNotes[spendDesc.nullifier].hash].mapSaplingNoteData[op];
-        auto pt = libzcash::SaplingNotePlaintext::decrypt(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
+        const SaplingOutPoint op = pwalletMain->mapSaplingNullifiersToNotes.at(spendDesc.nullifier);
+        const CWalletTx* parent = pwalletMain->GetWalletTx(op.hash);
+        auto ndIt = parent->mapSaplingNoteData.find(op);
+        // The spent note's plaintext, from the wallet's cache when it has it.
+        boost::optional<SaplingNoteCache> pt;
+        if (ndIt != parent->mapSaplingNoteData.end() && op.n < parent->vShieldedOutput.size()) {
+          const SaplingNoteData& nd = ndIt->second;
+          if (nd.cachedPlaintext) {
+            pt = nd.cachedPlaintext;
+          } else {
+            const OutputDescription& output = parent->vShieldedOutput[op.n];
+            auto both = libzcash::SaplingNotePlaintext::decrypt_with_note(output.encCiphertext,nd.ivk,output.ephemeralKey,output.cm);
+            if (both)
+              pt = SaplingNoteCache{ both->second, both->first.memo() };
+          }
+        }
 
         if (pt) {
-          auto note = pt.get();
-          auto pa = nd.ivk.address(note.d);
-          auto address = pa.get();
+          const libzcash::SaplingNote& note = pt->note;
+          const libzcash::SaplingPaymentAddress address(note.d, note.pk_d);
           shieldedSpendsSpentCount++;
           totalSpends += CAmount(-note.value());
           obj.push_back(Pair("address",EncodePaymentAddress(address)));
@@ -132,7 +246,7 @@ void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpen
 }
 
 
-void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress) {
+void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
 
   if(fZdebug)
     fprintf(stderr,"%s: txid=%s\n", __func__, wtx.GetHash().ToString().c_str() );
@@ -188,22 +302,18 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
   UniValue zsReceived(UniValue::VARR);
   if (isZsAddress || !filterByAddress) {
     for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
-      const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
       UniValue obj(UniValue::VOBJ);
       bool changeTx = false;
-      //Decrypt sapling incoming commitments using IVK
-      std::set<libzcash::SaplingPaymentAddress> addresses;
-      pwalletMain->GetSaplingPaymentAddresses(addresses);
-      for (auto addr : addresses) {
-        libzcash::SaplingExtendedSpendingKey extsk;
-        if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
-          auto pt = libzcash::SaplingNotePlaintext::decrypt(
-                  outputDesc.encCiphertext, extsk.expsk.full_viewing_key().in_viewing_key(), outputDesc.ephemeralKey, outputDesc.cm);
+      //Decrypt sapling incoming commitments using IVK: once per key, not once per address
+      OutputDecryptions decryptions(wtx, i);
+      for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
+        const libzcash::SaplingPaymentAddress& addr = a.addr;
+        {
+          const SaplingNoteCache* pt = decryptions.Get(a.ivk);
 
           if (pt) {
-            auto note = pt.get();
-            auto pt_unwrapped = pt.get();
-            auto memo = pt_unwrapped.memo();
+            const libzcash::SaplingNote& note = pt->note;
+            auto memo = pt->memo;
             obj.push_back(Pair("address",EncodePaymentAddress(addr)));
             obj.push_back(Pair("amount", ValueFromAmount(CAmount(note.value()))));
             obj.push_back(Pair("memo", HexStr(memo)));
@@ -219,10 +329,8 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
 
             //Check Change Status
             if (wtx.vShieldedSpend.size()!=0) {
-              std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
-              nullifierSet = pwalletMain->GetNullifiersForAddresses({addr});
               BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
-                if (nullifierSet.count(std::make_pair(addr, spendDesc.nullifier))) {
+                if (keys.IsNullifierOfAddress(a.walletIvk, spendDesc.nullifier)) {
                     changeTx = true;
                 }
               }
@@ -243,7 +351,7 @@ void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& total
 }
 
 
-void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress) {
+void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys) {
 
   LOCK2(cs_main, pwalletMain->cs_wallet);
 
@@ -296,6 +404,13 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
   //All Shielded Sends in the transaction
   UniValue zsSends(UniValue::VARR);
   if (isZsAddress || !filterByAddress) {
+    //The t to z shielding ovk comes from the HD seed: derive it once per tx, not once per output
+    boost::optional<uint256> seedOvk;
+    if (wtx.vShieldedSpend.size()==0 && wtx.vShieldedOutput.size()!=0) {
+      HDSeed seed;
+      if (pwalletMain->GetHDSeedForDerivation(seed))
+        seedOvk = ovkForShieldingFromTaddr(seed);
+    }
     for (int i = 0; i < wtx.vShieldedOutput.size(); i++) {
       const OutputDescription& outputDesc = wtx.vShieldedOutput[i];
       shieldedOutputCount++;
@@ -304,20 +419,14 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
 
       //Decrypt sapling outgoing t to z transaction using HDseed
       if (wtx.vShieldedSpend.size()==0) {
-        HDSeed seed;
-        if (pwalletMain->GetHDSeedForDerivation(seed)) {
-          auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
-                  outputDesc.outCiphertext,ovkForShieldingFromTaddr(seed),outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
+        if (seedOvk) {
+          {
+            auto sent = DecryptSent(outputDesc, *seedOvk);
 
-          if (opt) {
-            auto opt_unwrapped = opt.get();
-            auto pt = libzcash::SaplingNotePlaintext::decrypt(
-                    outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
-
-            if (pt) {
+            if (sent) {
               shieldedOutputDecryptedCount++;
-              auto pt_unwrapped = pt.get();
-              libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
+              const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
+              const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
               obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
               obj.push_back(Pair("amount", ValueFromAmount(CAmount(pt_unwrapped.value()))));
               obj.push_back(Pair("shieldedOutputIndex",i));
@@ -330,26 +439,22 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
           }
         }
 
-      //attempt Decryption of Outgoing Sapling using wallet extended spending keys
+      //attempt Decryption of Outgoing Sapling using wallet extended spending keys: once per ovk, not once per address
       } else {
-        std::set<libzcash::SaplingPaymentAddress> addresses;
-        pwalletMain->GetSaplingPaymentAddresses(addresses);
-        for (auto addr : addresses) {
-          libzcash::SaplingExtendedSpendingKey extsk;
-          if (pwalletMain->GetSaplingExtendedSpendingKey(addr, extsk)) {
-            auto opt = libzcash::SaplingOutgoingPlaintext::decrypt(
-                    outputDesc.outCiphertext,extsk.expsk.full_viewing_key().ovk,outputDesc.cv,outputDesc.cm,outputDesc.ephemeralKey);
+        std::map<uint256, boost::optional<std::pair<libzcash::SaplingNotePlaintext, libzcash::SaplingPaymentAddress>>> sentByOvk;
+        for (const SaplingHistoryKeys::SpendableAddress& a : keys.spendable) {
+          {
+            auto tried = sentByOvk.find(a.ovk);
+            if (tried == sentByOvk.end())
+              tried = sentByOvk.insert(std::make_pair(a.ovk, DecryptSent(outputDesc, a.ovk))).first;
+            const auto& sent = tried->second;
 
-            if (opt) {
-              auto opt_unwrapped = opt.get();
-              auto pt = libzcash::SaplingNotePlaintext::decrypt(
-                      outputDesc.encCiphertext,outputDesc.ephemeralKey,opt_unwrapped.esk,opt_unwrapped.pk_d,outputDesc.cm);
-
-              if (pt) {
-                auto pt_unwrapped = pt.get();
+            {
+              if (sent) {
+                const libzcash::SaplingNotePlaintext& pt_unwrapped = sent->first;
                 auto memo = pt_unwrapped.memo();
                 shieldedOutputDecryptedCount++;
-                libzcash::SaplingPaymentAddress sentAddr(pt_unwrapped.d, opt_unwrapped.pk_d);
+                const libzcash::SaplingPaymentAddress& sentAddr = sent->second;
                 obj.push_back(Pair("address",EncodePaymentAddress(sentAddr)));
                 obj.push_back(Pair("amount", ValueFromAmount(CAmount(-pt_unwrapped.value()))));
                 obj.push_back(Pair("memo", HexStr(memo)));
@@ -365,10 +470,10 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
 
                 //Check Change Status
                 if (wtx.vShieldedSpend.size()!=0) {
-                  std::set<std::pair<PaymentAddress, uint256>> nullifierSet;
-                  nullifierSet = pwalletMain->GetNullifiersForAddresses({sentAddr});
+                  libzcash::SaplingIncomingViewingKey sentIvk;
+                  pwalletMain->GetSaplingIncomingViewingKey(sentAddr, sentIvk);
                   BOOST_FOREACH(const SpendDescription& spendDesc, wtx.vShieldedSpend) {
-                    if (nullifierSet.count(std::make_pair(sentAddr, spendDesc.nullifier))) {
+                    if (keys.IsNullifierOfAddress(sentIvk, spendDesc.nullifier)) {
                         changeTx = true;
                     }
                   }
@@ -397,7 +502,7 @@ void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends,
 }
 
 
-void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType) {
+void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys) {
 
   LOCK2(cs_main, pwalletMain->cs_wallet);
 
@@ -469,7 +574,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
   }
   // Get Received
   if (returnType == 0 || returnType == 2) {
-    zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool);
+    zsTxReceivedToJSON(wtx, received, totalReceived, strAddress, fBool, keys);
     if (!fBool || totalReceived != 0) {
       tx.push_back(Pair("received",received));
     }
@@ -479,7 +584,7 @@ void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAd
   if (returnType == 0 || returnType == 3) {
     //Only include sends if there are spends that belong to the wallet.
     if (totalSpends != 0 || fBool) {
-      zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool);
+      zsTxSendsToJSON(wtx, sends, totalSends, strAddress, fBool, keys);
     }
     if (!fBool || totalSends != 0) {
       tx.push_back(Pair("sends",sends));

src/wallet/rpchushwallet.h

@@ -13,10 +13,35 @@ struct balancestruct {
   CAmount immature;
 };
 
+/** The wallet's Sapling keys as the history helpers use them, gathered once per RPC call. The helpers used to
+ *  look them up for every output and every address, re-deriving each full viewing key (an elliptic-curve
+ *  operation), and walked every note in the wallet for each change check. Build it with pwalletMain->cs_wallet
+ *  held, and keep holding it while the helpers run: that keeps the wallet's keys and notes fixed. The one thing
+ *  it can't stop is the walletpassphrase timer relocking the wallet mid-call; the call then reports the keys as
+ *  they were when it started, where the old per-output lookups switched over partway. */
+struct SaplingHistoryKeys {
+    struct SpendableAddress {
+        libzcash::SaplingPaymentAddress addr;
+        libzcash::SaplingIncomingViewingKey ivk;         //!< from its spending key: what received outputs are decrypted with
+        uint256 ovk;                                     //!< from its spending key: what outCiphertexts are decrypted with
+        libzcash::SaplingIncomingViewingKey walletIvk;   //!< GetSaplingIncomingViewingKey(addr), as the change check looked it up
+    };
+    //! Every wallet address whose spending key is available (never while the wallet is locked), in
+    //! GetSaplingPaymentAddresses order.
+    std::vector<SpendableAddress> spendable;
+    //! Whether GetNullifiersForAddresses({addr}) would contain (addr, nullifier), given addr's
+    //! GetSaplingIncomingViewingKey result (zero when the wallet doesn't know addr): nullifier belongs to a
+    //! wallet note received with that ivk. Walks the wallet's notes once, on first use.
+    bool IsNullifierOfAddress(const libzcash::SaplingIncomingViewingKey& addrIvk, const uint256& nullifier) const;
+private:
+    mutable boost::optional<std::set<std::pair<libzcash::SaplingIncomingViewingKey, uint256>>> noteNullifiers;
+};
+SaplingHistoryKeys GetSaplingHistoryKeys();
+
 void zsTxSpendsToJSON(const CWalletTx& wtx, UniValue& spends, CAmount& totalSpends, CAmount& filteredSpends, const std::string& strAddress, bool filterByAddress);
-void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress);
-void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress);
-void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType);
+void zsTxReceivedToJSON(const CWalletTx& wtx, UniValue& received, CAmount& totalReceived, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
+void zsTxSendsToJSON(const CWalletTx& wtx, UniValue& sends, CAmount& totalSends, const std::string& strAddress, bool filterByAddress, const SaplingHistoryKeys& keys);
+void zsWalletTxJSON(const CWalletTx& wtx, UniValue& ret, const std::string strAddress, bool fBool, const int returnType, const SaplingHistoryKeys& keys);
 
 
 #endif //HUSH_WALLET_RPCWALLET_H

src/wallet/rpcwallet.cpp

@@ -3230,6 +3230,7 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
         }
 
 
+        const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
         uint64_t t = GetTime();
         for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it)
         {
@@ -3250,7 +3251,7 @@ UniValue getalldata(const UniValue& params, bool fHelp,const CPubKey&)
               continue;
             }
 
-            zsWalletTxJSON(wtx, trans, "*", false, 0);
+            zsWalletTxJSON(wtx, trans, "*", false, 0, historyKeys);
             if (trans.size() >= nCount) break;
 
         }
@@ -3526,6 +3527,7 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
     }
 
 
+    const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
     uint64_t t = GetTime();
     //Reverse Iterate thru transactions
     for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
@@ -3550,7 +3552,7 @@ UniValue z_listreceivedaddress(const UniValue& params, bool fHelp,const CPubKey&
         }
 
         if (includeTransaction) {
-          zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2);
+          zsWalletTxJSON(wtx, ret, params[0].get_str() , true, 2, historyKeys);
         }
 
         if (ret.size() > nCount) break;
@@ -3711,6 +3713,7 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
     }
 
 
+    const SaplingHistoryKeys historyKeys = GetSaplingHistoryKeys();
     uint64_t t = GetTime();
     //Reverse Iterate thru transactions
     for (map<int64_t,const CWalletTx*>::reverse_iterator it = orderedTxs.rbegin(); it != orderedTxs.rend(); ++it) {
@@ -3737,7 +3740,7 @@ UniValue z_listsentbyaddress(const UniValue& params, bool fHelp,const CPubKey&)
       if (nFilterType == 2 && wtx.GetDepthInMainChain() > nFilter)
           continue;
 
-      zsWalletTxJSON(wtx, ret, "*", false, 0);
+      zsWalletTxJSON(wtx, ret, "*", false, 0, historyKeys);
 
 
       if (ret.size() >= nCount) break;