Upstream gitian updates
This PR pulls in all gitian-related PRs that have been merged upstream since 0.11.2. The only ones I left out were documentation-only PRs, because we removed `doc/gitian-building.md` at some point. Here are the commits applied here, in the order shown in `git log` (ie. last to first):
- bitcoin/bitcoin#7283
- fa42a67
- fa58c76
- bitcoin/bitcoin#8175
- 74c1347
- bitcoin/bitcoin#8167
- 7e7eb27
- ad38204
- b676f38
- bitcoin/bitcoin#7776
- f063863
- bitcoin/bitcoin#7424
- a81c87f ~ we already partly applied
- a8ce872
- f3d3eaf ~ we already partly applied
- 475813b
- ~~cd27bf5~~ X we already applied
- bitcoin/bitcoin#7060
- 3b468a0 ~ we removed doc/gitian-building.md
- ~~99fda26~~ X we removed doc/gitian-building.md
- bitcoin/bitcoin#7251
- fa09562
- bitcoin/bitcoin#6900
- ~~2cecb24~~ X we removed doc/gitian-building.md
- 957c0fd
- 2e31d74
- ~~0b416c6~~ X we removed QT
- 9f251b7
- bitcoin/bitcoin#6854
- 579b863 ~ we already partly applied
Part of #540
Initial packaging for Debian
This adds some source files, a script and skeleton for building a Debian package that installs the zcash daemon and CLI tool.
To test it:
Check out my branch.
Build Zcash binaries: `./zcutil/build.sh`
Build the package: `./zcutil/build-debian-package.sh`
You now have a .deb in the source root folder, so install it: `sudo dpkg -i zcash-1.0.0-rc1-amd64.deb`
Now you can run `zcashd`, `zcash-cli`, and `zcash-fetch-params` from `$PATH`.
This pull request will need further refinement and review, so let's leave it open for a while.
Return a more informative error message when trying to spend coinbase; select non-coinbase inputs when sending to a transparent output if needed
For #1373 and #1519
Code change:
- Extra parameter added to AvailableCoins to include or exclude Coinbase coins. Default value of parameter is 'true' as current behaviour is to include Coinbase coins.
- SelectCoins, used for sending taddr->taddr, will now exclude Coinbase coins.
Unit test:
Tried to write a test to focus on the extra parameter added to AvailableCoins but could not.
Empirical testing on Testnet:
Current behaviour is that upstream RPC commands sendfrom and sendtoaddress try to spend coinbase coins returned by AvailableCoins. So the user will see:
```
./zcash-cli sendtoaddress mrEGRmGJhmwAa4MQjzGd86ry63vrvovu9b 1000.0
error: {"code":-6,"message":"Insufficient funds"}
./zcash-cli sendtoaddress mrEGRmGJhmwAa4MQjzGd86ry63vrvovu9b 0.00003000
error: {"code":-4,"message":"Error: The transaction was rejected! This might happen if some of the coins in your wallet were already spent, such as if you used a copy of wallet.dat and coins were spent in the copy but not marked as spent here."}
./zcash-cli sendfrom "" mrEGRmGJhmwAa4MQjzGd86ry63vrvovu9b 0.00003000
error: {"code":-4,"message":"Error: The transaction was rejected! This might happen if some of the coins in your wallet were already spent, such as if you used a copy of wallet.dat and coins were spent in the copy but not marked as spent here."}
```
After fix is applied:
```
./zcash-cli sendtoaddress mrEGRmGJhmwAa4MQjzGd86ry63vrvovu9b 1000.0
error: {"code":-6,"message":"Insufficient funds"}
./zcash-cli sendtoaddress mrEGRmGJhmwAa4MQjzGd86ry63vrvovu9b 0.00003000
error: {"code":-4,"message":"Coinbase funds can only be sent to a zaddr"}
```
When non-coinbase UTXOs exist, they will now be selected and used:
```
./zcash-cli z_sendmany tnPJZHeVxegCg91utaquBRPEDBGjozfz9iLDHt7zvphFbZdspNgkTVLCGjDcadQBKNyUwKs8pNjDXuEZKrE1aNLpFwHgz4t '[{"address":"mx5fTRhLZwbYE7ZqhAPueZgQGSnwTbdvKU", "amount":0.01}]'
./zcash-cli sendtoaddress mrEGRmGJhmwAa4MQjzGd86ry63vrvovu9b 1000.0
error: {"code":-6,"message":"Insufficient funds"}
./zcash-cli sendtoaddress mrEGRmGJhmwAa4MQjzGd86ry63vrvovu9b 0.00003000
9818e543ac2f689d4ce8b52087607d73fecd771d45d316a1d9db092f0485aff2
./zcash-cli sendfrom "" mrEGRmGJhmwAa4MQjzGd86ry63vrvovu9b 0.00003000
899f2894823f51f15fc73b5e0871ac943edbe0ff88e1635f86906087b72caf30
```
Note that Coinbase maturity interval does not protect JoinSplits
Changed wording of Block Chain Reorganization section in security-warnings.md to note that we did not change Coinbase maturity #1387, but that this also does not protect JoinSplits from becoming invalidated in the case of a block chain reorg #953
Tor.md: change alpha to beta testnet, add new zcash hidden service address
Note: When @ageis and I checked this the other day, we were getting multiple peer connections. Think we said we'd change this before launch, so I opened issue #1510. Leaving documentation to expect a single peer.
Set wget retry options for fetching parameters
This will retry upon refused connections and similar errors, wait 1 second before the next retry, time out after 10 seconds if no data is received or the connection times out, plus try an infinite number of times.
Resolves#1196.
Inserts some notes related to testnet deployment
Should satisfy #959 for now. I don't want to get too detailed here, since the infrastructure automation is in a still non-public repository, but instead supply a basic overview. We can write more detailed documentation and attach it to that repo.
Enable high-priority alerts to put the RPC into safe mode
This reverts the changes in 986b5e257e and adds a priority check.
Continuation of #1337Closes#1106
Verify TLS certificates w/ wget in fetch-params.sh
Per NCC-2016-017, wget was being run with `--no-check-certificate`, which means that the connection can be man-in-the-middled, even if we are verifying the integrity of the parameters later via hash sums.
The rationale cited in the Bash comments does not stand up to scrutiny. There's really no persuasive reason not to verify certificates.
Fixes#1346.
Add a make command for checking expected failures
With this merged, we can add gtests prefixed with `DISABLED_`, and then configure Buildbot to run those tests but not require they pass in order to merge.
Improvements to fetch-params.sh
Two instances of the fetch-params.sh script running at once will result in corruption of the proving key and an error when the hashes are computed. This implements a lock to stop such a scenario from occurring.
We also terminate and don't create the symlinks if the downloaded parameters fail the checksum command.
Fixes#1306. This is the same as #1310 but now targeting master.
Fix casting error in GetFilteredNotes
Use int for minDepth like upstream instead of size_t which can lead to casting problems if a wallet tx has a depth of -1.
Also don't use FindMyNotes as mapNoteData has already been set on wallet tx.
@str4d As dicussed. This should be merged before other PRs related to wallet.
Upstream patch: Fix and improve relay from whitelisted peers
https://github.com/bitcoin/bitcoin/pull/7106
a9f3d3db5c0c8d1697998ed9b3e192ddbf9a31f4
An extra commit modifies the log message string, otherwise there are are a number of commits that need be to backported to add methods e.g. GetDebugMessage. These commits modify the interface in consensus/validation.h so there are conflicts to be resolved. e.g.
9003c7c
a9ac95c
5f12263
fbf44e6
Tweaks to difficulty adjustment algorithm
This PR changes the difficulty algorithm to adjust from the average difficulty over the
block window instead of from the last difficulty. It also removes the special rules for the
testnet, which are incompatible with difficulty averaging.
Closes#147 again.
Deallocate the public parameters during Shutdown.
This also has it deallocated in `GenerateParams` and `test_bitcoin`. The virtual destructor probably isn't necessary but I added it just in case we need it some other time.
Separate concepts of block difficulty and network difficulty in RPC
"Block difficulty" is the difficulty listed in a block's header, which in the
testnet can sometimes be min-difficulty (if time-since-last-block is too large).
"Network difficulty" is the difficulty that the network was trying to satisfy
at a particular block height. In mainnet this is always equal to the difficulty
of the solved block for that height, but in testnet the network difficulty is
derived from the last non-min-difficulty block difficulty.
This commit fixes the RPC APIs that are intended to show network difficulty, so
that on testnet they don't sometimes drop to 1.0, confusing users.
Closes#1181
Closes#1315 - change to RPC getblocksubsidy
For #1315
RPC getblocksubsidy height parameter is now optional
Returns result for future block
Test has been added to verify parameter input and results
Update payment-api.md
Add z_getoperationresult to documentation.
@ebfull This document might be updated again with a few notes, so let's merge closer to release. Thanks.
Output Equihash solution in RPC results as a hex string
When the Equihash solution was changed to a compact form in #1175, the block header changed from storing a vector of indices to a vector of chars. `blockToJSON()` was not updated, which meant that RPC calls like `getblock` returned the solution as a list of 1344 integers.
This PR changes `blockToJSON()` to output the Equihash solution as a hex string, identical to how it appears in the block header.
Undo debugging change from 5be6abbf84
This was unintentionally committed, and caused Equihash verification of blocks
without parents to be skipped. This only affects the genesis block on the test
network, but also causes the "time verifyequihash" benchmark to incorrectly
appear instantaneous.
RPC calls to manage zkey creation, management and backup
Rebased on PR #1205 for keystore functionality.
For #1197, implements:
z_getnewaddress
z_importkey
z_exportkey
For #1198, spending key support added to:
z_exportwallet
z_importwallet
Example dump file containing zkeys:
https://gist.github.com/bitcartel/d7013b30b19419c6a550bc5d6ff6cc47
Also adds:
z_listaddresses
Tests in:
gtest/test_wallet_zkeys
test/rpc_wallet_tests
Fix formatting of first paragraph of payment API document
The description of taddr and zaddr were bunched up one line. This PR fixes the issue.
Self-ACK. Please merge.
Fix inconsistent -O1/-O2, fix libzcash flags, add -fwrapv -fno-strict-aliasing
Closes#1168. In that ticket I decided the optimization flags for dependencies are out of scope, i.e. we go with whatever the upstream package maintainer chose.
Implement zkSNARK compression
Implement zkSNARK compression.
The serialization format of zkSNARKs has now been specified in our protocol specification. G1/G2 elements are now compressed into 33 byte and 65 byte formats, repsectively. `MONTGOMERY_OUTPUT` is enabled everywhere to make proving/verifying keys load faster and be consistent with the MPC.
Proofs are now 296 bytes. This PR also updates the proving and verifying keys.
Closes#1103. Closes#471.
See also https://github.com/zcash/libsnark/pull/3 and https://github.com/zcash/libsnark/pull/4
Improve speed and accuracy of zcbenchmark validatelargetx
The verification test, in a loop, passes `spending_tx` (a `CMutableTransaction`) to the constructor of `MutableTransactionSignatureChecker`, which immediately uses it to create a non-mutable `CTransaction` object, which is used for the actual verification process.
Since `spending_tx` is not mutated during the verification loop & process, we can instead convert it to a `CTransaction` just once, and use it with `TransactionSignatureChecker`.
This removes the time to create `CTransaction` objects from the benchmark itself.
Results show an improvement in running time to complete the verification phase of the test and consistent times across z7 and z8 releases.
```
Benchmarks on i3 processor:
z7 old 228.67205900 --> z7 new 49.27225200
z7 old 229.90048900 --> z7 new 48.38650700
z8 old 295.77963800 --> z8 new 48.37695100
z8 old 294.32640100 --> z8 new 49.93216100
```
Add gtest for non-malleable txids.
1. Test txid of normal transaction is double sha256 with sigscript input data removed
2. Test txid of coinbase transaction is double sha256 over whole tx
You can run a single test with:
./zcash-gtest --gtest_filter="txid_tests*"
A fix for transaction malleability
This PR fixes transaction malleability by not including the sigscript of transaction inputs and joinsplit sigs when hashing the txid.
This PR supercedes PR #1101 which was a minimal solution based on a new serialization flag.
This PR introduces GetTxid() to distinguish between getting a transaction id and the double sha256 hash.
The key changes are:
- Adding GetTxid() method to CTransaction which makes a copy of the transaction, clearing out the sigscript and joinsplitsig fields, before hashing.
- Verifying that every call to GetHash() actually wants a txid, and replacing with GetTxid().
- Renaming GetHash() to GetSerializeHash()
- Rationale: In future, upstream code we want to merge will use GetHash() but we don't know the intent. We should check to see if the intent is to receive a txid (most likely) in which case we replace with GetTxid(), or if upstream actually wants a double hash of the transaction we can use GetSerializeHash().
- Updated genesis data in chainparams.cpp
Note that coinbase transactions are excluded as they need the sigscript hashed to help avoid duplicate txids per BIP34:
- This modification is related to a question from @ebfull on PR #1101 - "Can we think of a way this change allows us to construct two transactions with the same txid which can simultaneously appear in the blockchain? My guess is it would be possible to construct a coinbase transaction of such a form... this surely breaks invariants."
This PR Passes all tests in test_bitcoin (test data was updated in bloom_tests, miner_tests and script_tests).
Remove the assumption that n/(k+1) is a multiple of 8
This version works, but generates the initial rows in a way that is probably
not what we want to specify.
Closes#1148
[WIP] Add more commands to run unit tests under valgrind.
This runs both zcash-gtest and test_bitcoin under valgrind. There's a corresponding PR to the buildbot config https://github.com/Electric-Coin-Company/bbotzc/pull/23. Closes#761.
Extend try catch block around call to libsnark verifier
As discussed in #1126.
@daira Per your [comment](https://github.com/zcash/zcash/pull/1126#issuecomment-234714939):
> I would like assertion errors during tests to cause a test failure (unless the test explicitly expects them). Can we split this into verify and verify_internal, where the former does the try/catch around a call to verify_internal, and the latter is called by verification unit tests?
Did you mean move everything inside the extended try/catch to verify_internal, or just the call to r1cs_ppzksnark_verifier_strong_IC?
Bump the (minimum) protocol version to avoid invoking legacy behavior
This sets the current protocol version to `170002` (an order of magnitude larger than Bitcoin's) and the minimum to the same.
Closes#1107
Collect all permutations of final solutions
This fixes a small bug where if there was a three-way (or more) collision in the final step, one or more valid solutions would be left out.
Remove the constraint system from the alpha proving key.
This removes the constraint system from the `z5-proving.key`, shortening it by about 35%. It appeared at the end of the file, which is why we didn't need to change public parameters in #1104.
Add security warnings doc with warning about side channels.
Closes#5. Closes#785. Closes#488. Closes#784.
Let's only merge this once we're sure the warning is at least as strong as it needs to be (and thus sufficient to close those tickets).
New private/public key pairs for broadcasting alert messages
Implements #424
Fixes and integrates method of sending alerts as described by upstream here:
- https://gist.github.com/laanwj/0e689cfa37b52bcbbb44
To send an alert:
- Copy private keys into alertkeys.h.
- Modify alert parameters and message found in sendalert.cpp
- Build and run to send the alert e.g. ./zcashd -printtoconsole -sendalert
Tested and verified with local nodes on alpha 6 testnet.
Rename things to match protocol specification
This is a pass through the code to eliminate usage of "pour" and "serial" and any other outdated terminology.
Closes#602
zkbot