How to Build a Bitcoin Wallet (Conceptually)

Wallets are a fundamental part of the Bitcoin ecosystem serving as the primary interface for most Bitcoin users to interact with the ecosystem. So a lot of user-facing applications contain a wallet tweaked to serve the application's needs. What follows is an introduction to BitcoinDevKit, a set of libraries which aim to provide a secure and feature-rich Bitcoin wallet SDK.

To truly understand how BDK works we will try to build a Wallet ourselves conceptually.

How to Read

This article represents a flow of thought and is therefore meant to be read in a linear fashion. It is written for budding contributors to the Bitcoin Tech ecosystem and curious individuals. The guide assumes knowledge of Bitcoin basics but at most places appropriate links are present for you to read more if you need. One last thing, the code snippets are in Rust but they are actually just meant to be pseudo-codes so knowledge of Rust is not required. Also if you want to understand BDK from the user's perspective, book-of-bdk is an awesome resource.

Let’s get our expectations in order first. Take a pause and think of what all things our wallet should do?

1. Create Transactions
2. Generate Addresses
3. Sign Transactions (-- or should it?)
4. Look at the blockchain for relevant transactions.
5. Store keys ( -- wait a minute?)
6. Buy Bitcoin ( -- really?)
7. Fee estimation (--maybe not?)
8. Mine Bitcoins (-- definitely not!)

Let's have a word over the list first.

Signing transactions can be performed by a wallet but this means that the wallet must know the private keys of the user which further implies that a bug in the wallet software or a virus in your device could possibly compromise the keys. Hence it's better if signing and storage is handled by special devices like H/W wallets and Cryobrick.

Ok, next up "Buy Bitcoin". That is the responsibility of an exchange! Thought exchanges use wallets in order to generate addresses/create transactions etc.

Also fee estimation is a cool but not necessary feature in a wallet since you could probably look at blockchain explorers or mempools to figure this out, so we'll skip this one.

And of course mining is done by ASICs and not wallets :sweat_smile: , although to construct coinbase transactions we would need addresses of the miners which we get from wallets.

Now let's go over each of the finalized requirements and think about what all do we need in order to implement these.

This is perhaps the first thing we do when we open our wallets. Assuming we know how addresses are created , let's think of how our wallet would generate addresses for us. It first needs a public key (note: not private key!) and the address type. But we need to be especially careful about address reuse!

Address Reuse

There are various entities (like ChainAnalysis) which actively try to associate Bitcoin addresses to individuals using heuristics. Address Reuse makes it easier for these entities to identify these associations and hence infringe privacy which in turn leads to censorship and targeted advertising. Also the Address Reuse becomes critical blow to security if ECDSA gets broken by advancements in Quantum Computing.Read More here

But it would be so cumbersome, if for every new address we need to keep track of a new public key! Enter BIP32 wallets.

BIP32 (Hierarchical Deterministic Wallets)

These wallets use a master key and a derivation path to deterministically generate keys. Read More

Ok we have A LOT keys here but what about the actual script to use? How do we know that? Enter Miniscript and Output Descriptors.

Miniscript

Miniscript is a language which compiles down to a subset Bitcoin Script. Miniscript helps us analyze if the script is valid (if it can be solved), whether its optimal etc. We use the policy language to specify the spending conditions. Policy is then converted to Miniscript language by the miniscript compiler, which performs optimization on the script for it to be solvable with the same conditions but the bytes we need to use to spend (satisfaction weight) becomes lesser etc. Read more

Output Descriptors

Decriptors consist of SCRIPT and KEY expressions and specify the scriptpubkey (in simple terms addresses) corresponding to our spending condition. In case of the newer descriptor types the SCRIPT can consist of policy. Read more

So with policy and output descriptors all our addresses can be generated from one single string that practically does not exhaust!

Now we have our addresses, except one thing! Remember address reuse? We do need to make sure we remember which all indexes of our descriptor we have already looked at! So we would need an Indexer to remember the last derived index of every keychain, let's call it last_revealed.

struct Indexer {
    last_revealed: HashMap <DescriptorHash, u32>
}

Also since our descriptors would generally have a wildpath (*), i.e., we would be able to generate a lot of keys from it, let's call this list of keys a keychain. To truly handle address reuse we must remember this last_revealed value across restarts! So we need some kind of persistence! More on that later.

Are we done? Hold on a bit! We talked a bit about entities which infringe privacy by using our addresses. A simple way to trick them is to use two keychains instead of one! One of the keychains will be External and the other would be Internal. And when somebody asks you for an address you give them one generated on your External keychain and you direct all your change to addresses on the Internal keychain. This way if the External (Public) keychain is compromised (maybe because it was stored on a server), the analysts still won't know half our addresses. So this trick solved our privacy issues, or does it? We'll revisit this in another section later on.

Finally done! Ready for the next requirement?

Monitoring the Blockchain#

So does our wallet need a node for this? Not really (as long as we can TRUST it -- any blockchain source works, it could be Bitcoin Core node, Electrum Client, Esplora Client or even a Compact Block Filter Client).

Electrum and Esplora

Electrum Protocol is the one used to communicate with Electrum servers. These servers index the blockchain in order to achieve faster information retrieval. Not to be confused with the electrum wallet (the wallet speaks the electrum protocol though). Esplora is a modification of the Electrs ( Rust implementation of the Electrum Server) and to communicate with Esplora servers we use Esplora Protocol. Read more about Electrum and Esplora.

Compact Block Filters (CBF)

This is the improved way of implementing a privacy preserving Simplified Verification Client, basically a way for wallets to obtain transactions of interest without trusting the peers on the Bitcoin P2P network. Read More

In most cases our Wallet would only be interested in transactions that either spend to us or from us and whether the transaction is in the chain of most Proof Of Work(POW) or not. We would want to remember the UTXOs that we have to be able to spend further.

So we need to check if a given transaction spends an output we control or pays to us. Remember the Indexer? We could have the Indexer also remember the scriptpubkey we derived (along with which index it was found on etc.) The Indexercan take the responsibility of remembering the transactions spending to us.

Also a lot of times, we would want to remember the transactions that spend to us or those we created. Perhaps to understand the conditions under which we can spend our outputs or to do a fee-bump later on. Let us store the transactions in a TxGraph, which maintains ancestry relationships between the transactions as well. Since we are storing the whole transactions anyways, our Indexer could be made lighter by just holding on to the Outpoints (txid, vout pairs for each output.) Actually it would better if we combine the Indexer and TxGraph into the IndexedTxGraph, in order to ensure these two are in sync for example we should not have OutPoint in Indexer which do not have corresponding transactions in TxGraph.

struct TxGraph {
    txs: HashMap<Txid, TxNodeInternal>
}

struct Indexer {
    last_revealed: HashMap<DescriptorHash, u32>,
    spks: BTreeMap<(Keychain, u32), ScriptBuf>,
    spk_txouts: BTreeSet<((Keychain, u32), OutPoint)>,
}

struct IndexedTxGraph {
    index: Indexer,
    graph: TxGraph,
}

But the way we query a chain source for information depends on its type.

SPK-based chain sources

These chain sources (contrary to block based sources) index information based on spk. Hence we can query transactions related to an spk easily. Electrum and Esplora are two examples.

Block based Chain sources

These chain sources provide whole blocks and mempool transactions when queried.

For an spk-based chain source we could provide the chain source the scriptpubkey (spk)s we are interested in.

For a block based source we could just query for new blocks or new transactions in mempool.

Now for every block or transaction the blockchain source tells us about through, we can update our IndexedTxGraph.

Just when we thought that life is easy, Bitcoin's distributed consensus reminds us of its presence. I mean that all this is good if there were no reorgs! In case of a reorg, the transactions we once thought were confirmed could get unconfirmed! ![[reorg.jpeg|650]]

Ok so every time we query our balance we need to ask the chain source whether each of the UTXOs actually exist in the current chain. And note we would need to do this for every UTXO, no matter what the reorg depth was. Crazy! Since in many applications, network speed is a bottleneck this design might cost us on performance. Also we might want our wallet to connect to the network as less as possible maybe because of limited network availability.

What if instead we could query the chain source for new block-headers and try to build up a view of the blockchain, say LocalChain ourselves offline?

For this every time we query the chain source, we can also ask it for the required block-headers. Also since not every block is of interest to us the LocalChain could be sparse, i.e., we can maintain CheckPoints (which contains the height, the hash of the block at this height and the previous CheckPoint).

struct CheckPoint {
    block_hash: BlockHash,
    height: u32,
    prev: Option<CheckPoint>,
}

But we started with the goal of finding if the transaction exists in the chain of most Proof Of Work(POW). For this we associate to each transaction an Anchor (pair of blockhash and block height) which anchors it in some chain. Now we just need to check if the Anchor corresponds to a block in the most POW chain (let's call it the Canonicalchain).

struct Anchor {
    block_hash: BlockHash,
    block_height: u32,
}

Also we could remember the information that the source tells us about the transaction, like when it was first_seen, last_seen from the mempool if its unconfirmed.

So our TxGraph now becomes:

struct TxGraph {
    txs: HashMap<Txid, Transaction>,
    anchors: HashMap<Txid, BTreeSet<Anchor>>,
    first_seen: HashMap<Txid, u64>,
    last_seen: HashMap<Txid, u64>,
}

Ok so now we know the UTXOs we control -- time to spend?!!

Wait a minute... For spk-based chain sources, we cannot just query the blockchain for all the spks, there are so many of them!!! Ok so we need to decide how many spks we would like to check. This brings us to two forms of querying:- Sync and Full Scan.

Full Scan and Sync

In case we have been querying the chain source frequently we can just query for scriptpubkeys that have been revealed, this is called sync. But if we have just created our wallet or are starting up after a long time, the wallet could check some extra (Lookahead many) scriptpubkeys along with those which have been revealed, this is called a full scan. Read more

In case of a block based sources we could just store some extra (lookahead many) scriptpubkeys apart from those revealed, when we add a descriptor to the Indexeror whenever we reveal a new address.

Now, we need to think of how the Wallet and the chain sources would interact. First of all we would need Clients which could speak the correct protocol (Electrum, Esplora, Bitcoin P2P etc.) . Since we want our Wallet to work with any chain source out there we need a wrapper over each of these clients which translates the Wallets requests to something that the source can understand and also the other way round.

For spk-based sources, we could have the Wallet send a SyncRequest and FullScanRequest for syncing and full-scan respectively. The SyncRequest should inform the chain source of the the spks we are interested in, our last_checkpoint.

struct SyncRequest {
    chain_tip: Option<CheckPoint>,
    spks: VecDeque<((DescriptorHash,u32), ScriptBuf)>,
}

Similarly the FullScanRequest should have the CheckPoint, the descriptors and the lookahead.

pub struct FullScanRequest {
    chain_tip: Option<CheckPoint>,
    last_revealed: BTreeMap<DescriptorHash, u32>,
    spks_by_keychain: BtreeMap<DescriptorHash,
}

Thinking about response we get for syncing we would get SyncResponse which should just give us the transactions and their timestamps and anchors along with blocks.

struct SyncResponse {
    pub txs: Vec<Transaction>,
    pub anchors: BTreeSet<(Anchor, Txid)>,
    pub seen_ats: HashSet<(Txid, u64)>,
    chain_update: Option<CheckPoint>,
}

Similarly the FullScanResponse could be (because remember during a FullScan we also update the last_revealed ):

struct FullScanResponse {
    pub txs: Vec<Transaction>,
    pub anchors: BTreeSet<(Anchor, Txid)>,
    pub seen_ats: HashSet<(Txid, u64)>,
    pub last_active_indices: BTreeMap<DescriptorHash, u32>,
    pub cTxs, Block Headershain_update: Option<CheckPoint>,
}

For the case of block based sources, we would just query the chain source wrapper for the next block and the current state of the mempool and the source could update us through a BlockEvent and a MempoolEvent respectively. The BlockEvent should contain the CheckPoint information while the MempoolEvent will have the transactions with their timestamps.

pub struct BlockEvent {
    pub checkpoint: CheckPoint,
}

struct MempoolEvent {
    pub update: Vec<(Transaction, u64)>,
}

and that's it for this section!

Transaction Creation#

Aah, finally... ! Before we go into how our Wallet would create transactions there is another big idea we need to discuss: Partially Signed Bitcoin Transactions (PSBTs)

PSBTs This is an (extensible) standard to represent a transaction in order to increase interoperability between wallets and signers. Every entity that interacts with PSBThas a role: Creator, Updator, Signer, Finalizer and Extractor. Read More

Creator and Updator Ok back to transaction building. Thankfully the last step told us what all UTXOs we can spend. So now we need to know what utxos the user wants to spend for sure(manually_selected), whom do we send the bitcoins to (recipient), the amountwe want to spend and what is the fee rate? Now it is possible that the manually selected utxos are not sufficient given the fee rate and the amount we want to send in that case we would like to use some other UTXOs too, if the user consents. But which ones? Does it matter, you ask? Yeah! Suppose the difference between the UTXOs we are spending and the ones we are creating after adjusting for fees is so small that the change output becomes a dust UTXO?

Dust outputs These are outputs which are uneconomical to spend, i.e., they cannot pay the fees for their relay. Transactions with such outputs are not relayed by Bitcoin Core nodes by default. Read More

Also the user needs to pay lesser fees if the transaction size is less so it is necessary to optimize for the size. Keeping these things in mind, we need to choose our coins carefully (this is called Coin Selection). Ok so we need a CoinSelector which given few Candidates (groups of inputs that need to be spent together) and target can decide (according to an algorithm) which Candidates to select.

What should the alogrithm be? It could be as simple as selecting randomly from the available UTXOs till we reach the desired target (called Single Random Draw). Or it could be as complicated as Branch and Bound algorithm as described in Murch's thesis. Or the LowestFee algorithm which optimizes for fees.

But in order to calculate the transaction size, we need to know the number of vbytes required to spend each output (satisfaction_weight) we are trying to spend. How do we do that? Remember this was exactly one of the goals of Miniscript !!! So we could simply ask our Miniscript library to generate a spending Plan for us when told about the signatures, hash preimages (Assets) we have! And we could query the Plan for the satisfaction_weight. After choosing the coins, using the Plan we will fill in the details of the PSBT that are needed for optimal spending.

Signer As discussed earlier, ideally the wallet shouldn't sign transactions itself. i.e., we could get our created PSBTs signed by external signers!

Finalizer We finally finalize the transaction, which means that we use the Wallet's descriptors to populate the final_scriptsig and the final_witness_script fields of the psbt.

Extract We can now extract the transaction and send it to the chain source!.

**Persistence **

It would be terrible to expect our users to remember their public descriptors, so we should persist these. Again note that we only persist the public descriptors since the security trade-off of persisting private descriptors is too much. Also the public descriptors are enough to back-up our whole wallet. That's why BIP-32 Wallets and Output Descriptors are GREAT ideas.

But the mainnet blockchain is huge and scanning it every time from the start whenever we restart the wallet program is a costly operation so we need to provide an option to persist other data like the transaction data we saw, the Anchors, the last_revealed, and the Local_Chain data.

For large wallets it will also be beneficial to persist the scriptpubkeys that our Indexerhas generated from the descriptors.

So how should we persist all this data? Clearly persisting the all the Wallet data every time we change the Wallet structures would be inefficient, so we need to record changes to each of the structures and persist these changes. We can have an entity, say ChangeSet for all of Wallet's structures which records changes to the structure. For the Indexer the ChangeSet could record the changes in the last_revealed for each descriptor and the new script_pubkeys to be added to the spk_cache.

It could look like this:

pub struct ChangeSet {
    pub last_revealed: BTreeMap<DescriptorHash, u32>,
    pub spk_cache: BTreeMap<DescriptorHash, BTreeMap<u32, ScriptPubKey>>,
}

Phew! Finally done! Now this is a lot of code that is common to almost all wallets. So why not have a library (or a set of libraries) which provides such structures? That is exactly what BDK is!!!

Built using the powerful rust-bitcoin (for Bitcoin primitives) and rust-miniscriptlibraries, BitcoinDevKit(BDK) provides a set of libraries which help wallet developers avoid all of the boilerplate code above.

To see how the conceptual wallet architecture we just designed translates directly into BDK libraries, look at the following:

Leveraging all this, bdk_wallet provides the Wallet structure we have been trying to build in this article!

As you must have noticed, BDK's modularity allows wallet developers to use their favourite client and their favourite storage format with the Wallet. In fact, if need be then developers could use the bare primitives provided by bdk_chain directly to create a wallet.

FFI All the libraries we have discussed are written in Rust but the Bitcoin ecosystem thrives on applications written in a variety of languages like Python, Java, React-Native. This brings in the need for bdk-ffi which is a wrapper around the bdk_wallet library and enables creating bindings to other languages using uniffi-rs. Using bdk-ffi, the bdk-android ,bdk-swift****, bdk_jvm, bdk-dart, bdk-rn and the community supported bdk-python libraries provide the bindings.

BDK-CLI#

Want to see BDK in action? Head over to the bdk-cli project which demonstrates the power of BDK using a simple command line application.

New Developments#

But as we talk about BDK, it continues to evolve! Remember we talked about client implementations? Recently the BDK team have been developing bdk-bitcoind-clientand electrum_streaming_client. Also work is being done on bdk_floresta which aims at processing data from the floresta chain source.

On the persistence side, bdk_sqlite and the community supported bdk_sqlx and bdk_redb are the new persistence backends.

Then we have bdk_sp that aims to introduce the silent-payment workflow into BDK.

Also for bdk_wallet 4.0 milestone we have some major changes coming up. Currently, the Wallet holds the signers inside it but these are planned to removed, along with the other signing APIs. Also bdk_wallet is about to introduce a new transaction building API which incorporates the new miniscript plan module (instead of the in-house policymodule), the new bdk-tx (transaction building library) and bdk_coin_select (coin-selection library).

Work is also going on a message-signing library and incorporating labels into Wallet.

Now something that I am currently helping with. Do you remember our discussion on using 2 keychains to increase privacy in the address generation section?

MultiKeychain#

If the wallet happens to use one of Bip-44,49,84,86 (which it usually would) then malicious entities could do a small change in the external public descriptor to find the internal one! In fact the only reason developers still stick to using 2 keychains is because changing it might screw up import/export of descriptors.

Also, some layer 2 protocols maintain descriptors other than the internal and external ones. This implies it is not necessary to have 2 keychains in fact it would be helpful if the Wallet could support an arbitrary number of descriptors. bdk_wallet v4.0 also plans to support this feature!

Ok, let's attempt to answer the most important question now if you are an aspiring Bitcoin FOSS contributor.

Why should you contribute to BDK?#

As we observed earlier, wallets are fundamental to the Bitcoin ecosystem as these are the objects that any Bitcoin user would interact with on a regular basis. So by contributing to BDK you make the whole Bitcoin ecosystem a happier place! for the user who is perhaps excited and nervous about spending bitcoin for the first time as well as the developer who wants to implement their awesome Bitcoin wallet idea.

Also, you get to work with the most awesome set of developers and learn a tonne about software engineering along the way (API design, importance of documentation, modularity etc.).

So are you ready to start?

Your Contribution Checklist#

• First of all, pat yourself at the back for reaching the end of this long article.
• Hop over to BDK's discord server and join the weekly dev-calls to see what all is being worked upon.
• Read the Contributing.md to understand the coding conventions used in the project.
• Go over the existing issues or work on some idea you got on the dev-call.
• Submit your PR/review and congrats! you are a BDK contributor :)

And that's it for the article! This guide was my attempt to explain things which initially confused me when I started contributing to BDK.

I hope this provides you a solid foundation to think about BDK and also makes you have more questions than were answered here! Feel free to shoot these questions in BDK's discord server.

Although the article ends here your journey into BDK has just begun!

See ya!!!

About me#

Well, I am a panda who followed the white rabbit into the Bitcoin rabbit hole. As a legendary coding panda fascinated by complex systems—especially those relating to freedom tech— I write the code and built a human bot to do the talking. The bot used to contribute to BDK as a Summer of Bitcoin intern and now continues as a Bitshala fellow, alongside TA-ing Programming Bitcoin. Ping the bot on the Bitshala or BDK discord server!

Acknowledgement#

No sensible Bitcoin educational resource can avoid acknowledging Greg Walker (the creator of learnmeabitcoin) who is the reason why so many people actually understand Bitcoin. Also a big thank you to the BDK team whose guidance has helped the author understand and contribute to this awesome set of libraries. And a big hug to the Bitshala team and SOB, who provided the author the opportunity to contribute Bitcoin Open Source.

Till then,

**Whoever you are, whatever you do, do it for Bitcoin - and Bitcoin will find you. **

*This article is written by codingp110 for bitshala_org.