The mempool (from "memory pool") is an important aspect of the bitcoin technology. Just as the name implies it stores many unconfirmed bitcoin transactions in memory. When a transaction is created, signed, and broadcasted, the nodes in the bitcoin network verify the transaction, add it to their mempool and pass it on to neighbouring nodes to propagate them on the network. Once the transaction is added to a block, it is removed from the mempool.
The mempool is a temporary home for valid transactions who have not yet found a block to live in.
Everything goes smoothly, transactions are added to the mempool, and then included in a block, but if the mempool is congested, the confirmation period may take longer, and since miners are privileged to choose which transactions they want to mine, they tend to prioritize transactions based on their fee rate i.e transaction fees, this way, transactions with lower transaction fees stay longer in the mempool.
Getting information about the mempool and the transactions present in it helps to estimate the average fee rate of transactions being confirmed (there is no specified fee rate for any transaction). It also helps to give an idea of the transaction fee we should pay so that our transaction might be confirmed faster, that way we don't overpay or underpay. If our transaction is already in the mempool we can use data from the mempool as a whole to estimate how soon (in how many blocks) our transaction will be confirmed.
In this article, we are going to access the mempool of a running node programmatically using python and get information about the fee rate of transactions in it.
Getting Started
To get started:
- ensure bitcoin core is installed on your machine, we will be running it on signet for this article
- ensure python is installed on your computer (python3 preferably)
- Open your terminal, and create a mempool-fee-analysis directory
mkdir mempool-fee-analysis
- cd into the folder, to create and activate the virtual environment
cd mempool-fee-analysis
pip install virtualenv
# to create virtual environment
virtualenv env
# to activate virtual environment
source env/bin/activate
- After activating the virtual environment, install the RPC library we will use
pip install python-bitcoinrpc
- Open the directory in a code editor to start writing code.
Let's get started.
Connecting to bitcoin core
The first thing to do is to connect to the bitcoin core already installed on your computer, this can be done by using RPC.
RPC works in a way that allows a computer program to call a procedure that is executed in a different space than its own, it is a function/ method call. RPC is used to call the bitcoin-cli commands needed, you however need to connect to bitcoin core first. We are going to make use of the python-bitcoinrpc library already installed.
from bitcoinrpc.authproxy import AuthServiceProxy
def ConnectToBtccore(network, rpc_user, rpc_pwd):
if network == "mainnet":
rpc_connection = AuthServiceProxy(
"http://%s:%s@localhost:8332" % (rpc_user, rpc_pwd)
)
elif network == "signet":
rpc_connection = AuthServiceProxy(
"http://%s:%s@localhost:38332" % (rpc_user, rpc_pwd)
)
elif network == "regtest":
rpc_connection = AuthServiceProxy(
"http://%s:%s@localhost:18443" % (rpc_user, rpc_pwd)
)
return rpc_connection
The function above allows us to connect to the bitcoin core using RPC. You can connect to a Bitcoin Core instance running on mainnet, signet or regtest, so you have to know which you are running. To get your RPC username and password, check your bitcoin.conf file and configure them in there if necessary. If the connection is successful, it returns a bitcoinrpc.authproxy.AuthServiceProxy object
<bitcoinrpc.authproxy.AuthServiceProxy object at 0x108c0efd0>
else, it returns an error.
NOTE: If you specify the wrong RPC username or password, you will still get a bitcoinrpc.authproxy.AuthServiceProxy object, however when you use the object on a method you get an error, so make sure you specify the correct RPC username and password.
Getting the list of transaction fees in the mempool
To get all the transaction fees of transactions in the mempool, the RPC method getrawmempool(True) is used, where the True parameter requests the verbose output. This will return details of all the transactions in the mempool including the fee rate of the transactions.
The output of the getrawmempool(True) looks like this:
"e1f1d91df1544e46aca6593167077aa2bf8398e54bc393125cffadee8a017131": {
"vsize": 153,
"weight": 609,
"time": 1652048146,
"height": 89454,
"descendantcount": 1,
"descendantsize": 153,
"ancestorcount": 1,
"ancestorsize": 153,
"wtxid": "5ad7c1a22ee628bc4af2d97be3773ad15a8efdb992dd0c9542ee7aa7b52a76db",
"fees": {
"base": 0.00000153,
"modified": 0.00000153,
"ancestor": 0.00000153,
"descendant": 0.00000153
},
"depends": [
],
"spentby": [
],
"bip125-replaceable": false,
"unbroadcast": false
}
"e1f1d91df1544e46aca6593167077aa2bf8398e54bc393125cffadee8a017131" is the transaction ID. We can see below what each of these fee rates is based upon.
Note that an ancestor transaction can be otherwise thought of as a "parent" of this transaction. In other words, this transaction is spending some outputs created in its ancestor (parent) transaction and therefore depends on the parent to be confirmed. Conversely, a descendant transaction is a "child" of this transaction; in other words, it is spending some outputs created in this transaction.
Miners will consider chains of unconfirmed transactions (ancestors and descendants) as "packages", and try to confirm the "packages" which yield them the maximum overall fee rate per byte of block space.
"fees" : {
"base" : n, (numeric) transaction fee in BTC
"modified": n, (numeric) transaction fee with fee deltas used for mining priority in BTC
"ancestor" : n, (numeric) modified fees (see above) of in-mempool ancestors (including this one) in BTC
"descendant" : n, (numeric) modified fees (see above) of in-mempool descendants (including this one) in BTC }
Since we do not need all the information of the getrawmempool(True) output, we loop through it to get what we need and output it.
def GetTransactionFees():
rpc_connect =ConnectToBtccore('signet','bitcoin','bitcoin')
mempool_list = rpc_connect.getrawmempool(True)
transaction_fees = [{i: str(mempool_list[i]["fees"]["base"])} for i in mempool_list]
return transaction_fees
The function above will output the transaction ID with the transaction fee, the output of the previous function that connects to the bitcoin core will be used with the getrawmempool method.
It works basically like the bitcoin-cli command bitcoin-cli getrawmempool true. The output of the ConnectToBtccore function is a bitcoinrpc.authproxy.AuthServiceProxy object which acts as the bitcoin-cli, and the getrawmempool(True) acts as the getrawmempool true command.
The output of the GetTransactionFees function will look like this:
[
{'49efd31e8db77ed42394fd70ef48ce417dcc073a7e4a26953fa2a8e353c4121a': '0.00000153'},
{'d9f07b0e83d0647abba0bbe232244d0c3a75f400a7bd43fcdcae03ff55711cd4': '0.0000026'},
{'51cc8724bf902fadc382bd69d7d08c9b2ba86f150dee75e41a540db010c53903': '0.0000053'},
{'883d7667372b614662d84e880ef6aa43243a9a79fde8a976c7685e78e14b719e': '0.00000153'},
{'f063adfa3ab5aa7c34b38a20335a3c745acda5c0ffea77982271ad6d5aa91c00': '0.00000153'},
{'1443a3ce094aa2c637a13bd44f6510e6270033e15dbc098175b12f769c18595b': '0.00000137'},
{'a7d83ad666b8706dc9d32c8b8d176b1bbdebf28166ce5741d0f1a648ad60effd': '0.00000153'},
{'04ff38fbadda70a7f9d7bf10e9bf4b92ee3cadb8bc56515eab639f0678c64c32': '0.00000150'},
{'e7bb5fd75aa952251e752ce28164e7ab039f55df914835e53a863a804acc2a74': '0.00000153'},
{'ddfecaa545772647e5a56881dc6d5db0d0df116cc4a25fd13992c4d9dd219bf2': '0.00000153'}
]
//{transaction ID:transaction fee}
Now, we have the list of the transaction fees of all transactions, we can do anything we want with it, say we use it to decide what we will pay as the transaction fee for our transaction to ensure it is quickly confirmed.
getrawmempool(True)will output the transactions with fee rates and other details, whilegetrawmempool()will output just the list of transactions in the mempool bytxid.
Getting the lowest and highest transaction fee in the mempool
You might want to get the lowest and highest transaction fee in the mempool, instead of having to check through the list of transactions and their details, you can always write a script that will return the lowest and highest fee.
def GetLowestandHighestFees():
rpc_connect = ConnectToBtccore('signet','bitcoin','bitcoin')
mempool_list = rpc_connect.getrawmempool(True)
txn_fees = [str(mempool_list[i]["fees"]["base"]) for i in mempool_list]
txn_fees=set(txn_fees)
highlow = [{"lowest fee": min(txn_fees)}, {"highest fee": max(txn_fees)}]
return highlow
//output:[{'lowest fee': '0.00000153'}, {'highest fee': '0.00000200'}]
After getting all the transaction fees, using the min and max functions in python will help get the lowest and highest fee. min() function returns the lowest number in a list, which will be the lowest fee in this case, while the max() function returns the highest number in a list which is the highest fee.
Yay! You did it 💪💪.
In this article, we have been able to access the mempool to get information concerning the transaction fees of the unconfirmed transactions using python. There is much information to get from the bitcoin mempool in your python code by using RPC commands, think of any other information and try to write a function to get it or convert these functions to endpoints and call on the frontend to get a nice visual of the information.
Enjoy Coding ❤