Setup Guide

Complete walkthrough for setting up an Agave validator on Solana testnet.

This is a guide for getting your validator setup on the Solana testnet cluster for the first time. Testnet is a Solana cluster used for performance testing of the software before the software is used on mainnet. Since testnet is stress tested daily, it is a good cluster to practice validator operations.

Although the guide is specific to testnet, it can be adapted to mainnet or devnet as well.

Prerequisites

Before starting, make sure you’ve reviewed the system requirements and have the necessary hardware and network setup.

1. Open The Terminal Program
2. You will be running commands on your trusted computer, not on the remote machine that you plan to use for validator operations. Locate the terminal program on your trusted computer:
4. Install The Solana CLI Locally
5. Validator operators are required to install the tools included in the Solana CLI using the installation instructions.
6. Once the Solana CLI is installed, verify the installation:
7. solana --version
   

8. You should see output similar to:
9. solana-cli 1.14.17 (src:b29a37cf; feat:3488713414)
   

10. Verify the agave-validator binary is installed:
11. agave-validator --version
    

12. You should see output similar to:
13. agave-validator 2.3.1 (src:e3eca4c1; feat:3640012085, client:Agave)
    

14. Configure Solana CLI for Testnet
15. Change your config to make requests to the testnet cluster:
16. solana config set --url https://api.testnet.solana.com
    

17. Verify your config:
18. solana config get
    

19. You should see: RPC URL: https://api.testnet.solana.com
20. Create Keypairs
21. On your local computer, create the 3 keypairs that you will need to run your validator.
22. Some operators choose to make vanity keypairs for their identity and vote account using the grind sub command.

23. Create the validator identity keypair:
24. solana-keygen new -o validator-keypair.json
    

25. Create the vote account keypair:
26. solana-keygen new -o vote-account-keypair.json
    

27. Create the authorized withdrawer keypair:
28. solana-keygen new -o authorized-withdrawer-keypair.json
    

29. The authorized-withdrawer-keypair.json should be considered very sensitive information. Many operators choose to use a multisig, hardware wallet, or paper wallet for the authorized withdrawer keypair. The withdrawer keypair should never be stored on the remote machine that the validator software runs on.

30. Create a Vote Account
31. Set the default keypair that the Solana CLI uses:
32. solana config set --keypair ./validator-keypair.json
    

33. Verify your account balance (should be 0):
34. solana balance
    

35. Airdrop some SOL to create your vote account:
36. solana airdrop 1
    

37. The airdrop sub command does not work on mainnet, so you will have to acquire SOL and transfer it into this keypair’s account if you are setting up a mainnet validator.

38. Create the vote account:
39. solana create-vote-account -ut \
        --fee-payer ./validator-keypair.json \
        ./vote-account-keypair.json \
        ./validator-keypair.json \
        ./authorized-withdrawer-keypair.json
    

40. The -ut flag tells the CLI command that we would like to use the testnet cluster. --fee-payer specifies the keypair that will be used to pay the transaction fees.

41. Save the Withdrawer Keypair Securely
42. Make sure your authorized-withdrawer-keypair.json is stored in a safe place. If you have chosen to create a keypair on disk, you should first backup the keypair and then delete it from your local machine.
43. If you lose your withdrawer keypair, you will lose control of your vote account. You will not be able to withdraw tokens from the vote account or update the withdrawer. Make sure to store the authorized-withdrawer-keypair.json securely before you move on.

44. SSH To Your Validator
45. Connect to your remote server:
46. ssh user@<server.hostname>
    

47. You will have to check with your server provider to get the correct user account and hostname.
48. Update Ubuntu Packages
49. Make sure you have the latest package versions on your server:
50. sudo apt update
    sudo apt upgrade
    

51. Create Sol User
52. Create a new Ubuntu user, named sol, for running the validator:
53. sudo adduser sol
    

54. It is a best practice to always run your validator as a non-root user, like the sol user we just created.

55. Setup Hard Drives
56. Verify you have at least 2TB of disk space mounted:
57. df -h
    

58. To see available hard disk devices:
59. lsblk -f
    

60. You may see some devices in the list that have a name but do not have a UUID. Any device without a UUID is unformatted.
61. Format the Ledger Drive
62. Assuming you have an nvme drive at /dev/nvme0n1, format the drive:
63. sudo mkfs -t ext4 /dev/nvme0n1
    

64. Verify the UUID was created:
65. lsblk -f
    

66. Mount the Ledger Drive
67. Create a directory for mounting:
68. sudo mkdir -p /mnt/ledger
    

69. Change ownership to the sol user:
70. sudo chown -R sol:sol /mnt/ledger
    

71. Mount the drive:
72. sudo mount /dev/nvme0n1 /mnt/ledger
    

73. Format and Mount the Accounts Drive
74. Assuming you have a device at /dev/nvme1n1, format the device:
75. sudo mkfs -t ext4 /dev/nvme1n1
    

76. Verify the UUID:
77. lsblk -f
    

78. Create a directory for mounting:
79. sudo mkdir -p /mnt/accounts
    

80. Change ownership:
81. sudo chown -R sol:sol /mnt/accounts
    

82. Mount the drive:
83. sudo mount /dev/nvme1n1 /mnt/accounts
    

84. Apply System Tuning
85. Optimize sysctl knobs:
86. sudo bash -c "cat >/etc/sysctl.d/21-agave-validator.conf <<EOF
    # Increase max UDP buffer sizes
    net.core.rmem_max = 134217728
    net.core.wmem_max = 134217728
    
    # Increase memory mapped files limit
    vm.max_map_count = 1000000
    
    # Increase number of allowed open file descriptors
    fs.nr_open = 1000000
    EOF"
    
    sudo sysctl -p /etc/sysctl.d/21-agave-validator.conf
    

87. Increase systemd and session file limits:
88. sudo bash -c "cat >/etc/security/limits.d/90-solana-nofiles.conf <<EOF
    # Increase process file descriptor count limit
    * - nofile 1000000
    # Increase memory locked limit (kB)
    * - memlock 2000000
    EOF"
    

89. Close all open sessions (log out then, in again) for the changes to take effect.

90. Copy Keypairs to Validator
91. On your personal computer, securely copy your keypair files to the validator server:
92. scp validator-keypair.json sol@<server.hostname>:
    scp vote-account-keypair.json sol@<server.hostname>:
    

93. The vote-account-keypair.json does not have any function other than identifying the vote account to potential delegators. Only the public key of the vote account is important once the account is created.

94. Switch to Sol User
95. On the validator server, switch to the sol user:
96. su - sol
    

97. Install Agave Validator
98. Your remote machine will need agave-validator installed. For simplicity, install the application as the sol user. Refer to the build from source instructions.
99. Create Validator Startup Script
100. In your sol home directory, create a folder called bin and a startup script:
101. mkdir -p /home/sol/bin
     touch /home/sol/bin/validator.sh
     chmod +x /home/sol/bin/validator.sh
     

102. Open the file for editing:
103. nano /home/sol/bin/validator.sh
     

104. Copy and paste the following contents:
105. #!/bin/bash
     exec agave-validator \
         --identity /home/sol/validator-keypair.json \
         --vote-account /home/sol/vote-account-keypair.json \
         --known-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on \
         --known-validator 7XSY3MrYnK8vq693Rju17bbPkCN3Z7KvvfvJx4kdrsSY \
         --known-validator Ft5fbkqNa76vnsjYNwjDZUXoTWpP7VYm3mtsaQckQADN \
         --known-validator 9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv \
         --only-known-rpc \
         --log /home/sol/agave-validator.log \
         --ledger /mnt/ledger \
         --accounts /mnt/accounts \
         --rpc-port 8899 \
         --dynamic-port-range 8000-8020 \
         --entrypoint entrypoint.testnet.solana.com:8001 \
         --entrypoint entrypoint2.testnet.solana.com:8001 \
         --entrypoint entrypoint3.testnet.solana.com:8001 \
         --expected-genesis-hash 4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY \
         --wal-recovery-mode skip_any_corrupted_record \
         --limit-ledger-size
     

106. Refer to agave-validator --help for more information on what each flag does. This startup script is specifically intended for testnet.

107. Verify Validator is Working
108. Test that your validator.sh file runs properly:
109. /home/sol/bin/validator.sh
     

110. In a new terminal window, SSH into your server and verify the process is running:
111. ps aux | grep agave-validator
     

112. Check the Logs
113. Tail the logs to verify proper operation:
114. su - sol
     tail -f agave-validator.log
     

115. Keep an eye out for any lines that say ERROR. Assuming you do not see any error messages, exit out of the command.
116. Verify Gossip Registration
117. Identify your validator’s pubkey:
118. solana-keygen pubkey ~/validator-keypair.json
     

119. Check that your validator is in gossip:
120. solana gossip | grep <pubkey>
     

121. You should see a single line with your validator’s IP address and pubkey.
122. Verify in Validators List
123. After you have staked some SOL to your validator and the stake has activated, verify your validator is ready:
124. solana validators | grep <pubkey>
     

125. Monitor Catchup Progress
126. The solana catchup command shows how quickly your validator is processing blocks:
127. solana catchup <pubkey>
     

128. Once you are happy that the validator can start up without errors, stop the validator by pressing CTRL+C.
129. Create System Service
130. Create the systemd service file at /etc/systemd/system/sol.service:
131. [Unit]
     Description=Solana Validator
     After=network.target
     StartLimitIntervalSec=0
     
     [Service]
     Type=simple
     Restart=always
     RestartSec=1
     User=sol
     LimitNOFILE=1000000
     LimitMEMLOCK=2000000000
     LogRateLimitIntervalSec=0
     Environment="PATH=/bin:/usr/bin:/home/sol/.local/share/solana/install/active_release/bin"
     ExecStart=/home/sol/bin/validator.sh
     
     [Install]
     WantedBy=multi-user.target
     

132. Implement log rotation:
133. cat > logrotate.sol <<EOF
     /home/sol/agave-validator.log {
       rotate 7
       daily
       missingok
       postrotate
         systemctl kill -s USR1 sol.service
       endscript
     }
     EOF
     sudo cp logrotate.sol /etc/logrotate.d/sol
     systemctl restart logrotate.service
     

134. Start your validator:
135. sudo systemctl enable --now sol
     

136. Verify it’s running:
137. tail -f /home/sol/agave-validator*.log
     

Next Steps

Now that your validator is running:

1. Set up monitoring with agave-watchtower
2. Review security best practices
3. Learn about day-to-day operations
4. Understand troubleshooting common issues

Common Issues

Out of Disk Space

Make sure your ledger is on a drive with at least 2TB of space.

Validator Not Catching Up

This could be a networking/hardware issue, or you may need to get the latest snapshot from another validator node.

PoH Hashes/Second Rate is Slower Than Cluster Target

If you are using agave-validator built from source, ensure that you are using a release build and not a debug build.

Ensure that your machine’s CPU base clock speed is 2.8GHz or faster. Use lscpu to check your clock speed.

Set performance governor:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Force minimum frequency to maximum:

# Example if your maximum GHz is 2.8
echo 2850000 | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq