TL;DR

1. We sincerely apologize for a program incident leading to the sudden closure of the OptiFi mainnet program and we could not recover it
2. The total fund of 661K USDC is all locked in the PDAs on-chain
3. We will compensate all users’ funds and prevent it from happening again
4. We appeal to all Solana developers to carefully use ‘solana program close’
5. We suggest Solana officials add the warning regarding the irrecoverable result of closing a program in Solana Docs and Command-Line Interface (CLI) to help Solana devs understand the function

Background

On 29th August around 06:00 UTC, we had an update to our Solana program code, so our deployer tried to upgrade the OptiFi program on Solana mainnet.

However we accidentally used the ‘solana program close’ command, resulting in our OptiFi program on mainnet being unfortunately closed. All user’s funds and open positions on OptiFi locked in PDAs, $661K in total (AMM vault, user account…) and it’s not recoverable at the moment of writing.

Fully compensation for all users

We promise that we will return all users’ deposits and settle all user positions manually according to Pyth oracle at 8 AM Sep 2nd UTC. All transactions and deposits will be based on Solscan. The estimated process time will be two weeks. Please head to our discord to get the latest update. We’ll be there to answer questions from all users.

How did it happen?

• OptiFi Program ID: optFiKjQpoQ3PvacwnFWaPUAqXCETMJSz2sz8HwPe9B
• Deployer address: DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M

1. At around 06:00 UTC Aug 29th, we started the deployment using the following command line with our deployment address (our Program is built with Anchor).

$ anchor deploy
Deploying workspace: https://api.mainnet-beta.solana.com
Upgrade authority: {PATH_TO}/DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M.json
Deploying program “Optifi”…
Program path: {PATH_TO}/OptiFi/Optifi/target/deploy/optifi.so…
⠉ [0/1] Finalizing transaction 4RRh5kHWf6oh5VfewVevVYT1V53ktC1tK6airHnBN7
^C

2. The deployment transaction took much longer than usual, instead of returning a Success sign quite soon - probably it’s because of bad network status, so we aborted the deployment process with ‘CTL+C’.

3. Later we found that 17.2023808 SOL had been transferred into the new buffer account in the deployment tx below, with 14.96709408 SOL left in our deployment wallet.
https://explorer.solana.com/tx/4RRh5kHWf6oh5VfewVevVYT1V53ktC1tK6airHnBN7Tuo4TvhiPYgdTixwJohfJWSTPJp247unCSuVc9dK1KM6Rk

4. This means a new buffer account was created but was not going to be used by the program account, so we wanted to get the SOL tokens in the new buffer account back.

5. We ran the following command line:

$ solana program close -u mainnet-beta --recipient DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M -k ~/solana-keys/DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M.json --authority~/solana-keys/DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M.jsonerror: The following required arguments were not provided:
 —-buffers

The command line shows ‘--buffers’ were missing, so we thought the memory phrases of the created new buffer account should be provided after the ‘--buffers’ flag. However, we didn’t get the buffer account’s memory phrases from the last step due to the process of deployment tx being interrupted, so we thought that we cannot get the 17.2023808 SOL in the buffer account back.

6. Then we remembered that we had succeeded in collecting SOL tokens in the buffer account without using buffer account’s memory phrases on Solana devnet. The approach was to close the program account and buffer accounts’ SOL balance can be collected back.

7. So we ran the following code to close the program:

$ solana program close -u mainnet-beta optFiKjQpoQ3PvacwnFWaPUAqXCETMJSz2sz8HwPe9B
--recipient DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M -k
~/solana-keys/DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M.json --authority 
~/solana-keys/DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M.json

8. At 06:07 UTC - The command line was executed successfully and the OptiFi program was closed. https://explorer.solana.com/tx/4dPWDPhDHPJhCjqcxoFosa8pbYzdvpR5LhKZ9EYjK9YpvgBTWsKhX37U9jSV1qyj3xbjvm5mpzStTiNaexVaN3jg

9. We got enough SOL balance (43.3883674 SOL) to deploy after closing the program, so we started deploying the program for the second time:

$ anchor deploy
Deploying workspace: https://api.mainnet-beta.solana.com
Upgrade authority: {PATH_TO}/DpLY9ZAomC35AXfWQmJvu7kVJgaSYyUVyCm9miZ6tj4M.json
Deploying program “Optifi”…
Program path: {PATH_TO}/OptiFi/Optifi/target/deploy/optifi.so…Error: Program optFiKjQpoQ3PvacwnFWaPUAqXCETMJSz2sz8HwPe9B has been closed, use a new Program Id
There was a problem deploying: Output { status: ExitStatus(unix_wait_status(256)), stdout: “”, stderr: “” }.

10. We didn’t realize that the program was closed permanently until the error above showed up. It’s basically saying that our deployed program has been closed and cannot be re-deployed, unless a new program id is used.

12. We immediately searched the relevant topic on Solana discord, and also asked a core dev from Solana, but the answers we got are overwhelmingly indicating that we are unable to re-deploy our program to the previous program id optFiKjQpoQ3PvacwnFWaPUAqXCETMJSz2sz8HwPe9B anymore.

Here it turned out that we didn’t really understand the impact and risk of this closing program command line. ‘solana program close’ is actually for closing the program permanently and sending the SOL tokens in the buffer account used by the program back to the recipient wallet.

Impacts

1. Our OptiFi program with id optFiKjQpoQ3PvacwnFWaPUAqXCETMJSz2sz8HwPe9B is closed permanently, and no more callable.
2. ALL the users’ margin accounts USDC tokens, option tokens, and AMMs USDC vaults are locked in where they are, because they are using PDAs, which are bound to optFiKjQpoQ3PvacwnFWaPUAqXCETMJSz2sz8HwPe9B.
3. According to our TVL on Defi Llama, the total locked USDC amount is around 661K USDC.
4. Participants of the AMM contest will not be affected. We’ll announce winners on Sep 5th.

Lessoned we learned harshly

EVERY DEPLOYMENT NEEDS A RIGOROUS PROCESS AND SINGLE POINT FAILURE CAN BE AVOIDED. PLEASE DON’T RUSH LIKE WHAT WE DID, ESPECIALLY FOR DEFI PROJECTS.

Next — How can we do better to avoid such a situation?

Strictly execute peer-surveillance approach that requires at least 3 peers to engage in the deployment process

• We will adopt a peer-surveillance approach which requires at least 3 peers to engage in the deployment process. They have the responsibility to remind the main deployer of any potential risk, and make sure each step complies with the deployment guides and norms.
• In case anything abnormal happens during the deployment process, such as bad network status or insufficient deployment fee, we should calm down and have a discussion with peers to make sure each operation is safe. Meanwhile, we should mark down every command line and returned message for further reference.

We will separate capital pools (AMM) from our Main program to minimize the impact of such mistakes from program design

Proposals to Solana Officials

Advice for solana dev docs and Command-Line Interface (CLI) to prevent reckless deployment from devs

There are tutorials about how to close programs and buffer accounts on Solana’s official doc website, but it doesn’t mention the potential risks of doing so. Thus, we suggest Solana officials add descriptions in the solana docs to warn the result of closing the program.

Source: https://docs.solana.com/cli/deploy-a-program

Double confirmation for closing a program

When running ‘solana program close’, there are no warnings or reminders for developers to clearly understand what they are doing by running this command line. Therefore, we recommend to add two-steps confirmation for running the ‘solana program close’.

Retweet this thread

Please share this information to prevent such an incident from happening again.

Open question to all devs

How do you manage your deployment risk?

Please comment in the tweet and share with us. We’re open to discussing and accepting all suggestions! Here are proposals to follow:

Stay connected for the latest update

🌐 Website: https://www.optifi.app
🕊 Twitter: https://twitter.com/OptifiLabs
👾 Discord: https://discord.gg/qJ7rumqBQJ
📚 GitBook: https://optifi.gitbook.io/optifi/