Bug in program.account.<account>.fetch() when account contains a field that is an array of Option<CustomStruct>, and CustomStruct contains Optionals.

gistfile1.md

Update

Gist #2 below provides a more concise explanation of what I believe is going wrong and causing decoding to fail. I am leaving gist #1 up, but would recommend readers to read Gist #2 first to get a better understanding of the problem.

Problem

Breaking code:

    let primaryStruct = await program.account.primaryStruct.fetch(stateAccount);

The error:

Error: Invalid option undefined
      at OptionLayout.decode (node_modules/@project-serum/borsh/src/index.ts:146:11)
      at Sequence.decode (node_modules/buffer-layout/lib/Layout.js:1090:34)
      at Structure.decode (node_modules/buffer-layout/lib/Layout.js:1234:32)
      at WrappedLayout.decode (node_modules/@project-serum/borsh/src/index.ts:100:37)
      at Structure.decode (node_modules/buffer-layout/lib/Layout.js:1234:32)
      at AccountsCoder.decode (node_modules/@project-serum/anchor/src/coder/accounts.ts:53:19)
      at AccountClient.fetchNullable (node_modules/@project-serum/anchor/src/program/namespace/account.ts:153:33)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at AccountClient.fetch (node_modules/@project-serum/anchor/src/program/namespace/account.ts:165:18)

Example code that breaks

#[account(zero_copy)]
pub struct PrimaryStruct {
    pub owner: Pubkey,
    pub is_initialized: bool,
    pub example_arr: [Option<FakeStruct>; 10],

    pub some_bump: u8,
    pub some_bump_2: u8,
}

#[derive(AnchorSerialize, AnchorDeserialize, Clone, Default, Copy)]
pub struct FakeStruct {
  pub aU8: u8,
  pub anOptionalU8: Option<u8>,
}

Example code that works (removed pub anOptionalU8: Option<u8> from the FakeStruct)

#[account(zero_copy)]
pub struct PrimaryStruct {
    pub owner: Pubkey,
    pub is_initialized: bool,
    pub example_arr: [Option<FakeStruct>; 10],

    pub some_bump: u8,
    pub some_bump_2: u8,
}

#[derive(AnchorSerialize, AnchorDeserialize, Clone, Default, Copy)]
pub struct FakeStruct {
  pub aU8: u8,
}

Example code that also works (keep the FakeStruct optional, but set example_arr to not be optional:

#[account(zero_copy)]
pub struct PrimaryStruct {
    pub owner: Pubkey,
    pub is_initialized: bool,
    pub example_arr: [FakeStruct; 10],

    pub some_bump: u8,
    pub some_bump_2: u8,
}

#[derive(AnchorSerialize, AnchorDeserialize, Clone, Default, Copy)]
pub struct FakeStruct {
  pub aU8: u8,
  pub anOptionalU8: Option<u8>,
}

Why this is a problem: I cannot currently fetch my program's state account data due to this deserialization bug.

Additional notes:

This is the bit of code in node_modules/@project-serum/borsh/src/index.ts that is failing. I threw in some logs above the error:

    decode(b, offset = 0) {
        const discriminator = this.discriminator.decode(b, offset);
        if (discriminator === 0) {
            return null;
        }
        else if (discriminator === 1) {
            return this.layout.decode(b, offset + 1);
        }
        console.log(this);
        console.log(this.discriminator.decode(b, offset));
        throw new Error('Invalid option ' + this.property);
    }

and got the following:

OptionLayout {
  span: -1,
  property: undefined,
  layout: Structure {
    span: 20,
    property: undefined,
    fields: [
      [UInt],          [UInt],
      [UInt],          [WrappedLayout],
      [WrappedLayout], [WrappedLayout],
      [WrappedLayout], [WrappedLayout],
      [WrappedLayout], [WrappedLayout],
      [UInt],          [UInt],
      [BNLayout]
    ],
    decodePrefixes: false
  },
  discriminator: UInt { span: 1, property: undefined }
}

second log:

2

Additional info 2

It appears this problem isn't only related to nested Option types. I've removed all Option types from my non-example code in an attempt to circumvent this error, yet still receive the error. My non-example code currently contains 5 u8s, 7 bools, and a u64, yet is still failing to fetch with the same error.

gistfile2.md

Update

After doing some more testing, it seems this isn't as straightforward as the initial gist entailed. This problem is not wholey to do with Option types. I have removed all form of Option types from my non-example code and issues with decoding persists.

In my non-example code, I have isolated the problem down to the CustomStruct (InnerStruct in code below) containing any form of unsigned int greater than u8

At least, currently. Last night I was seeing decoding failures when I was working with just 5 u8s and 7 bools - but I do not have that proof in front of me anymore.

I have noticed a strange descrepency in account size requirement which could be shining light on the problem:

#[account(zero_copy)]
pub struct PrimaryAccount{
    pub owner: Pubkey,
    pub is_initialized: bool,
    pub state_arr: [InnerStruct; 5000],

    pub some_bump1: u8,
    pub some_bump2: u8,
}

#[derive(AnchorSerialize, AnchorDeserialize, Clone, Default, Copy)]
pub struct InnerStruct {
    pub bool1: bool,
    pub escrow_account: Pubkey,
    pub 1_mint_id: u8,
    pub 2_mint_id: u8,

    pub 1_bump: u8,
    
    pub bool2: bool,
    pub bool3: bool,
    pub bool4: bool,
    pub bool5: bool,
    pub bool6: bool,
    pub bool7: bool,
    pub bool8: bool,

    pub 3_mint_id: u8,
    
    pub 4_mint_id: u8,
    
    // BREAKING
    pub a_u32: u32,
}

Leaving out the final a_u32 from my struct, the size of the PrimaryAccount is 225035 + 8 (8 being the anchor discriminator). 225035 is derived from std::mem::size_of::<HuntState>(). This works. I can fetch the account data with no errors

Adding a_u32 would be expected to add 4 * 5000 bytes to my PrimaryAccount, creating a size of 245035 + 8. However, calculating final size once again with std::mem::size_of::<HuntState>() comes to a confusing result of 260035. This is 15000 bytes more than anticipated - it is adding 3 extra bytes per u32.

I believe this discrepency in byte requirement is causing decoding problems.

Perhaps these additional required bytes can be explained, but that stands out to me as incorrect. Sure enough, if I provide an account of size 245043, the program panicked due to an incorrect account size being provided. If the account is provided the (seemingly incorrect) account size of 260043, the account creates sucessfully, but the decoding gives the following error when trying to decode:

Error: Invalid bool: 217
      at WrappedLayout.decodeBool [as decoder] (node_modules/@project-serum/borsh/src/index.ts:180:10)
      at WrappedLayout.decode (node_modules/@project-serum/borsh/src/index.ts:100:17)
      at Structure.decode (node_modules/buffer-layout/lib/Layout.js:1234:32)
      at Sequence.decode (node_modules/buffer-layout/lib/Layout.js:1090:34)
      at Structure.decode (node_modules/buffer-layout/lib/Layout.js:1234:32)
      at WrappedLayout.decode (node_modules/@project-serum/borsh/src/index.ts:100:37)
      at Structure.decode (node_modules/buffer-layout/lib/Layout.js:1234:32)
      at AccountsCoder.decode (node_modules/@project-serum/anchor/src/coder/accounts.ts:53:19)
      at AccountClient.fetchNullable (node_modules/@project-serum/anchor/src/program/namespace/account.ts:153:33)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at AccountClient.fetch (node_modules/@project-serum/anchor/src/program/namespace/account.ts:165:18)

When I set this up with a slightly different configuration last night, I was receiving Invalid bool: 200. I am not sure where the 200 vs 217 are coming from. In addition, testing by changing a_u32 to a u16 and a u64 see similar problems: A larger-than-expected required account size, and decoding errors. Converting a_u32 to a u8 works, presumably because it does not require extra bytes incorrectly (again, this is entirely assumption). The account size calculation comes out as expected and fetch returns properly.

Conclusion

Certain types in a nested struct create decoding issues in the current version of anchor. This renders some types of program functionality impossible and is difficult to work around in clean ways. I believe this problem warrants high priority for investigation by the anchor team to resolve.

Thanks for reading.