Manipulate Neural Fields

This is part of my journey of learning NeRF.

2.5. Manipulate Neural Fields

Neural fields is ready to be a prime representation, similar as point clouds or meshes, that is able to be manipulated.

image-20221212211525928

You can either edit the input coordinates, or edit the parameters θ.

On the other axis, you can edit through an explicit geometry, or an implicit neural fields.

image-20221212213802209

The following examples 落在不同的象限。

Editing the input via Explicit geometry (left-up)

• You can represent each object using a separated neural field (local frame), and then compose them together in different ways.

• If you want to manipulate not only spatially, but also temporaly, it is also possible. You can add a time coordinate as the input of the neural field network, and transform the time input.

• You can also manipulate (especially human body) via skeleton.

  
  image-20221212212838893

  • Beyond human, we can also first estimate different moving parts of an object, to form some skeleton structure, and then do the same.

    
    Noguchi etal, CVPR22

• Beyond rigid, we can also manipulate via mesh. coz we have plenty of manipulation tools on mesh. The deformation on mesh can be re-mapped as the deformation on the input coordinate

  
  image-20221212213601773

Editing the input via Neural Flow Fields (left-down)

image-20230104183222294

We use the fi→j to edit the ri→j to represent one ray into another one.

We need to define the consistency here, so that the network can learn through forward and backward:

image-20230104183453487

Editing network parameters via Explicit geometry (right-up)

The knowledge is already in the network. So instead of editing the inputs, we can directly edit the network parameters for generating new things.

image-20230104185014312

• This proposed solution makes use of an encoder. The encoder learns to represent the rotated input as a high-dimensional latent code Z, with the same rotation R, in 3-dim space. The the following network use the latent code to generate the fθ

image-20230104185544623

• In this work, the key idea is to map the high-resolutional object and the similar but lower resolutional object into the same latent space. Then, you can easily manipulate the lower resolutional object, and it should also affect the higher resolutional one. Then, the shared latent space are put into the following neural field network, which outputs high resolutional results.

image-20230104202425695

image-20230104202625346

• This work (Yang et al. NeurlPS'21) about shape editing is "super important" but the speaker does not have enough time... Basically it shows that the tools that we use to manipulate a mesh can also be used on a neural field, where we can keep some of the network parameters to make sure the basic shape of the object the same, and then the magical thing is the "curvature manipulation" item. Given the neural field is differentiable, this can be achieved.

image-20230104203311551

• Obeying the points (a.k.a generalization). It makes sure the manipulation done on the input points are reconstructed.

Editing network parameters via Neural Fields (right-down)

image-20230104204330741

• This work constructs a reasonable latent space of the object, then do interpolation of different objects.
• Beyond geometry, we can also manipulate color

image-20230104204738067

It decomposes the network into shape and color networks, and we can edit each independently.

image-20230104204937204

• This is the stylization work. It mainly depends on a different loss function, which does not search for the exact feature of the vgg, but somehow the nearest neighbor.