傅里叶特征学习高频：Fourier 相关工作+实验分析+代码实现

最新推荐文章于 2024-04-10 21:33:27 发布

FakeOccupational

最新推荐文章于 2024-04-10 21:33:27 发布

阅读量3.2k

点赞数 3

分类专栏：算法文章标签：神经网络

本文链接：https://blog.csdn.net/ResumeProject/article/details/123290194

版权

算法专栏收录该内容

54 篇文章 4 订阅

订阅专栏

在这里插入图片描述
$个人感觉这个任务还是有一定特点的：输入数据应该呈现类似均匀分布$

基于坐标的MLP的相关实验

由于基于坐标的MLP易于基于梯度的优化和机器学习，并且可以比网格采样表示更紧凑，因此该策略具有吸引力。并在许多方面各种任务中取得了最先进的结果。

表示图像（合成图案生成网络）
- Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. CVPR, 2015.
- Kenneth O. Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic Programming and Evolvable Machines, 2007.
体积密度
- Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934, 2020.
占用率
- Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3D reconstruction in function space. CVPR, 2019.
符号距离
- Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. CVPR,2019.
形状表示
- Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. CVPR,2019.
- Boyang Deng, JP Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi. Neural articulated shape approximation. arXiv preprint arXiv:1912.03207, 2019.
- Kyle Genova, Forrester Cole, Aaron Sarna Daniel Vlasic, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. ICCV, 2019.
- Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3D shape. CVPR, 2020.
- Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3D scenes. CVPR, 2020.
- Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. ICCV, 2019.
- Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. CVPR,2019.
纹理合成
- Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. Learning a neural 3d texture space from 2d exemplars. CVPR, 2020.
- Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger.Texture fields: Learning texture representations in function space. ICCV, 2019.
图像的形状推断
- Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc Pollefeys, and Zhaopeng Cui. Dist:Rendering deep implicit signed distance function with differentiable sphere tracing. CVPR,2020.
- Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li. Learning to infer implicit surfaces without 3D supervision. NeurIPS, 2019.
新颖的视图合成
- Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi,and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934, 2020.
- Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3D representations without 3D supervision. CVPR, 2020.
- Shunsuke Saito, , Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. ICCV, 2019.
- Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. Scene representation networks: Continuous 3D-structure-aware neural scene representations. NeurIPS, 2019.

在这里插入图片描述
在第6节中，我们继续进行高维实验的高斯分布，并将尺度作为超参数，在验证数据集上进行调整.在映射时需要在高斯分布中随机采样参数，分布的std对实验结果的影响如下：

在这里插入图片描述

在这里插入图片描述
除了3Dshape其他的评价指标都为PSNR,3Dshape使用IoU (higher is better for all).

On the Spectral Bias of Deep Neural Networks笔记
 深度神经网络为什么不易过拟合？固有频谱偏差On the Spectral Bias of Neural Networks
github代码
 论文 18 Jun 2020
论文笔记1
论文笔记2
video
相关链接：
Nerf
https://dellaert.github.io/NeRF/
ICLR 2019 有什么值得关注的亮点？我们终于证明了，只要足够宽，随机初始化的神经网络+梯度下降真的可以拟合所有数据!!!
Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Model and training code

官方代码

二维图像回归。

以下为官方实现的核心代码：this example,for each input image coordinate $(x, y)$ , the model predicts the associated color $(r, g, b)$ .
在这个任务中，我们训练MLP从2D输入像素坐标回归到图像的相应RGB值。对于每个测试图像，我们在一个有规则间隔的屏幕上训练一个MLP
包含1/4像素的网格，并报告剩余像素的测试错误。我们比较了自然图像数据集和文本图像数据集上的输入映射。

B_dict = {}
# Standard network - no mapping
B_dict['none'] = None
# Basic mapping
B_dict['basic'] = np.eye(2)
# Three different scales of Gaussian Fourier feature mappings
B_gauss = random.normal(rand_key, (mapping_size, 2))
for scale in [1., 10., 100.]:
    B_dict[f'gauss_{scale}'] = B_gauss * scale

# This should take about 2-3 minutes
outputs = {}
for k in tqdm(B_dict):
    outputs[k] = train_model(network_size, learning_rate, iters, B_dict[k], train_data, test_data)

init_fn, apply_fn = make_network(*network_size)
apply_fn(params, input_mapping(x, B)))

# JAX network definition
def make_network(num_layers, num_channels):
    layers = []
    for i in range(num_layers - 1):
        layers.append(stax.Dense(num_channels))
        layers.append(stax.Relu)
    layers.append(stax.Dense(3))
    layers.append(stax.Sigmoid)
    return stax.serial(*layers)

项目https://github.com/Atom-101/FourierFeat-Siren

Pytorch implementation and comparison of Fourier Feature Networks and Sinusoidal Representation Networks

MLP(
  (layers): Sequential(
    (0): Linear(in_features=512, out_features=256, bias=True)
    (1): ReLU(inplace=True)
    (2): Linear(in_features=256, out_features=256, bias=True)
    (3): ReLU(inplace=True)
    (4): Linear(in_features=256, out_features=256, bias=True)
    (5): ReLU(inplace=True)
    (6): Linear(in_features=256, out_features=3, bias=True)
  )
)

def map_x(x,B):
    xp = torch.matmul(2*math.pi*x,B)
    return torch.cat([torch.sin(xp),torch.cos(xp)],dim=-1)
def map_x(x,B):
    xp = torch.matmul(2*math.pi*x,B)
    return torch.sin(xp)

====================================================

# Image taken from authors' colab demo: https://colab.research.google.com/github/tancik/fourier-feature-networks/blob/master/Demo.ipynb
import numpy as np
import matplotlib.pyplot as plt
# from tqdm.notebook import tqdm as tqdm
from tqdm import *
import os, imageio
from imageio import imread,imsave
import torch
import torch.nn as nn
import torch.nn.functional as F
import math

# Download image, take a square crop from the center
image_url = 'https://live.staticflickr.com/7492/15677707699_d9d67acf9d_b.jpg'
img = imageio.imread(image_url)[..., :3] / 255.
c = [img.shape[0]//2, img.shape[1]//2]
r = 256
img = img[c[0]-r:c[0]+r, c[1]-r:c[1]+r]

# plt.imshow(img)
# plt.show()

# Create input pixel coordinates in the unit square
coords = np.linspace(0, 1, img.shape[0], endpoint=False)
x_test = np.stack(np.meshgrid(coords, coords), -1)
test_data = [x_test, img]
train_data = [x_test[::2,::2], img[::2,::2]]

#%%

class MLP(nn.Module):
    def __init__(self,depth=4,mapping_size=512,hidden_size=256):
        super().__init__()
        layers = []
        layers.append(nn.Linear(mapping_size,hidden_size))
        layers.append(nn.ReLU(inplace=True))
        # for _ in range(depth-2):
        #     layers.append(nn.Linear(hidden_size,hidden_size))
        #     layers.append(nn.ReLU(inplace=True))
        layers.append(nn.Linear(hidden_size,3))
        self.layers = nn.Sequential(*layers)
    def forward(self,x):
        return torch.sigmoid(self.layers(x))

#%%

xb,yb = torch.tensor(train_data[0]).reshape(-1,2),torch.tensor(train_data[1]).reshape(-1,3)
x_test,y_test = torch.tensor(test_data[0]).reshape(-1,2),torch.tensor(test_data[1]).reshape(-1,3)
xb,yb,x_test,y_test = xb.float().cuda(),yb.float().cuda(),x_test.float().cuda(),y_test.float().cuda()


#%% md

# Original Mapping

#%%

def map_x(x,B):
    xp = torch.matmul(2*math.pi*x,B)
    return torch.cat([torch.sin(xp),torch.cos(xp)],dim=-1)

#%%

model = MLP().cuda()
opt = torch.optim.Adam(model.parameters(),lr=1e-4)
loss = nn.MSELoss()
B = torch.randn(2,256).cuda() * 10
xt = map_x(xb,B)  # [65536,2] - [2,256] -> [65536,256]
for i in tqdm(range(1000)):
    print(i)
    ypred = model(xt)
    l = loss(ypred,yb)
    opt.zero_grad()
    l.backward()
    opt.step()
    model.eval()
    with torch.no_grad():
        ypreds = model(map_x(x_test, B))
        ypreds = ypreds.reshape(512, 512, 3)
        imsave('gaussion\\gaussion'+str(i)+'.png', (ypreds * 255).cpu().numpy())


# Preds
model.cpu().eval()
with torch.no_grad():
    ypreds = model(map_x(x_test,B.cpu()))
    ypreds = ypreds.reshape(512,512,3)
    imsave('gaussion.png',(ypreds*255).numpy() )


















========================================================================

# Image taken from authors' colab demo: https://colab.research.google.com/github/tancik/fourier-feature-networks/blob/master/Demo.ipynb
import numpy as np
import matplotlib.pyplot as plt
# from tqdm.notebook import tqdm as tqdm
from tqdm import *
import os, imageio
from imageio import imread,imsave
import torch
import torch.nn as nn
import torch.nn.functional as F
import math

# Download image, take a square crop from the center
image_url = 'https://live.staticflickr.com/7492/15677707699_d9d67acf9d_b.jpg'
img = imageio.imread(image_url)[..., :3] / 255.
c = [img.shape[0]//2, img.shape[1]//2]
r = 256
img = img[c[0]-r:c[0]+r, c[1]-r:c[1]+r]

# plt.imshow(img)
# plt.show()

# Create input pixel coordinates in the unit square
coords = np.linspace(0, 1, img.shape[0], endpoint=False)
x_test = np.stack(np.meshgrid(coords, coords), -1)
test_data = [x_test, img]
train_data = [x_test[::2,::2], img[::2,::2]]

#%%

class MLP(nn.Module):
    def __init__(self,depth=4,mapping_size=512,hidden_size=256):
        super().__init__()
        layers = []
        layers.append(nn.Linear(mapping_size,hidden_size))
        layers.append(nn.ReLU(inplace=True))
        # for _ in range(depth-2):
        #     layers.append(nn.Linear(hidden_size,hidden_size))
        #     layers.append(nn.ReLU(inplace=True))
        layers.append(nn.Linear(hidden_size,3))
        self.layers = nn.Sequential(*layers)
    def forward(self,x):
        return torch.sigmoid(self.layers(x))

#%%

xb,yb = torch.tensor(train_data[0]).reshape(-1,2),torch.tensor(train_data[1]).reshape(-1,3)
x_test,y_test = torch.tensor(test_data[0]).reshape(-1,2),torch.tensor(test_data[1]).reshape(-1,3)
xb,yb,x_test,y_test = xb.float().cuda(),yb.float().cuda(),x_test.float().cuda(),y_test.float().cuda()


#%% md

# Original Mapping

#%%

def map_x(x,B):
    xp = torch.matmul(2*math.pi*x,B)
    return torch.cat([torch.sin(xp),torch.cos(xp)],dim=-1)

#%%

model = MLP().cuda()
opt = torch.optim.Adam(model.parameters(),lr=1e-4)
loss = nn.MSELoss()

xb = torch.randn(65536,2).cuda()
B = torch.randn(2,256).cuda() * 10
xt = torch.randn(65536,256*2).cuda()#map_x(xb,B)  # [65536,2] - [2,256] -> [65536,256*2]
for i in tqdm(range(1000)):
    print(i)
    ypred = model(xt)
    l = loss(ypred,yb)
    opt.zero_grad()
    l.backward()
    opt.step()
    model.eval()
    with torch.no_grad():
        ypreds  = model(xt)#model(map_x(x_test, B))
        ypreds = ypreds.reshape(256, 256, 3)
        imsave('gaussion\\gaussion'+str(i)+'.png', (ypreds * 255).cpu().numpy())





class MLP(nn.Module):
    def __init__(self,outdim=3,mapping_size=512,hidden_size=256):
        super().__init__()
        layers = []
        layers.append(nn.Linear(mapping_size,hidden_size))
        layers.append(nn.ReLU(inplace=True))
        # for _ in range(depth-2):
        #     layers.append(nn.Linear(hidden_size,hidden_size))
        #     layers.append(nn.ReLU(inplace=True))
        layers.append(nn.Linear(hidden_size,outdim))
        self.layers = nn.Sequential(*layers)
    def forward(self,x):
        return torch.sigmoid(self.layers(x))