EfficientFormer是纯Transformer模型，经优化设计，在移动设备上表现优异。最快的L1在ImageNet-1K准确率79.2%，iPhone 12延迟1.6毫秒，与MobileNetv2×1.4速度相当，证明合理设计的Transformer能兼顾低延迟与高性能。

☞☞☞AI 智能聊天, 问答助手, AI 智能搜索, 免费无限量使用 DeepSeek R1 模型☜☜☜

EfficientFormer: 速度上可以与MobileNet媲美的视觉Transformer

摘要

        视觉Transformer(ViT)在计算机视觉任务中取得了迅速的进展，在各种基准上都取得了很好的结果。 然而，由于ViT模型的参数和模型设计，例如注意力机制，其速度通常比轻量级卷积网络慢几倍。 因此，面向实时应用的ViT部署尤其具有挑战性，尤其是在资源受限的硬件上，如移动设备。 近年来的研究试图通过网络结构搜索或与MobileNet块的混合设计来降低ViT的计算复杂度，但推理速度仍不尽如人意。 这就引出了一个重要的问题：Transformer能在获得高性能的同时运行得像MobileNet一样快吗？ 为了回答这个问题，我们首先回顾基于ViT的模型中使用的网络架构和运算符，并识别出低效设计。 然后我们介绍了一个维度一致的纯Transformer（没有MobileNet块）作为设计范例。 最后，我们进行延迟驱动的裁剪，得到一系列最终的模型，称为EfficientFormer。 通过大量的实验，证明了该算法在移动设备性能和速度上的优越性。 我们最快的模型EfficientFormer-L1在ImageNet-1K上的准确率达到79.2%，在iPhone 12（用CoreML编译）上的推理延迟仅为1.6毫秒，运行速度与MobileNetv2×1.4（1.6毫秒，74.7%Top-1)一样快。我们最大的模型EfficientFormer-L7在ImageNet-1K上的准确率达到83.3%，延迟仅为7.0毫秒。 我们的工作证明，适当设计的Transformer可以在移动设备上达到极低的延迟，同时保持高性能。

1. EfficientFormer

1.1 对轻量化视觉Transformer的一些思考

        从图2可以得到如下轻量化视觉Transformer的观察：

1. 大内核、大步幅的Patch嵌入是移动设备上的一个速度瓶颈
2. 一致的特征尺寸对于选择Token Mixer很重要。 MHSA不一定是速度瓶颈
3. Conv-BN比LN(GN)-Linear更有利于时延，精度下降一般可以接受（在推理阶段，BN可以通过重参数化技术融合到Conv中）
4. 非线性的延迟与硬件和编译器有关

       

1.2 EfficientFormer

        基于以上的观察，本文设计了一个新的轻量化视觉Transformer——EfficientFormer，从宏观上看，主要包含两种结构：Patch Embedding和Meta Transformer Block，用公式表示为：

$$\begin{array}{c}\mathcal{Y}={\prod }_i^m{\mathrm{M}\mathrm{B}}_i\left(\text{ PatchEmbed }\left({\mathcal{X}}_0^{B,3,H,W}\right)\right)\\ {\mathcal{X}}_{i+1}={\mathrm{M}\mathrm{B}}_i\left({\mathcal{X}}_i\right)=\mathrm{MLP}\left(\text{ TokenMixer }\left({\mathcal{X}}_i\right)\right)\end{array}$$Y=∏imMBi( PatchEmbed (X0B,3,H,W))Xi+1=MBi(Xi)=MLP( TokenMixer (Xi))

Motiff妙多

Motiff妙多是一款AI驱动的界面设计工具，定位为“AI时代设计工具”

334 查看详情

        为了在早期捕获局部特征，本文使用类似于PoolFormer的架构（实际使用DWConv更好，但是本文想提出一个纯Transformer架构，因此没用），为了在后期捕获全局特征，本文使用原始的Transformer架构。同时，为了保证一致特征维度，早期是四维的使用卷积操作，后期是三维的使用线性层操作。

1. ${MB}^{4D}$MB4D :

$$\begin{array}{c}{\mathcal{I}}_i=\mathrm{Pool}\left({\mathcal{X}}_i^{B,C_j,\frac{H}{2^{j+1}},\frac{W}{2j+1}}\right)+{\mathcal{X}}_i^{B,C_j,\frac{H}{2j+1},\frac{W}{2^{j+1}}},\\ {\mathcal{X}}_{i+1}^{B,C_j,\frac{H}{2j+1},\frac{W}{2j+1}}={}_B\left({}_{B,G}\left({\mathcal{I}}_i\right)\right)+{\mathcal{I}}_i,\end{array}$$Ii=Pool(XiB,Cj,2j+1H,2j+1W)+XiB,Cj,2j+1H,2j+1W,Xi+1B,Cj,2j+1H,2j+1W=ConvB(ConvB,G(Ii))+Ii,

2. ${MB}^{3D}$MB3D :

$$\begin{array}{c}{\mathcal{I}}_i=\mathrm{Linear}(\mathrm{MHSA}(\mathrm{Linear}(\mathrm{LN}\left({\mathcal{X}}_i^{B,\frac{HW}{4^{j+1}},C_j}\right))))+{\mathcal{X}}_i^{B,\frac{HW}{4^{j+1}},C_j}\\ {\mathcal{X}}_{i+1}^{B,\frac{HW}{4^{j+1}},C_j}=\text{ Linear }\left(\text{ Linear }G(\mathrm{LN}\left({\mathcal{I}}_i\right))\right)+{\mathcal{I}}_i\\ \mathrm{MHSA}(Q,K,V)=\mathrm{Softmax}(\frac{Q\cdot K^T}{\sqrt{C_j}}+b)\cdot V\end{array}$$Ii=Linear(MHSA(Linear(LN(XiB,4j+1HW,Cj))))+XiB,4j+1HW,CjXi+1B,4j+1HW,Cj= Linear ( Linear G(LN(Ii)))+IiMHSA(Q,K,V)=Softmax(CjQ⋅KT+b)⋅V

       

2. 代码复现

2.1 下载并导入所需的库

In [ ]

!pip install paddlex

    In [ ]

%matplotlib inlineimport paddleimport paddle.fluid as fluidimport numpy as npimport matplotlib.pyplot as pltfrom paddle.vision.datasets import Cifar10from paddle.vision.transforms import Transposefrom paddle.io import Dataset, DataLoaderfrom paddle import nnimport paddle.nn.functional as Fimport paddle.vision.transforms as transformsimport osimport matplotlib.pyplot as pltfrom matplotlib.pyplot import figureimport paddleximport itertools

   

2.2 创建数据集

In [3]

train_tfm = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.6, 1.0)),
    transforms.ColorJitter(brightness=0.2,contrast=0.2, saturation=0.2),
    transforms.RandomHorizontalFlip(0.5),
    transforms.RandomRotation(20),
    paddlex.transforms.MixupImage(),
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])

test_tfm = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])

    In [4]

paddle.vision.set_image_backend('cv2')# 使用Cifar10数据集train_dataset = Cifar10(data_file='data/data152754/cifar-10-python.tar.gz', mode='train', transform = train_tfm, )
val_dataset = Cifar10(data_file='data/data152754/cifar-10-python.tar.gz', mode='test',transform = test_tfm)print("train_dataset: %d" % len(train_dataset))print("val_dataset: %d" % len(val_dataset))

       

train_dataset: 50000
val_dataset: 10000

        In [5]

batch_size=256

    In [6]

train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, drop_last=True, num_workers=4)
val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False, drop_last=False, num_workers=4)

   

2.3 模型的创建

2.3.1 标签平滑

In [7]

class LabelSmoothingCrossEntropy(nn.Layer):
    def __init__(self, smoothing=0.1):
        super().__init__()
        self.smoothing = smoothing    def forward(self, pred, target):

        confidence = 1. - self.smoothing
        log_probs = F.log_softmax(pred, axis=-1)
        idx = paddle.stack([paddle.arange(log_probs.shape[0]), target], axis=1)
        nll_loss = paddle.gather_nd(-log_probs, index=idx)
        smooth_loss = paddle.mean(-log_probs, axis=-1)
        loss = confidence * nll_loss + self.smoothing * smooth_loss        return loss.mean()

   

2.3.2 DropPath

In [8]

def drop_path(x, drop_prob=0.0, training=False):
    """
    Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
    the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...
    See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ...
    """
    if drop_prob == 0.0 or not training:        return x
    keep_prob = paddle.to_tensor(1 - drop_prob)
    shape = (paddle.shape(x)[0],) + (1,) * (x.ndim - 1)
    random_tensor = keep_prob + paddle.rand(shape, dtype=x.dtype)
    random_tensor = paddle.floor(random_tensor)  # binarize
    output = x.divide(keep_prob) * random_tensor    return outputclass DropPath(nn.Layer):
    def __init__(self, drop_prob=None):
        super(DropPath, self).__init__()
        self.drop_prob = drop_prob    def forward(self, x):
        return drop_path(x, self.drop_prob, self.training)

   

2.3.3 EfficientFormer模型的创建

In [9]

class Attention(nn.Layer):
    def __init__(self, dim=384, key_dim=32, num_heads=8,
                 attn_ratio=4,
                 resolution=7):
        super().__init__()
        self.resolution = resolution
        self.num_heads = num_heads
        self.scale = key_dim ** -0.5
        self.key_dim = key_dim
        self.nh_kd = nh_kd = key_dim * num_heads
        self.d = int(attn_ratio * key_dim)
        self.dh = int(attn_ratio * key_dim) * num_heads
        self.attn_ratio = attn_ratio
        h = self.dh + nh_kd * 2
        self.N = resolution ** 2
        self.N2 = self.N
        self.qkv = nn.Linear(dim, h)
        self.proj = nn.Linear(self.dh, dim)

        points = list(itertools.product(range(self.resolution), range(self.resolution)))
        N = len(points)
        self.N = N
        attention_offsets = {}
        idxs = []        for p1 in points:            for p2 in points:
                offset = (abs(p1[0] - p2[0]), abs(p1[1] - p2[1]))                if offset not in attention_offsets:
                    attention_offsets[offset] = len(attention_offsets)
                idxs.append(attention_offsets[offset])
        self.attention_biases = self.create_parameter((len(attention_offsets), num_heads), default_initializer=nn.initializer.Constant(0.0))
        self.attention_bias_idxs = idxs    def forward(self, x):  # x (B,N,C)
        B, N, C = x.shape
        qkv = self.qkv(x)
        q, k, v = qkv.reshape((B, N, self.num_heads, -1)).split([self.key_dim, self.key_dim, self.d], axis=3)
        q = q.transpose((0, 2, 1, 3))
        k = k.transpose((0, 2, 1, 3))
        v = v.transpose((0, 2, 1, 3))

        attn = (q @ k.transpose((0, 1, 3, 2))) * self.scale
        attn = attn + self.attention_biases[self.attention_bias_idxs].transpose((1, 0)).reshape((1, self.num_heads, self.N, self.N))
        
        attn = F.softmax(attn, axis=-1)
        x = (attn @ v).transpose((0, 2, 1, 3)).reshape((B, N, self.dh))
        x = self.proj(x)        return x

    In [10]

# Conv Stemdef stem(in_chs, out_chs):
    return nn.Sequential(
        nn.Conv2D(in_chs, out_chs // 2, kernel_size=3, stride=2, padding=1),
        nn.BatchNorm2D(out_chs // 2),
        nn.ReLU(),
        nn.Conv2D(out_chs // 2, out_chs, kernel_size=3, stride=2, padding=1),
        nn.BatchNorm2D(out_chs),
        nn.ReLU())

    In [11]

class Embedding(nn.Layer):
    """
    Patch Embedding that is implemented by a layer of conv.
    Input: tensor in shape [B, C, H, W]
    Output: tensor in shape [B, C, H/stride, W/stride]
    """

    def __init__(self, patch_size=16, stride=16, padding=0,
                 in_chans=3, embed_dim=768, norm_layer=nn.BatchNorm2D):
        super().__init__()

        self.proj = nn.Conv2D(in_chans, embed_dim, kernel_size=patch_size,
                              stride=stride, padding=padding)
        self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()    def forward(self, x):
        x = self.proj(x)
        x = self.norm(x)        return x

    In [12]

class Flat(nn.Layer):

    def __init__(self, ):
        super().__init__()    def forward(self, x):
        x = x.flatten(2).transpose((0, 2, 1))        return x

    In [13]

class Pooling(nn.Layer):
    """
    Implementation of pooling for PoolFormer
    --pool_size: pooling size
    """

    def __init__(self, pool_size=3):
        super().__init__()
        self.pool = nn.AvgPool2D(
            pool_size, stride=1, padding=pool_size // 2)    def forward(self, x):
        return self.pool(x) - x

    In [14]

class LinearMlp(nn.Layer):
    """ MLP as used in Vision Transformer, MLP-Mixer and related networks
    """

    def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.):
        super().__init__()
        out_features = out_features or in_features
        hidden_features = hidden_features or in_features

        self.fc1 = nn.Linear(in_features, hidden_features)
        self.act = act_layer()
        self.drop1 = nn.Dropout(drop)
        self.fc2 = nn.Linear(hidden_features, out_features)
        self.drop2 = nn.Dropout(drop)    def forward(self, x):
        x = self.fc1(x)
        x = self.act(x)
        x = self.drop1(x)
        x = self.fc2(x)
        x = self.drop2(x)        return x

    In [15]

class Mlp(nn.Layer):
    """
    Implementation of MLP with 1*1 convolutions.
    Input: tensor with shape [B, C, H, W]
    """

    def __init__(self, in_features, hidden_features=None,
                 out_features=None, act_layer=nn.GELU, drop=0.):
        super().__init__()
        out_features = out_features or in_features
        hidden_features = hidden_features or in_features
        self.fc1 = nn.Conv2D(in_features, hidden_features, 1)
        self.act = act_layer()
        self.fc2 = nn.Conv2D(hidden_features, out_features, 1)
        self.drop = nn.Dropout(drop)

        self.norm1 = nn.BatchNorm2D(hidden_features)
        self.norm2 = nn.BatchNorm2D(out_features)    def forward(self, x):
        x = self.fc1(x)

        x = self.norm1(x)

        x = self.act(x)
        x = self.drop(x)
        x = self.fc2(x)

        x = self.norm2(x)

        x = self.drop(x)        return x

    In [16]

class Meta3D(nn.Layer):

    def __init__(self, dim, mlp_ratio=4.,
                 act_layer=nn.GELU, norm_layer=nn.LayerNorm,
                 drop=0., drop_path=0.,
                 use_layer_scale=True, layer_scale_init_value=1e-5):

        super().__init__()

        self.norm1 = norm_layer(dim)
        self.token_mixer = Attention(dim)
        self.norm2 = norm_layer(dim)
        mlp_hidden_dim = int(dim * mlp_ratio)
        self.mlp = LinearMlp(in_features=dim, hidden_features=mlp_hidden_dim,
                             act_layer=act_layer, drop=drop)

        self.drop_path = DropPath(drop_path) if drop_path > 0. \            else nn.Identity()
        self.use_layer_scale = use_layer_scale        if use_layer_scale:
            self.layer_scale_1 = self.create_parameter([dim], default_initializer=nn.initializer.Constant(layer_scale_init_value))
            self.layer_scale_2 = self.create_parameter([dim], default_initializer=nn.initializer.Constant(layer_scale_init_value))    def forward(self, x):
        if self.use_layer_scale:
            x = x + self.drop_path(self.layer_scale_1 * self.token_mixer(self.norm1(x)))
            x = x + self.drop_path(self.layer_scale_2 * self.mlp(self.norm2(x)))        else:
            x = x + self.drop_path(self.token_mixer(self.norm1(x)))
            x = x + self.drop_path(self.mlp(self.norm2(x)))        return x

    In [17]

class Meta4D(nn.Layer):

    def __init__(self, dim, pool_size=3, mlp_ratio=4.,
                 act_layer=nn.GELU,
                 drop=0., drop_path=0.,
                 use_layer_scale=True, layer_scale_init_value=1e-5):
        super().__init__()

        self.token_mixer = Pooling(pool_size=pool_size)
        mlp_hidden_dim = int(dim * mlp_ratio)
        self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim,
                       act_layer=act_layer, drop=drop)

        self.drop_path = DropPath(drop_path) if drop_path > 0. \            else nn.Identity()
        self.use_layer_scale = use_layer_scale        if use_layer_scale:
            self.layer_scale_1 = self.create_parameter([1, dim, 1, 1], 
                                default_initializer=nn.initializer.Constant(layer_scale_init_value))
            self.layer_scale_2 = self.create_parameter([1, dim, 1, 1], 
                                default_initializer=nn.initializer.Constant(layer_scale_init_value))    def forward(self, x):
        if self.use_layer_scale:

            x = x + self.drop_path(self.layer_scale_1 * self.token_mixer(x))
            x = x + self.drop_path(self.layer_scale_2 * self.mlp(x))        else:
            x = x + self.drop_path(self.token_mixer(x))
            x = x + self.drop_path(self.mlp(x))        return x

    In [18]

def meta_blocks(dim, index, layers,
                pool_size=3, mlp_ratio=4.,
                act_layer=nn.GELU, norm_layer=nn.LayerNorm,
                drop_rate=.0, drop_path_rate=0.,
                use_layer_scale=True, layer_scale_init_value=1e-5, vit_num=1):
    blocks = []    if index == 3 and vit_num == layers[index]:
        blocks.append(Flat())    for block_idx in range(layers[index]):
        block_dpr = drop_path_rate * (
                block_idx + sum(layers[:index])) / (sum(layers) - 1)        if index == 3 and layers[index] - block_idx <= vit_num:
            blocks.append(Meta3D(
                dim, mlp_ratio=mlp_ratio,
                act_layer=act_layer, norm_layer=norm_layer,
                drop=drop_rate, drop_path=block_dpr,
                use_layer_scale=use_layer_scale,
                layer_scale_init_value=layer_scale_init_value,
            ))        else:
            blocks.append(Meta4D(
                dim, pool_size=pool_size, mlp_ratio=mlp_ratio,
                act_layer=act_layer,
                drop=drop_rate, drop_path=block_dpr,
                use_layer_scale=use_layer_scale,
                layer_scale_init_value=layer_scale_init_value,
            ))            if index == 3 and layers[index] - block_idx - 1 == vit_num:
                blocks.append(Flat())

    blocks = nn.Sequential(*blocks)    return blocks

    In [19]

class EfficientFormer(nn.Layer):

    def __init__(self, layers, embed_dims=None,
                 mlp_ratios=4, downsamples=None,
                 pool_size=3,
                 norm_layer=nn.LayerNorm, act_layer=nn.GELU,
                 num_classes=1000,
                 down_patch_size=3, down_stride=2, down_pad=1,
                 drop_rate=0., drop_path_rate=0.,
                 use_layer_scale=True, layer_scale_init_value=1e-5,
                 vit_num=0,
                 distillation=False):
        super().__init__()

        self.num_classes = num_classes

        self.patch_embed = stem(3, embed_dims[0])

        network = []        for i in range(len(layers)):
            stage = meta_blocks(embed_dims[i], i, layers,
                                pool_size=pool_size, mlp_ratio=mlp_ratios,
                                act_layer=act_layer, norm_layer=norm_layer,
                                drop_rate=drop_rate,
                                drop_path_rate=drop_path_rate,
                                use_layer_scale=use_layer_scale,
                                layer_scale_init_value=layer_scale_init_value,
                                vit_num=vit_num)
            network.append(stage)            if i >= len(layers) - 1:                break
            if downsamples[i] or embed_dims[i] != embed_dims[i + 1]:                # downsampling between two stages
                network.append(
                    Embedding(
                        patch_size=down_patch_size, stride=down_stride,
                        padding=down_pad,
                        in_chans=embed_dims[i], embed_dim=embed_dims[i + 1]
                    )
                )

        self.network = nn.LayerList(network)        # Classifier head
        self.norm = norm_layer(embed_dims[-1])
        self.head = nn.Linear(
            embed_dims[-1], num_classes) if num_classes > 0 \            else nn.Identity()
        self.dist = distillation        if self.dist:
            self.dist_head = nn.Linear(
                embed_dims[-1], num_classes) if num_classes > 0 \                else nn.Identity()

        self.apply(self.cls_init_weights)    # init for classification
    def cls_init_weights(self, m):
        tn = nn.initializer.TruncatedNormal(std=.02)
        kaiming = nn.initializer.KaimingNormal()
        zero = nn.initializer.Constant(0.)
        one = nn.initializer.Constant(1.)        if isinstance(m, nn.Linear):
            tn(m.weight)            if isinstance(m, nn.Linear) and m.bias is not None:
                zero(m.bias)        
        if isinstance(m, nn.Conv2D):
            kaiming(m.weight)            if isinstance(m, nn.Conv2D) and m.bias is not None:
                zero(m.bias)        
        if isinstance(m, (nn.BatchNorm2D, nn.LayerNorm)):
            one(m.weight)
            zero(m.bias)    def forward_tokens(self, x):
        outs = []        for idx, block in enumerate(self.network):
            x = block(x)        return x    def forward(self, x):
        x = self.patch_embed(x)

        x = self.forward_tokens(x)

        x = self.norm(x)        if self.dist:
            cls_out = self.head(x.mean(-2)), self.dist_head(x.mean(-2))            if not self.training:
                cls_out = (cls_out[0] + cls_out[1]) / 2
        else:
            cls_out = self.head(x.mean(-2))        # for image classification
        return cls_out

   

2.3.4 模型的参数

In [20]

EfficientFormer_width = {    'l1': [48, 96, 224, 448],    'l3': [64, 128, 320, 512],    'l7': [96, 192, 384, 768],
}

EfficientFormer_depth = {    'l1': [3, 2, 6, 4],    'l3': [4, 4, 12, 6],    'l7': [6, 6, 18, 8],
}def efficientformer_l1(pretrained=False, **kwargs):
    model = EfficientFormer(
        layers=EfficientFormer_depth['l1'],
        embed_dims=EfficientFormer_width['l1'],
        downsamples=[True, True, True, True],
        num_classes=10,
        vit_num=1)    return modeldef efficientformer_l3(pretrained=False, **kwargs):
    model = EfficientFormer(
        layers=EfficientFormer_depth['l3'],
        embed_dims=EfficientFormer_width['l3'],
        downsamples=[True, True, True, True],
        num_classes=10,
        vit_num=4)    return modeldef efficientformer_l7(pretrained=False, **kwargs):
    model = EfficientFormer(
        layers=EfficientFormer_depth['l7'],
        embed_dims=EfficientFormer_width['l7'],
        downsamples=[True, True, True, True],
        num_classes=10,
        vit_num=8)    return model

    In [ ]

# EfficientFormer-L1model = efficientformer_l1()
paddle.summary(model, (1, 3, 224, 224))

   

       

In [ ]

# EfficientFormer-L3model = efficientformer_l3()
paddle.summary(model, (1, 3, 224, 224))

   

       

In [ ]

# EfficientFormer-L7model = efficientformer_l7()
paddle.summary(model, (1, 3, 224, 224))

   

       

2.4 训练

In [24]

learning_rate = 0.001n_epochs = 100paddle.seed(42)
np.random.seed(42)

    In [ ]

work_path = 'work/model'# EfficientFormer-L1model = efficientformer_l1()

criterion = LabelSmoothingCrossEntropy()

scheduler = paddle.optimizer.lr.CosineAnnealingDecay(learning_rate=learning_rate, T_max=50000 // batch_size * n_epochs, verbose=False)
optimizer = paddle.optimizer.Adam(parameters=model.parameters(), learning_rate=scheduler, weight_decay=1e-5)

gate = 0.0threshold = 0.0best_acc = 0.0val_acc = 0.0loss_record = {'train': {'loss': [], 'iter': []}, 'val': {'loss': [], 'iter': []}}   # for recording lossacc_record = {'train': {'acc': [], 'iter': []}, 'val': {'acc': [], 'iter': []}}      # for recording accuracyloss_iter = 0acc_iter = 0for epoch in range(n_epochs):    # ---------- Training ----------
    model.train()
    train_num = 0.0
    train_loss = 0.0

    val_num = 0.0
    val_loss = 0.0
    accuracy_manager = paddle.metric.Accuracy()
    val_accuracy_manager = paddle.metric.Accuracy()    print("#===epoch: {}, lr={:.10f}===#".format(epoch, optimizer.get_lr()))    for batch_id, data in enumerate(train_loader):
        x_data, y_data = data
        labels = paddle.unsqueeze(y_data, axis=1)

        logits = model(x_data)

        loss = criterion(logits, y_data)

        acc = paddle.metric.accuracy(logits, labels)
        accuracy_manager.update(acc)        if batch_id % 10 == 0:
            loss_record['train']['loss'].append(loss.numpy())
            loss_record['train']['iter'].append(loss_iter)
            loss_iter += 1

        loss.backward()

        optimizer.step()
        scheduler.step()
        optimizer.clear_grad()
        
        train_loss += loss
        train_num += len(y_data)

    total_train_loss = (train_loss / train_num) * batch_size
    train_acc = accuracy_manager.accumulate()
    acc_record['train']['acc'].append(train_acc)
    acc_record['train']['iter'].append(acc_iter)
    acc_iter += 1
    # Print the information.
    print("#===epoch: {}, train loss is: {}, train acc is: {:2.2f}%===#".format(epoch, total_train_loss.numpy(), train_acc*100))    # ---------- Validation ----------
    model.eval()    for batch_id, data in enumerate(val_loader):

        x_data, y_data = data
        labels = paddle.unsqueeze(y_data, axis=1)        with paddle.no_grad():
          logits = model(x_data)

        loss = criterion(logits, y_data)

        acc = paddle.metric.accuracy(logits, labels)
        val_accuracy_manager.update(acc)

        val_loss += loss
        val_num += len(y_data)

    total_val_loss = (val_loss / val_num) * batch_size
    loss_record['val']['loss'].append(total_val_loss.numpy())
    loss_record['val']['iter'].append(loss_iter)
    val_acc = val_accuracy_manager.accumulate()
    acc_record['val']['acc'].append(val_acc)
    acc_record['val']['iter'].append(acc_iter)    
    print("#===epoch: {}, val loss is: {}, val acc is: {:2.2f}%===#".format(epoch, total_val_loss.numpy(), val_acc*100))    # ===================s*e====================
    if val_acc > best_acc:
        best_acc = val_acc
        paddle.s*e(model.state_dict(), os.path.join(work_path, 'best_model.pdparams'))
        paddle.s*e(optimizer.state_dict(), os.path.join(work_path, 'best_optimizer.pdopt'))print(best_acc)
paddle.s*e(model.state_dict(), os.path.join(work_path, 'final_model.pdparams'))
paddle.s*e(optimizer.state_dict(), os.path.join(work_path, 'final_optimizer.pdopt'))

   

       

2.5 结果分析

In [26]

def plot_learning_curve(record, title='loss', ylabel='CE Loss'):
    ''' Plot learning curve of your CNN '''
    maxtrain = max(map(float, record['train'][title]))
    maxval = max(map(float, record['val'][title]))
    ymax = max(maxtrain, maxval) * 1.1
    mintrain = min(map(float, record['train'][title]))
    minval = min(map(float, record['val'][title]))
    ymin = min(mintrain, minval) * 0.9

    total_steps = len(record['train'][title])
    x_1 = list(map(int, record['train']['iter']))
    x_2 = list(map(int, record['val']['iter']))
    figure(figsize=(10, 6))
    plt.plot(x_1, record['train'][title], c='tab:red', label='train')
    plt.plot(x_2, record['val'][title], c='tab:cyan', label='val')
    plt.ylim(ymin, ymax)
    plt.xlabel('Training steps')
    plt.ylabel(ylabel)
    plt.title('Learning curve of {}'.format(title))
    plt.legend()
    plt.show()

    In [27]

plot_learning_curve(loss_record, title='loss', ylabel='CE Loss')

       

<Figure size 1000x600 with 1 Axes>

                In [28]

plot_learning_curve(acc_record, title='acc', ylabel='Accuracy')

       

<Figure size 1000x600 with 1 Axes>

                In [29]

import time
work_path = 'work/model'model = efficientformer_l1()
model_state_dict = paddle.load(os.path.join(work_path, 'best_model.pdparams'))
model.set_state_dict(model_state_dict)
model.eval()
aa = time.time()for batch_id, data in enumerate(val_loader):

    x_data, y_data = data
    labels = paddle.unsqueeze(y_data, axis=1)    with paddle.no_grad():
        logits = model(x_data)
bb = time.time()print("Throughout:{}".format(int(len(val_dataset)//(bb - aa))))

       

Throughout:856

        In [30]

def get_cifar10_labels(labels):  
    """返回CIFAR10数据集的文本标签。"""
    text_labels = [        'airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog',        'horse', 'ship', 'truck']    return [text_labels[int(i)] for i in labels]

    In [31]

def show_images(imgs, num_rows, num_cols, pred=None, gt=None, scale=1.5):  
    """Plot a list of images."""
    figsize = (num_cols * scale, num_rows * scale)
    _, axes = plt.subplots(num_rows, num_cols, figsize=figsize)
    axes = axes.flatten()    for i, (ax, img) in enumerate(zip(axes, imgs)):        if paddle.is_tensor(img):
            ax.imshow(img.numpy())        else:
            ax.imshow(img)
        ax.axes.get_xaxis().set_visible(False)
        ax.axes.get_yaxis().set_visible(False)        if pred or gt:
            ax.set_title("pt: " + pred[i] + "\ngt: " + gt[i])    return axes

    In [32]

work_path = 'work/model'X, y = next(iter(DataLoader(val_dataset, batch_size=18)))
model = efficientformer_l1()
model_state_dict = paddle.load(os.path.join(work_path, 'best_model.pdparams'))
model.set_state_dict(model_state_dict)
model.eval()
logits = model(X)
y_pred = paddle.argmax(logits, -1)
X = paddle.transpose(X, [0, 2, 3, 1])
axes = show_images(X.reshape((18, 224, 224, 3)), 1, 18, pred=get_cifar10_labels(y_pred), gt=get_cifar10_labels(y))
plt.show()

       

Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

       

<Figure size 2700x150 with 18 Axes>

                In [ ]

!pip install interpretdl

    In [34]

import interpretdl as it

    In [35]

work_path = 'work/model'model = efficientformer_l1()
model_state_dict = paddle.load(os.path.join(work_path, 'best_model.pdparams'))
model.set_state_dict(model_state_dict)

    In [36]

X, y = next(iter(DataLoader(val_dataset, batch_size=18)))
lime = it.LIMECVInterpreter(model)

    In [44]

lime_weights = lime.interpret(X.numpy()[3], interpret_class=y.numpy()[3], batch_size=100, num_samples=10000, visual=True)

       

100%|██████████| 10000/10000 [00:50<00:00, 196.29it/s]

       

<Figure size 640x480 with 1 Axes>

               

以上就是EfficientFormer: 速度上可以与MobileNet媲美的ViT的详细内容，更多请关注其它相关文章！

# python  # git  # 石渠营销外包推广怎么样  # 肇庆东莞网站推广  # 营销推广网站源码  # 黄页网站推广  # 博景网络营销怎么做推广  # 团购自助营销推广文案  # 是在  # 后期  # 率达到  # 官网  # 轻量化  # 仅为  # 运算符  # 高性能  # 一言  # 中文网  # type  # fig  # latte  # igs  # red  # cos  # ai  # iphone  # 电商网站怎么推广好呢  # 江西咨询网站建设哪家好  # 新兴区seo培训  # 巴彦淖尔商城网站建设 

相关栏目： 【 行业资讯67740 】 【 技术百科0 】 【 网络运营39195 】

相关推荐： drawing是什么意思  ao3镜像网站永久地址入口  5r是多少钱  面包车收音机power是什么意思  苹果16新增哪些功能  春运抢票如何快速抢到票  如何去除计算器的命令  电动车power灯亮红灯是什么意思  typescript中文怎么读  光刻机的分类及特点  新版路由器如何设置路由命令  记录仪power灯亮是什么意思  华为交换机 配置 如何复制命令行  单片机软件keil怎么运行  如何操作fixup命令  j*a整形怎么转数组  空调控制面板power灯一直亮是什么意思  5G手机导航怎么旋转  如何用dos命令启动u盘  单片机log怎么看  type-c接口接地是什么意思  一天多少分钟  平仓是什么意思?  typescript如何使用viewer  typescript是什么时候出来的  5G类似微信的聊天软件有哪些  科技型企业成长＂十步法＂  微波炉power中文是什么意思  老电脑如何装固态硬盘  市盈率市净率是什么意思  12306退票手续费最新规定  命令不执行如何处理  typescript学会要多久  360f4怎么取消百变壁纸  怎么确定手机是5g  如何在命令提示符播放音频  春运抢票哪个城市好抢  如何在命令行写j*a程序  夸克绑定设备是什么意思  vi命令如何退出  如何用命令查看数据库日志文件  ssd固态硬盘如何选择  怎么自学typescript  市盈率动亏损是什么意思  交管12123协议头不完整是啥意思  eraser是什么意思  typescript怎么理解的  苹果16哪些功能好用  如何给电脑加装固态硬盘  苹果16有哪些亮点功能