一、本文介绍本文给大家介绍使用AdaFDAdaptive Frequency-Decoupled Fusion模块改进RT-DETR多模态网络模型能够有效改善其在小目标检测、复杂背景下的性能。提升RT-DETR多模态融合目标检测的频域特征表达能力本文引入AdaFD自适应频率解耦融合模块对特征中的低频语义信息与高频细节信息进行分离建模并通过动态生成高低通滤波器实现自适应融合。该模块能够根据不同目标和场景特征调整频率响应强化小目标边缘、纹理及结构信息同时抑制背景噪声和无效高频干扰。相比传统空间域融合方式AdaFD可有效缓解目标与背景相似导致的特征混淆在保持全局语义完整性的同时提升细粒度特征表达能力从而增强RT-DETR在小目标、多尺度目标及复杂背景场景下的定位精度、检测鲁棒性和泛化能力。欢迎订阅我的专栏、带你学习使用最新-最前沿-独家RT-DETR多模态创新改进RT-DETR多模态改进专栏目录《独家RT-DETR多模态改进专栏目录 》全网独家创新多模态融合改进教程包含早期融合、中期融合、后期融合、损失函数改进、二次创新模块、独家创新等几百种创新点改进答疑群提供完整项目永久更新中RT-DETR多模态创新订阅地址RT-DETR多模态创新改进专栏—轻松跑实验、多模态项目较容易冲顶会顶刊永久更新中本文目录一、本文介绍二、AdaFD自适应频率解耦融合模块介绍GSFANet 详细网络结构图2.1 AdaFD自适应频率解耦融合模块结构图2.2 AdaFD自适应频率解耦融合模块的作用2.3 AdaFD自适应频率解耦融合模块的原理2.4 AdaFD自适应频率解耦融合模块的优势三、完整核心代码四、手把手教你配置模块和修改tasks.py文件1.首先在ultralytics/nn/newsAddmodules创建一个.py文件2.在ultralytics/nn/newsAddmodules/__init__.py中引用3.修改tasks.py文件五、创建涨点yaml配置文件 中期融合创新改进: rtdetr-r18-midfusion-AdaFDFusion.yaml 中后期融合创新改进: rtdetr-r18-mid-to-late-AdaFDFusion.yaml 后期融合创新改进: rtdetr-r18-latefusion-AdaFDFusion.yaml六、正常运行二、AdaFD自适应频率解耦融合模块介绍摘要红外小目标检测IRSTD在空间域学习方面取得了显著进展。然而单帧图像的有限空间语义使得目标与相似噪声之间的区分变得困难同时也增加了大规模目标的完整性检测难度。为了解决这些问题我们提出了全球空间–频率注意力网络GSFANet它通过频率域的角度增强目标与噪声之间的分布差异同时保持空间信息的完整性。核心创新包括三个模块1参数化小波下采样PWD在频率细化过程中保留小目标细节以防止特征碎片化2层次化门控核注意力HGKA通过跨通道核注意力C2K捕捉跨层频率关系并通过跨空间门控注意力CSG保持空间一致性有效地桥接层之间的语义差距3自适应频率解耦融合AdaFD动态融合与目标相关的频率成分同时抑制噪声。我们进一步提出了AdaFL Loss以平衡多尺度目标的梯度并稳定训练。在三个基准数据集上的实验结果表明GSFANet在复杂场景中的检测性能和分割鲁棒性优于现有的最先进方法SOTA。不同方法的检测结果其中红色圆圈表示正确检测蓝色虚线圆圈表示漏检目标黄色圆圈表示误报。角落中显示了放大视图。图像 (a) 和 (b) 选自 SIRST。(c)、(d) 和 (g) 来自 IRSTD-1k。(e) 和 (f) 来自 NUDT-SIRST。(c) 和 (g) 中也展示了所提出方法的误检测示例。GSFANet详细网络结构图GSFANet 的架构通过频率下采样编码过程和自适应频率融合解码过程实现了联合的空间-频率学习。它通过 HGKA 模块在编码器和解码器的每个层级促进了充分的混合和交互。2.1 AdaFD自适应频率解耦融合模块结构图2.2 AdaFD自适应频率解耦融合模块的作用AdaFD模块的主要作用是通过频率解耦的方式将图像的低频和高频特征分别进行处理并通过自适应融合的方式优化目标与背景的分离特别是在小目标检测任务中增强了对目标的细节表达和对噪声的抑制。2.3 AdaFD自适应频率解耦融合模块的原理频率解耦在传统的卷积神经网络中通常将图像中的所有特征进行融合但这往往忽略了不同频率成分对目标和背景的不同影响。AdaFD通过生成低通滤波器低频成分和高通滤波器高频成分分别处理这两种频率成分。低频成分通常包含更宏观的目标信息而高频成分则包含更细致的边缘和纹理信息。自适应滤波器生成AdaFD模块根据输入特征的频率重要性动态生成自适应的高通和低通滤波器。这些滤波器会根据图像的特性动态调整以便在不同的层级中有效分离和融合目标相关的特征。特征融合通过自适应地调整高频和低频特征AdaFD模块能够将目标的细节特征高频与语义特征低频进行有效融合从而提供更精确的目标检测结果。2.4AdaFD自适应频率解耦融合模块的优势提高目标细节表达通过频率解耦AdaFD能够更好地保留目标的细节特别是在小目标检测中能够增强对细小目标边界和纹理的感知避免这些重要细节在常规下采样或融合过程中丢失。有效抑制噪声高频成分通常包含较多的噪声信息而低频成分则提供更稳定的目标语义信息。AdaFD通过自适应的频率融合方法能够抑制背景噪声同时保留有效的目标信息减少误检和假阳性。提升多尺度目标检测性能AdaFD模块通过频率解耦可以更好地处理不同尺度的目标特征在大尺度目标的全局语义和小尺度目标的细节特征之间实现平衡从而提高了多尺度目标的检测能力。增强对复杂背景的适应性在复杂的背景环境下尤其是含有高噪声的场景AdaFD的频率解耦和自适应滤波器能够有效区分目标和背景提升模型在不同背景下的鲁棒性和准确性。AdaFD模块通过自适应的频率解耦和融合策略有效增强了模型在小目标检测中的细节表达能力减少了噪声干扰并优化了多尺度目标的检测性能。这使得它在处理复杂背景和多种尺度的目标时表现出显著的优势尤其是在红外图像和其他噪声较多的场景中。三、完整核心代码import torch import torch.nn as nn import torch.nn.functional as F __all__ [AdaFDFusion] from ultralytics.nn.modules import Conv class LowPassConvGenerator(nn.Module): def __init__(self, in_channel, num_k2, ratio8): super(LowPassConvGenerator, self).__init__() self.num_k num_k self.avg_pool nn.AdaptiveAvgPool2d(1) self.max_pool nn.AdaptiveMaxPool2d(1) self.fc1_list nn.ModuleList( [nn.Conv2d(in_channel // num_k * 2, in_channel // ratio, 1, biasFalse) for _ in range(num_k)]) self.fc2_list nn.ModuleList( [nn.Conv2d(in_channel // ratio, 9 * in_channel, 1, biasFalse) for _ in range(num_k)]) self.relu nn.ReLU(inplaceTrue) def forward(self, x): b, c, h, w x.shape kernels [] for i in range(self.num_k): start_idx i * (c // self.num_k) end_idx (i 1) * (c // self.num_k) x_i x[:, start_idx:end_idx, :, :] avg_feature_x self.avg_pool(x_i) # b, c/num_k, 1, 1 max_feature_x self.max_pool(x_i) # b, c/num_k, 1, 1 fc1 self.fc1_list[i] fc2 self.fc2_list[i] combined_feature torch.cat([avg_feature_x, max_feature_x], dim1) kernel fc2(self.relu(fc1(combined_feature))) # (b, 9c, 1, 1) kernel kernel.reshape(b, c, 9, 1) kernel F.softmax(kernel, dim2) kernel kernel.reshape(b, c, 3, 3)[0].unsqueeze(1) # (c/num_k, 1, 3, 3) kernels.append(kernel) final_kernel torch.cat(kernels, dim0) # (num_k * c, 1, 3, 3) return final_kernel class HighPassConvGenerator(nn.Module): def __init__(self, in_channel, num_k2, kernel_size3, ratio8): super(HighPassConvGenerator, self).__init__() self.num_k num_k self.avg_pool nn.AdaptiveAvgPool2d(1) self.max_pool nn.AdaptiveMaxPool2d(1) self.fc1_list nn.ModuleList( [nn.Conv2d(in_channel // num_k * 2, in_channel // ratio, 1, biasFalse) for _ in range(num_k)]) self.fc2_list nn.ModuleList( [nn.Conv2d(in_channel // ratio, 9 * in_channel, 1, biasFalse) for _ in range(num_k)]) self.relu nn.ReLU(inplaceTrue) self.identity_kernel nn.Parameter(torch.tensor([[0, 0, 0], [0, 1, 0], [0, 0, 0]], dtypetorch.float32).unsqueeze(0).unsqueeze(0), requires_gradFalse) def forward(self, x): b, c, h, w x.shape # 64,64,64 kernels [] for i in range(self.num_k): start_idx i * (c // self.num_k) end_idx (i 1) * (c // self.num_k) x_i x[:, start_idx:end_idx, :, :] avg_feature_x self.avg_pool(x_i) # b, c/num_k, 1, 1 max_feature_x self.max_pool(x_i) # b, c/num_k, 1, 1 fc1 self.fc1_list[i] fc2 self.fc2_list[i] combined_feature torch.cat([avg_feature_x, max_feature_x], dim1) kernel fc2(self.relu(fc1(combined_feature))) # (b, 9c, 1, 1) kernel kernel.reshape(b, c, 9, 1) kernel F.softmax(kernel, dim2) kernel kernel.reshape(b, c, 3, 3)[0].unsqueeze(1) # (c/num_k, 1, 3, 3) kernels.append(self.identity_kernel - kernel) final_kernel torch.cat(kernels, dim0) # (num_k * c, 1, 3, 3) return final_kernel class AdaFDFusion(nn.Module): def __init__(self, c_out, c_low,num_k2): super(AdaFDFusion, self).__init__() # assert c_high c_low * 2, c_high must be 2 * c_low c_high 2*c_low self.num_k num_k self.c_high c_high self.c_low c_low self.c_out c_out self.init_high_conv nn.Conv2d(c_high, c_high // 2, 1, 1, paddingsame) self.y_conv1 nn.Conv2d(c_high, c_low, 1, 1) self.x_conv2 nn.Conv2d(c_high, c_low, 1, 1) self.HPG_conv nn.Sequential(nn.Conv2d(c_low, c_low, 7, 1, paddingsame, groupsc_low), nn.BatchNorm2d(c_low), nn.ReLU(inplaceTrue)) self.LPG_conv nn.Sequential(nn.Conv2d(c_low, c_low, 3, 1, paddingsame, groupsc_low), nn.BatchNorm2d(c_low), nn.ReLU(inplaceTrue)) self.Cat_conv nn.Sequential( nn.Conv2d(c_high c_low, c_high c_low, 3, 1, paddingsame, groupsc_high c_low), nn.BatchNorm2d(c_high c_low), nn.ReLU(inplaceTrue), nn.Conv2d(c_high c_low, c_out, 1, 1, paddingsame), nn.BatchNorm2d(c_out), nn.ReLU(inplaceTrue) ) self.res_conv nn.Conv2d(c_low, c_low * 2, 1, 1, padding0) self.HPG HighPassConvGenerator(c_low, num_kself.num_k) # (c_low, 1, 3, 3) self.LPG LowPassConvGenerator(c_low, num_kself.num_k) # (c_low, 1, 3, 3) self.x_conv_fhnorm nn.BatchNorm2d(self.c_low * 2) self.y_conv_flnorm nn.BatchNorm2d(self.c_low * 2) self.plus_conv nn.Sequential( nn.Conv2d(c_low, c_out, 3, 1, paddingsame), nn.BatchNorm2d(c_out), nn.ReLU(inplaceTrue), ) self.out_conv nn.Sequential( nn.Conv2d( c_out, c_out, 3, 1, paddingsame), nn.BatchNorm2d(c_out), nn.ReLU(inplaceTrue), nn.Conv2d(c_out, c_out, 3, 1, paddingsame), nn.BatchNorm2d(c_out), nn.ReLU(inplaceTrue), ) self.conv Conv(c_low,c_high,1) def forward(self, data): y_high,x_low data if y_high.shape[1] x_low.shape[1]: y_high self.conv(y_high) else: y_high y_high if y_high.shape[2] ! x_low.shape[2]: #如果y_high和x_low二者特征图大小不一样此时会对高频y_high进行上采样 y_high_up F.interpolate(y_high, scale_factor2, modebilinear, align_cornersTrue) else: y_high_up y_high cat_fuse torch.cat([x_low, y_high_up], dim1) # c_lowc_high,64,64 cat_fuse self.Cat_conv(cat_fuse) x_h self.HPG_conv(x_low) x_l self.LPG_conv(x_low) kernel_L self.LPG(x_l) kernel_H self.HPG(x_h) # 64,1,3,3 res x_low x_low F.conv2d(x_low, kernel_H, padding1, groupskernel_H.shape[0] // self.num_k) # 64,64,64 x_low self.x_conv_fhnorm(x_low self.res_conv(res)) x_low self.x_conv2(x_low) y_high_up self.init_high_conv(y_high_up) y_high_up F.conv2d(y_high_up, kernel_L, padding1, groupskernel_L.shape[0] // self.num_k) # 64,64,64 y_high_up self.y_conv_flnorm(y_high_up) y_high_up self.y_conv1(y_high_up) plus_fuse x_low y_high_up plus_fuse self.plus_conv(plus_fuse) fuse plus_fuse cat_fuse fuse self.out_conv(fuse) return fuse四、手把手教你配置模块和修改tasks.py文件1.首先在ultralytics/nn/newsAddmodules创建一个.py文件2.在ultralytics/nn/newsAddmodules/__init__.py中引用3.修改tasks.py文件在tasks.py中找到这个参数方法 def parse_model(d, ch, verboseTrue):elif m in { AdaFDFusion,}: c1 ch[f[1]] c2 ch[f[0]] args [c2, c1]五、创建涨点yaml配置文件展示 rtdetr-r18的改进 rtdetr-r50和rtdetr-resnet50的yaml文件可以参考rtdetr-r18的去配置 中期融合创新改进: rtdetr-r18-midfusion-AdaFDFusion.yaml# Ultralytics YOLO , AGPL-3.0 license # RT-DETR-l object detection model with P3-P5 outputs. For details see https://docs.ultralytics.com/models/rtdetr nc: 80 # number of classes scales: # model compound scaling constants, i.e. modelyolov8n-cls.yaml will call yolov8-cls.yaml with scale n # [depth, width, max_channels] l: [1.00, 1.00, 1024] #来自:Ai缝合怪 改进 ch: 6 backbone: # [from, repeats, module, args] #visible - [ -1, 1, Silence, [ ] ] # 0-P1/2 - [ 0, 1, SilenceChannel, [ 0,3 ] ] # 1-P1/2 - [-1, 1, ConvNormLayer, [32, 3, 2, None, False, relu]] # 2-P1/2 - [-1, 1, ConvNormLayer, [32, 3, 1, None, False, relu]] # 3 - [-1, 1, ConvNormLayer, [64, 3, 1, None, False, relu]] # 4 - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 5-P2/4 # [ch_out, block_type, block_nums, stage_num, act, variant] - [-1, 1, Blocks, [64, BasicBlock, 2, 2, relu]] # 6 - [-1, 1, Blocks, [128, BasicBlock, 2, 3, relu]] # 7-P3/8 - [-1, 1, Blocks, [256, BasicBlock, 2, 4, relu]] # 8-P4/16 - [-1, 1, Blocks, [512, BasicBlock, 2, 5, relu]] # 9-P5/32 # [from, repeats, module, args] #infrared - [ -1, 1, Silence, [ ] ] # 10-P1/2 - [ 0, 1, SilenceChannel, [3,6 ] ] # 11-P1/2 - [ -1, 1, ConvNormLayer, [ 32, 3, 2, None, False, relu ] ] # 12-P1/2 - [ -1, 1, ConvNormLayer, [ 32, 3, 1, None, False, relu ] ] # 13 - [ -1, 1, ConvNormLayer, [ 64, 3, 1, None, False, relu ] ] # 14 - [ -1, 1, nn.MaxPool2d, [ 3, 2, 1 ] ] # 15-P2/4 # [ch_out, block_type, block_nums, stage_num, act, variant] - [-1, 1, Blocks, [64, BasicBlock, 2, 2, relu]] # 16 - [-1, 1, Blocks, [128, BasicBlock, 2, 3, relu]] # 17-P3/8 - [-1, 1, Blocks, [256, BasicBlock, 2, 4, relu]] # 18-P4/16 - [-1, 1, Blocks, [512, BasicBlock, 2, 5, relu]] # 19-P5/32 head: # visible 可见光 - [9, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 20 input_proj.2 - [-1, 1, AIFI, [1024, 8]] # 21 - [-1, 1, Conv, [256, 1, 1]] # 22, Y5, lateral_convs.0 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 23 - [8, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 24 input_proj.1 - [[-2, -1], 1, Concat, [1]] # 25 - [-1, 3, RepC3, [256,0.5]] # 26, fpn_blocks.0 - [-1, 1, Conv, [256, 1, 1]] # 27, Y4, lateral_convs.1 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 28 - [7, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 29 input_proj.0 - [[-2, -1], 1, Concat, [1]] # 30 cat backbone P4 - [-1, 3, RepC3, [256,0.5]] # X3 (31), fpn_blocks.1 - [-1, 1, Conv, [256, 3, 2]] # 32, downsample_convs.0 - [[-1, 27], 1, Concat, [1]] # 33 cat Y4 - [-1, 3, RepC3, [256,0.5]] # F4 (34), pan_blocks.0 - [-1, 1, Conv, [256, 3, 2]] # 35, downsample_convs.1 - [[-1, 22], 1, Concat, [1]] # 36 cat Y5 - [-1, 3, RepC3, [256,0.5]] # F5 (37), pan_blocks.1 # infrared 红外光 - [19, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 38 input_proj.2 - [-1, 1, AIFI, [1024, 8]] # 39 - [-1, 1, Conv, [256, 1, 1]] # 40, Y5, lateral_convs.0 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 41 - [18, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 42 input_proj.1 - [[-2, -1], 1, Concat, [1]] # 43 - [-1, 3, RepC3, [256,0.5]] # 44, fpn_blocks.0 - [-1, 1, Conv, [256, 1, 1]] # 45, Y4, lateral_convs.1 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 46 - [17, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 47 input_proj.0 - [[-2, -1], 1, Concat, [1]] # 48cat backbone P4 - [-1, 3, RepC3, [256,0.5]] # X3 (49), fpn_blocks.1 - [-1, 1, Conv, [256, 3, 2]] # 50, downsample_convs.0 - [[-1, 45], 1, Concat, [1]] # 51 cat Y4 - [-1, 3, RepC3, [256,0.5]] # F4 (52), pan_blocks.0 - [-1, 1, Conv, [256, 3, 2]] # 53, downsample_convs.1 - [[-1, 40], 1, Concat, [1]] # 54 cat Y5 - [-1, 3, RepC3, [256,0.5]] # F5 (55), pan_blocks.1 - [[31, 49], 1, AdaFDFusion, [256]] # cat backbone P3 56 - [[34, 52], 1, AdaFDFusion, [256]] # cat backbone P4 57 - [[37, 55], 1, AdaFDFusion, [256]] # cat backbone P5 58 - [[56, 57, 58], 1, RTDETRDecoder, [nc, 256, 300, 4, 8, 6]] # Detect(P3, P4, P5) 中后期融合创新改进: rtdetr-r18-mid-to-late-AdaFDFusion.yaml# Ultralytics YOLO , AGPL-3.0 license # RT-DETR-l object detection model with P3-P5 outputs. For details see https://docs.ultralytics.com/models/rtdetr nc: 80 # number of classes scales: # model compound scaling constants, i.e. modelyolov8n-cls.yaml will call yolov8-cls.yaml with scale n # [depth, width, max_channels] l: [1.00, 1.00, 1024] #来自:Ai缝合怪 改进 ch: 6 backbone: # [from, repeats, module, args] #visible - [ -1, 1, Silence, [ ] ] # 0-P1/2 - [ 0, 1, SilenceChannel, [ 0,3 ] ] # 1-P1/2 - [-1, 1, ConvNormLayer, [32, 3, 2, None, False, relu]] # 2-P1/2 - [-1, 1, ConvNormLayer, [32, 3, 1, None, False, relu]] # 3 - [-1, 1, ConvNormLayer, [64, 3, 1, None, False, relu]] # 4 - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 5-P2/4 # [ch_out, block_type, block_nums, stage_num, act, variant] - [-1, 1, Blocks, [64, BasicBlock, 2, 2, relu]] # 6 - [-1, 1, Blocks, [128, BasicBlock, 2, 3, relu]] # 7-P3/8 - [-1, 1, Blocks, [256, BasicBlock, 2, 4, relu]] # 8-P4/16 - [-1, 1, Blocks, [512, BasicBlock, 2, 5, relu]] # 9-P5/32 # [from, repeats, module, args] #infrared - [ -1, 1, Silence, [ ] ] # 10-P1/2 - [ 0, 1, SilenceChannel, [3,6 ] ] # 11-P1/2 - [ -1, 1, ConvNormLayer, [ 32, 3, 2, None, False, relu ] ] # 12-P1/2 - [ -1, 1, ConvNormLayer, [ 32, 3, 1, None, False, relu ] ] # 13 - [ -1, 1, ConvNormLayer, [ 64, 3, 1, None, False, relu ] ] # 14 - [ -1, 1, nn.MaxPool2d, [ 3, 2, 1 ] ] # 15-P2/4 # [ch_out, block_type, block_nums, stage_num, act, variant] - [-1, 1, Blocks, [64, BasicBlock, 2, 2, relu]] # 16 - [-1, 1, Blocks, [128, BasicBlock, 2, 3, relu]] # 17-P3/8 - [-1, 1, Blocks, [256, BasicBlock, 2, 4, relu]] # 18-P4/16 - [-1, 1, Blocks, [512, BasicBlock, 2, 5, relu]] # 19-P5/32 head: # visible 可见光 - [9, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 20 input_proj.2 - [-1, 1, AIFI, [1024, 8]] # 21 - [-1, 1, Conv, [256, 1, 1]] # 22, Y5, lateral_convs.0 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 23 - [8, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 24 input_proj.1 - [[-2, -1], 1, Concat, [1]] # 25 - [-1, 3, RepC3, [256,0.5]] # 26, fpn_blocks.0 - [-1, 1, Conv, [256, 1, 1]] # 27, Y4, lateral_convs.1 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 28 - [7, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 29 input_proj.0 - [[-2, -1], 1, Concat, [1]] # 30 cat backbone P4 - [-1, 3, RepC3, [256,0.5]] # X3 (31), fpn_blocks.1 # infrared 红外光 - [19, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 32 input_proj.2 - [-1, 1, AIFI, [1024, 8]] # 33 - [-1, 1, Conv, [256, 1, 1]] # 34, Y5, lateral_convs.0 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 35 - [18, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 36 input_proj.1 - [[-2, -1], 1, Concat, [1]] # 37 - [-1, 3, RepC3, [256,0.5]] # 38, fpn_blocks.0 - [-1, 1, Conv, [256, 1, 1]] # 39, Y4, lateral_convs.1 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 40 - [17, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 41 input_proj.0 - [[-2, -1], 1, Concat, [1]] # 42 cat backbone P4 - [-1, 3, RepC3, [256,0.5]] # X3 (43), fpn_blocks.1 #在此处进行多模态特征融合 - [[22, 34], 1, AdaFDFusion, [256]] # cat backbone P5 44 - [[27, 39], 1, AdaFDFusion, [256]] # cat backbone P4 45 - [[31, 43], 1, AdaFDFusion, [256]] # cat backbone P3 46 - [-1, 1, Conv, [256, 3, 2]] # 47, downsample_convs.0 - [[-1, 45], 1, Concat, [1]] # 48 cat Y4 - [-1, 3, RepC3, [256,0.5]] # F4 (49), pan_blocks.0 P4 - [-1, 1, Conv, [256, 3, 2]] # 50, downsample_convs.1 - [[-1, 44], 1, Concat, [1]] # 51 cat Y5 - [-1, 3, RepC3, [256,0.5]] # F5 (52), pan_blocks.1 P5 - [[46, 49, 52], 1, RTDETRDecoder, [nc, 256, 300, 4, 8, 6]] # Detect(P3, P4, P5) 后期融合创新改进: rtdetr-r18-latefusion-AdaFDFusion.yaml# Ultralytics YOLO , AGPL-3.0 license # RT-DETR-l object detection model with P3-P5 outputs. For details see https://docs.ultralytics.com/models/rtdetr nc: 80 # number of classes scales: # model compound scaling constants, i.e. modelyolov8n-cls.yaml will call yolov8-cls.yaml with scale n # [depth, width, max_channels] l: [1.00, 1.00, 1024] #来自:Ai缝合怪 改进 ch: 6 backbone: # [from, repeats, module, args] #visible - [ -1, 1, Silence, [ ] ] # 0-P1/2 - [ 0, 1, SilenceChannel, [ 0,3 ] ] # 1-P1/2 - [-1, 1, ConvNormLayer, [32, 3, 2, None, False, relu]] # 2-P1/2 - [-1, 1, ConvNormLayer, [32, 3, 1, None, False, relu]] # 3 - [-1, 1, ConvNormLayer, [64, 3, 1, None, False, relu]] # 4 - [-1, 1, nn.MaxPool2d, [3, 2, 1]] # 5-P2/4 # [ch_out, block_type, block_nums, stage_num, act, variant] - [-1, 1, Blocks, [64, BasicBlock, 2, 2, relu]] # 6 - [-1, 1, Blocks, [128, BasicBlock, 2, 3, relu]] # 7-P3/8 - [-1, 1, Blocks, [256, BasicBlock, 2, 4, relu]] # 8-P4/16 - [-1, 1, Blocks, [512, BasicBlock, 2, 5, relu]] # 9-P5/32 # [from, repeats, module, args] #infrared - [ -1, 1, Silence, [ ] ] # 10-P1/2 - [ 0, 1, SilenceChannel, [3,6 ] ] # 11-P1/2 - [ -1, 1, ConvNormLayer, [ 32, 3, 2, None, False, relu ] ] # 12-P1/2 - [ -1, 1, ConvNormLayer, [ 32, 3, 1, None, False, relu ] ] # 13 - [ -1, 1, ConvNormLayer, [ 64, 3, 1, None, False, relu ] ] # 24 - [ -1, 1, nn.MaxPool2d, [ 3, 2, 1 ] ] # 15-P2/4 # [ch_out, block_type, block_nums, stage_num, act, variant] - [-1, 1, Blocks, [64, BasicBlock, 2, 2, relu]] # 16 - [-1, 1, Blocks, [128, BasicBlock, 2, 3, relu]] # 17-P3/8 - [-1, 1, Blocks, [256, BasicBlock, 2, 4, relu]] # 18-P4/16 - [-1, 1, Blocks, [512, BasicBlock, 2, 5, relu]] # 19-P5/32 - [[7, 17], 1, AdaFDFusion, [128]] # cat backbone P3 20 - [[8, 18], 1, AdaFDFusion, [256]] # cat backbone P4 21 - [[9, 19], 1, AdaFDFusion, [512]] # cat backbone P5 22 head: - [-1, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 23 input_proj.2 - [-1, 1, AIFI, [1024, 8]] # 24 - [-1, 1, Conv, [256, 1, 1]] # 25, Y5, lateral_convs.0 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 26 - [21, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 27 input_proj.1 - [[-2, -1], 1, Concat, [1]] # 28 - [-1, 3, RepC3, [256,0.5]] # 29, fpn_blocks.0 - [-1, 1, Conv, [256, 1, 1]] # 30, Y4, lateral_convs.1 - [-1, 1, nn.Upsample, [None, 2, nearest]] # 31 - [20, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 32 input_proj.0 - [[-2, -1], 1, Concat, [1]] # 33 cat backbone P4 - [-1, 3, RepC3, [256,0.5]] # X3 (34), fpn_blocks.1 - [-1, 1, Conv, [256, 3, 2]] # 35, downsample_convs.0 - [[-1, 30], 1, Concat, [1]] # 36 cat Y4 - [-1, 3, RepC3, [256,0.5]] # F4 (37), pan_blocks.0 - [-1, 1, Conv, [256, 3, 2]] # 38, downsample_convs.1 - [[-1, 25], 1, Concat, [1]] # 39 cat Y5 - [-1, 3, RepC3, [256,0.5]] # F5 (40), pan_blocks.1 - [[34, 37, 40], 1, RTDETRDecoder, [nc, 256, 300, 4, 8, 6]] # Detect(P3, P4, P5)六、正常运行engine\trainer: taskdetect, modetrain, model./ultralytics/cfg/models/rt-detr_add/rtdetr-r18-latefusion-AdaFDFusion.yaml, data./datasets/data_FLIR.yaml, epochs30, timeNone, patience100, batch2, imgsz640, saveTrue, save_period-1, cacheFalse, device0, workers0, projectruns/, nameFLIR13, exist_okFalse, pretrainedTrue, optimizerAdamW, verboseTrue, seed0, deterministicTrue, single_clsFalse, rectFalse, cos_lrFalse, close_mosaic10, resumeFalse, ampFalse, fraction1.0, profileFalse, freezeNone, multi_scaleFalse, overlap_maskTrue, mask_ratio4, dropout0.0, valTrue, splitval, save_jsonFalse, save_hybridFalse, confNone, iou0.7, max_det300, halfFalse, dnnFalse, plotsTrue, sourceNone, vid_stride1, stream_bufferFalse, visualizeFalse, augmentFalse, agnostic_nmsFalse, classesNone, retina_masksFalse, embedNone, showFalse, save_framesFalse, save_txtFalse, save_confFalse, save_cropFalse, show_labelsTrue, show_confTrue, show_boxesTrue, line_widthNone, formattorchscript, kerasFalse, optimizeFalse, int8False, dynamicFalse, simplifyTrue, opsetNone, workspaceNone, nmsFalse, lr00.0001, lrf0.001, momentum0.937, weight_decay0.0005, warmup_epochs3.0, warmup_momentum0.8, warmup_bias_lr0.1, box7.5, cls0.5, dfl1.5, pose12.0, kobj1.0, nbs64, hsv_h0.015, hsv_s0.7, hsv_v0.4, degrees0.0, translate0.1, scale0.5, shear0.0, perspective0.0, flipud0.0, fliplr0.5, bgr0.0, mosaic0.0, mixup0.0, copy_paste0.0, copy_paste_modeflip, auto_augmentrandaugment, erasing0.4, crop_fraction1.0, cfgNone, trackerbotsort.yaml, channels6, use_simotmRGBIR6C, brightness0.2, save_dirruns\FLIR13 Overriding model.yaml nc80 with nc3 WARNING ⚠️ no model scale passed. Assuming scalel. from n params module arguments 0 -1 1 0 torch.nn.modules.linear.Identity [] 1 0 1 0 ultralytics.nn.modules.conv.SilenceChannel [0, 3] 2 -1 1 928 ultralytics.nn.modules.block.ConvNormLayer [3, 32, 3, 2, None, False, relu] 3 -1 1 9280 ultralytics.nn.modules.block.ConvNormLayer [32, 32, 3, 1, None, False, relu] 4 -1 1 18560 ultralytics.nn.modules.block.ConvNormLayer [32, 64, 3, 1, None, False, relu] 5 -1 1 0 torch.nn.modules.pooling.MaxPool2d [3, 2, 1] 6 -1 1 152192 ultralytics.nn.modules.block.Blocks [64, 64, class ultralytics.nn.modules.block.BasicBlock, 2, 2, relu] 7 -1 1 525568 ultralytics.nn.modules.block.Blocks [64, 128, class ultralytics.nn.modules.block.BasicBlock, 2, 3, relu] 8 -1 1 2099712 ultralytics.nn.modules.block.Blocks [128, 256, class ultralytics.nn.modules.block.BasicBlock, 2, 4, relu] 9 -1 1 8393728 ultralytics.nn.modules.block.Blocks [256, 512, class ultralytics.nn.modules.block.BasicBlock, 2, 5, relu] 10 -1 1 0 ultralytics.nn.modules.conv.Silence [] 11 0 1 0 ultralytics.nn.modules.conv.SilenceChannel [3, 6] 12 -1 1 928 ultralytics.nn.modules.block.ConvNormLayer [3, 32, 3, 2, None, False, relu] 13 -1 1 9280 ultralytics.nn.modules.block.ConvNormLayer [32, 32, 3, 1, None, False, relu] 14 -1 1 18560 ultralytics.nn.modules.block.ConvNormLayer [32, 64, 3, 1, None, False, relu] 15 -1 1 0 torch.nn.modules.pooling.MaxPool2d [3, 2, 1] 16 -1 1 152192 ultralytics.nn.modules.block.Blocks [64, 64, class ultralytics.nn.modules.block.BasicBlock, 2, 2, relu] 17 -1 1 525568 ultralytics.nn.modules.block.Blocks [64, 128, class ultralytics.nn.modules.block.BasicBlock, 2, 3, relu] 18 -1 1 2099712 ultralytics.nn.modules.block.Blocks [128, 256, class ultralytics.nn.modules.block.BasicBlock, 2, 4, relu] 19 -1 1 8393728 ultralytics.nn.modules.block.Blocks [256, 512, class ultralytics.nn.modules.block.BasicBlock, 2, 5, relu] 20 9 1 131584 ultralytics.nn.modules.conv.Conv [512, 256, 1, 1, None, 1, 1, False] 21 -1 1 789760 ultralytics.nn.modules.transformer.AIFI [256, 1024, 8] 22 -1 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1] 23 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, nearest] 24 8 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1, None, 1, 1, False] 25 [-2, -1] 1 0 ultralytics.nn.modules.conv.Concat [1] 26 -1 3 657920 ultralytics.nn.modules.block.RepC3 [512, 256, 3, 0.5] 27 -1 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1] 28 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, nearest] 29 7 1 33280 ultralytics.nn.modules.conv.Conv [128, 256, 1, 1, None, 1, 1, False] 30 [-2, -1] 1 0 ultralytics.nn.modules.conv.Concat [1] 31 -1 3 657920 ultralytics.nn.modules.block.RepC3 [512, 256, 3, 0.5] 32 -1 1 590336 ultralytics.nn.modules.conv.Conv [256, 256, 3, 2] 33 [-1, 27] 1 0 ultralytics.nn.modules.conv.Concat [1] 34 -1 3 657920 ultralytics.nn.modules.block.RepC3 [512, 256, 3, 0.5] 35 -1 1 590336 ultralytics.nn.modules.conv.Conv [256, 256, 3, 2] 36 [-1, 22] 1 0 ultralytics.nn.modules.conv.Concat [1] 37 -1 3 657920 ultralytics.nn.modules.block.RepC3 [512, 256, 3, 0.5] 38 19 1 131584 ultralytics.nn.modules.conv.Conv [512, 256, 1, 1, None, 1, 1, False] 39 -1 1 789760 ultralytics.nn.modules.transformer.AIFI [256, 1024, 8] 40 -1 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1] 41 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, nearest] 42 18 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1, None, 1, 1, False] 43 [-2, -1] 1 0 ultralytics.nn.modules.conv.Concat [1] 44 -1 3 657920 ultralytics.nn.modules.block.RepC3 [512, 256, 3, 0.5] 45 -1 1 66048 ultralytics.nn.modules.conv.Conv [256, 256, 1, 1] 46 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, nearest] 47 17 1 33280 ultralytics.nn.modules.conv.Conv [128, 256, 1, 1, None, 1, 1, False] 48 [-2, -1] 1 0 ultralytics.nn.modules.conv.Concat [1] 49 -1 3 657920 ultralytics.nn.modules.block.RepC3 [512, 256, 3, 0.5] 50 -1 1 590336 ultralytics.nn.modules.conv.Conv [256, 256, 3, 2] 51 [-1, 45] 1 0 ultralytics.nn.modules.conv.Concat [1] 52 -1 3 657920 ultralytics.nn.modules.block.RepC3 [512, 256, 3, 0.5] 53 -1 1 590336 ultralytics.nn.modules.conv.Conv [256, 256, 3, 2] 54 [-1, 40] 1 0 ultralytics.nn.modules.conv.Concat [1] 55 -1 3 657920 ultralytics.nn.modules.block.RepC3 [512, 256, 3, 0.5] 56 [31, 49] 1 2982153 ultralytics.nn.newaddmodules.AdaFDFusion_TGRS2025.AdaFDFusion[256, 256] 57 [34, 52] 1 2982153 ultralytics.nn.newaddmodules.AdaFDFusion_TGRS2025.AdaFDFusion[256, 256] 58 [37, 55] 1 2982153 ultralytics.nn.newaddmodules.AdaFDFusion_TGRS2025.AdaFDFusion[256, 256] 59 [56, 57, 58] 1 7308017 ultralytics.nn.modules.head.RTDETRDecoder [3, [256, 256, 256], 256, 300, 4, 8, 6] rtdetr-r18-latefusion-AdaFDFusion summary: 988 layers, 48,584,652 parameters, 48,584,625 gradients, 158.92 GFLOPs