If you are still worried about "wanting both" in edge computing, then this article may be worth reading.

Abstract

In the design of lightweight convolutional neural networks, how to balance "local receptive field" and "global spatial awareness" under limited FLOPs has always been a core challenge.

Traditional $3 \times 3$ convolution is limited by receptive field size, while conventional SE-Block attention mechanisms cause spatial position information collapse due to global pooling operations. For this reason, a new operator was specially developed. This structure innovatively fuses $5 \times 5$ large kernel depthwise convolution with Coordinate Attention, and through forced residual strategy and GroupNorm optimization, successfully constructs a hardware-friendly feature extraction paradigm with robust position encoding capabilities.

1. Design Motivation and Theoretical Background

Before analyzing the code, we need to understand the three core pain points this module attempts to solve:

• Limitations of Effective Receptive Field (ERF): Traditional lightweight networks overly rely on $3 \times 3$ convolution stacking. According to research, the actual effective receptive field of deep networks often follows a Gaussian distribution and decays with depth, making it difficult to capture large-scale semantic targets.
• Spatial Semantic Misalignment: Standard SE modules compress feature maps to $1 \times 1 \times C$ through Global Average Pooling. While this strengthens channel dependencies, it completely loses object spatial coordinate information.
• Micro-Batch Instability: When performing transfer learning or fine-tuning on edge devices, limited by memory, Batch Size is often extremely small (such as 2 or 4). At this time, BatchNorm's statistical estimation will produce huge deviations, leading to training divergence.

And the attention-fused convolution kernel constructed this time is precisely the solution proposed based on the above theoretical background.

2. Core Architecture Breakdown

This module is not a simple layer stacking, but a carefully designed feature reconstruction closed loop. The following combines code logic for in-depth analysis step by step:

2.1 Inductive Bias from Large Kernel Depthwise Convolution

Code implementation:

self.dw_conv = nn.Conv2d(c1, c1, kernel_size=5, stride=s, padding=2, groups=c1, bias=False)

Design: The convolution kernel is upgraded from $3 \times 3$ to $5 \times 5$. From an information theory perspective, this increases the "visible area" of a single neuron.

Theoretical advantages: The receptive field area of $5 \times 5$ convolution is $25/9 \approx 2.78$ times that of $3 \times 3$. In lightweight networks (such as MobileNetV3), this large kernel depthwise convolution can effectively simulate the Token Mixer behavior in Transformers, enhancing the ability to capture textures and shapes, and inference frameworks like NCNN already have extremely high Winograd algorithm optimization support for $5 \times 5$ DW operators.

2.2 Orthogonal Feature Decomposition and Coordinate Attention

This is the "soul" of this module. Unlike SE's global pooling, this module uses two orthogonal 1D Global Pooling operations to decompose spatial information.

Step I: Orthogonal Projection

x_h = self.pool_h(feat)  # Output: (N, C, H, 1)
x_w = self.pool_w(feat)  # Output: (N, C, 1, W)

• Mathematical representation: Input tensor $X$ is aggregated along horizontal coordinate $X$ and vertical coordinate $Y$ respectively. This operation generates two direction-aware feature maps, enabling the network to capture long-range dependencies along one spatial direction while preserving precise position information in the other direction.

Step II: Cross-Dimensional Interaction and Dimensionality Reduction

y = torch.cat([x_h, x_w], dim=2)
y = self.conv_pool(y)
y = self.gn(y) # GroupNorm for stability

• Optimization strategy: A bottleneck layer with reduction=16 is introduced here to reduce model complexity.

• Improvement: The introduction of GroupNorm is the finishing touch. In the middle layer of the attention branch, feature channels are compressed and often accompanied by extremely small Batch Size. GN normalizes by grouping channels, and its statistics do not depend on Batch Size, thus solving the "statistical drift" problem caused by BN layers in fine-tuning tasks.

Step III: Attention Recalibration

a_h = self.conv_h(x_h).sigmoid()
a_w = self.conv_w(x_w).sigmoid()
out = identity_feat * a_w * a_h

• Feature fusion: The final output feature map is obtained through Hadamard Product of the original features with attention maps in both directions. This is equivalent to assigning an "importance weight" calculated based on global context to each pixel $(i, j)$ on the feature map.

2.3 Forced Residual Flow

if self.use_res:
    return x + out

• Gradient flow protection: Attention mechanism is essentially a "Soft Gating". In the early training stage, attention weights may be close to zero. Forced residual connection builds an identity mapping pathway, ensuring that in the worst case (attention layer failure), the module degenerates into a standard convolution layer, thus guaranteeing effective backpropagation of gradients in deep networks and avoiding gradient vanishing.

3. Detailed Execution Flow and Tensor Evolution

To more clearly demonstrate the internal data flow of this module, we formalize the Forward process into the following detailed steps:

1. Spatial Feature Extraction:

• Input $X \in \mathbb{R}^{N \times C_1 \times H \times W}$.

• After $5 \times 5$ DWConv $\rightarrow$ $1 \times 1$ PWConv $\rightarrow$ BN $\rightarrow$ Hardswish.

• Output intermediate feature $F \in \mathbb{R}^{N \times C_2 \times H \times W}$.

2. Coordinate Information Encoding:

   • H-Pooling: Compress $F$ into $Z^h \in \mathbb{R}^{N \times C_2 \times H \times 1}$.
   • W-Pooling: Compress $F$ into $Z^w \in \mathbb{R}^{N \times C_2 \times 1 \times W}$.

3. Transformation and Activation:

• Concatenate $Z^h$ and $Z^w$ and reduce dimension to $C_{mid}$ through $1 \times 1$ convolution.
• Apply GroupNorm(1, mip) for normalization (Group=1 here is equivalent to LayerNorm, but for channel dimension).
• Apply Non-linear activation function.

4. Decoding and Reweighting:

• Split the feature tensor back into spatially aware weight vectors $A^h$ and $A^w$.
• $Y = F \odot A^h \odot A^w$ (where $\odot$ represents element-wise multiplication under broadcasting mechanism).

5. Feature Reconstruction (Reconstruction):

• Final output $O = X + Y$ (if residual condition is satisfied).

4. Experimental Validation and Data Visualization

To verify the effectiveness of this convolution kernel in real scenarios, we conducted rigorous comparative experiments in a controlled environment.

Experimental Setup:

• Dataset: Custom detection dataset (containing high similarity categories such as batons, flashlights, knives, etc.).

• Training Strategy: SGD optimizer, Cosine LR scheduler, training for 5000 Epochs (to ensure complete model convergence).

Baseline: Only replace this module with standard 3x3 DWConv, keeping all other network architectures completely identical.

4.1 Overall Performance Evaluation: Trade-off between Computational Cost and Accuracy

4.2 Hard Example Mining and Fine-Grained Classification

The biggest challenge in testing is "inter-class similarity".

For example, long and thin "batons" and "flashlights" are extremely difficult to distinguish at low resolution.

We extracted the model's Top-1 Accuracy on these specific categories for comparative analysis:

5. Conclusion

This operator demonstrates a highly forward-looking design approach for lightweight networks.

1. Introduces stronger spatial inductive bias through $5 \times 5$ convolution.
2. Solves the problem of standard CNNs lacking position awareness through Coordinate Attention.
3. Demonstrates excellent engineering implementation awareness through GroupNorm and Hardswish, making it highly practical for few-shot fine-tuning and edge-side inference scenarios.

This module is not only a plug-and-play component, but also provides a standard spatial-channel decoupling paradigm for subsequent lightweight detection network design.