Developing Algorithms to Apply Machine Learning to Tabular Data 

Improving scientific analysis and network intrusion detection applications.  

Note: Case studies shown are a combination of Northern Analysis engagements and professional projects from our founding team’s careers. No confidential, proprietary, or sensitive information is disclosed. Certain details may be generalized or omitted to protect client, employer, and stakeholder confidentiality.

Summary

Artificial Intelligence based on Deep Learning models have achieved remarkable accuracy in image recognition, however, applying them effectively to structured tabular data – the kind of data used in business and science — has led to unreliable results.

Darrin co-developed two novel tabular-to-image (T2I) conversion algorithms — Binary Image Encoding (BIE) and Correlated Binary Image Encoding (cBIE) — which convert structured data into images using each feature’s native binary representation, preserving all original information without normalization. The algorithms were evaluated against the previous state-of-the-art method, IGTD, across four benchmark datasets using ResNet-50. BIE and cBIE consistently matched or outperformed the current state of the art models in all metrics.  On the most complex dataset — a large multiclass network intrusion detection problem — BIE outperformed not only IGTD but also the leading traditional machine learning approach, XGBoost.  

Darrin’s system outperformed industry benchmarks in scientific analysis and network intrusion, advancing the field of machine learning advancements and computer vision to tabular data.  This was published in the Journal of Data Science in February 2025. 

Situation

Deep learning has transformed fields like computer vision and natural language processing, but its application to tabular data — the most common format in business, government, and scientific analysis — hasunderperformed. The core difficulty is that convolutional neural networks are designed to exploit spatial relationships between neighbouring data points, a property that raw tabular data does not have. 

To bridge this gap, Darrin and team developed tabular-to-image algorithms that convert the feature vectors of tabular data into 2D images the chosen deep learning model can process. The leading method, IGTD, improved on earlier approaches by arranging features spatially according to their correlations. However, it still required min-max normalization of the input data, making it sensitive to outliers and prone to information loss. It also produced lossy pixel representations, with each pixel mapping to a single aggregated feature value. No existing method fully preserved the original data while also capturing spatial correlation structure in both dimensions of the generated image. 

Solution

Darrin and his co-authors developed BIE, which encodes each feature as its native binary representation — converting floating-point values into their IEEE 754 sign, exponent, and mantissa bits, and integers and booleans into their corresponding binary strings. These bit vectors are stacked vertically to form a 2D grayscale image, where each row corresponds to a feature and each column to a bit position. Because the encoding uses the native structure of each data type, no normalization is required and no information is lost in translation. The resulting images are compact, computationally efficient to generate, and compatible with standard pre-trained CNN architectures such as ResNet-50. 

The team also developed cBIE, an extension that reorders the feature rows by their Pearson correlation with the target label before encoding. This correlation-based ordering causes each row to be spatially adjacent to the most similar features, replicating the neighborhood structure that CNNs are designed to exploit in natural images. Together, BIE and cBIE addressed the core limitations of prior methods — normalization dependence, information loss, and one-dimensional spatial structure — without requiring domain knowledge or complex pre-processing. 

Results

The algorithms were evaluated on four benchmark problems spanning biological classification, chemical analysis, medical diagnosis, and network intrusion detection.  

On three of the four datasets, BIE and cBIE achieved perfect accuracy, F1-scores, and AUC. This outperformed IGTD, which fell short of perfect scores on two of those three. On the most challenging dataset, a 13-class DDoS attack detection problem with hundreds of thousands of samples, BIE achieved 77.04% accuracy compared to IGTD’s 67.83% — and also exceeded XGBoost,  the strongest traditional benchmark, which achieved 74.08% on the same task. BIE and cBIE also encoded images substantially faster than IGTD on smaller datasets, an important consideration for real-time and resource-constrained applications. 

The system’s outperformance against the leading industry benchmarks in scientific analysis and network intrusion detection verified the algorithm’s effectiveness. It established binary encoding as a viable and competitive framework for applying deep learning to structured data across a wide range of classification problems.   

The research was published in the Journal of Data Science in February 2025, with code and datasets released publicly to support further development.