Cardiac function state recognition model based on bimodal time–frequency representation

Keywords: Multi-modal, Phonocardiogram signal, Electrocardiogram signal, Feature encoding, Heart disease screening

Cardiovascular disease remains one of the leading causes of death and illness worldwide, currently accounting for 16% of all deaths [1]. The prevalence of cardiovascular disease in China continues to rise, with an estimated 330 million people now affected [2]. Phonocardiogram (PCG) and electrocardiogram (ECG) serve as critical physiological parameters for evaluating cardiac function. PCG mainly reflect the mechanical and acoustic features of valve movement and blood flow, while ECG records the heart’s electrophysiological activity [3, 4]. 

Current research has widely examined single-modality classification of PCG and ECG [5, 6]. Advances in sensing and signal-processing technologies have made multimodal technology an emerging direction, as the two modalities together provide complementary information [7]. 

Chakir et al. extracted ten time-domain statistical features from ECG and PCG [8]. Li et al. divided the PCG signal into four frequency bands and used convolutional neural networks and long short-term memory for feature extraction [9]. Sun et al. improved coronary artery disease detection by decoupling ECG and PCG to create coupled signals, and extracted entropy features and recursive graph depth features [10]. Li et al. enhanced the Dempster-Shafer evidence theory to fuse classification results, which improved the accuracy of the final decision [11]. Zhang et al. proposed an end-to-end Co-learning-assisted Progressive Dense Fusion Network with a three-branch interleaved architecture, which showed strong performance on both public and private datasets [12]. Zhu et al. proposed Dual-Scale Deep Residual Network, which uses a dual-scale feature aggregation module to merge low-level features from different scales [13]. Liu et al. used the Vision Transformer to detect coronary heart disease [14]. However, several challenges remain, including low positive detection rates for medical conditions, limited feature representations, and complex network architectures. 

This study proposes a cardiac function state recognition method based on dual-modality time–frequency representations. The time–frequency maps provide a comprehensive view of the cardiac signals. A neural network module then automatically extracts features from these maps, and finally, a machine learning model performs classification.

2.1 Overall architecture

To address the aforementioned issues, this study proposes a cardiac function state recognition model based on dual-modality PCG and ECG data.

The proposed model framework consists of four main modules: the signal preprocessing module, the time–frequency information conversion module, the feature vector extraction module, and the feature fusion and classification module.

2.2 Data preprocessing

Given the characteristics of the signals and the presence of noise, this study first performed preprocessing and denoising on the PCG and ECG signals separately.

Noise in PCG signals usually comes from friction sounds caused by a subject’s breathing or movements [15]. To denoise the PCG signals, this study adopted the Complete Ensemble Empirical Mode Decomposition with Adaptive Noise proposed in the literature [16]. First, the method was used to extract multi-timescale intrinsic mode function (IMF) components. Next, detrend fluctuation analysis metrics were calculated for these IMF components. Finally, adaptive dual-threshold wavelet analysis was applied to the IMF components based on these metrics to perform denoising and signal reconstruction, producing the cleaned PCG signal. Following preprocessing, the PCG signal was normalized to the 0–1 range.

Figure 1 shows the PCG signals before and after noise reduction. The noise was effectively removed without altering the original waveform shape.

The main sources of noise in ECG signals are baseline drift caused by a subject’s respiration and power-line interference from electrical equipment [17]. To remove baseline drift, this study applied a fourth-order Butterworth high-pass filter with a cutoff frequency of 0.5 Hz. To eliminate 60 Hz power-line interference and its 120 Hz harmonic, a second-order infinite impulse response comb filter was used on the ECG signal. Figure 2 shows the ECG signals before and after noise reduction. The noise was also eliminated.

2.3 Time-frequency information conversion module

A spectrogram is a time–frequency representation of a signal, converting a one-dimensional time-domain signal into a two-dimensional image that contains both time and frequency information. This allows for a more comprehensive analysis of non-stationary signals. Compared with one-dimensional time-series signals, time–frequency plots preserve both temporal and frequency details, helping models capture the dynamic evolution of signals more effectively. Unlike traditional spectral analysis, which provides only global frequency information, time–frequency plots show when specific frequencies occur or change, offering greater flexibility and precision.

The time–frequency plot shows time on the horizontal axis and frequency on the vertical axis. Color or brightness represents the signal’s power or energy, with brighter colors indicating higher energy at a specific time and frequency. This representation provides neural networks with input that carries rich physical meaning.

2.3.1 Mel-frequency cepstral coefficients (MFCC) transformation of PCG signals

PCG signals are non-stationary, with frequencies that change over time. This study uses MFCC as the time–frequency representation for one-dimensional PCG signals. MFCC has the advantage of simulating the human ear’s sensitivity to low frequencies using Mel filter banks, which map the linear spectrum onto the Mel scale.

The logarithmic Mel-spectrogram is created by first filtering a signal through a set of Mel-scale filters, followed by logarithmic compression applied to the resulting power spectrum.

First, the power spectral density of the signal P(k) is calculated:

where X(k) denotes the frequency component of the signal in the k band, and N represents the number of sample points.

Next, the Mel filter bank is calculated.

The Mel-scale frequency intervals for the Mel filter bank are:

where M denotes the number of Mel filters. The Mel filter bank covers a minimum frequency of ɸmin and a maximum frequency of ɸmax.

Therefore, the centre frequencies of the Mel scale for the Mel filters in Group M are:

The formula for converting frequency from hertz to the Mel scale is:

The formula for converting frequency from the Mel scale back to hertz is:

The frequency response formula for an equal-height triangular filter bank is:

where f(m) denotes the centre frequency of the m-th filter bank in hertz.

Next, the logarithmic spectral energy distribution of the Mel frequencies for each frame is calculated:

In this study, the MFCC window length was set to 25 milliseconds to balance temporal and frequency resolution. The overlap length was set to 15 milliseconds to reduce information loss between frames and to smooth time–frequency variations. The number of Mel filters was set to 40, providing higher resolution in the low-frequency range while still covering the high-frequency range. The fast Fourier transform length was set to 512 to balance efficiency and accuracy. Since PCG signals have frequency components between 0 and 500 Hz, the frequency range was set to 0–500 Hz [18].

The MFCCs of normal and abnormal PCG signals are shown in Figure 3A and 3B, respectively.

2.3.2 Fourier synchrosqueezing transform (FSST) transformation of ECG signals

ECG signals are also non-stationary. FSST allows visualization of their frequency information over time. When ECG signals have abrupt changes or transient frequency components, FSST can identify their instantaneous frequency characteristics [19]. By synchronously compressing the results of the short-time Fourier transform (STFT), FSST concentrates dispersed energy toward the instantaneous frequency curve, producing a clearer time–frequency representation.

First, the STFT of the input signal is computed:

where f represents frequency, τ represents the time parameter, and g(τ-t) represents the window function.

The STFT is limited by the principle of time–frequency uncertainty. Signal energy spreads in the time–frequency plane due to the window function, causing frequency blurring. To estimate instantaneous frequency, the partial derivative with respect to time is calculated:

The coefficients V(t, f) obtained from the STFT are first mapped along the frequency axis to the estimated instantaneous frequency . Synchronous compression is then applied to redistribute the energy, producing the amplitude of the FSST:

where ω represents the frequency after synchronous compression, g(0) denotes the window function at time zero, and δ represents the Dirac delta function.

The FSST tracings for normal and abnormal ECG signals are shown in Figure 4A and 4B respectively.

Normal ECG signals show cleaner and more concentrated energy in the FSST, whereas abnormal ECG signals display chaotic and dispersed energy across the spectrum.

In this study, the FSST used a Kaiser window function with a length of 64. Since ECG signals contain frequency components between 0 and 250 Hz, the frequency range was set to 0–250 Hz accordingly [14].

2.4 Feature vector extraction module

2.4.1 PCG feature vector extraction module

This study uses VGG16 as the baseline model for fine-tuning and modification. The network architecture and parameters are shown in Table 1 and Figure 5.

Additionally, this study incorporates the Dual Relation-Aware Attention Network Module (DA Module) to improve classification accuracy [20]. The architecture of the DA Module is shown in Figure 6.

The DA Module contains three components: the Position Attention Module, the Channel Attention Module, and the additive fusion layer. Both the input size and the output size are C×H×W.

The structures of the Position Attention Module and Channel Attention Module are shown in Figure 7A and 7B respectively.

Within the Position Attention Module, the input feature map A first passes through a convolutional layer to extract features. The module then replicates these features into three components: Query, Key, and Value. Each component is reshaped before the positional attention calculation begins. After reshaping, the Query feature map B is transposed to size HW×C, while the Key feature map C is reshaped to size C×HW. The attention scores between these two maps are then computed, reducing the feature map size to HW×HW. The module multiplies this encoded positional attention by the Value feature map D, and finally reshapes the result back to C×H×W. The computational process is as follows:

After obtaining the feature map K modulated by the positional attention mechanism, the module multiplies it by the weighting coefficient α. It then fuses this result with the original input feature map A through addition:

where α is set to 0.5.

Similarly, within the Channel Attention Module, the input feature map A is first reshaped to C×HW. The module then multiplies A by its transpose AT, producing a matrix of size C×C. After that, the feature is normalised using the Softmax function to generate attention scores between channels. These attention scores are then multiplied by A and finally reshaped back to C×H×W. The computational process is as follows:

After obtaining the feature map U modulated by the channel attention mechanism, the module multiplies it by the weighting coefficient β and fuses it with the input feature map A through summation:

The outputs from the two attention modules are fused through summation, enriching the spatial contextual information and strengthening the discriminative power of the channel features.

2.4.2 ECG feature vector extraction module

This study uses ResNet18 as the baseline model for fine-tuning and modification. The network architecture and parameters are shown in Table 2 and Figure 8.

Here, SC represents the residual connections, DSConv represents depthwise separable convolutions, and MidConv denotes the convolutional layers where the input and output dimensions are identical.

To reduce the number of parameters and computational load, this study replaces the convolutional layers within residual blocks with depthwise separable convolutions. Depthwise separable convolutions decompose standard convolution operations into two lighter, decoupled steps: depthwise convolution and pointwise convolution. The depthwise convolution uses fixed-size kernels to independently convolve each channel and stack the resulting feature maps. The subsequent pointwise convolution employs a 1×1 kernel to aggregate information from the depthwise stage, enabling linear combinations across channels. To further enhance inter-channel information fusion, a MidConv layer is introduced to strengthen feature interactions among different channels.

Additionally, this study incorporates the Selective Kernel Module (SK Module) to improve classification accuracy [21]. The architecture of the SK Module is shown in Figure 9.

The SK Module includes three components: Split, Fuse, and Select.

In the Split section, two convolutional kernels of different sizes are first applied to the input feature map X, producing Ũ and Û, respectively. These two outputs are then added together to generate U. The computational process is as follows:

The Fuse component uses global average pooling to extract key information from the overall context:

Subsequently, a fully connected layer is used to compress the key information S into a compact representation Z:

The Select component then processes Z usingtwo fully connected layers of size d×C. The resulting outputs are concatenated and normalized via the softmax function to generate the weights 

Finally, the first dimension is multiplied by weight W and applied to Ũ, while the second dimension is multiplied by weight W and applied to Û. This process produces the feature map V, which is modulated by the attention mechanism with selectable kernels:

2.5 Feature fusion and recognition

This study employed a feature vector extraction module to obtain 64-dimensional features from each modality. These features were then concatenated to form a 128-dimensional feature vector. Finally, classification was performed using a Support Vector Machine (SVM) model.

3.1 Experimental data

To validate the effectiveness of the proposed method, data from the publicly available Physio+Net/Computing in Cardiology Challenge 2016 dataset were used. The training-a subset of this dataset contains 409 samples. After excluding samples without paired ECG recordings and those officially annotated as unclassifiable due to excessive noise, 388 samples remained. Each sample includes synchronously recorded PCG and ECG signals and is labeled as either normal or abnormal. The sampling frequency is 2,000 Hz. Statistical analysis indicates that the sample durations in the training-a set range from 9 to 36 seconds. The 388 samples were divided into training and test sets at a ratio of 9:1. To prevent data leakage, segments derived from the same recording were assigned exclusively to either the training set or the test set. To better reflect real-world conditions, record-level classification results were used instead of segment-level results. Specifically, record-level predictions were obtained by averaging the segment-level output probabilities.

3.2 Experimental setup

The model was trained on a system equipped with a single NVIDIA RTX 4060 GPU (16 GB VRAM). The implementation was based on the PyTorch deep learning framework (version 2.5.1). During training, the Adam optimizer was employed for 60 epochs with a learning rate of 1×10-5. The batch size was set to 32, and the cross-entropy loss function was used.

3.3 Evaluation criteria

This study uses accuracy (Acc), sensitivity (Sen), specificity (Spe), and F1 score to evaluate model performance. The definitions of these four metrics are as follows:

where TP, FP, TN, and FN represent the number of true positives, false positives, true negatives, and false negatives in the confusion matrix, respectively.

In binary classification, sensitivity measures the proportion of actual positive samples correctly predicted as positive. Higher sensitivity indicates a stronger ability to identify positive samples, resulting in fewer missed detections. Specificity measures the proportion of actual negative samples correctly predicted as negative. Higher specificity indicates a stronger ability to exclude negative examples, resulting in fewer false positives. Precision measures the proportion of true positives among all samples predicted as positive, reflecting the accuracy of the model’s predictions. The F1 score is the harmonic mean of sensitivity and precision, providing a balanced metric for overall model performance.

As shown in Table 3, the improved PCG model achieved higher Acc, Sen, Spe, and F1 score. The PCG model with the Axial Attention Module achieved the highest Spe. However, its overall Acc remained lower than that of the model with the DA Module, which reached 85.3%. As shown in Table 4, the improved ECG model incorporating the SK Module achieved the highest Acc, Sen, and F1 scores. Table 5 presents a comparative assessment of dual-modality versus single-modality results. The PCG and ECG dual-modality network outperformed the single-modality approach across all four metrics, with particularly strong complementary improvement in specificity. While time–frequency images effectively detect abnormal cardiac function, the dual-modality approach increases the model’s confidence in negative predictions and reduces false alarms. The “Benchmark” in Tables 3 and 4 refers to the backbone network without any additional modules introduced.

Table 6 compares different classifiers. The decision tree model achieved the highest specificity, but its overall accuracy was relatively low. The SVM achieved the highest Acc, Sen, and F1 score.

This study proposed a cardiac function state recognition model based on dual-modality time–frequency representations of PCG and ECG signals. First, both PCG and ECG signals underwent preprocessing for noise reduction and segmentation. Next, the one-dimensional time-series signals were converted into time–frequency plots to better capture pathological information. For these two-dimensional time–frequency plots, distinct automatic feature extraction neural networks were designed. Accuracy and sensitivity were improved by refining the baseline model and integrating attention modules. Since minimizing missed diagnoses is crucial in disease screening, this study prioritized high sensitivity. Finally, the 64-dimensional feature vectors from each modality were concatenated and input into an SVM, achieving a classification accuracy of 95.4% and a sensitivity of 97.4%.

Compared with state-of-the-art methods, the approach developed in this study achieved higher Acc and Sen on public datasets. In future work, this research will focus on refining the methodology and models to improve their generalization capabilities, allowing them to be applied in scenarios with missing modalities.

Author contributions

Mingzhi Zhang was responsible for data processing, time–frequency conversion, model construction, and manuscript writing. Piding Li made important contributions to data analysis.

Funding

This research received no external funding.

Data availability

This study utilised only publicly available datasets.

Ethics approval and consent to participate

Not applicable.

Consent for publication

The authors agree to the publication of this manuscript in both print and electronic formats in ZENTIME and grant the publisher permission to make necessary editorial and formatting revisions.

Competing interests

The authors declare that they have no competing interests.

Acknowledgements

The authors confirm that this manuscript is original, unpublished, and not under consideration elsewhere. All authors have approved the manuscript and confirm adherence to ethical guidelines, including disclosure of potential conflicts of interest.

[1] World Health Organization. WHO reveals leading causes of death and disability worldwide: 2000-2019 [Internet]. Geneva (Switzerland): WHO; 2020 [updated 2020 Dec 9; cited 2025 Nov 20]. Available from: https://www.who.int/news/item/09-12-2020-who-reveals-leading-causes-of-death-and-disability-worldwide-2000-2019

[2] National Center for Cardiovascular Diseases, The Writing Committee of the Report on Cardiovascular Health and Diseases in China. Report on cardiovascular health and diseases in China 2023: An updated summary. Chin Circ J. 2024 Jul 27;39(7):625-660. https://doi.org/10.3969/j.issn.1000-3614.2024.07.001

[3] Karhade J, Dash S, Ghosh SK, Dash DK, Tripathy RK. Time-frequency-domain deep learning framework for the automated detection of heart valve disorders using PCG signals. IEEE Trans Instrum Meas. 2022 Mar 29;71:1-11. https://doi.org/10.1109/TIM.2022.3163156

[4] Houssein EH, Ibrahim IE, Neggaz N, Hassaballah M, Wazery YM. An efficient ECG arrhythmia classification method based on manta ray foraging optimization. Expert Syst Appl. 2021 Nov 1;181:115131. https://doi.org/10.1016/j.eswa.2021.115131

[5] Liu C, Springer D, Li Q, Moody B, Juan RA, Chorro FJ, et al. An open access database for the evaluation of heart sound algorithms. Physiol Meas. 2016 Dec;37(12):2181. https://doi.org/10.1088/0967-3334/37/12/2181

[6] Han H, Huang X, Chang H, Fan H, Chen P, Chen J. Review of self-supervised learning methods in field of ECG. J Front Comput Sci Technol. 2024;18(7):1683-1704. https://doi.org/10.3778/j.issn.1673-9418.2310043

[7] Cosoli G, Poli A, Scalise L, Spinsante S. Measurement of multimodal physiological signals for stimulation detection by wearable devices. Measurement. 2021 Nov;184:109966. https://doi.org/10.1016/j.measurement.2021.109966

[8] Chakir F, Jilbab A, Nacir C, Hammouch A. Recognition of cardiac abnormalities from synchronized ECG and PCG signals. Phys Eng Sci Med. 2020 Jun;43(2):673-677. https://doi.org/10.1007/s13246-020-00875-2

[9] Li P, Hu Y, Liu ZP. Prediction of cardiovascular diseases by integrating multi-modal features with machine learning methods. Biomed Signal Process Control. 2021 Apr;66:102474. https://doi.org/10.1016/j.bspc.2021.102474

[10] Sun C, Liu X, Liu C, Wang X, Liu Y, Zhao S, et al. Enhanced CAD detection using novel multi-modal learning: Integration of ECG, PCG, and coupling signals. Basel. 2024 Oct 30;11(11):1093. https://doi.org/10.3390/bioengineering11111093

[11] Li J, Ke L, Du Q, Chen X, Ding X. Multi-modal cardiac function signals classification algorithm based on improved D-S evidence theory. Biomed Signal Process Control. 2022 Jan;71(Part A):103078. https://doi.org/10.1016/j.bspc.2021.103078

[12] Zhang H, Zhang P, Lin F, Chao L, Wang Z, Ma F, et al. Co-learning-assisted progressive dense fusion network for cardiovascular disease detection using ECG and PCG signals. Expert Syst Appl. 2024 Mar 15;238(Part F):122144. https://doi.org/10.1016/j.eswa.2023.122144

[13] Zhu J, Liu H, Liu X, Chen C, Shu M. Cardiovascular disease detection based on deep learning and multi-modal data fusion. Biomed Signal Process Control. 2025 Jan;99:106882. https://doi.org/10.1016/j.bspc.2024.106882

[14] Liu X, You L, Lv C, Chen M, Wei L, Zheng Y, et al. Integrating ECG and PCG signals through a Dual-Modal ViT for coronary artery disease detection. IEEE Journal of Biomedical and Health Informatics 2025. https://doi.org/10.1109/JBHI.2025.3589257

[15] Xu W, Yu K, Ye J, Li H, Chen J, Yin F, et al. Automatic pediatric congenital heart disease classification based on heart sound signal. Artif Intell Med. 2022 Apr;126:102257. https://doi.org/10.1016/j.artmed.2022.102257

[16] Lu G, Tang T, Qi J, Wang Y, Zhao Y. Heart sound denoising based on CEEMDAN and wavelet analysis with adaptive dual thresholds. J Nanjing Univ Posts Telecommun (Nat Sci). 2025 Aug;45(4):36-47. https://doi.org/10.14132/j.cnki.1673-5439.2025.04.005

[17] Kabir MA, Shahnaz C. Denoising of ECG signals based on noise reduction algorithms in EMD and wavelet domains. Biomed Signal Process Control. 2012 Sep;7(5):481-489. https://doi.org/10.1016/j.bspc.2011.11.003

[18] Suryady Z, Rahman AWA, Oktarina D. Human emotion classification based on phonocardiography signals (PCG). AIP Conf Proc. 2024 Feb 7;2729(1):020002. https://doi.org/10.1063/5.0168132

[19] Daubechies I, Lu J, Wu HT. Synchrosqueezed wavelet transforms: An empirical mode decomposition-like tool. Appl Comput Harmon A. 2011 Mar;30(2):243-261. https://doi.org/10.1016/j.acha.2010.08.002

[20]Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, et al. Dual attention network for scene segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition; 2019. https://doi.org/10.1109/cvpr.2019.00326

[21]Li X, Wang W, Hu X, Yang J. Selective Kernel Networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 2019 June 15-20; Long Beach, CA, USA. IEEE. p. 510-519. https://doi.org/10.1109/CVPR.2019.00060

[22] Hettiarachchi R, Haputhanthri U, Herath K, Kariyawasam H, Munasinghe S, Wickramasinghe K, et al. A novel transfer learning-based approach for screening pre-existing heart diseases using synchronized ECG signals and heart sounds. 2021 IEEE International Symposium on Circuits and Systems (ISCAS); 2021 May 22-28; Daegu, Korea. IEEE; 2021. p. 1-5. https://doi.org/10.1109/ISCAS51556.2021.9401093