A Novel Nonlinear Equalizer for Probabilistic Shaping 64-QAM Based on Constellation Segmentation and Support Vector Machine

Abstract

The probability distribution of probabilistic shaping 64 quadrature amplitude modulation (PS-64QAM) is uneven. The traditional M-ary support vector machine (SVM) algorithm is incompatible with the data set with uneven distribution. In order to solve the problem, we propose a novel nonlinear equalizer (NLE) for PS-64QAM based on constellation segmentation (CS) and SVM, called CS M-ary SVM NLE. The performance of CS M-ary SVM NLE has been demonstrated in a 120 Gb/s PS-64QAM coherent optical communication system. The experimental results show that after employing the proposed scheme, the launched optical power dynamic range (LOPDR) of PS-64QAM can be increased by 1.6 dBm compared with the situation without NLE. In addition, aided by the proposed scheme, the LOPDR of PS-64QAM is increased by 0.6 dBm than M-ary SVM NLE. Compared with employing M-ary SVM NLE and without employing NLE, when employing the proposed scheme, the Q factor is improved about 0.50 dB and 0.96 dB, respectively. The number of support vectors (SVs) and CPU running time for both NLE schemes are collected to measure the complexity of the two NLE schemes. The results show that the complexity of the proposed scheme is lower than that of the M-ary SVM scheme under the entire measured launched optical power range from −5 dBm to +5 dBm.

1. Introduction

In the past several decades, data traffic has expanded with explosive growth due to the development of big data, cloud computing, Internet of Things, and other services. Inspired by the ever-increasing demands for transmission bandwidth and capacity, high-order modulation formats have been extensively applied in optical fiber communication systems [1]. However, high-order modulation formats are extremely susceptible to nonlinear impairments caused by devices and fiber transmission links in optical fiber communication systems. Furthermore, a higher optical signal-to-noise ratio (OSNR) is required to achieve the desired bit error ratio (BER).

As an attractive solution to approach the Shannon limit in optical fiber communication systems, PS has recently become a research hot spot. PS can effectively reduce the average power of the constellation by increasing the transmission probabilities of inner constellation points while reducing the transmission probabilities of outer constellation points. Aided with PS technology, optical fiber communication systems will have a better ability to restrain nonlinearity effects and get better BER performance [2,3,4,5]. In the Ref. [6], based on probabilistically shaped PDM 4096-QAM, a transmission of a 10-subcarrier superchannel with a record-high intra-channel spectral efficiency of 17.3 bit/s/Hz over 50 km fiber is demonstrated. Joint channel coding and constellation shaping based on polar-coded modulations is also proposed, which offers more than 0.6 dB gain without the need for an external distribution matcher and an increase of decoding complexity [7]. A feedback-based PS-16QAM modulation method, which is used to adaptively modify the distribution of transmitted symbols based on errors at the receiver, is proposed in the Ref. [8]. The error feedback enables solving an optimization problem to find the distribution that maximizes the mutual information between the input and output of the channel without knowledge of the channel itself. It shows superior performance in a variety of application scenarios, and realizes ultra-high performance improvement of reducing the system bit error rate by 50%. In addition, a probabilistic shaping distribution matching algorithm employing uneven segmentation is proposed, which achieves a 0.4 dB performance improvement compared with the single forward error correction (FEC) system under the condition of maintaining low complexity of encoding and decoding [9].

Besides, with the development of coherent detection, many digital signal processing (DSP) algorithms have been proposed to compensate the nonlinear effects. Digital back-propagation (DBP), which is a traditional DSP algorithm, is effective at mitigating nonlinearity but with large complexity in real implementations [10,11]. In recent years, machine learning (ML) techniques have attracted more and more attention to deal with nonlinear effects in optical communication systems. For example, some ML clustering algorithms are used to cluster constellation points such as k-means [12] and Gaussian mixture models (GMM) [13]. In addition, some ML classifier algorithms such as SVM [14] and K-nearest neighbor (KNN) [15] have been used for classifying constellation points. Among various ML algorithms, SVM has shown prominent proficiency for mitigating fiber nonlinearity. The M-ary SVM is introduced as a nonparameter nonlinear phase noise (NLPN) mitigation approach for the coherent optical systems [16]. A fast Newton-based SVM nonlinear equalizer (NLE) is experimentally demonstrated in 40 Gb/s 16-quadrature amplitude modulated coherent optical orthogonal frequency division multiplexing at 2000 km of transmission whose computational load is ∼6 times lower than traditional SVM [17]. In addition, a SVM-based machine learning method to mitigate modulation nonlinearity distortion for PAM-4 and PAM-8 vertical cavity surface emitter laser multi-mode fiber (VCSEL-MMF) optical link has been demonstrated in the Ref. [18].

Recently, some schemes combining PS and ML algorithms have been proposed to improve the performance of the optical communication systems. In the Ref. [19], a weighted K-means scheme for a PS 64QAM signal was proposed in order to locate the decision points more accurately and enhance the robustness of clustering algorithm. In addition, the combined benefit of artificial neural network-based nonlinearity compensation and probabilistic shaping was experimentally demonstrated in the Ref. [20].

In this paper, we propose a novel nonlinear equalizer for PS-64QAM based on CS and SVM. This scheme can effectively mitigate the fiber nonlinearity with lower complexity. The performance of CS M-ary SVM NLE has been demonstrated in a 120 Gb/s PS-64QAM coherent optical communication system. The experimental results show that the LOPDR can be increased by 1.6 dBm and 0.6 dBm by employing the proposed scheme compared with the situation without NLE and employing the traditional M-ary SVM NLE, respectively. Moreover, by using the proposed scheme, the Q factor of PS-64QAM is improved by about 0.50 dB and 0.96 dB compared with employing M-ary SVM NLE and without employing NLE, respectively. We use the number of SVs and CPU running time for both schemes to evaluate the complexity of the schemes. The results show that the complexity of the proposed scheme is lower than that of the M-ary SVM scheme under the entire measured launched optical power range from −5 dBm to +5 dBm.

This paper is organized as follows. We begin in Section 2 by discussing the principles of PS, M-ary SVM and CS M-ary SVM. In Section 3, we describe the experimental system. In Section 4, we discuss the experimental results. Finally, we provide some concluding remarks in Section 5.

2. Principle of the Proposed Scheme

In this section, we will introduce the principle of PS in Section 2.1. In addition, the principle of M-ary SVM is introduced in Section 2.2. Finally, the principle of CS M-ary SVM is introduced in Section 2.3.

2.1. Principle of PS

The probability distribution of PS-64QAM follows the Maxwell-Boltzmann (MB) distribution, which can be expressed as:

$$P_{x_i}=\frac{exp(-v(Re{\left(x_i\right)}^2+Im{\left(x_i\right)}^2))}{{\sum }_{i=1}^{64}exp(-v(Re{\left(x_i\right)}^2+Im{\left(x_i\right)}^2))},$$

(1)

where $x_i$ stands for the ith PS-64QAM constellation point and v denotes the shaping parameter. With the aid of PS, the PS-64QAM constellation will have a lower average power compared with the uniform 64QAM constellation by increasing the transmission probabilities of the points with lower energy while decreasing the transmission chances of the points with higher energy. When $v=0.05$, the probability distribution of PS-64QAM is shown in Figure 1. According to the probability mass function (PMF) $P_{x_i}$, the source entropy H of PS-64QAM can be calculated by:

$$H=-\sum _{i=1}^{64}{P}_{x_i}lo{g}_2{P}_{x_i}.$$

(2)

2.2. Principle of M-ary SVM

As a binary classifier, the basic idea of SVM is to find the separating hyperplane which can correctly divide the training data set and has the maximum margin. Figure 2 shows the example of linearly separating data from two groups in two dimensions. The separating hyperplane can be expressed as:

$$y\left(\mathbf{x}\right)={\mathbf{w}}^T\mathbf{x}+b,$$

(3)

where the vector $\mathbf{w}=\left(w_1;w_2;\cdots ;w_d\right)$ and the scalar b are the parameters of the hyperplane. The hyperplane function $y\left(\mathbf{x}\right)$ is learned from N training vectors ${\mathbf{x}}_1,\cdots ,{\mathbf{x}}_N$ with the corresponding labels $l_1,\cdots ,l_N$, where $l_n\in \left\{-1,+1\right\}$, and the testing data are classified according to the sign of $y\left(\mathbf{x}\right)$.

However, the data cannot always be separated linearly. For the linearly inseparable situation, a kernel function can be used to map the data from a lower dimensional feature space to a higher dimensional feature space, which is shown in Figure 3. In the higher dimensional space, a hyperplane can be obtained to linearly separate the data. In this paper, the Gaussian radial basis function (RBF) was chosen as the kernel function:

$$\kappa ({\mathbf{x}}_i,{\mathbf{x}}_j)=exp(-\frac{{∥{\mathbf{x}}_i-{\mathbf{x}}_j∥}^2}{2{\sigma }^2}).$$

(4)

SVM is a binary classifier which can only divide the data into two classes. For PS-64QAM signals, each constellation cluster should be considered as a class. Therefore, it is inevitable to build a multiclass classifier by combining many binary SVMs to divide the data into different classes. In order to solve the problem, a “one-against-all” approach is proposed in the Ref. [21], which is a multiclass classifier whose number of SVMs is the same as the number of the classes. Therefore, the PS-64QAM signal requires 64 SVMs to divide the data into 64 classes. Another commonly used scheme for the multiclass classifier is called “one-against-one” [22]. In this scheme, $K(K-1)/2$ (K is the number of classes) different binary classifier models are trained on all possible pairs of classes. Then the data are detected according to which class has the highest number of “votes”. However, the above approaches are time-consuming because they need more training data to obtain a large number of classifiers.

A M-ary SVM scheme for the K-class signal was proposed in the Ref. [16]. In this scheme, only $lo{g}_{2}K$ binary SVMs are required for the K-class signal. For PS-64QAM, each signal carries 6 bits and each bit is modeled by a binary SVM. The classification strategy for PS-64QAM is shown in Figure 4. The six SVMs from SVM1 to SVM6 are used to detect the first bit to the sixth bit of the PS-64QAM constellation points. For PS-64QAM, each symbol is detected by six SVMs. We take one constellation point (010010) of PS-64QAM as an example to illustrate the classification strategy. Firstly, the test data (010010) are detected by SVM1, and the first bit “0” is labeled as “−1”. Next, the SVM2 is used to detect the second bit and the second bit “1” is labeled as “+1”. Then SVM3, SVM4, SVM5 and SVM6 are used to detect the third, fourth, fifth and sixth bits, respectively. The third to the sixth bits of the example constellation point (010010) were labeled as “−1", “−1", “+1” and “−1", respectively. Therefore, PS-64QAM can be detected accurately by only six SVMs aided by the M-ary SVM scheme. This scheme can effectively reduce the training time and calculation quantity of the multiclass classification SVM model.

2.3. Principle of CS M-ary SVM

As we already know, the distribution of PS-64QAM constellation points is uneven. For the SVM algorithm, an uneven distribution constellation will impel the hyperplane to move to the lower probability constellation points as close as possible. Therefore, a large number of outer constellation points will be mistakenly considered as inner constellation points. In order to solve the problem, we proposed the CS M-ary SVM NLE scheme. The process of the CS M-ary SVM NLE scheme is shown in Figure 5. Firstly, the constellation of PS-64QAM is divided into two subsets by a SVM. In this step, the training data (60% of the data set) are used to obtain a hyperplane, and the testing data (40% of the data set) are divided into two subsets according to the hyperplane. Then the inner and outer sets are detected by the M-ary SVM scheme, respectively. Figure 6 shows the Q factor as a function of launched optical power for 120 Gb/s PS-64QAM with different constellation segment methods. The Q factor can be calculated by BER and the relationship between them can be expressed as $Q=20lo{g}_{10}\left[\sqrt{2}erf{c}^{-1}\left(2BER\right)\right]$ [23]. For PS-64QAM, the constellation points have nine different amplitudes and the points with the same amplitude have the same probability, which can be calculated by the Formula (1). The amplitude and probability distributions of the constellation points are shown in

Table 1. The amplitude and probability distributions of the standard PS-64QAM constellation ($v=0.05$).

The Number of Constellation Points	The Amplitude of Each Constellation Point	The Probability of Each Constellation Point
4	$\sqrt{2}$	0.058564
8	$\sqrt{10}$	0.039446
4	$\sqrt{18}$	0.026569
8	$\sqrt{26}$	0.017666
8	$\sqrt{34}$	0.011899
12	$\sqrt{50}$	0.005324
8	$\sqrt{58}$	0.003586
8	$\sqrt{74}$	0.001606
4	$\sqrt{98}$	0.000484

. In the step of the constellation segment, the training data are labeled as “0” or “1” to identify the points belonging to the inner set or the outer set. We label the training data according to the amplitudes of the corresponding standard constellation points which have not been transmitted in the optical communication system. The hyperplane obtained by the training data is used to divide the testing data into two subsets. According to the amplitudes of the standard constellation points, there are eight constellation segment methods. As can be seen in Figure 6, when the number of inner points is 12 and the number of outer points is 52, the PS-64QAM system has the best performance. In addition, according to the probability distribution of PS-64QAM, when the number of inner points is 12, the sum of the probabilities of inner constellation points is 0.549824, which is the closest to 0.5 compared with the other seven constellation segment methods. Therefore, in this scheme, we pick the situation in which the number of inner constellation points is 12 and the number of outer constellation points is 52 as the constellation segment method.

For the constellation segment method we choose in this paper, the number of inner constellation points is 12 and the number of outer constellation points is 52. For the inner constellation points, the second bits of all constellation points are 1, while the fifth bits of all constellation points are 1 as well. Therefore, only 4 SVMs (Inner-SVM1, Inner-SVM3, Inner-SVM4 and Inner-SVM6) are required to detect the symbol of the inner subset whose classification strategy is shown in Figure 7a. However, there are six SVMs needed for the outer set detection, whose classification strategy is shown in Figure 7b. Finally, we obtain the mapping symbol and calculate the BER.

3. Experimental Setup

Figure 8a depicts the experimental setup for a 120 Gb/s PS-64QAM coherent optical communication system. At the transmitter side, the pseudo-random binary sequence (PRBS) is generated by a Matlab program. A constant composition distribution matcher (CCDM) [24,25] is used to generate non-uniform 64QAM signals with a Maxwell–Boltzmann distribution. In this paper, we set v as 0.05. Then the non-uniform 64QAM signals are mapped to a PS-64QAM constellation. The analog signals are sent into a 25 GSa/s arbitrary waveform generator (AWG). The signal output from the AWG is amplified by two electric amplifiers (EAs) and fed into the I/Q modulator to generate modulated signals. An external cavity laser (ECL) is used to generate a continuous optical wave with a linewidth of 100 kHz and wavelength of 1550 nm. The polarization-division-multiplexing (PDM) module includes a polarization-maintaining optical coupler (PM-OC), a polarization controller (PC), optical delay line, and a polarization beam combiner (PBC). An erbium-doped fiber amplifier (EDFA) is set before the variable optical attenuator (VOA) to amplify the signals. The VOA is also used to adjust the signal power.

The transmission link consists of five spans of G.652D single-mode fiber (SMF) with a length of 75 km. There is an EDFA following each span to compensate for the fiber loss. There are five spans in this system. At the receiver side, an ECL with a linewidth of 100 kHz is used as the local oscillator (LO) for coherent detection. Then, the optical signals are detected by the coherent receiver. A 100 GSa/s digital phosphor oscilloscope (DPO) is used to digitize the signals. The offline digital signal processing (DSP) flow chart is shown in Figure 8b. The offline DSP includes the low-pass filter, amplitude normalization, chromatic dispersion compensation (CDC), clock recovery, resampling, Gram–Schmidt orthogonalizing process (GSOP), constant modulus algorithm (CMA) equalization, frequency offset estimation (FOE), carrier phase recovery (CPR), CS M-ary SVM NLE, PS-64QAM demapping and BER calculation.

4. Results and Discussion

In this section, we will discuss the performance of the CS M-ary SVM NLE scheme in compensating the nonlinear effects of the 120 Gb/s PS-64QAM coherent optical communication system.

Figure 9 shows the Q factor and the number of SVs versus the ratio of training data when launched optical power is 0 dBm. The size of the data set is 32,768. With the increase of the ratio of training data, both the Q factor and the number of SVs are increasing. The complexity of SVM only depends on the number of SVs. Therefore, the complexity increases with the increase of the ratio of training data. When the ratio of training data is greater than 0.6, although the number of SVs is increasing, the performance of the system is no longer improved. The Q factor of the system has been convergent. Therefore, we pick 60% data as training data while 40% of data are testing data.

Figure 10 shows the Q factor as a function of launched optical power for 120 Gb/s PS-64QAM with CS M-ary SVM NLE, M-ary SVM NLE, and without (w/o) employing NLE. The Q factor of the soft-decision FEC (SD-FEC) limit is 6.25 dB, which corresponds to the BER of $2\times {10}^{-2}$ [26]. As shown in Figure 10, when the launched optical power is in the range of −5 dBm to −3 dBm, the Q factor increases with the increase of launched optical power. This is because in this range, the OSNR of the received signal is increasing with the increase of the launched optical power. In this situation, amplified spontaneous emission (ASE) is the main element affecting the performance of the system. However, when the launched optical power is higher than −3 dBm, the Q factor will decrease as the increase of launched optical power. In this launched optical power range, the system is mainly affected by the fiber nonlinearity instead of ASE. The nonlinearity of the system is more and more serious as the launched optical power increasing, especially for the outer constellation points with higher power.

The LOPDR, which represents the difference between the two power values corresponding to the points where Q factors are equal to the Q factor of the SD-FEC limit (6.25 dB) is employed to evaluate the performance of the NLE schemes. As for the results shown in Figure 10, the LOPDR is 9.2 dBm (−5 dBm ∼+4.2 dBm) under CS M-ary SVM NLE. In addition, the LOPDRs are 8.6 dBm (−5 dBm ∼+3.6 dBm) and 7.6 dBm (−4.8 dBm ∼+2.8 dBm) under M-ary SVM NLE and without NLE, respectively. Therefore, when employing the proposed scheme, the LOPDR is 0.6 dBm larger than when the M-ary SVM NLE scheme is employed. In addition, with the aid of the proposed scheme, the LOPDR of PS-64QAM is 1.6 dBm larger than without the NLE situation. At the optimal launched optical power of −3 dBm, compared with employing M-ary SVM NLE and without employing NLE, the proposed CS M-ary SVM scheme can obtain about 0.50 dB and 0.96 dB Q factor gain, respectively. The constellation diagrams of PS-64QAM detected by CS M-ary SVM NLE and without employing NLE are illustrated in Figure 10a,b, respectively.

The complexity of the SVM algorithm is only dependent on the number of support vectors (SVs). The number of SVs is measured to compare the complexity of the two NLE schemes. The number of SVs for the two NLE schemes is shown in Figure 11a. As can be seen in the results, the number of SVs is the least at the optimal launched optical power of −3 dBm. Due to the ASE noise, the number of SVs is increasing with the decrease of the launched optical power when the optical power is lower than −3 dBm. However, when the launched optical power is higher than −3 dBm, the number of SVs is gradually increasing as the launched optical power is increasing. Throughout the entire measured launched optical power range from −5 dBm to +5 dBm, the number of SVs of CS M-ary NLE is always lower than that of M-ary NLE.

In addition, the CPU running time of the two NLE schemes is depicted in Figure 11b. The results show that the CPU running time is the lowest when the launched optical power is −3 dBm. Throughout the entire measured launched optical power range from −5 dBm to +5 dBm, the CPU running time of CS M-ary NLE is always lower than that of M-ary NLE.

5. Conclusions

In this paper, we have proposed a novel CS M-ary SVM NLE scheme to mitigate the nonlinearity of the optical communication system. This scheme firstly divided the PS-64QAM constellation into two subsets by a SVM. Then the inner and outer sets were detected by M-ary SVM NLE scheme, respectively. This scheme improves the Q factor of the PS-64QAM signal with lower complexity. Experimental verification of the proposed CS M-ary NLE scheme was demonstrated in a 120 Gb/s PS-64QAM coherent optical communication system with a transmission distance of 375 km. The experimental results show that for the proposed scheme, the LOPDR of PS-64QAM is 0.6 dBm and 1.6 dBm higher than that of the M-ary SVM scheme and without the NLE scheme, respectively. In addition, for the proposed scheme, the Q factor of PS-64QAM is improved by about 0.50 dB and 0.96 dB compared with employing M-ary SVM NLE and without employing NLE, respectively. In terms of complexity, we used the number of SVs to measure the complexity of the two NLE schemes. The results show that the complexity of the proposed scheme is lower than that of the M-ary SVM scheme under the entire measured launched optical power range from −5 dBm to +5 dBm.