Hybrid Constellation Shaping 64QAM Based on Hexagonal Lattice of Constellation Subset

Abstract

Increasing demand for higher-speed and large-capacity data communications has driven the development of constellation shaping technology. This paper proposes a hybrid constellation shaping scheme for 64-quadrature amplitude modulation (64QAM) based on hexagonal lattice of a constellation subset. The proposed scheme aims to enhance the nonlinear tolerance of higher-order modulated signals and further improve the constellation shaping gain. The initial quantitative characterization of the constellation is firstly performed based on the hexagonal lattice structure. Then, the objective function of maximizing constellation figure of merits (CFM) is utilized to determine the position distribution of constellation points, resulting in the generation of the geometric shaping-64QAM (GS-64QAM) signal. Finally, according to concentric hexagonal layers, all constellation points are divided into multiple subsets where points within the same subset are assigned the same probability, and the hybrid shaping-64QAM (HS-64QAM) signal is generated. To validate the effectiveness of the proposed scheme, the experimental verification was demonstrated in a 120 Gbit/s multi-span coherent optical communication system. Experimental results indicate that, at the soft-decision forward error correction threshold, HS-64QAM achieves an optical signal-to-noise ratio (OSNR) gain of 1.9 dB and 4.1 dB over uniform GS-64QAM in back-to-back and 375 km transmission scenarios, respectively. Furthermore, HS-64QAM achieves an OSNR gain of 2.7 dB and 7.6 dB over uniform Square-64QAM in back-to-back and 375 km transmission scenarios, respectively.

1. Introduction

With the rapid development of global communication, new diversified technologies such as artificial intelligence, cloud computing, and big data have been promoted and integrated [1,2,3]. Optical fiber communication systems are faced with the increasing demands for higher-speed and large-capacity data communication [4,5,6]. To meet the demands mentioned above, higher-order modulation technology has been widely adopted to improve spectral efficiency and channel capacity without taking up additional spectrum [7,8]. However, with increasing the modulation order, the constellation points get closer to each other. Traditional higher-order modulation constellations are particularly sensitive to nonlinearities, resulting in limited transmission distances [9,10]. Among various emerging technologies, constellation shaping technology achieves the shaping gain by designing the position or probability distribution of constellation points, which is an effective approach to improve the system capacity [11,12,13].

Constellation shaping mainly includes geometric shaping (GS) and probabilistic shaping (PS). On the one hand, GS obtains a larger minimum Euclidean distance (min-ED) through equiprobable and non-equidistant constellation points. By flexibly designing arbitrarily shaped constellations, GS enables better utilization of the constellation plane and reduces the symbol error rate [14,15,16]. On the other hand, PS achieves the shaping gain by designing the probability distribution of equidistant constellation points, thereby decreasing the gap between the system capacity and the Shannon limit. At a given transmitted optical power (TOP), the signal energy is concentrated toward the constellation center. PS improves noise tolerance to achieve higher spectral efficiency and larger transmission capacity [17,18,19,20]. For example, Pavel et al. proposed Huffman-coded sphere shaping as a method for probabilistic constellation shaping, which provided an improved tolerance to fiber nonlinearities in single-span links [17]. Ding et al. introduced pairwise optimization and probabilistic fold shaping to optimize geometrically shaped and probabilistically shaped 32-ary quadrature amplitude modulation (QAM), respectively [14]. Tong et al. proposed a joint PS/precoding method that used PS to decrease the probability of constellation points with high power, which significantly improved the resistance to nonlinear distortions [18]. In addition, compared with the constellation structures such as square, cross, and star, the hexagonal constellation can obtain additional positions of constellation points when min-ED is fixed. For example, Hu et al. proposed a hexagonal constellation with a larger min-ED, which combined with rotation coding achieved around 1 dB gain of Q factor than square-16QAM [15]. To obtain a higher shaping gain and better transmission performance, hybrid shaping (HS) technology has become a research subject, which combines GS with PS [21,22,23,24]. Wu et al. proposed a PS method based on reduced-exponentiation subset indexing and honeycomb-structured constellation optimization, which compressed the number of signal constellation points and significantly decreased the average signal power [21]. Andrej et al. proposed a robust and pilot-free modulation scheme by leveraging joint geometric and probabilistic constellation shaping, which achieved a gain of at least 0.1 bit/symbol over square QAM constellations with neural demappers [22]. To the best knowledge of the authors, there has been much less research on HS than that on only PS or only GS over the past few decades.

In this paper, a novel hybrid constellation shaping scheme for 64QAM based on hexagonal lattice of constellation subset is proposed to enhance the nonlinear tolerance of higher-order modulated signals and further improve the constellation shaping gain. Firstly, the initial quantitative characterization of the 64QAM constellation is based on the hexagonal lattice structure. Then, the maximization of constellation figure of merits (CFM) is taken as the objective function, which obtains the position distribution of corresponding constellation points to generate the GS-64QAM signal. Finally, the HS-64QAM signal is obtained by dividing all constellation points into multiple subsets according to concentric hexagonal layers, where points within the same subset are assigned the same probability. In addition, the experimental verification of the proposed HS-64QAM scheme was demonstrated in a 120 Gbit/s multi-span coherent optical communication system. The performance of HS-64QAM is analyzed for the optical signal-to-noise ratio (OSNR) range from 16 dB to 21.5 dB in the back-to-back (BTB) scenario, and for the OSNR range from 17 dB to 30 dB and the TOP range from −11 dBm to −4 dBm in a 375 km transmission scenario. Experimental results indicate that HS-64QAM invariably outperforms uniform GS-64QAM and uniform Square-64QAM (S-64QAM) under various circumstances.

2. Theory and Principle

The proposed HS-64QAM process consists of three main steps, including the initial quantitative characterization of the 64QAM constellation, the generation of the GS-64QAM signal, and the generation of the HS-64QAM signal.

2.1. The Initial Quantitative Characterization of the 64QAM Constellation

The hexagonal structure can achieve denser constellation points in the constellation plane with min-ED fixed. Moreover, increasing min-ED reduces the probability of erroneous decisions and improves the error-tolerant performance of the communication system. Given the above reasons, the 64QAM constellation is initialized with a hexagonal lattice structure, where the centroid of each lattice is the initial position of the constellation points, as shown in Figure 1.

Assuming that there are $m$-layer constellation points in the optimized constellation, where $m=1,2,\cdots ,M_m$, $M_m$ is a positive integer. Moreover, the most central point is initialized to the first layer, which typically adopts the origin. Then, the constellation point set ${\mathrm{Point}}_{m,k}(x,y)$ at the m-th layer can be expressed as:

$$\begin{array}{cc}\hspace{1em}{S}_m\hfill & =\left\{{\mathrm{Point}}_{m,k}(x,y)\right\}=\left\{r_{m,k}\exp [j{\phi }_{m,k}(k)]\right\}\hfill \\ \hfill & =\left\{r_{m,k}\exp [j({\theta }_i+\frac{2\pi }{H_m}(H_m-k))]\right\}\hfill \\ \hfill & =\left\{\sqrt{(x_{m,k}^2+y_{m,k}^2)}\exp [j({\theta }_i+\frac{2\pi }{H_m}(H_m-k))]\right\},k=1,2,\cdots ,H_m\hfill \end{array},$$

(1)

where $(x_{m,k},y_{m,k})$ indicates the k-th constellation point on the m-th layer, $r_{m,k}$ and ${\phi }_{m,k}$ represent the radius and phase information of the constellation point $(x_{m,k},y_{m,k})$, respectively, $r_{m,k}=\sqrt{(x_{m,k}^2+y_{m,k}^2)}$, and ${\theta }_i$ denotes the minimum angle from the positive real axis between the constellation points on the m-th layer.

The number H_m of all constellation points on the m-th layer satisfies the following conditions:

$$H_m=\left\{\begin{array}{c}\begin{array}{cc}1& m=1\end{array}\\ \begin{array}{cc}6(m-1)& 1<m\le M_m-2\end{array}\\ \begin{array}{cc}h& m=M_m-1\end{array}\\ \begin{array}{cc}M-{\displaystyle \sum _{m=1}^{M_m-1}{H}_m}& m=M_m\end{array}\end{array}\right.,$$

(2)

where the variable h is the number of all constellation points on the second layer inwards from the outside, and M is the total number of 64QAM constellation points (M = 64).

Therefore, the set $S_C$ of all 64QAM constellation points can be obtained as follows:

$$S_C=\left\{S_1,S_2,\cdots ,S_m,\cdots ,S_{M_m}\right\}$$

(3)

2.2. Generation of the GS-64QAM Signal

To better measure the performance of the constellation C, the ratio of min-ED $d_{\min }(C)$ to the average power $P_{mean}^{}(C)$ is typically used to define $CFM$, which is expressed as:

$$CFM(C)=\frac{d_{\min }^2(C)}{P_{mean}^{}(C)}$$

(4)

GS optimizes the position distribution of constellation points, which improves the $CFM$ without changing the signal transmitted power. To improve the nonlinear tolerance of the 64QAM signal, the GS-64QAM is designed with the maximization of $CFM$ as the objective function. In addition, the normalization of min-ED is commonly adopted for evaluating the constellation performance by $CFM$.

Combined with Equations (1), (3), and (4), the objective function can be calculated as follows:

$$CF{M}^{\prime }(S_C)=\frac{1}{{\displaystyle \sum _{m=1}^{M_m}{\displaystyle \sum _{k=1}^{H_m}{H}_m{r}_{m,k}^2}/64}}=\frac{1}{{\displaystyle \sum _{m=1}^{M_m}{\displaystyle \sum _{k=1}^{H_m}{H}_m(x_{m,k}^2+y_{m,k}^2)}/64}}$$

(5)

The specific value of h and corresponding constellation coordinates are obtained after several iterations with the goal of achieving the maximum value of $CF{M}^{\prime }(S_C)$. Figure 2a shows the 64QAM constellation structure with adjusted coordinates. Since the number of points adjacent to a random constellation point ranges from two to six, gray coding cannot be strictly followed. To retain the superiorities of gray coding as much as possible, quasi-gray (QG) coding is adopted to minimize the Hamming distance of all adjacent constellation points, which decreases the damage of errors. QG coding preferentially encodes the case where the Hamming distance of the innermost point is exactly one and satisfies the requirement that the Hamming distance of the innermost point is smaller than the Hamming distance of the outermost point.

Then, the GS-64QAM signal is generated according to the QG coding rules. The GS-64QAM constellation diagram is shown in Figure 2b. To achieve symbols of normalized energy, the adjusted coordinates of constellation points in Figure 2a are normalized. Moreover, the normalized coordinates of the GS-64QAM constellation points are obtained, as illustrated in

Table 1. The bits under QG coding and normalized coordinates of GS-64QAM constellation points.

Point Index	QG-Bit	Normalized Coordinates	Point Index	QG-Bit	Normalized Coordinates	Point Index	QG-Bit	Normalized Coordinates
1	000000	0 + 0i	23	010110	0.1770 + 0.9197i	45	101100	−1.2390 + 0.3066i
2	000001	−0.3540 + 0i	24	010111	−0.7080 − 0.6131i	46	101101	1.2390 − 0.3066i
3	000010	0.3540 + 0i	25	011000	−0.7080 + 0.6131i	47	101110	1.2390 + 0.3066i
4	000011	−0.1770 − 0.3066i	26	011001	0.7080 − 0.6131i	48	101111	−0.8850 − 0.9197i
5	000100	−0.1770 + 0.3066i	27	011010	0.7080 + 0.6131i	49	110000	−0.8850 + 0.9197i
6	000101	0.1770 − 0.3066i	28	011011	−0.8850 − 0.3066i	50	110001	0.8850 − 0.9197i
7	000110	0.1770 + 0.3066i	29	011100	−0.8850 + 0.3066i	51	110010	0.8850 + 0.9197i
8	000111	−0.5310 − 0.3066i	30	011101	0.8850 − 0.3066i	52	110011	−0.3540 − 1.2263i
9	001000	−0.5310 + 0.3066i	31	011110	0.8850 + 0.3066i	53	110100	−0.3540 + 1.2263i
10	001001	0.5310 − 0.3066i	32	011111	−0.5310 − 0.9197i	54	110101	0.3540 − 1.2263i
11	001010	0.5310 + 0.3066i	33	100000	−0.5310 + 0.9197i	55	110110	0.3540 + 1.2263i
12	001011	0 − 0.6131i	34	100001	0.5310 − 0.9197i	56	110111	1.2390 − 0.9197i
13	001100	0 + 0.6131i	35	100010	0.5310 + 0.9197i	57	111000	1.4160 − 0.6131i
14	001101	−0.7080 + 0i	36	100011	−1.0620 + 0i	58	111001	0.7080 − 1.2263i
15	001110	0.7080 + 0i	37	100100	1.0620 + 0i	59	111010	0.7080 + 1.2263i
16	001111	−0.3540 − 0.6131i	38	100101	0 − 1.2263i	60	111011	1.5930 − 0.3066i
17	010000	−0.3540 + 0.6131i	39	100110	0 + 1.2263i	61	111100	1.4160 + 0i
18	010001	0.3540 − 0.6131i	40	100111	−1.0620 − 0.6131i	62	111101	1.5930 + 0.3066i
19	010010	0.3540 + 0.6131i	41	101000	−1.0620 + 0.6131i	63	111110	1.4160 + 0.6131i
20	010011	−0.1770 − 0.9197i	42	101001	1.0620 − 0.6131i	64	111111	1.2390 + 0.9197i
21	010100	−0.1770 + 0.9197i	43	101010	1.0620 + 0.6131i
22	010101	0.1770 − 0.9197i	44	101011	−1.2390 − 0.3066i

. Based on the QG coding rules, the sum of the Hamming distances between a selected constellation point and its neighbors is as small as possible. QG coding is a reliable coding method that reduces the misjudgment rate between constellation points. The bits under the QG coding rules and the normalized coordinates of corresponding GS-64QAM constellation points are illustrated in Table 1.

2.3. Generation of the HS-64QAM Signal

PS can achieve flexibility in terms of entropy and transmission performance by designing the probability distribution of the constellation. To further improve the constellation shaping gain, PS is optimized on top of GS to achieve a superposition of PS and GS dominance.

In the GS-64QAM constellation as shown in Figure 2b, all constellation points are divided into $M_m$($M_m=6$) subsets (layers) according to concentric hexagonal layers centered at the origin, as illustrated in Figure 3. The number of points within the same subset (on the same layer) is $H_m$. In addition, constellation points within the same subset are assigned the same probability. The designation of a probability distribution is completed by directly adjusting the probability of each subset (each layer). Then, constant composition distribution matcher (CCDM) converts uniform bit streams to symbol sequences with expected probability distribution, and HS-64QAM signals are generated.

The probability of constellation points $x_i$ on the i-th layer can be described as:

$$P_{CS}(x_i)=\frac{\exp (-\upsilon P_{mean}^i)}{{\displaystyle \sum _{k=1}^{M_m}{H}_k\exp (-\upsilon P_{mean}^k)}},\upsilon \ge 0$$

(6)

where $\upsilon$ represents the shaping parameter and determines the degree of PS, $H_k$ denotes the number of constellation points on the k-th layer, $P_{mean}^i$ and $P_{mean}^k$ indicate the average power of the i-th hexagonal layer and the k-th hexagonal layer, respectively.

$P_{mean}^i$ can be given by the ratio of the power of all points on the i-th layer to the total number $H_i$ of constellation points on the i-th layer, which can be given by:

$$P_{mean}^i=\frac{1}{H_i}{\displaystyle \sum _j^{H_i}(\mathrm{Re}{(P{o}_{i,j})}^2+\mathrm{Im}{(P{o}_{i,j})}^2)},i=1,2,\cdots ,M_m$$

(7)

where $P{o}_{i,j}$ represents the j-th constellation points on the i-th layer, $\mathrm{Re}(\cdot )$ and $\mathrm{Im}(\cdot )$ represent the real and imaginary parts of a complex constellation point $P{o}_{i,j}$, respectively, and the corresponding entropy H can be calculated by:

$$H=-{\displaystyle \sum _{i=1}^{M_m}{P}_{CS}(x_i)}\cdot {\log }_2{P}_{CS}(x_i)(\mathrm{bits}/\mathrm{symbol}).$$

(8)

The change of the value $\upsilon$ can achieve the adjustment of the signal entropy and further realize the comprehensive consideration of the transmission performance and signal entropy $H$ according to practical requirements of the system. By combining Equations (6) and (7), the optimized probability distribution of the GS-64QAM constellation is obtained. The symbol sequence of the expected probability distribution is further obtained by CCDM, and then the HS-64QAM signal is generated.

3. Experimental Setup and Results

3.1. Experimental Setup

To verify the effectiveness of the proposed scheme, an experiment was conducted on a 120 Gbit/s transmission over a 5 × 75 km single mode fiber (SMF) with the proposed HS-64QAM in a coherent optical fiber communication system, as illustrated in Figure 4. The transmission performance of HS-64QAM is compared with that of uniform S-64QAM and uniform GS-64QAM.

At the transmitter end, the pseudo-random bit sequence (PRBS) was generated and used to generate different 64QAM signal sequences (HS-64QAM with H = 5.7 bits/symbol, and uniform S-64QAM, uniform GS-64QAM). The shaping factor $v$ of HS-64QAM was 0.1308. To ensure a net rate of 120 Gbit/s, the baud rate multiplied by the entropy $H$ should be the same. The above three 64QAM signals should have a baud rate of 21.06 GBaud, 20 GBaud, and 20 GBaud, respectively. The above sequences were successively imported into the arbitrary waveform generator (AWG) to complete the digital-to-analog conversion, where the sampling rate of the AWG was set to 50 GSa/s, and the peak value of the AWG output voltage was 200 mVpp. Analog signals were then amplified by two electrical amplifiers (EA) (SHF809) and fed into the IQ modulator (FTM7960). An external cavity laser (ECL) with 100 kHz linewidth and 15 dBm signal output power was used to generate the continuous optical wave. Optical waves at 1550 nm were loaded into an IQ modulator for electrical-to-optical conversion. Finally, an erbium-doped fiber amplifier (EDFA) with a noise figure of 5.5 dB and a variable optical attenuator (VOA) were used to adjust the TOP transmitted into the fiber link.

The transmission link was a 375 km SMF consisting of five spans G.652 fiber, and the length of each span was 75 km. Moreover, each span was equipped with an 18 dB gain EDFA to compensate for the power loss due to signal attenuation over the span.

At the receiver end, an ECL with a linewidth of 100 kHz was utilized as the local oscillator (LO). The received signal was detected by the integrated coherent receiver (ICR) to achieve the phase diversity of the signal. After that, ICR and LO achieved optical-to-electrical conversion. The digital phosphor oscilloscope (DPO72004C) with a sampling rate of 100 GSa/s was used to implement analog-to-digital conversion. Then, the sampled data sequences were processed by offline digital signal processing (DSP). At the same time, an optical spectrum analyzer (OSA) was employed to monitor the OSNR of received optical signals.

Figure 5a shows the DSP flow chart for uniform received signals (uniform S-64QAM and uniform GS-64QAM). The processing flow includes lowpass filtering, amplitude normalization, Gram–Schmidt orthogonalizing process (GSOP), chromatic dispersion (CD)/nonlinear compensation, clock recovery using Gardner algorithm, adaptive equalization using constant modulus algorithm (CMA), frequency offset estimation (FOE), carrier phase estimation (CPE) based on blind phase search (BPS), synchronization, 64QAM demapping, and bit error rate (BER) calculation. Figure 5b shows the DSP flow chart for received HS-64QAM signals, which consists of lowpass filtering, normalization, GSOP, CD/nonlinear compensation, clock recovery using Gardner, adaptive equalization using CMA, FOE, CPE, BPS, synchronization, 64QAM demapping, de-distribution matcher (de-DM), and BER calculation.

3.2. Results and Analysis

Figure 6 shows the BERs of 64QAM (uniform S-64QAM, uniform GS-64QAM, and HS-64QAM with H = 5.7 bits/symbol) versus OSNR in the 120 Gbit/s coherent optical communication system for the BTB situation. A VOA and an EDFA were applied to adjust the OSNR for the BER measurement. The recovered constellations for the three 64QAMs at 20 dB OSNR are plotted in the insets of Figure 6a–c. It is shown that constellations of uniform GS-64QAM and uniform S-64QAM are not conducive to the decision compared to the proposed HS-64QAM with the same transmission power. When the OSNR is 20 dB, the BER in Figure 6a–c is 0.0237, 0.0158, and 0.0029, respectively. For all the three modulation formats, the BER decreases as the OSNR gradually increases. It can be clearly seen that HS-64QAM outperforms the uniform GS-64QAM and the uniform S-64QAM in the measured OSNR range of 16 dB to 21.5 dB for the BTB situation. Furthermore, the soft-decision forward error correction (SD-FEC) threshold of $2\times {10}^{-2}$ and hard-decision forward error correction (HD-FEC) threshold of $3.8\times {10}^{-3}$ are commonly used to evaluate BER performance for optical communication systems. Therefore, these two values are applied to measure the performance in the experimental verification.

As shown in Figure 6, with an OSNR of 19.6 dB, the BER of uniform GS-64QAM can reach below the SD-FEC threshold, while the BER performance of HS-64QAM and uniform S-64QAM is 0.0043 and 0.03, respectively. It can be clearly seen that a considerably better OSNR sensitivity can be obtained by introducing the proposed scheme. Taking the SD-FEC threshold as a reference, the HS-64QAM signal obtains an OSNR gain of 1.9 dB and 2.7 dB, respectively, compared to the uniform GS-64QAM signal and the uniform S-64QAM signal. When BER = 10⁻², HS-64QAM obtains an OSNR gain of 2 dB and 2.8 dB over uniform GS-64QAM and uniform S-64QAM, respectively. When the OSNR is greater than 19.7 dB, the BER of HS-64QAM reaches below the HD-FEC threshold. Utilizing the proposed scheme can enhance the system performance for 64QAM. This is because the proposed HS-64QAM scheme realizes the combination of GS and PS dominance, which further improves the constellation shaping gain.

Figure 7 shows the BER curves of 64QAM (uniform S-64QAM, uniform GS-64QAM, and HS-64QAM with H = 5.7 bits/symbol) versus OSNR in the 120 Gbit/s coherent optical communication system for the 375 km transmission situation. The recovered constellations for the three 64QAMs at 23 dB OSNR are plotted in the insets of Figure 7a–c. When the OSNR is 23 dB, the BERs of the proposed HS-64QAM with H = 5.7 bits/symbol, uniform GS-64QAM, and uniform S-64QAM are 0.0024, 0.019, and 0.032. It is also shown that the recovered constellation for the proposed HS-64QAM at the same transmission power is more favorable to be distinguished compared to uniform S-64QAM and uniform GS-64QAM.

As shown in Figure 7, the BER of the three 64QAMs decreases as the OSNR gradually increases. It can be clearly seen that the proposed HS-64QAM scheme is superior to the uniform GS-64QAM and the uniform S-64QAM throughout the entire OSNR range from 17 dB to 30 dB for the 375 km transmission situation. Since the uniform GS-64QAM constellation is optimized, the min-ED of the uniform GS-64QAM constellation is the largest. Probabilistically shaped on the basis of uniform GS-64QAM, the proposed HS-64QAM can effectively reduce the transmission power. Compared with uniform GS-64QAM and uniform S-64QAM, HS-64QAM obtains 4.1 dB and 7.6 dB OSNR gains, respectively, at the BER of the SD-FEC threshold. When BER = 10⁻², HS-64QAM obtains a 5.6 dB OSNR gain over uniform GS-64QAM. In addition, when the OSNR is 21.9 dB, the BER of HS-64QAM reaches the HD-FEC threshold, while the best BER performance of uniform GS-64QAM and uniform S-64QAM is 0.026 and 0.04, respectively. Experimental results in both BTB and 375 km transmission scenarios indicate that the proposed HS-64QAM scheme is more reliable than the uniform GS-64QAM and the uniform S-64QAM when adapted to transmission systems with the same transmission power.

Figure 8 shows the BER curves of 64QAM (uniform S-64QAM, uniform GS-64QAM, and HS-64QAM with H = 5.7 bits/symbol) versus TOP in the 120 Gbit/s coherent optical communication system for the 375 km transmission situation. The TOP transmitted into the SMF fiber link was managed by a VOA and an EDFA to measure BER curves. The recovered constellations for the three 64QAMs at −6 dBm TOP are plotted in the insets of Figure 8a–c. When the TOP into the SMF is −6 dBm, the BERs of the proposed HS-64QAM with H = 5.7 bits/symbol, uniform GS-64QAM, and uniform S-64QAM are 0.0115, 0.028, and 0.036. It is worth observing from the diagrams of recovered constellations that the points in the corners of uniform S-64QAM are blended together and hard to distinguish. However, the outermost constellation points of the proposed scheme have higher discrimination, which again indicates the superiority of the proposed scheme. BER decreases as TOP increases in the measured TOP range from −11 dBm to −4 dBm. The reason is that the OSNR of the received signal is positively correlated with TOP transmitted into the SMF, and the amplifier spontaneous emission noise is the main factor affecting BER over the entire range of TOP measurements.

As illustrated in Figure 8, when achieving the same BER performance, the TOP required for HS-64QAM is optimal, followed by uniform GS-64QAM and then uniform S-64QAM. Taking the $4\times {10}^{-2}$ BER threshold as a reference, the TOP performance of the HS-64QAM outperforms uniform GS-64QAM and uniform S-64QAM by 2.11 dBm and 2.71 dBm, respectively. When TOP is −8.8 dBm, BER of HS-64QAM reaches the SD-FEC threshold, while the best BER performance is 0.043 and 0.048 for uniform GS-64QAM and uniform S-64QAM, respectively. Furthermore, the BER of HS-64QAM reaches the ${10}^{-2}$ threshold at TOP = −4.9 dBm. This is due to the superiority of the proposed HS-64QAM scheme, which reduces signal transmission power and confirms high-capacity transmission performances.

4. Discussion

The reason why the proposed HS-64QAM scheme can achieve a high performance is that the proposed scheme combines the advantages of GS and PS to achieve a higher constellation shaping gain. By utilizing the hexagonal lattice structure, which is the densest pattern in the complex plane, denser constellation points are achieved within a fixed min-ED constellation plane for 64QAM. This densification enhances the error tolerance compared to the traditional S-64QAM. Moreover, the transmission link is limited in distance due to the non-desired tolerance to nonlinearity in the traditional 64QAM scheme. The proposed HS-64QAM scheme optimizes the coordinates and probabilistic distribution of constellation points. Compared with the traditional S-64QAM, GS-64QAM achieves a certain CFM gain. On the basis of GS, HS-64QAM realizes probabilistic optimization and the average-signal-power reduction. In this way, not only is the resistance to nonlinearity improved, but also constellation shaping gain is enhanced. Consequently, we can conclude that the proposed scheme exhibits superior transmission performance compared to S-64QAM.

As can be seen from Figure 6 and Figure 7, the proposed HS-64QAM scheme consistently achieves superior transmission performance as the number of OSNRs increases in both BTB and 375 km transmission scenarios. In addition, it can also be seen from Figure 8 that the BER performance of the proposed scheme can satisfy the SD-FEC threshold with TOP above −8.8 dBm. These results highlight the significant potential of the proposed HS-64QAM scheme in enhancing the performance of high-capacity coherent optical communication systems. With the in-depth development of future communication technologies such as the fifth/sixth generation, the explosion of network traffic will require further improvement of transmission capacity. In this case, the proposed HS-64QAM scheme has been demonstrated to be superior and feasible. Therefore, the proposed scheme holds substantial application value in long-haul, high-sensitivity, high-speed, and even ultra-high-speed optical communications.

To address the diverse requirements of practical application scenarios, the proposed HS-64QAM scheme can be extended to other modulation orders. More schemes such as the HS-16QAM scheme, the HS-32QAM scheme, and the HS-128QAM scheme will be explored. In order to verify the effectiveness of the proposed schemes, we will perform verifications based on actual scenarios in subsequent studies. At the same time, different application scenarios will be considered to ensure the reliability of the above schemes. Moreover, machine learning technology has gained widespread adoption in the field of optical communications, presenting extensive opportunities for achieving higher-speed and more flexible data transmission. The utilization of data training sets allows for advanced optimization and enhanced performance in optical communication systems. By combining the proposed HS-64QAM scheme with machine learning technology, the intelligent and adaptive constellation shaping can be realized at the transmitter side. The approach will involve adaptively adjusting the geometric and probabilistic distributions of the constellation based on channel characteristics. Therefore, combining the proposed hybrid constellation shaping scheme with machine learning technology is also one of the promising directions for future research.

5. Conclusions

A novel HS-64QAM based on hexagonal lattice of a constellation subset is proposed to enhance the nonlinear tolerance of higher-order modulated signals and further improve the constellation shaping gain. Firstly, the initial quantitative characterization of the constellation is performed on the basis of the hexagonal lattice structure. Then, the position distribution of corresponding constellation points is obtained by maximizing the CFM as the objective function to generate GS-64QAM. Finally, HS-64QAM is obtained by dividing all constellation points into multiple subsets according to concentric hexagonal layers, where points within the same subset are endowed with the same probability. In addition, the experimental verification of the proposed HS-64QAM scheme was demonstrated in a 120 Gbit/s multi-span coherent optical communication system. Experimental results indicate that HS-64QAM achieves an OSNR gain of 1.9 dB and 4.1 dB compared to uniform GS-64QAM in the BTB and transmission scenarios, respectively, taking the SD-FEC threshold as a reference. Moreover, HS-64QAM achieves an OSNR gain of 2.7 dB and 7.6 dB compared with uniform S-64QAM in BTB and transmission scenarios, respectively. Therefore, the proposed scheme has demonstrated its superiority and can be used to realize high-speed optical transmission over longer distance and with higher sensitivity, providing a viable means for the future development of ultra-high-speed optical communication networks.