S2IGAN: Speech-to-Image Generation via Adversarial Learning
Table 1: Performance of S2IGAN compared to other methods. † means that the results are taken from the original paper. The best performance is shown in bold.
['Evaluation Metric', 'Input', 'CUB (Bird) mAP', 'CUB (Bird) FID', 'CUB (Bird) IS', 'Oxford-102 (Flower) mAP', 'Oxford-102 (Flower) FID', 'Oxford-102 (Flower) IS']
First, our method outperformed [li2020direct] on all evaluation metrics and datasets. Compared with the StackGAN-v2 that took our speech embedding as input, our S2IGAN also achieved higher mAP and lower FID on both datasets. These results indicate that our method is effective in generating high-quality and semantically consistent images on the basis of spoken descriptions. The comparison of our S2IGAN with three state-of-the-art T2IG methods show that the S2IGAN method is competitive, and thus establishes a solid new baseline for the S2IG task. Speech input is generally considered to be more difficult to deal with than text because of its high variability, its long duration, and the lack of pauses between words. Therefore, S2IG is more challenging than T2IG. However, the comparison of the performances of StackGAN-v2 on the S2IG and T2IG tasks shows that StackGAN-v2 generated better images using speech embeddings learned by our SEN. Moreover, the StackGAN-v2 based on our learned speech embeddings outperforms [li2020direct] on almost all evaluation metrics and datasets, except for the slightly higher FID on CUB dataset. These results confirm that our learned speech embeddings are competitive compared to text input and the speech embeddings in [li2020direct], showing the effectiveness of our SEN module.