A Monolingual Approach to Contextualized Word Embeddingsfor Mid-Resource Languages
Table 7: Average power draw (Watts), training times (in both hours and days), mean power consumption (KWh) and CO2 emissions (kg) for each ELMo model trained.
['Language', 'Power', 'Hours', 'Days', 'KWh⋅PUE', 'CO2e']
In our set-up we used two different machines, each one having 4 NVIDIA GeForce GTX 1080 Ti graphic cards and 128GB of RAM, the difference between the machines being that one uses a single Intel Xeon Gold 5118 processor, while the other uses two Intel Xeon E5-2630 v4 processors. One GeForce GTX 1080 For the DRAM we can use the work of Desrochers et al. Having this information, we can now use the formula proposed by Strubell et al. in order to compute the total power required to train one ELMo model: pt=1.58t(cpc+pr+gpg)1000 Where c and g are the number of CPUs and GPUs respectively, pc is the average power draw (in Watts) from all CPU sockets, pr the average power draw from all DRAM sockets, and pg the average power draw of a single GPU. We estimate the total power consumption by adding GPU, CPU and DRAM consumptions, and then multiplying by the Power Usage Effectiveness (PUE), which accounts for the additional energy required to support the compute infrastructure. We use a PUE coefficient of 1.58, the 2018 global average for data centers (Strubell et al. We can further estimate the CO2 emissions in kilograms of each single model by multiplying the total power consumption by the average CO2 emissions per kWh in France (where the models were trained). According to the RTE (Réseau de transport d’électricité / Electricity Transmission Network)