ChatGPT is capable of writing (an incomplete) implementation of a biLSTM language model in pytorch (with general instructions on how to write the rest of it). We might be getting closer to bootstrapping language models.