On (1) I think anytime something is communicated that combines text and visual meaning. This is already happening to an extent with e.g. ascii diagrams in ChatGPT and multimodal prompts (see VIMA and the motivations/use