A system of two non-linear matrix equations

Let $A,B\in M_n(\mathbb{C})$ s.t. $A^3+B^3=0,AB-B^2A^2=I_n$.

According to

Prove $BA-A^2B^2=I$

one can deduce that $A,B$ are invertible and $BA-A^2B^2=I_n$.

Now, when $n=2$, one has: (*) $A^6=I_2$ and $AB-BA$ is nilpotent.

I prove (*) by a PC computation using the Grobner basis theory.

Question 1. Prove (*) with hand.

Question 2. Does (*) remain true when $n>2$ ?

Here is a proof when $n=2$. The case $n>2$ seems to be much more difficult.

Case 1. $AB=BA$. Alors $A^6=I$ in $M_n(\mathbb{C})$. Indeed $(AB)^2-(AB)+I=0$ implies that $AB$ is similar to $diag(-jI_p,-j^2I_q)$ where $j=\exp(2i\pi/3)$ and $p+q=n$; then $(AB)^3=A^3B^3=-I$ and $A^6=I$.

Case 2. $AB\not= BA$. Necessarily, the system $\{I,A,B\}$ is free. Let $A^2=aI+\alpha A,B^2=bI+\beta B$. Then $AB-BA=B^2A^2-A^2B^2=\alpha\beta(BA-AB)$ and $\alpha\beta=-1$. Moreover, $A^3+B^3=(\alpha a+\beta b)I+(a+\alpha^2)A+(b+\beta^2)B=0$ implies $\alpha a+\beta b=a+\alpha^2=b+\beta^2=0$.

We deduce that $\alpha^6=1,\beta=-1/\alpha$ and finally $A^3=-\alpha^3 I$.

Now we show that $trace((AB-BA)^2)=0$.

$BABA=B^3A^3+BA$ implies $(BA)^2-(BA)+I=0$; thus $BA$ is diagonalizable and $spectrum(BA)=\{-j,-j^2\}$ (because $AB\not= BA$); consequently, $spectrum(AB)=\{-j,-j^2\}$.

Finally $tr((AB-BA)^2)=2tr((AB)^2)-2tr(B^2A^2)=$

$2tr((AB)^2)-2tr(AB-I)=2(j+j^4)-2(-j-1-j^2-1)=0$.

Conclusion. $AB-BA$ is nilpotent and $A,B$ are simultaneously triangularizable.
{"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 1, "mathjax_display_tex": 0, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0, "katex": 0, "math-container": 0, "wp-katex-eq": 0, "align": 0, "equation": 0, "x-ck12": 0, "texerror": 0, "url": "https://math.stackexchange.com/questions/2451396/a-system-of-two-non-linear-matrix-equations", "fetch_time": 1576283463000000000, "content_mime_type": "text/html", "warc_filename": "crawl-data/CC-MAIN-2019-51/segments/1575540569332.20/warc/CC-MAIN-20191213230200-20191214014200-00424.warc.gz", "warc_record_offset": 424470397, "warc_record_length": 30155, "token_count": 637, "char_count": 1462, "score": 3.6875, "int_score": 4, "crawl": "CC-MAIN-2019-51", "snapshot_type": "latest", "language": "en", "language_score": 0.6121800541877747, "file_path": "/sailhome/yjruan/project/latent-lm/data/processed/finemath-4plus/finemath-4plus/train-00005-of-00064.parquet"}