On the "Rise" of "AI"

So here we are, being bombarded with article after article of LLMs being able to solve difficult math problems. So it's pretty clear that the sky is falling, right?

I've had some questions and opinions on these LLMs in math and want to make this post so pick the brains of the users here, as I'm really not sure where the hype ends and the miracles/bullshit begins.

Let me explain my biases and presuppositions really quick so we're on even footing. I'm skeptical of the coming of AGI and ASI (indeed, if both are possible, why isn't ChatGPT or Claude or what have you already AGI?). I have trouble imagining a future where humans don't still control things like we do now. I have no idea why some people seem to think we'll just hand it over to AI. If you want to address these presuppositions and how wrong you think they are, go ahead.

  1. Aren't these models still fundamentally next-word predictors? I see people here all the time saying they aren't but how so? I'm not trying to undermine how big these models are.
  2. How are these problems being solved? Are they being solved in completely novel (i.e., unthought of before) ways, or are there methods from one area of math being applied to a different area?
  3. Assume that LLMs are this good at math. How will humans not be needed to at least understand what the digital God is outputting? Terrance Tao needed to verify that the proof of Erdos problem 1196 was correct, didn't he?
  4. If the answer to 3 is something along the lines of "Eventually the AI will get so good that it will no longer need a human", how? How will that happen eventually, and why can't the AI do it now?
  5. Why does any of this seem to make people think that the end of mathematics is near? Why wouldn't this just allow us to do more?
  6. A common sentiment here is that eventually AI will get so advanced that the math it outputs will be incomprehensible to us. How exactly does that matter? Why would math incomprehensible to us be useful to us? Wouldn't we spend time learning the math required to understand the incomprehensible math?

Rip me apart and make me reconsider my life choices in pursuing math.

Author: Dandon314