Live LLM Commentator

I connected a LLM to the game, using this package: https://github.com/undreamai/LLMUnity.

I used the qwen2.5-7B-instruct-1M-gguf checkpoint, with Q8 quantization. The inference speed on my local machine is 52.76 tok/sec, consuming less than 8GB of GPU memory.

For the character prompt, I used:

"You are a professional commentator for racing games. During the race, you will be input with a sequence of leaderboards, with time stamps until now, as well as a marble in focus. Each leaderboard is a ordered list of racer names.

Each marble has their own background stories:

...... (descriptions)

Important requirements: Only answer in ONE SHORT sentence of comment for the race, pay special attention to the marble in focus, while taking their characters into account. Pay special attention to potential overtaking attempts or events!!!"

I convert the leader board of marble names into a string, and input a sequence of strings to the LLM. The LLM output will be shown at the bottom center of the UI. Meanwhile, I used a TTS (text-to-speech) algorithm to read the comments aloud. After the reading finished, a new batch of strings will be fed to the LLM.

Note that one limitation is that the LLM only know about the leader board, without information about the state of each marble. Thus some racing events might be ignored by the LLM.

Meanwhile, the usage of LLM will affect the frame rate, even if I use coroutine. One possible solution is to use a smaller LLM. I tried qwen3 0.6B, 1.4B, and 4B, but none of them work as well as qwen2.5-7B-instruct.

"The God's Hand"

The player could change the direction of the gravity of the whole game world with key inputs. This could change the route off all marble racers, and could potentially change the game results.

One critical problem is how to actually decide the rotation coordinate? Since the user is shown a camera view following the marble, directly rotating at the world coordinate is counter intuitive.

My solution is handle the rotation input in the camera coordinate. I used the unity built in function to convert the rotation vector from camera coordinate to the world coordinate.

To visualize the input, I added a reference line at the top. As shown in the figure below, the white line will stay horizontal, and the yellow line indicated the user input.

The interaction could be made more fun with other inputs rather than keyboards. However, I met issues connecting the external controls in my unity game so I gave it up.