Introduction to LLM Skirmish
LLM Skirmish is a benchmark that pits LLMs (Large Language Models) against each other in a series of 1v1 real-time strategy games. The goal of this benchmark is to test the in-context learning capabilities of LLMs. In-context learning refers to the ability of a model to learn from the context of a situation and adapt its behavior accordingly.
For example, in a game of chess, a model that can learn from the context of the game and adjust its strategy accordingly would be demonstrating in-context learning. Similarly, in LLM Skirmish, models are given the objective of eliminating their opponent’s spawn and must write a script to achieve this goal.
How LLM Skirmish Works
Each LLM Skirmish tournament consists of five rounds. In each round, each LLM is asked to write a script implementing its strategy. For all rounds after the first, each LLM can see the results of all its matches from the previous round and use that information to make changes to the script it submits for the next round.
This allows the models to learn from their mistakes and adapt their strategies over time. The tournament setup is designed to test the in-context learning capabilities of the models, as they must be able to adjust their strategies based on the results of previous rounds.
Agent Setup and Prompt Structure
LLM Skirmish was conducted using OpenCode, an open-source general-purpose agentic coding harness. Each LLM agent runs in an isolated Docker container with OpenCode providing the coding environment.
The orchestrator coordinates the tournament by sending prompts to each agent, which then uses OpenCode’s tools to write and submit their game scripts. The prompt structure consists of two files: OBJECTIVE.md and NEXT_ROUND.md.
OBJECTIVE.md contains the game rules, API documentation, and instructions for writing a game script. NEXT_ROUND.md contains instructions for reviewing match logs from the previous round, which is only provided for rounds 2-5.
Results and Analysis
The results of the LLM Skirmish tournament show that four of the five models evaluated have notable increases in average win rate between round 1 and round 5. This suggests that these models are able to learn from the context of the game and adapt their strategies accordingly.
However, the performance of Gemini 3 Pro presents an anomaly. Its round 1 average win rate was 70%, which is higher than all four other evaluated models. But its round 2-5 average win rate was 15%, which is lower than all four other evaluated models.
This suggests that Gemini 3 Pro may not be able to learn from the context of the game as effectively as the other models. Meanwhile, the other models are able to adjust their strategies and improve their performance over time.
Conclusion and Future Work
In conclusion, LLM Skirmish is a benchmark that tests the in-context learning capabilities of LLMs. The results of the tournament show that four of the five models evaluated are able to learn from the context of the game and adapt their strategies accordingly.
However, the performance of Gemini 3 Pro presents an anomaly that requires further investigation. Additionally, future work could involve running the tournament with different models and evaluating their performance.
For example, running the tournament with a larger number of models could provide more insight into the in-context learning capabilities of LLMs. Furthermore, evaluating the performance of different models could help to identify the strengths and weaknesses of each model.
Frequently Asked Questions
Q: What is LLM Skirmish?
LLM Skirmish is a benchmark that pits LLMs against each other in a series of 1v1 real-time strategy games. The goal of this benchmark is to test the in-context learning capabilities of LLMs.
Q: How does LLM Skirmish work?
Each LLM Skirmish tournament consists of five rounds. In each round, each LLM is asked to write a script implementing its strategy. For all rounds after the first, each LLM can see the results of all its matches from the previous round and use that information to make changes to the script it submits for the next round.
Q: What is the prompt structure for LLM Skirmish?
The prompt structure consists of two files: OBJECTIVE.md and NEXT_ROUND.md. OBJECTIVE.md contains the game rules, API documentation, and instructions for writing a game script. NEXT_ROUND.md contains instructions for reviewing match logs from the previous round.
Q: What are the results of the LLM Skirmish tournament?
The results of the LLM Skirmish tournament show that four of the five models evaluated have notable increases in average win rate between round 1 and round 5. However, the performance of Gemini 3 Pro presents an anomaly.
Q: What are the implications of the results?
The results suggest that four of the five models evaluated are able to learn from the context of the game and adapt their strategies accordingly. However, the performance of Gemini 3 Pro requires further investigation.
