reevebot — a Minecraft AI with a body and eyes

A Minecraft AI that runs on a modded server and can see the world.

It was built for a Forge 1.20.1 server with thirty-odd mods, where the existing options do not run. Projects built on mineflayer are protocol-level bots posing as clients; their docs list vanilla versions and say nothing about modded servers, and in practice the bot is kicked at the Forge handshake. The alternative is writing a mod, and most of those are Fabric-side. Fabric and Forge are incompatible loaders.

Underneath both routes sits an assumption worth naming: the model receives a sentence — "stone block ahead, oak tree left, zombie behind" — and works from a list of coordinates it has to imagine.

Two halves, both in this repo:

• mod/ — a Forge 1.20.1 server mod that spawns a genuine ServerPlayer. It appears in the player list. Mobs target it. It takes fall damage, drowns, starves, and dies. It walks on forward input with a jump on collision. There is no teleporting.
• mcp/ — an MCP server exposing 124 tools to any MCP-capable model (Claude Desktop, Claude Code, anything that speaks the protocol). One tool call sets a goal; the mod spends the next few thousand ticks pursuing it.

What makes it different

1. It can see

mc_see returns an actual image — a first-person render of what the bot is looking at, produced entirely server-side:

• a ray is cast per pixel and walked through the voxel grid (DDA)
• colours come from BlockState.getMapColor(), so modded blocks render for free with no colour table to maintain
• creatures are drawn from vanilla rigs with vanilla textures — a zombie renders as a zombie, a villager as a villager, and players wear their real skin PNG, sampled through the standard UV unwrap
• face shading, distance fog, sky gradient, crosshair

320×180 takes about 0.4 s on a CPU. No GPU, no client, no X server. The model receives it as an MCP image block and looks at it the way you would.

There is also mc_map (top-down) and mc_look_around (labelled block slices with a legend), for when coordinates beat pixels.

2. It has ground truth when things go wrong

mc_world_dump returns every block of the 2×2 chunk square around the bot — id, state properties, exact coordinates, losslessly. Naive per-block output would be tens of megabytes; a palette plus run-length columns brings four whole chunks to about 19 KB, small enough to hand a model in one go.

CHUNKS x=[80..111] z=[48..79] y=[36..76] dim=overworld bot=(96,63,64)
PALETTE
0=air
1=stone
2=oak_stairs[facing=north,half=bottom]
COLUMNS
96 64: 1*24 0*17

There is also mc_bot_step: move exactly one block, and when it refuses it names the block in the way — blocked: stone wall at 98, 62, 64. Between the two, a model can work its way out of a stuck automation on facts.

3. The unit of instruction is a goal

Beating Minecraft takes tens of thousands of actions. No model can afford one tool call per swing, so the long jobs live in Java and report back:

mc_bot_mine_for  ore=diamond count=24   → branch-mines at y=-58, follows veins,
                                          torches behind itself, avoids lava,
                                          banks junk in a chest it crafts, swaps
                                          in a fresh pickaxe, comes home
mc_bot_travel    x=2000 z=2000          → paths in legs, bores through hills,
                                          swims, shelters underground at night
mc_bot_hunt      type=blaze count=6     → bow at range, sword up close
mc_bot_brew      potion=fire_resistance → drives a real brewing stand, waits out
                                          the real 400-tick cycles

Everything is honest: crafting checks real recipes and really consumes materials, the furnace is a real furnace, enchanting pays 3 lapis and 3 levels for a real level-30 roll. Nothing is conjured.

The 124 tools, by family

family	examples
senses	mc_see mc_map mc_look_around mc_world_dump mc_block_at mc_bot_find_block mc_bot_find_entity
body	mc_bot_spawn mc_bot_goto mc_bot_step mc_bot_travel mc_bot_dimension mc_bot_follow mc_bot_mount
hands	mc_bot_mine mc_bot_gather mc_bot_place mc_bot_use_at mc_bot_craft mc_bot_smelt mc_bot_equip
containers	mc_bot_container mc_bot_take mc_bot_put mc_bot_deposit mc_bot_withdraw mc_bot_stash
survival	mc_bot_eat mc_bot_sleep mc_bot_farm mc_bot_fish mc_bot_breed mc_bot_shear mc_bot_milk
building	mc_bot_bridge mc_bot_tower mc_bot_tunnel mc_bot_dig_down mc_bot_light_up mc_bot_clear_area
combat	mc_bot_attack mc_bot_shoot mc_bot_hunt mc_bot_kill_hostiles mc_bot_guard mc_bot_patrol
late game	mc_bot_portal mc_bot_barter mc_bot_portal_room mc_bot_fill_portal
magic	mc_bot_enchant mc_bot_brew mc_bot_anvil
social	mc_say mc_whisper mc_villager_trades mc_bot_trade mc_bot_give_item
server	mc_watch mc_players mc_time mc_weather mc_find_structure mc_run

mc_run refuses stop, op, ban, whitelist, kick and reload; combat tools never target players.

Getting it running

Requirements

This runs on a VPS, or any always-on server box you administer. The mod and the MCP server must sit on the same machine and share a directory for renders and world dumps, and the game server has to stay up for the bot to live in. A laptop running a LAN world will not do.

• a Forge 1.20.1 dedicated server with RCON enabled
• Java 17, Python 3.10+
• shell access to that machine

# 1. build the mod
cd mod && ./gradlew build          # jar lands in build/libs/
cp build/libs/reevebot-1.0.0.jar /path/to/server/mods/

# 2. server.properties
#    enable-rcon=true
#    rcon.password=<something>
#    rcon.port=25575

# 3. point the MCP server at it
export RCON_PW=<the same password>
export REEVEBOT_OUT=/path/both/halves/can/write   # renders + world dumps land here

# 4. generate and run
cd mcp && python3 gen_tools.py && python3 helpers.py && python3 verify.py
python3 mc_mcp.py                  # stdio MCP; or mc_http.py for a remote connector

Then in your MCP client:

{ "mcpServers": { "minecraft": { "command": "python3", "args": ["/path/to/mcp/mc_mcp.py"] } } }

Say hello with mc_bot_spawn, then mc_bot_status, then mc_see.

The tool table is generated

Don't hand-edit the TOOLS block in mc_mcp.py. Tools are one row each in gen_tools.py; helpers.py injects the hand-written query helpers, and verify.py statically checks that every tool has a dispatch entry and no name is duplicated. Re-run all three after any change.

Four things that cost days to work out

Anyone building a server-side fake player will hit these:

1. placeNewPlayer loads player data before creating the packet listener, and permission mods message the player during that load → NPE on a null connection. Construct ServerGamePacketListenerImpl(server, conn, bot) yourself, first.
2. Forge's NetworkFilters needs a netty pipeline. Override Connection.channel() to return an EmbeddedChannel, and add a handler literally named packet_handler — Forge does addBefore on that name.
3. ConnectionType.forVersionFlag does not null-check. Set the netty attribute AttributeKey.valueOf("fml:netversion") = "FML3". (Forge's own constant is package-private; the key string is lowercase with a colon, which is not what the field name suggests.)
4. A fake player has no client feeding it jump physics. The jump-and-place pillaring every tutorial teaches silently does nothing: the bot rises and lands with no block placed. Move it up a block outright, then fill the space it left.

Two more that only show up in play: canSeeSky is false forever under a tree, so "reached the surface" also has to accept the no-leaves heightmap; and a cross-dimension /tp does nothing to a bot, because the move is delivered by a respawn packet nobody receives — move the entity between levels directly.

Where this will hit a wall

The body and the senses here are stronger than in the projects below; the brain barely exists, and those projects have the opposite balance. docs/DESIGN-NOTES.md is an honest account of the four flaws that follow from that.

Prior art

mindcraft (Mineflayer, protocol-level bot) and Voyager (skill library, code as action) shaped the action vocabulary here. The stuck-watchdog and the branch-mining layout are adapted from mc_aiplayer (MIT). What this project adds is the body and the eyes: a real ServerPlayer on a modded server, and a renderer the model can look through.

Status — roughly 75% done

This is my first project of this kind. Expect unpredictable bugs, rough edges, and behaviour that has only ever been exercised on one server.

Runs daily on a private modded server. Verified live on that server: branch-mining to a full set of diamond armour; nether portal build and light; blaze hunting with a bow; villager trading; potion brewing; enchanting; piglin bartering; farming, breeding, shearing, milking; bridges, towers, tunnels; guard and patrol duty.

Licence

MIT. See LICENCE.