Your AI Agents Can't See Each Other's Rewards — This Trick Made Them Cooperate Anyway

Imagine a team of delivery robots sharing one warehouse. Each robot can see what the others are doing — who carried which box, who waited at the door — but none of them can see the others' scorecards. Robot A doesn't know whether Robot B just got rewarded or penalised. Can a team like that ever learn to cooperate, or is it doomed to selfish chaos?

For years, the standard answer in multi-agent reinforcement learning (MARL) has been: give every agent access to everyone else's rewards, and shape those rewards so that selfish behaviour hurts. It works in the lab. It collapses in the real world, because private reward signals are usually exactly that — private. A new paper from Mohamed Ayman Mohamed, Harshil Kotamreddy, and Marcos Menon Jose asks a beautifully human question: what if agents did what people do, and judged other agents' outcomes through their own eyes?

What the researchers did

The paper, "Self-Referenced Social Preferences: Cooperation without Observing Others Rewards" (arXiv:2610.07881, submitted 6 October 2026, revised 7 October 2026), introduces a method called self-referenced social preferences. Here is the idea in plain English:

Each agent first learns a model of its own reward — what good and bad outcomes look like from its own point of view. Then, when it watches another agent act, it runs that peer's observed transitions through its own reward model and asks: "If that had happened to me, how good would it have been?" That self-referenced assessment is then fed into standard social preferences — the same fairness- and benevolence-style shaping researchers already use — so the agent learns to care about outcomes it can never directly observe.

The team tested two ways of wiring those assessments into learning:

• Modify the learning reward: blend the self-referenced assessment of peers into the reward the agent trains on.
• Weight policy updates: use the assessments to weight how strongly each experience changes the agent's policy.

They evaluated the approach on three sequential social dilemmas, each standing in for a classic cooperation problem: Escape Room (volunteering — someone has to take the costly action so everyone escapes), Clean Up (public-good contribution — someone has to clean the river so apples grow), and Commons Harvest (resource restraint — don't eat the whole commons or nothing regrows).

The results are striking. Across all three environments, agents learned cooperative behaviour without ever observing others' rewards — including in settings where independent learners failed to cooperate at all. And in a twist the authors flag themselves, the self-referenced agents frequently achieved more equitable divisions of jointly produced returns than agents that could see the true rewards. The right integration point turned out to depend on the social preference: inequity aversion worked best in the reward together with a value look-ahead, while a purely benevolent preference benefited from policy-update weighting. Under partial observability — the realistic case, where agents can't even see everything — the policy-update approach kept supporting cooperation.

Why it matters

This matters because almost every real multi-agent deployment is partially observed and privacy-constrained. Warehouse robots run by different vendors, traffic controllers for competing fleets, energy agents balancing a shared grid, customer-service bots handing off to one another — none of these can or should open up their internal reward functions to their peers. Until now, that has been treated as a hard blocker for cooperative MARL: no peer rewards, no social preferences, no cooperation.

Self-referenced preferences remove that blocker without new infrastructure. An agent needs only what it already has — its own reward model and observations of peer behaviour — to start acting prosocially. That is a much cheaper deployment story than building a trusted reward-sharing layer between agents that may belong to different owners. It also reframes a design question every agent builder faces: cooperation doesn't require omniscience, it requires a reasonable way to imagine the other side's outcome.

There is a caution worth keeping, too. The paper's evidence comes from three benchmark dilemmas, not production fleets. "More equitable than true-reward agents" is a result about these environments, not a guarantee for yours.

How YOU can use it

You don't need to retrain a research lab's MARL stack to steal the core idea. If you build multi-agent systems — LangGraph crews, AutoGen teams, game agents, robot fleets — here is a practical recipe inspired by the paper:

Step 1: Give each agent its own scorecard, in writing

Before any cooperation logic, define each agent's reward model explicitly: what counts as a good outcome for that agent, in numbers or in a rubric. For LLM-agent teams, this can be a scoring prompt ("Rate this trajectory 0–10 for task completion, cost, and side effects"). You cannot do self-referenced assessment until "my perspective" is a real, runnable function.

Step 2: Score peers through your own lens

After each shared episode, have each agent score every peer's observed trajectory using its own scorecard — not the peer's. In code, the shape is simple:

# self-referenced assessment, in pseudocode
for peer in teammates:
    peer_outcome_from_my_view = my_reward_model(peer.observed_transitions)
    social_signal[peer] = peer_outcome_from_my_view
# then feed social_signal into your existing shaping:
#  option A: add it to the learning reward
#  option B: use it to weight policy / prompt updates

For LLM-agent builders without RL training, the equivalent is a reflection step: after a run, each agent writes a short assessment of each teammate's contribution as if it had done that work itself, and those assessments feed the next run's planning prompt.

Step 3: Pick your integration point deliberately

Follow the paper's guidance: if your goal is fairness (inequity aversion — nobody hogs the reward), blend the assessments into the reward signal and pair them with a value look-ahead, so agents consider future consequences, not just the last step. If your goal is pure helpfulness (benevolence — maximise the team's good), weight your updates by the assessments instead. Test each and measure which one your environment rewards.

Step 4: Test on a dilemma, not a demo

Copy the paper's evaluation discipline: build a small test where cooperation is actually costly. A shared-budget task (agents draw from one token budget), a shared-cleanup task (someone must do the unglamorous work), and a restraint task (agents must not exhaust a shared resource) will tell you more about your team than ten happy-path demos. Measure two things: does cooperation emerge, and is the division of returns equitable? If independent agents fail your dilemma but self-referenced ones pass it, you have replicated the paper's headline result in your own system.

Key takeaways

• Cooperative multi-agent learning does not require access to peers' private reward signals.
• Agents can assess peers' outcomes through their own learned reward model — self-referenced social preferences.
• The method worked across volunteering, public-good, and resource-restraint dilemmas, including where independent learners failed.
• Integration point matters: inequity aversion suits reward shaping with a value look-ahead; benevolence suits policy-update weighting.
• Under partial observability, the policy-update approach kept cooperation alive — the realistic setting for real deployments.

Read the original paper: Mohamed Ayman Mohamed, Harshil Kotamreddy, and Marcos Menon Jose, "Self-Referenced Social Preferences: Cooperation without Observing Others Rewards", arXiv:2610.07881 — https://arxiv.org/abs/2610.07881