Did You Just Talk AI Into Agreeing With You?

This summer we put our AI connection to Rent Manager in front of real beta customers. Before that, we'd run it through more than 30 controlled test questions ourselves: clean, unambiguous, one right answer apiece. It got every one of them right.

Then a customer asked about occupancy, and I watched the whole thing play out live.

• Question 1: "What was our occupancy last month?"
• Answer 1: A number, based on the AI's own reading of "occupancy." It included units in make-ready.
• Reaction: "That seems low."
• Question 2: "What was occupancy last month, excluding vacant-not-rentable units?"
• Answer 2: A higher number. This one felt right, so that's where the conversation stopped.

Both answers were correct. The AI did exactly what it was asked, twice. Nobody set out to game anything. The second question just got reverse-engineered from the number the customer wanted to see, and once the number matched, nobody asked why the definition underneath it had changed.

That's the risk we didn't fully appreciate until we saw it happen. Not the model getting the math wrong. A person, with the best of intentions, steering the question until it agreed with them.

Why This Isn't a Knock on Anyone, Including the Model

We've watched this happen more than once now, and it's worth saying plainly what it is and isn't. It isn't the AI being unreliable: the underlying answer was accurate both times, for the question actually asked. It isn't the customer doing anything wrong either. Rewording a question until the answer feels right is a completely ordinary human instinct. All of us do it the moment a number surprises us.

The problem is something else. An open-ended connection to live data gives that instinct room to run. A one-off question quietly turns into a repeated experiment, and the experiment stops the moment the numbers look expected instead of the moment they're actually understood.

The same pattern shows up in collections. "What's our collections rate?" comes back lower than the number in last month's board deck. Rather than asking why the two don't match, the next question adds a condition: "not counting tenants who've already moved out." The number lines up. Both answers were correct for the question asked. The deck's definition and the AI's definition of "collections rate" were never actually reconciled; the second prompt just found the wording that made them look the same.

If this were an accuracy problem, the fix would be a better model. It isn't an accuracy problem, so that's not the fix. The fix is deciding, ahead of time, which questions are allowed to stay open to interpretation and which ones aren't.

Govern the Answer That's About to Become a Decision

If you're the one who has to stand behind a number in a board deck or a lender report, here's a scene you'll recognize even without AI involved: it's two hours before a lender call, the controller has pulled the same collections report three times, and the third version is the one that goes in the deck because it's the one that "looks right." That's not new. What's new is how fast an AI conversation gets you there, and how easy it is to lose track of the rewrites once the chat has scrolled past them.

We've had a name for the fix since long before AI showed up: the Metrics Specification document. It's the list of judgment calls a portfolio has already made, written down once so nobody has to re-litigate them in a chat window:

• Which unit statuses count toward occupancy
• Which GL accounts belong in CapEx
• How a move-in reconciles against a signed lease
• Whether a moved-out tenant still counts in a collections metric

Decide those once, put them in the spec, and the ambiguity that makes a question reword-able disappears before the conversation starts. The press has started calling this kind of thing "governance." We've adopted that word too, but the actual work behind it is just the Metrics Specification, kept current.

That's the difference between the two lanes we build into Asset Management Studio's approach to AI and your portfolio data: a fast, live connection for exploring, and a governed foundation, the Metrics Spec and everything behind it, for anything about to leave your desk.

What to Check on Your Own Rent Manager Data

If you're testing an AI connection to your Rent Manager data yourself, there's a simple way to catch this before it costs you anything. Ask the same financial question two different ways and see if you get two different "right" answers:

• "Collections rate," phrased plainly, versus phrased with your own exclusions built in
• "Occupancy," with and without units in make-ready
• "CapEx spend," with and without a GL account your team has always treated as a gray area

If the two phrasings disagree, you haven't found a bug. You've found exactly the kind of ambiguity that belongs in a Metrics Spec, not left to whichever way someone happens to phrase the question that day.

None of this is an argument for asking fewer questions. Most of what your team asks an AI about your portfolio is genuinely one-off: a quick check before a call, a hunch worth ten seconds to confirm. Reword one of those as many times as you want. The moment to stop and check the definition is the moment a number stops being a curiosity and starts being something you'll repeat: a monthly review, a lender quote, a decision.

The Question That Actually Matters

An AI that can answer questions about your portfolio isn't the hard part anymore. Knowing which answers you can stand behind is. Before you repeat a number anywhere, ask yourself one thing: did you ask that question once, or did you ask it until it agreed with you?