In short
- A demo on the vendor's data shows the happy path, and every product has one.
- Test with your own project, as an ordinary user, and ask to be refused.
- Score only what you saw done live on your own data.
The demo is on Thursday. The vendor shares a screen with a tidy programme. Every project has an owner, every milestone has a date, and the dashboard fills in as the presenter clicks. The room nods. The next vendor comes on Friday and shows the same thing in different colours.
By the third demo the scoring sheet has three columns of ticks, and nobody can say what separates them.
I have run that process from the buyer's side, and I now sit on the vendor's side of the table. A demo shows the happy path, and every product has one. What separates products is what they do when something goes wrong, and you only see that on your own data.
What goes wrong
A demo is built to pass. Three things about it hide the differences a buyer needs to see.
The data is clean. No evidence is missing and no number is disputed. A real programme is mostly exceptions, and the demo workspace has none.
The presenter is the administrator. Every button works for them. Nobody watches the system refuse a person who lacks the authority, so nobody learns whether it would.
Time only runs forward. The demo shows today's figures. Nobody asks what the record looked like last quarter, or who changed it since.
So the scoring sheet ends up counting features. Most products have a Gantt chart, a KPI page, a risk register and an AI assistant, and each one gets the tick. The sheet cannot tell a system of record from a well-drawn slide.
What good looks like
Change what the vendor is asked to show. Four moves do it, and none of them needs a budget.
Bring your own programme
Send each vendor the same small pack a week ahead: one real project with its schedule file, one KPI with a year of readings, and your delegation of authority. Ask them to load it. What they cannot load is your first finding.
Sit in an ordinary user's seat
Ask for the tests to be run as a project manager, with the administrator logged out. Most of the tests below end with the system saying no to someone. You want to watch it say no.
Ask about the past
Any system can show today. Ask it to show a figure as it stood when the committee approved it, and what has changed since. A programme is audited on its history.
Score only what you saw
Give a tick when the thing was done live, on your data, in the session. A slide scores nothing, and neither does the roadmap. Put the same seven tests to every vendor, so that the sheet compares like with like.
| The test | What to ask for | What a good answer shows |
|---|---|---|
| 1. A green that is wrong | Mark a milestone complete with nothing attached, and report the project green. | The claim is held short of complete until someone else approves it, and a screen lists the greens with nothing behind them. |
| 2. A change above my authority | Raise a budget change larger than I am allowed to approve. | The request goes to the approver named for that amount. I cannot approve it myself, and I can see where it is waiting. |
| 3. A target changed mid-year | Lower a KPI target in June. | The change needs approval. The old target, the new one, the date and the name stay on the record. |
| 4. One task, up to the board | Pick one task and follow it upwards. | Its progress moves the activity, the project, the initiative and the pillar, and you can see which target that work serves. |
| 5. A change made by AI | Ask the assistant to move a date. | It proposes and a person approves. The change carries that person's name, follows the usual approval route and is logged. |
| 6. My real schedule file | Load our own Microsoft Project or Primavera schedule, in the session. | The vendor says which file formats it reads. The schedule arrives with its structure and links, and can be exported again. |
| 7. Last quarter, as approved | Show the second-quarter figures as the committee approved them. | The approved figures and the files behind them are frozen. A later correction shows as a correction. |
The first test has a post of its own, Green, and wrong. Schedule files and what to do with them are covered in After Project Online.
Expect the tests to be managed
A vendor who knows the tests in advance will prepare for them, as any of us would. Four habits are worth watching for.
The rehearsed path through your data. Your pack goes out a week early and comes back as a polished demo. Change one thing on the day: a different amount, an approver who is on leave, a second KPI.
"That can be configured." It may well be true. Ask for it to be configured in the session, by the person presenting, and note how long it takes.
The quiet switch to administrator. "Let me log in as admin for this step" is the answer to the test. Write down which step it was.
The recording sent afterwards. A test run offline cannot be questioned. Score it as not seen.
None of this is bad faith. A demo team is there to show the product at its best, and your job is to see it on an ordinary day.
Put the same tests to every vendor, us included
These are the tests we ask buyers to put to Lodestar, on their own data and from an ordinary user's seat. One real project with its schedule, one KPI with a year of readings and your delegation of authority are enough to run all seven. Where the right answer is a refusal, ask to watch it happen. Send us the pack when you are ready.

Questions for your next review
- Which of our own projects, KPIs and approval limits will we give every vendor before the demo?
- Who on our side will sit in the ordinary user's seat and try to be refused?
- How many ticks on the current scoring sheet came from something we watched happen on our own data?
- Which of the seven tests would our present tool pass today?
A vendor who is confident in the product will welcome the seven tests. One who steers you back to the prepared demo has answered them.
