↑↓ to moveEnter to openAll results →

    Seven tests for strategy execution software

    Every demo shows the happy path, so every demo looks the same. Products differ in what they refuse, who they send a change to and what they keep. Seven tests, each run on your own data, show that before you sign.

    CHOOSING SOFTWARESeven testsbefore you sign

    In short

    • A demo on the vendor's data shows the happy path, and every product has one.
    • Test with your own project, as an ordinary user, and ask to be refused.
    • Score only what you saw done live on your own data.

    The demo is on Thursday. The vendor shares a screen with a tidy programme. Every project has an owner, every milestone has a date, and the dashboard fills in as the presenter clicks. The room nods. The next vendor comes on Friday and shows the same thing in different colours.

    By the third demo the scoring sheet has three columns of ticks, and nobody can say what separates them.

    I have run that process from the buyer's side, and I now sit on the vendor's side of the table. A demo shows the happy path, and every product has one. What separates products is what they do when something goes wrong, and you only see that on your own data.

    What goes wrong

    A demo is built to pass. Three things about it hide the differences a buyer needs to see.

    The data is clean. No evidence is missing and no number is disputed. A real programme is mostly exceptions, and the demo workspace has none.

    The presenter is the administrator. Every button works for them. Nobody watches the system refuse a person who lacks the authority, so nobody learns whether it would.

    Time only runs forward. The demo shows today's figures. Nobody asks what the record looked like last quarter, or who changed it since.

    So the scoring sheet ends up counting features. Most products have a Gantt chart, a KPI page, a risk register and an AI assistant, and each one gets the tick. The sheet cannot tell a system of record from a well-drawn slide.

    What good looks like

    Change what the vendor is asked to show. Four moves do it, and none of them needs a budget.

    Bring your own programme

    Send each vendor the same small pack a week ahead: one real project with its schedule file, one KPI with a year of readings, and your delegation of authority. Ask them to load it. What they cannot load is your first finding.

    Sit in an ordinary user's seat

    Ask for the tests to be run as a project manager, with the administrator logged out. Most of the tests below end with the system saying no to someone. You want to watch it say no.

    Ask about the past

    Any system can show today. Ask it to show a figure as it stood when the committee approved it, and what has changed since. A programme is audited on its history.

    Score only what you saw

    Give a tick when the thing was done live, on your data, in the session. A slide scores nothing, and neither does the roadmap. Put the same seven tests to every vendor, so that the sheet compares like with like.

    The testWhat to ask forWhat a good answer shows
    1. A green that is wrongMark a milestone complete with nothing attached, and report the project green.The claim is held short of complete until someone else approves it, and a screen lists the greens with nothing behind them.
    2. A change above my authorityRaise a budget change larger than I am allowed to approve.The request goes to the approver named for that amount. I cannot approve it myself, and I can see where it is waiting.
    3. A target changed mid-yearLower a KPI target in June.The change needs approval. The old target, the new one, the date and the name stay on the record.
    4. One task, up to the boardPick one task and follow it upwards.Its progress moves the activity, the project, the initiative and the pillar, and you can see which target that work serves.
    5. A change made by AIAsk the assistant to move a date.It proposes and a person approves. The change carries that person's name, follows the usual approval route and is logged.
    6. My real schedule fileLoad our own Microsoft Project or Primavera schedule, in the session.The vendor says which file formats it reads. The schedule arrives with its structure and links, and can be exported again.
    7. Last quarter, as approvedShow the second-quarter figures as the committee approved them.The approved figures and the files behind them are frozen. A later correction shows as a correction.

    The first test has a post of its own, Green, and wrong. Schedule files and what to do with them are covered in After Project Online.

    Middle row: what the demo shows. Numbered boxes: where the tests leave it. 6 My schedule file 1 A wrong green 2 Above my authority 7 Last quarter Plan loaded Status updated Change made Pack produced 4 Task to target 5 An AI change 3 A moved target each on your own data
    The demo runs straight along the middle. Each test steps off it at the point where a real programme does.

    Expect the tests to be managed

    A vendor who knows the tests in advance will prepare for them, as any of us would. Four habits are worth watching for.

    The rehearsed path through your data. Your pack goes out a week early and comes back as a polished demo. Change one thing on the day: a different amount, an approver who is on leave, a second KPI.

    "That can be configured." It may well be true. Ask for it to be configured in the session, by the person presenting, and note how long it takes.

    The quiet switch to administrator. "Let me log in as admin for this step" is the answer to the test. Write down which step it was.

    The recording sent afterwards. A test run offline cannot be questioned. Score it as not seen.

    None of this is bad faith. A demo team is there to show the product at its best, and your job is to see it on an ordinary day.

    Put the same tests to every vendor, us included

    These are the tests we ask buyers to put to Lodestar, on their own data and from an ordinary user's seat. One real project with its schedule, one KPI with a year of readings and your delegation of authority are enough to run all seven. Where the right answer is a refusal, ask to watch it happen. Send us the pack when you are ready.

    Project schedule: activities, tasks and sub-tasks with dependencies, the critical path and the today line
    Project schedule. Activities down to sub-tasks, with the critical path outlined.

    Questions for your next review

    1. Which of our own projects, KPIs and approval limits will we give every vendor before the demo?
    2. Who on our side will sit in the ordinary user's seat and try to be refused?
    3. How many ticks on the current scoring sheet came from something we watched happen on our own data?
    4. Which of the seven tests would our present tool pass today?

    A vendor who is confident in the product will welcome the seven tests. One who steers you back to the prepared demo has answered them.

    Hany Amer
    Written byHany AmerFounder and CEO, Lodestar

    Hany is a strategy and transformation executive with a technical background in software and AI. He has advised C-suites at a top-three strategy consulting firm (MBB) and independently, and has run transformation programmes of more than USD 1 billion.

    Connect on LinkedIn

    See it on your own programme.

    Send us a slice of your portfolio and we load it before the session, from strategy down to task.