In short
- A pilot on demo data proves the demo; one on your own data proves the fit.
- Write three to five success criteria, each tied to a decision, before the pilot starts.
- Fix the decision date and its three outcomes: buy, extend for a named question, or stop.
The pilot started well. The vendor set up a sandbox with a sample programme, the PMO team learned the screens in a week, and the dashboards looked better than anything the office had. The steering committee liked what it saw.
Eight weeks later nobody can say whether it worked. There were no written criteria, so each person judged it on what they had seen. The project owners never used it, because their projects were not in it. The security review has not started, because the sandbox was the vendor's. The pilot is extended by a month, then by another.
A pilot that ends without a decision has cost the organisation a quarter and taught it very little.
What goes wrong
A pilot is the cheapest moment to learn whether a platform fits. Most of them are set up in a way that avoids the lesson.
It runs on someone else's data. A sample programme is clean, complete and well linked. Your programme has missing dates, odd structures and owners who report late. A platform that handles the sample proves it can be demonstrated. Only your data shows whether it fits.
The criteria are written afterwards. Without criteria agreed at the start, the pilot is judged on impressions. Those favour whatever was most visible, usually the dashboards, and say little about whether the platform will change how decisions are made.
Only the PMO takes part. The PMO is the buyer, but the owners who update the work and the approvers who sign it are the people who decide whether the platform is used. A pilot without them tests the reporting and skips the adoption, which is where most rollouts fail. I wrote about that in a platform nobody opens.
It runs somewhere production will not. The vendor's environment, the vendor's sign-in, the vendor's AI model. After the pilot, the security review starts from zero with the real hosting, the real identity service and the model the organisation is allowed to use. Weeks are lost, and sometimes the features that impressed in the pilot do not work under the real constraints.
Nobody set an end. With no decision date and no defined outcomes, the easiest choice is always another month.
What good looks like
A pilot is a short, controlled test of one question: will this platform change how we run the programme? Five moves.
Pick one real slice, on your own data
Choose one initiative or one department: big enough to show the joins between projects, small enough to set up in days. Bring its real schedules, KPIs, risks and budget in, as they are. The awkward parts of your data are the most useful part of the test.
Make sure the slice includes at least one dependency on work owned by another team, and at least one change that needs an approval. Those two moments, a join between teams and a decision that needs a signature, show more about a platform than any number of dashboards.
Write the criteria first, each tied to a decision
Agree three to five criteria before the pilot starts, each one a decision the platform should make easier. For example: the monthly steering pack is produced from the record, not rebuilt by hand; owners update their work without being chased; a slipped milestone shows its effect on the KPI it supports; approvals run on the route the organisation set. Each criterion should be observable in the pilot, not promised for later.
Bring in the people who will decide adoption
Include a handful of project owners who will update the work, and the approvers who will sign it. Watch whether they use it in the second week without a reminder. That is the strongest evidence a pilot can produce.
Run it where production will run
If the platform will run in a particular region, in your own data centre or air-gapped, run the pilot there, with sign-in through your own identity service and the AI on the model you will be allowed to use. The security review then happens during the pilot, and the features you judge are the features you will get. For organisations with sovereignty requirements this is the step that saves the most time. The questions to put to the vendor are in data sovereignty and PMO software.
Fix the date and the three outcomes
Set the decision date before the pilot starts, and name the three outcomes: buy, extend for one named question with its own date, or stop. Stopping is a good result if the criteria were not met. It costs far less than finding out after a full rollout.
Lodestar pilots run on the client's own data. Schedules come in from Microsoft Project XML and Primavera P6 PMXML, and a re-import updates the plan in place, so the pilot follows the live programme rather than a copy frozen on day one. A pilot can be requested on the pilot page.

Questions for your next review
- Which slice of our own programme would show whether this platform fits?
- What three to five criteria would we accept as proof, and is each tied to a decision?
- Which owners and approvers will take part, and how will we know they used it?
- On what date will we decide, and what are the three outcomes we will choose between?
A pilot is a decision with a deadline, not a trial with no end. Set it up that way, and it will tell you what you need to know.
