Hopes are "slim" that Cosmos-1 made it into orbit
But they said the apparent detection of signals from the craft by tracking stations remained to be explained.
The privately funded Cosmos-1 craft was launched on Tuesday on a Russian rocket from a submarine in the Barents Sea.
Russian officials said the modified missile carrying the craft failed during firing of its first stage.
Despite announcements by the Russian navy and space agency that the vehicle had been lost, the Cosmos-1 team held out hope throughout Tuesday that the launch had succeeded.
However, that hope has now all but faded. In a statement, mission sponsors the Planetary Society accepted the Russians' conclusion was probably correct.
But, it continued, "there are some inconsistent indications from information received from other sources".
Apparent signals were detected by three ground stations, at Petropavlovsk in Kamchatka, Majuro in the Marshall Islands and Panska Ves in the Czech Republic, the Planetary Society said. It added that doppler data was also detected over one of these stations.
Slim chance
But Jiri Simunek, a scientist at Panska Ves, told the BBC News website that no signal had been detected by the Czech tracking station, "just noise".
Nevertheless, Cosmos-1 scientists said there was still a slim possibility that the craft made it into orbit, though a lower one than expected.
The spacecraft was launched from a Russian submarine
Cosmos-1: How does it work?
A spokesperson for the Russian space agency said the Volna rocket booster carrying the spacecraft had failed 83 seconds after launch due to a problem with the first stage engine of the three-stage booster.
"The booster's failure means that the solar sail vehicle was lost," said agency spokesperson Vyacheslav Davidenko.
"The Russian navy is searching the area for the debris of the booster and the vehicle."
Rocket failure
The Russian-built Cosmos-1 was launched aboard a modified Volna intercontinental ballistic missile (ICBM) from the nuclear submarine Borisoglebsk.
The $4m (£2.1m) experimental craft uses "solar sails" for power.
The sail reflects particles of light, or photons, from the Sun, gaining momentum in the opposite direction.
Some think solar sails offer a cheaper, faster form of spacecraft propulsion.
"Solar sailing is really the only known technology that could potentially take us to the stars one day because it does not have to carry fuel with it and because it can keep accelerating - even at incredible distances," the Planetary Society's Amir Alexander told the BBC.
The acceleration from sunlight is very small; but the advantage of solar sailing over chemical propulsion is that the acceleration is sustained.
Cosmos-1 would have got faster and faster - and climbed higher in orbit - as time went on.
The 100kg (220lbs) craft had been scheduled to reach an 800km- (500 mile-) high orbit.
It was then to have taken pictures of Earth for four days before unfurling its eight aluminium-backed plastic sail blades into a 30m (100ft) circle.
The US, European, Japanese and Russian space agencies also have solar sail programmes in the offing.
The Planetary Society have stated that their experimental solar sail craft Cosmos 1 is probably lost, but intermittent signals have given them some hope the mission has made orbit.

The 100kg vehicle was launched atop a converted ICBM from a Russian nuclear submarine, however it is believed the first stage of the Volna booster rocket failed 83 seconds into the flight. The team behind the $4m craft have a slim hope that it managed to reach a low orbit, and efforts to pick up the tracking beacon are continuing.

Signals have apparently been received by at least two tracking stations around the globe - in the Czech Republic, and in the Marshall Islands. The Planetary Society are being assisted by the US Strategic Command in an effort to find the spacecraft, if it did make orbit.

However a scientist from the Czech station has said they have received only noise.

If Cosmos-1 is in orbit, the onboard computer may still be functioning and could begin to unfurl the 30 meter diameter sail in three days time.

A solar sail harnesses pressure exerted upon the sail by photons from the sun to push the craft along. Although its acceleration is very slow, it can go on nearly indefinitely (as long as the Sun exists). It was hoped Cosmos-1 would use a 30 m diameter sail to reach an orbit 800km above the Earth, as a demonstration of the potential of the technology.