More than 4,000 prisoners with serious mental illness are being held in solitary confinement in US prisons, according to new research, despite the knowledge that holding people in isolation exacerbates mental problems and can even trigger them.

A survey by Yale law researchers together with the Association of State Correctional Administrators (ASCA) has revealed the shocking prevalence of solitary confinement among prisoners struggling with profound mental health issues. They are kept in total isolation for at least 22 hours a day for 15 continuous days or more.

The ASCA and Arthur Liman Center at Yale law school found that most states in America are holding mentally ill individuals in isolation. Of the 33 states that responded to the survey with details, only one state, Texas, said it had no such inmates in solitary confinement.

Thirteen of the states that replied – more than a third – revealed that at least 10% of their male prisoners classified as having mental health problems were being held in isolation. Missouri had the highest number – 703 inmates – while New Mexico had the highest proportion, with some 64% of its mentally ill prisoners being kept in solitary.

“This is tragic,” said Judith Resnik, a law professor at Yale who is founding director of the Arthur Liman Center. “Solitary confinement is a disabling setting that is harmful for human health and safety. It can do harm for people who are mentally OK and inflict terrible damage on people who are already mentally ill.”

There have been numerous studies underlining the damage wrought by solitary confinement on the mental health of prisoners. A 2014 study carried out in New York city’s jail system, for instance, found that the prevalence of self harm among inmates held in isolation was seven times that of those in general population.

The evidence of harm is so well established that the American Correctional Association has issued standards forbidding states from holding mentally ill prisoners in isolation cells, also known as segregation, for prolonged periods.

On Tuesday the US supreme court justice Sonia Sotomayor issued an opinion in a case which the court declined to hear in which she said she was “deeply troubled” by the ongoing practice. She wrote that “solitary confinement imprints on those that it clutches a wide range of psychological scars”.

Referencing Charles Dickens’s A Tale of Two Cities, Sotomayor lamented that prisoners were being held in conditions of “near-total isolation from the living world”, in what she said “comes perilously close to a penal tomb”.

Some states have made strides in reducing, or even eliminating, the use of solitary for mentally ill prisoners. Colorado has a new rule that when a prisoner is found to have psychological or other problems they are immediately redirected to treatment rather than segregated lock-up.

Rick Raemisch, executive director of the state’s department of corrections, told the Yale survey that since implementing the new rule the number of prisoners being treated in mental health facilities had drastically fallen. “It is too early to tell if the reason for this is because we have stopped manufacturing or multiplying mental illness by the overuse of segregation,” he said.

Colorado is one of the success stories contained in the Yale/ASCA report. In 2017, it became the first and only state in the US to ban the practice of holding prisoners in isolation cells for longer than 15 days.

Overall, the survey found that the number of solitary confinement prisoners has begun to decline as states start to reassess the practice. They used to see it as a crucial means of ensuring control over volatile inmates, but now the report notes “they see it as a problem to be solved”.

The total segregated population has fallen from up to 100,000 in 2014 to about 61,000 last year. But progress remains patchy – while many states have reduced their tallies, in 11 states the numbers have actually gone up.

In Louisiana, almost one in five prisoners continue to be held in solitary confinement. Louisiana had the dubious distinction of being home to the longest-standing isolation prisoner in America: Albert Woodfox, who was released in 2016 having served 43 years entirely alone in a concrete cell measuring 6ft by 9ft.

The amount of time spent in isolation also remains at crisis point. The survey found that almost 2,000 prisoners have been held in segregation for more than six years – a length of time that can only be described as extreme given that being locked up in isolation for 22 hours a day can drive people to distraction within a matter of days.