The Guardian’s Ed Pilkington, accompanied the United Nation’s Philip Alston on a two-week mission into the dark heart of the world’s richest nation. After this hew [published the flowing story and says, Philip Alston wants to know why 41 million Americans are living in poverty.

Los Angeles, California, 5 December

“You got a choice to make, man. You could go straight on to heaven. Or you could turn right, into that.”

We are in Los Angeles, in the heart of one of America’s wealthiest cities, and General Dogon, dressed in black, is our tour guide. Alongside him strolls another tall man, grey-haired and sprucely decked out in jeans and suit jacket. Professor Philip Alston is an Australian academic with a formal title: UN special rapporteur on extreme poverty and human rights.

General Dogon, himself a veteran of these Skid Row streets, strides along, stepping over a dead rat without comment and skirting round a body wrapped in a worn orange blanket lying on the sidewalk.

The two men carry on for block after block after block of tatty tents and improvised tarpaulin shelters. Men and women are gathered outside the structures, squatting or sleeping, some in groups, most alone like extras in a low-budget dystopian movie.

We come to an intersection, which is when General Dogon stops and presents his guest with the choice. He points straight ahead to the end of the street, where the glistening skyscrapers of downtown LA rise up in a promise of divine riches.

Then he turns to the right, revealing the “black power” tattoo on his neck, and leads our gaze back into Skid Row bang in the center of LA’s downtown. That way lies 50 blocks of concentrated human humiliation. A nightmare in plain view, in the city of dreams.

So begins a two-week journey into the dark side of the American Dream. The spotlight of the UN monitor, an independent arbiter of human rights standards across the globe, has fallen on this occasion on the US, culminating on Friday with the release of his initial report in Washington.

His fact-finding mission into the richest nation the world has ever known has led him to investigate the tragedy at its core: the 41 million people who officially live in poverty.

Of those, nine million have zero cash income – they do not receive a cent in sustenance.

Alston’s epic journey has taken him from coast to coast, deprivation to deprivation. Starting in LA and San Francisco, sweeping through the Deep South, travelling on to the colonial stain of Puerto Rico then back to the stricken coal country of West Virginia, he has explored the collateral damage of America’s reliance on private enterprise to the exclusion of public help.

The Guardian had unprecedented access to the UN envoy, following him as he crossed the country, attending all his main stops and witnessing the extreme poverty he is investigating first hand.

Think of it as payback time. As the UN special rapporteur himself put it: “Washington is very keen for me to point out the poverty and human rights failings in other countries. This time I’m in the US.”

The tour comes at a critical moment for America and the world. It began on the day that Republicans in the US Senate voted for sweeping tax cuts that will deliver a bonanza for the super wealthy while in time raising taxes on many lower-income families. The changes will exacerbate wealth inequality that is already the most extreme in any industrialized nation, with three men – Bill Gates, Jeff Bezos and Warren Buffet – owning as much as half of the entire American people.

A few days into the UN visit, Republican leaders took a giant leap further. They announced plans to slash key social programs in what amounts to an assault on the already threadbare welfare state.

“Look up! Look at those banks, the cranes, the luxury condos going up,” exclaimed General Dogon, who used to be homeless on Skid Row and now works as a local activist with Lacan. “Down here, there’s nothing. You see the tents back to back, there’s no place for folks to go.”

California made a suitable starting point for the UN visit. It epitomizes both the vast wealth generated in the tech boom for the 0.001%, and the resulting surge in housing costs that has sent homelessness soaring. Los Angeles, the city with by far the largest population of street dwellers in the country, is grappling with crisis numbers that increased 25% this past year to 55,000.

Ressy Finley, 41, was busy sterilizing the white bucket she uses to slop out in her tent in which she has lived on and off for more than a decade. She keeps her living area, a mass of worn mattresses and blankets and a few motley possessions, as clean as she can in a losing battle against rats and cockroaches. She also endures waves of bed bugs, and has large welts on her shoulder to prove it. She cannot simply visit https://www.pestcontrolexperts.com/ like the rest of us would, for a quick solution; she has to live with it every day. I can’t imagine the frustration and powerlessness she must feel.

She receives no formal income, and what she makes on recycling bottles and cans is no way enough to afford the average rents of $1,400 a month for a tiny one-bedroom. A friend brings her food every couple of days, the rest of the time she relies on nearby missions.

She cried twice in the course of our short conversation, once when she recalled how her infant son was taken from her arms by social workers because of her drug habit (he is now 14; she has never seen him again). The second time was when she alluded to the sexual abuse that set her as a child on the path towards drugs and homelessness.

Given all that, it’s remarkable how positive Finley remains. What does she think of the American Dream, the idea that everyone can make it if they try hard enough? She replies instantly: “I know I’m going to make it.”

A 41-year-old woman living on the sidewalk in Skid Row going to make it?

“Sure I will, so long as I keep the faith.”

What does “making it” mean to her?

“I want to be a writer, a poet, an entrepreneur, a therapist.”

Ressy Finley, who lives in a tent on 6th Street in Downtown LA. Photograph: Dan Tuffs for the Guardian

Robert Chambers occupies the next patch of sidewalk along from Finley’s. He’s created an area around his tent out of wooden pallets, what passes in Skid Row for a cottage garden.

He has a sign up saying, “Homeless Writers Coalition”, the name of a group he runs to give homeless people dignity against what he calls the “animalistic” aspects of their lives. He’s referring not least to the lack of public bathrooms that forces people to relieve themselves on the streets.

LA authorities have promised to provide more access to toilets, a critical issue given the deadly outbreak of Hepatitis A that began in San Diego and is spreading on the west coast claiming 21 lives mainly through lack of sanitation in homeless encampments. At night local parks and amenities are closed specifically to keep homeless people out.

Skid Row has had the use of nine toilets at night for 1,800 street-faring people. That’s a ratio well below that mandated by the UN in its camps for Syrian refugees.

“It’s inhuman actually, and eventually in the end you will acquire animalistic psychology,” Chambers said.

He has been living on the streets for almost a year, having violated his parole terms for drug possession and in turn being turfed out of his low-cost apartment. Possession of drugs is a very serious charge, and anyone who is arrested for this will be in need of professional help, similar to what lawyer Adam Krischer can provide you, so you are able to receive the lightest sentence. But as Chambers violated his parole, he has ended up in an even worse situation, as he is now homeless. There’s no help for him now, he said, no question of “making it”.

“The safety net? It has too many holes in it for me.”

Of all the people who crossed paths with the UN monitor, Chambers was the most dismissive of the American Dream. “People don’t realize – it’s never getting better, there’s no recovery for people like us. I’m 67, I have a heart condition, I shouldn’t be out here. I might not be too much longer.”

That was a lot of bad karma to absorb on day one, and it rattled even as seasoned a student of hardship as Alston. As UN special rapporteur, he’s reported on dire poverty and its impact on human rights in Saudi Arabia and China among other places. But Skid Row?

“I was feeling pretty depressed,” he told the Guardian later. “The endless drumbeat of horror stories. At a certain point you do wonder what can anyone do about this, let alone me.”

And then he took a flight up to San Francisco, to the Tenderloin district where homeless people congregate, and walked into St Boniface church.

What he saw there was an analgesic for his soul.

San Francisco, California, 6 December

The Gubbio project at St Boniface in San Francisco. The church opens its doors every weekday at 6am to allow homeless people to rest until 3pm. Photograph: David Levene for the Guardian

About 70 homeless people were quietly sleeping in pews at the back of the church, as they are allowed to do every weekday morning, with worshippers praying harmoniously in front of them. The church welcomes them in as part of the Catholic concept of extending the helping hand.

“I found the church surprisingly uplifting,” Alston said. “It was such a simple scene and such an obvious idea. It struck me – Christianity, what the hell is it about if it’s not this?”

It was a rare drop of altruism on the west coast, competing against a sea of hostility. More than 500 anti-homeless laws have been passed in Californian cities in recent years. At a federal level, Ben Carson, the neurosurgeon who Donald Trump appointed US housing secretary, is decimating government spending on affordable housing.

Perhaps the most telling detail: apart from St Boniface and its sister church, no other place of worship in San Francisco welcomes homeless people. In fact, many have begun, even at this season of goodwill, to lock their doors to all comers simply so as to exclude homeless people.

As Tiny Gray-Garcia, herself on the streets, described it to Alston, there is a prevailing attitude that she and her peers have to contend with every day. She called it the “violence of looking away”.

Coy Catley, 63, in her homeless box made of cardboard sheets on a sidewalk of Tenderloin, San Francisco. Photograph: Ed Pilkington for the Guardian

That cruel streak – the violence of looking away – has been a feature of American life since the nation’s founding. The casting off the yoke of overweening government (the British monarchy) came to be equated in the minds of many Americans with states’ rights and the individualistic idea of making it on your own – a view that is fine for those fortunate enough to do so, less happy if you’re born on the wrong side of the tracks.

Countering that has been the conviction that society must protect its own against the vagaries of hunger or unemployment that informed Franklin Roosevelt’s New Deal and the Great Society of Lyndon Johnson. But in recent times the prevailing winds have blown strongly in the “you’re on your own, buddy” direction. Ronald Reagan set the trend with his 1980s tax cuts, followed by Bill Clinton, whose 1996 decision to scrap welfare payments for low-income families is still punishing millions of Americans.

The cumulative attack has left struggling families, including the 15 million children who are officially in poverty, with dramatically less support than in any other industrialized economy. Now they face perhaps the greatest threat of all.

As Alston himself has written in an essay on Trump’s populism and the aggressive challenge it poses to human rights: “These are extraordinarily dangerous times. Almost anything seems possible.”

Lowndes County, Alabama, 9 December

Trump’s undermining of human rights, combined with the Republican threat to pare back welfare programs next year in order to pay for some of the tax cuts for the rich they are rushing through Congress, will hurt African Americans disproportionately.

Black people are 13% of the US population, but 23% of those officially in poverty and 39% of the homeless.

The racial element of America’s poverty crisis is seen nowhere more clearly than in the Deep South, where the open wounds of slavery continue to bleed. The UN special rapporteur chose as his next stop the “Black Belt,” the term that originally referred to the rich dark soil that exists in a band across Alabama but over time came to describe its majority African American population.

The link between soil type and demographics was not coincidental. Cotton was found to thrive in this fertile land, and that in turn spawned a trade in slaves to pick the crop. Their descendants still live in the Black Belt, still mired in poverty among the worst in the union.

You can trace the history of America’s shame, from slave times to the present day, in a set of simple graphs. The first shows the cotton-friendly soil of the Black Belt, then the slave population, followed by modern black residence and today’s extreme poverty – they all occupy the exact same half-moon across Alabama.

There are numerous ways you could parse the present parlous state of Alabama’s black community. Perhaps the starkest is the fact that in the Black Belt so many families still have no access to sanitation. Thousands of people continue to live among open sewers of the sort normally associated with the developing world.

The crisis was revealed by the Guardian earlier this year to have led to an ongoing endemic of hookworm, an intestinal parasite that is transmitted through human waste. It is found in Africa and South Asia, but had been assumed eradicated in the US years ago.

Yet here the worm still is, sucking the blood of poor people, in the home state of Trump’s US attorney general Jeff Sessions.

A disease of the developing world thriving in the world’s richest country.

The open sewerage problem is especially acute in Lowndes County, a majority black community that was an epicenter of the civil rights movement having been the setting of Martin Luther King’s Selma to Montgomery voting rights march in 1965.

Philp Alston talks to a resident. Many families in Butler and Lowndes counties choose to live with open sewer systems made from PVC pipe. Photograph: Bob Miller for the Guardian

Despite its proud history, Catherine Flowers estimates that 70% of households in the area either “straight pipe” their waste directly onto open ground, or have defective septic tanks incapable of dealing with heavy rains.

When her group, Alabama Center for Rural Enterprise (Acre), pressed local authorities to do something about it, officials invested $6m in extending waste treatment systems to primarily white-owned businesses while bypassing overwhelmingly black households.

“That’s a glaring example of injustice,” Flowers said. “People who cannot afford their own systems are left to their own devices while businesses who do have the money are given public services.”

Walter, a Lowndes County resident who asked not to give his last name for fear that his water supply would be cut off as a reprisal for speaking out, lives with the daily consequences of such public neglect. “You get a good hard rain and it backs up into the house.”

That’s a polite way of saying that sewage gurgles up into his kitchen sink, hand basin and bath, filling the house with a sickly-sweet stench.

Given these circumstances, what does he think of the ideology that anyone can make it if they try?

“I suppose they could if they had the chance,” Walter said. He paused, then added: “Folks aren’t given the chance.”

Had he been born white, would his sewerage problems have been fixed by now?

After another pause, he said: “Not being racist, but yeah, they would.”

Round the back of Walter’s house the true iniquity of the situation reveals itself. The yard is laced with small channels running from neighboring houses along which dark liquid flows. It congregates in viscous pools directly underneath the mobile home in which Walter’s son, daughter-in-law and 16-year-old granddaughter live.

It is the ultimate image of the lot of Alabama’s impoverished rural black community. As American citizens they are as fully entitled to life, liberty and the pursuit of happiness. It’s just that they are surrounded by pools of excrement.

This week, the Black Belt bit back. On Tuesday a new line was added to that simple graphic, showing exactly the same half-moon across Alabama except this time it was not black but blue.

It depicted the army of African American voters who turned out against the odds to send Doug Jones to the US Senate, the first Democrat from Alabama to do so in a generation. It delivered a bloody nose to his opponent, the alleged child molester Roy Moore, and his puppet masters Steve Bannon and Donald Trump.

It was arguably the most important expression of black political muscle in the region since King’s 1965 march. If the previous entries in the graphic could be labeled “soil”, “slavery” and “poverty”, this one should be captioned “empowerment”.

Guayama, Puerto Rico, 10 December

So how does Alston view the role of UN rapporteur and his visit? His full report on the US will be released next May before being presented to the UN human rights council in Geneva.

Nobody expects much to come of that: the world body has no teeth with which to enforce good behavior on recalcitrant governments. But Alston hopes that his visit will have an impact by shaming the US into reflecting on its values.

“My role is to hold governments to account,” he said. “If the US administration doesn’t want to talk about the right to housing, healthcare or food, then there are still basic human rights standards that have to be met. It’s my job to point that out.”

Alston’s previous investigations into extreme poverty in places like Mauritania pulled no punches. We can expect the same tough love when it comes to his analysis of Puerto Rico, the next stop on his journey into America’s dark side.

Three months after Maria, the devastation wrought by the hurricane has been well documented. It tore 70,000 homes to shreds, brought industry to a standstill and caused a total blackout of the island that continues to cause havoc.

But Puerto Rico’s plight long predates Maria, rooted in the indifference with which it has been regarded since being acquired as a spoil of war in 1898. Almost half of Americans have no idea that the 3.5 million Puerto Ricans on the island are US citizens, which adds insult to the injury of the territory having no representation in Congress while its fiscal policies are dictated by an oversight board imposed by Washington. What was that about casting off the yoke of overweening government?

Nor do most people appreciate that the island has twice the proportion of people in poverty (44%) than the lowliest US state, including Alabama (19%). And that was before the hurricane, which some estimates suggest has pushed the poverty rate up to 60%.

“Puerto Rico is a sacrifice zone,” said Ruth Santiago, a community rights lawyer. “We are ruled by the United States, but we are never consulted – we have no influence, we’re just their plaything.”

The UN monitor was given a sense of what being a plaything of the US means in practice when he travelled south to Guayama, a town of 42,000 close to where Maria made landfall. Devastation was everywhere – houses mangled, roofs missing, power lines drooping alarmingly overhead.If Lyndon Johnson declared a war on poverty, then Trump is waging a war on the poor

Looming over the community is a coal-fired power plant built by the Puerto Rican branch of AES Corporation, a Virginia-headquartered multinational. The plant’s smoke stack dominates the horizon, as does a huge mound of residue from the combusted coal that rises to at least 70ft like a giant sandcastle.

The mound is exposed to the elements and local people complain that toxins from it leach into the sea, destroying the livelihoods of fishermen through mercury poisoning. They also fear that dust coming off the pile causes health problems, a concern shared by local doctors who told the UN monitor that they see a high incidence of respiratory disease and cancer.

“It kills the leaves of my mango tree,” said Flora Picar Cruz, 82. She was lying in bed at midday, breathing with difficulty through an oxygen mask.

Norma Judith Colón stands in front of her damaged home after hurricane María. Photograph: José Jiménez-Tirado/Getty Image

Studies of the pile have found perilous levels of toxic substances including arsenic, boron, chloride and chromium. Even so, the Trump administration is in the process of easing the relatively lax regulations on monitoring dangerous effluents from it.

AES Puerto Rico told the Guardian that there was nothing to worry about, as the plant was one of the cleanest in the US having been purpose built to avoid any run-off into air or sea. That’s not what the people of Guayama think. They fear that the age-old pattern of being taken for granted by the US colonizer is about to rise to the next level.

When such attitudes are replicated across the island it helps explain why so many Puerto Ricans are voting with their feet: almost 200,000 have packed their bags and quit for Florida, New York and Pennsylvania since the hurricane, adding to the more than 5m who were already on the US mainland. Which gives a whole new meaning to the American Dream – anyone can make it, so long as they abandon their families, their homes, and their culture and head off into a strange and forbidding land.

Charleston, West Virginia, 13 December

“You’re an amazing people! We’re going to take care of a lot of years of horrible abuse, OK? You can count on it 100%.”

Donald Trump’s promise to the white voters of West Virginia was made just as he was securing the Republican presidential nomination in May 2016. Six months later, his audience handsomely repaid him with a landslide victory.

It is not surprising that white families in West Virginia should have responded positively to Trump’s charm offensive, given that he offered them the world – “We’re going to put the miners back to work!” After all, numerically a majority of all those living in poverty nationwide – 27 million people – are white.

In West Virginia in particular, white families have a lot to feel sore about. Mechanization and the decline of coal mining have decimated the state, leading to high unemployment and stagnant wages. The transfer of jobs from the mines and steel mills to Walmart has led to male workers earning on average $3.50 an hour less today than they did in 1979.

What is surprising is that so many proud working folk should have entrusted their dreams to a (supposed) billionaire who built his real estate empire on the back of handouts from his father.

Before he ran for the presidency, Trump showed scant interest in the struggles of low-income families, white or otherwise. After almost a year in the Oval Office, there is similarly little sign of those campaign promises being kept.

Quite the contrary. When the UN rapporteur decamped in Charleston, West Virginia on Wednesday as the final stop in his tour, he was inundated with evidence that the president is turning the screws on the very people who elected him.

That same day, Republicans in the Senate and House were fusing their plans for tax cuts ahead of a final vote next week. Many West Virginians will be lulled into believing that the changes are designed to help them, as initially everybody in the state will pay less tax.

But come 2027 when deficit-saving changes kick in, the bottom 80% of the population will pay more, while the top 1% will continue to enjoy a $21,000 bonanza.

“Trump’s policies will exacerbate inequality, suppress wages and make it harder for low-income families to seek assistance,” said Ted Boettner, executive director of the non-partisan West Virginia Center on Budget and Policy.

If sewage is the abiding image of the burden of the Black Belt, then a mouthful of rotting teeth is West Virginia’s.

A patient who came into the clinic needing all 30 of his teeth to have root canal surgery. Photograph: Doctors at Health Right

Doctors at Health Right, a volunteer-based medical center in Charleston that treats 21,000 low-income working people free of charge, presented the UN monitor with a photograph of one of its dentistry clients.

The man is only 32, but when he opened his mouth he turned into one of Macbeth’s witches. His few remaining rotting teeth and greenish-blue gums looked like the festering broth in their burning cauldrons.

Adult dentistry is uncovered by Medicaid unless it is an emergency, and so people do the logical thing – they do nothing until their abscesses erupt and they have to go to ER. One woman seen by the center’s mobile dentistry clinic was found to have nothing but 30 roots in her mouth, all of which needed surgery. Leaving dental issues until they become a medical emergency is not the wisest course of action. Most people would enquire about dental implants in Rochester long before the situation detoriated to this point. However, this is not possible for everyone.

In other briefings, Alston was given a picture of life under siege for West Virginia’s low-income families. If Lyndon Johnson declared a war on poverty, then Trump is waging a war on the poor.

People are jailed for years because they cannot afford bail awaiting trial; private detectives are used to snoop on disability benefit claimants; mandatory minimum drug sentences are back in fashion; Jeff Sessions is scrapping federal rehabilitation schemes for those released from prison; tenants in subsidized housing are living in fear that they will be evicted for the slightest infraction – the list goes on and on.

And the result of this relentless drubbing? “People end up fighting each other,” said Eli Baumwell, policy director of the ACLU in West Virginia. “You become so obsessed with what you’ve got and what your neighbor has got that you become resentful. That’s what Trump is doing – turning one against the other.”

And so it was that Philip Alston boarded one last plane and headed for Washington, carrying with him the distilled torment of the American people.

At one point in the trip Alston revealed that he had had a sleepless night, reflecting on the lost souls we had met in Skid Row.

He wondered about how a person in his position – “I’m old, male, white, rich and I live very well” – would react to one of those homeless people. “He would look at him and see someone who is dirty, who doesn’t wash, who he doesn’t want to be around.”

Then Alston had an epiphany.

“I realized that’s how government sees them. But what I see is the failure of society. I see a society that let that happen, that is not doing what it should. And it’s very sad.”