The area surrounding his Milton farm includes a string of coastal towns that have gone through a property boom since the COVID-19 pandemic heralded a once-in-a-lifetime exodus to the regions.
"One of the biggest issues that we face in a really nice place, rents have gone up ... and therefore if you're looking for staff to work in your business you can't find them," Mr Miller said.
"We've lost four full-time staff, so that's over 400 cows we've had to remove from the herd because we don't have the staff."
For the fifth-generation dairy farmer, his frustration is exacerbated by the knowledge that he's getting the highest price he's ever received for milk.
His July contract included a 25 per cent increase on last year's prices.
"I've never had it better in the sense of income coming into the farm, it's really really positive — but I need staff, I can't do it by myself."
Normally, Mr Miller would advertise out of the area for workers, but he said because of the housing shortage there was little point.
"I've lost some good young staff, because what I was paying them wasn't enough to be able to rent and live here on the south coast anymore."
The 59-year-old said the accommodation shortage was in part due to the lingering impact of the Black Summer bushfires.
More than 100 homes were razed when fire tore through the nearby Conjola area on New Year's Eve in 2019.
But Mr Miller said things got worse during the pandemic as people moved to the region from Sydney and Canberra, driving up demand.
Similar problems with housing affordability are playing out across the country, according to the Real Estate Institute of Australia, although rental vacancy rates in regions along the east coast are particularly low.
Institute president Hayden Groves said the housing shortage was part of a "vicious cycle" creating economic issues nationwide.
"We've got vacancy rates in Ballarat and Bendigo as low as point one of one per cent ... that is a savage shortage of rental stock in those regional areas," he said.
"In the last two years, in all jurisdictions around Australia, there's been a shortening of supply in available rental stock.
"Previous city dwellers are now taking up those [regional] homes and not leaving much for the locals."
Mr Groves wants all levels of government to give more attention to the housing shortage, although he acknowledged it would not be easy to increase supply given the lack of labour and building materials.
Farming groups said the worker shortfall would only worsen later in the year during the crucial harvest period.
National Farmers' Federation president Fiona Simson agreed the labour shortage was being exacerbated by the lack of accommodation for rural workers.
"Farm businesses need two main types of accommodation — permanent housing options for skilled, long-term workers, who often also have families, and temporary housing for seasonal workers," she said.
"Just like other businesses, housing availability and affordability is connected to farmers’ ability to attract and retain employees."
Data from independent think tank the Regional Australia Institute (RAI) highlights the scale of the labour shortage in regional areas.
The latest RAI figures point to 83,300 job vacancies in regional Australia in June, although the tally is a slight dip on the number of jobs advertised in May.
RAI chief executive Liz Ritchie said regional areas were experiencing a moment unequalled in the country's history.
"In the decade to 2020, (regional) population grew by an average of 76,500 people per annum," she said.
"A life in the regions is now more attractive than ever before."
But she said the surge in popularity for regional living brought with it "growing pains" that needed to be addressed, with jobs and housing two of the significant areas the institute has been targeting.
The difficulty for employers in filling roles has been identified as one of the biggest impediments to further regional growth.
Bendigo Bank head economist David Robertson said with demand for workers across the country at a record high, many regional businesses had been left struggling for staff.
He said a lack of migrant workers and students, combined with a shortage of people on other working visas, was putting extra strain on an already stretched labour force as the regional economy outperforms capital cities.
"Demand is exceeding supply for property, for rental property, for labour in general," he said.
"You're seeing regional dwelling prices rising at the same time that capital city prices are falling, and that's the best evidence that demand for property is even higher in regional Australia than the capitals."
With the official jobless rate dropping to 3.5 per cent in June — the lowest since 1974 — there are almost the same number of job vacancies as there are people seeking work.
Mr Robertson wants to see more social and affordable housing built and said the Federal Government could incentivise state and local governments to increase that supply.
Back in Milton, some short-term relief may be on the way as local real estate agents report a recent bump in property listings.
For Mr Miller it's simple: more staff means milking more cows and a better bottom line.
"I've given all my staff major pay rises this year just to keep them," he said.
"We need young people to come into the dairy industry. It's a physical game, and therefore it's crucial to have that accommodation for them."
20°C