The area now covered by the Audley recreational complex was originally inhabited by the Dharawal tribe. The tribes economy was based on the rich marine and estuarine resources. The stability of these resources allowed for long-term, semi-sedentary summer occupation, which has left evidence in the form of middens, rock shelter deposits, burials, rock engravings, rock art, axe-grinding groo5045515ves and open campsites (Evans in McDonald 1987:2).
Bass and Flinders first explored the area in March 1796, but it remained un-utilised as it was unsuitable for pastoralism and only spasmodic attempts at timber getting were initiated.
The land was dedicated as Royal Park in April 1879 for the use of the New South Wales Zoological Society who were intent on introducing non-native flora and fauna. Work on the Park began in 1880 and consisted of clearing land and damming the Georges River. By 1883 a dam across the Hacking River created a lake-like environment for pleasure craft and such activities became one of the longest-lived activities at Audley.
In 1884 the main camp was established on the confluence of the Hacking River and Kangaroo Creek and was dubbed 'Audley', in commemoration of Lord Audley's survey camp. Audley had surveyed the area in 1864. The camp initially consisted of a dock, boat house, jetty, weatherboard pavilion, stables, stores, outhouses, smithy, forge and plant. Paddocks were also fenced and the number was increased soon after. Bridges, culverts, new cottages for the workers and a boat slip were the next priority. Plantings of ornamental trees began very early and continued through to 1890. Plantings included Red Cedar (Juniperus virginiana), Catapla Oaks (Catalpa bignonioides), Moreton Bay Figs (Ficus macrophylla) and Lilli-pilli (Acmena smithii). Between 1887 and 1889 unemployed men, under the supervision of the Casual Labour Board, cleared large areas of land. To fund the Park and the Society's aims attempts were made to exploit the coal, clay, gravel, timber, grass and oysters, but was largely unsuccessful.
Between 1891 and 1893 the focus was on making the park visitor-friendly. Cleared land along the river was sown with grass, directional signs were erected, a bridge built over Kangaroo Creek and an aviary constructed. Other works included placing rustic tables around the park, the construction of facilities for the hiring of boats, a public kitchen, a windmill to supply water and accommodation for 30 guests. The accommodation was first known as the public pavilion, then 'The Rest' and for most of its history as 'Allambie House'. In 1893 the first guide book was published. Visitor facilities were much enhanced in 1895 when toilets were provided and in the following year a refreshment room was constructed.
In 1903 the focus of management of the Park altered somewhat with a shift towards resource extraction. A sawmill was constructed to facilitate logging of sections of the forest. During this period weekend camps were allowed. During the depression large numbers of illegal squatters occupied the Park and were living off the Park's resources, industries included the sale of wildflowers, timber and soil. In 1933 34 camps were identified at Audley. Attempts were made to remove the illegal camps, but eight still remained by 1941.
World War II halted development of Audley, but at the end of the War funds became available to update the Parks facilities. Extensive roadworks had to be undertaken to repair damage done by army exercises in the Park. Buildings that had been neglected during the war were renovated and renewed. In 1954 Queen Elizabeth approved the addition of "Royal" to the name of the park.
The formation of the National Parks and Wildlife Service in 1967 effected the management of the Park, which was previously controlled by a Trust. The Trust was retained as an advisory committee. The NPWS brought a philosophical shift in the management of the Park with the aim of returning the Park to its natural state: cars were restricted, many huts and former buildings were removed and the ornamental gardens were no longer maintained. From the inception of the Park its primary function had been to provide recreational facilities to the overcrowded city dwellers. With the advent of the conservation movement, beginning in the 1920s, this aim came under attack and was slowly eroded.
In 1973 Allambie House kiosk and the swimming pool kiosk were demolished. Two years later Allambie House its self was burnt to the ground. Demolition of structures at Audley continued throughout the 1970s. In 1979 a new administrative centre was constructed at Loftus Heights, replacing a temporary setup on the top floor of the former the Dance Hall. Landscaping, path building and facilities upgrade begun in the previous year continued. A new kiosk was constructed on the picnic flat in 1980.