The Cayley crest
Some of the Cayleys who settled in Australia were descended from a Norfolk family. It is quite possible that these Cayleys were descendants of the medieval Cailly family who had extensive landholdings in Norfolk, and from whom the Cayley baronets of Brompton, Yorkshire descended - but this is pure speculation.
The earliest Cayley ancestor of this particular family whom I have traced is Thomas Cayley (b. c.1821 in Shimpling Norfolk; d. 1905 in Ballarat, Victoria, Australia). He moved to Guisborough in Yorkshire, where the 1871 census records him as being an ironstone miner. He must have emigrated to Australia after that, and almost certainly before 1885 when his daughter Elizabeth married in Australia. His wife was Rachel Chatten (b. c.1829 in Shimpling, Norfolk, d. in Guisborough, Yorkshire in 1886).
Thomas and Rachel had at least five children:
Henry Cayley (b. c.1849 in Shimpling), who, like his father, worked as an ironstone miner at Guisborough in Yorkshire. He married Margaret Dobson, and they had at least four children, all born in Guisborough: Elizabeth Cayley (b. c.1878), Albert E Cayley (b. c.1880), Frederick Caley (b. 1886), and Joseph Caley (b. c.1890). It seems likely that at least Frederick and Joseph adopted the spelling "Caley" for their surname.
James Cayley (b. 1854 in Shimpling, Norfolk), who in 1871 was working as a driver at an ironstone mine at Guisborough, Yorkshire. He married a woman called Fanny.
Elizabeth Cayley, known as Betsy (b. 1861 in Shimpling, Norfolk), who was working in Guisborough as a nurse in 1871, and who afterwards emigrated to Australia, where in 1885 she married Richard Stafford Cook in Victoria
Clara Cayley (b. c.1863 in Shimpling, Norfolk)
John Cayley (b. 1867 in Brotton, Yorkshire; d. 1911 in Waratah, Tasmania, Australia), who emigrated to Australia, where in 1892 he married Blanche Maude Crick at Ballarat, Victoria.
St George's Church, Shimpling, Norfolk