Welcome Creamcheese. Feel free to introduce yourself.
Your question isn't about guitars, but we will bring you into the fold...!
Why don't you start with old components for yard sales, goodwill, your local recycling spot perhaps. Ask neighbors and family! There are hundreds of system good and bad sitting in attics, closets...Do not pay (much) for some of this stuff, capacitors and such do age, even though what I found has been fine so far. Some early 80s Japanese hifi components were very well built - I just cleaned a Sony reciever and the innards were superb. Some purists (not me!) seek the warm tones these components produce.
Old truntables are a bit tough to get a hold of, apparently. I think a lot did not age well and got tossed. Delicate equipment, so treat old ones with care. There are good, inexpensive all-purpose cartridges on the market, 35-40 dollars gets you started.