"I don't have that big a collection," he admitted. "I've had more records in the past, and things happen, you don't always get to keep hold of all your possessions. You can get a little neurotic about the actual possession of objects - it's whether it's in your heart. I've never been too bothered about the rarity of a particular label or record as long as I could hear the music. Although some of the records invoke memories; like an album I picked up from my father when he was playing with a band. It's wonderful that I actually have that record."

Elvis Costello likes to have very strong control over the release and marketing of his records - he held off Kojak Variety for five years while he released other work; later he sacked his manager, Jake Riviera, and now manages himself. Then he overruled Warner Bros and insisted on releasing one single per week from All This Useless Beauty over a period of one month.
He is a man who feels the need to control his own marketing.

By his own definition, what's important isn't "the actual possession of objects - it's whether it's in your heart." and that what is important is "as long as I could hear the music."

Despite his frequent arguments with record labels, controversy and his "angry young man becomes grumpy curmudgeon" image, Costello is, in financial respects at least, a typical record company lackey.
He makes his money from selling records, and wants control over that process. Which is fair enough; he has a mutual relationship with the record company, but obviously wants it to work to his financial and creative advantage.
He won't be dictated to about what kind of record to write - anything from punk to classical, through blues, country and most other genres. He won't be dictated to about marketing strategy (and when they offered $1000 to promote an album in the US, it's not difficult to understand why).

But, as both a fan and artist, he wants to have his cake and eat it, on three levels; he wants:

• Complete artistic control over the music
• Significant marketing control over the music
• Not to to perceive other people's music as "product"

In this, Costello actually sees no place for record companies: their objectives are:

• Significant artistic control over the product
• Complete marketing control over the product
• Everyone to perceive the music as "product"

So in his own career, he sees little need for the record industry (only in so far as they deal with the boring logistics of getting his music to his fans, and to write him a monthly cheque for the privilege), and as a fan, he feels no neurosis about having to physically own a record in order to enjoy it.

Admittedly, he says "it's whether it's in your heart," not "it's whether it's on your hard drive". Also the phrase, "I've never been too bothered about the rarity of a particular label or record" could be interpreted as meaning that if he loses a particularly rare record he isn't too bothered that it will cost £50 to replace, however in context it appears to mean that what doesn't bother him is keeping hold of such physical items in the first place - he is taking "in your heart" as offering some kind of moral right to hear the music, whether or not the listener currently owns a physical copy of the recording. There are three main ways of hearing a piece of music which is in your heart, but which you don't physically own (in decreasing order of audio quality):

1. Borrowing a copy from a friend
2. Playing an MP3 or similar file format downloaded over the internet
3. Playing back the memory in your head

Of these, the first and second are illegal in the US, while only the second is illegal in the UK. As far as I know, the third is perfectly legal anywhere, but probably because it's impossible to police rather than from any altruistic motives on the part of the record labels.
Depending on your circumstances, the practicality of each of these options will vary:

• If you are in a student residence, it may be quite feasible to pop down the hall to borrow a CD because "I just fancy hearing that right now."
• If you don't know anybody nearby who has a copy, or they aren't available, downloading it from the internet is a convenient way of accessing the music.
• Whilst driving, one may have to settle for hearing it in your head, unless it happens to be on the radio at the moment.

How does Costello balance his desire for revenue from his own music with his desire to listen to the music of others, of which he does not own a recording? Costello is the classic fan; he has more influences than most people have heard of. But he also depends on per-copy or per-listen payments for his own income.