Adrian,
You are correct a certain amount of this has to do with belief and expectation. I have been asked where this info on frequencies comes from. Some comes from EEG and other research. Some comes from spiritual and philosophical beliefs.
There is science that says brainwaves can be entrained with binaural and isochronic beats. It is possible to have your brainwaves in the range normally seen during sleep, yet you are awake. We call that an altered state of consciousness.
Generally delta range is considered to be healing and theta is associated with the range monks are in when in deep meditation.
It is science that says relaxation and lowering stress is beneficial to health. So is a positive outlook. Meditation and the various aids to it can help to relax and reduce stress.
As for any specific meaning such as astral projection, lucid dreaming etc. People report having better results for certain things with certain frequencies. I started out with the idea of just making stuff and stating what beats and frequencies were used. But then I realized most people are not going to do the amount of reading on the subject I am doing, and most are not going to look through a list of frequencies and try to make heads or tails of it. They will not get into the technicalities. They just find this stuff works and want someone to make it for them with a bit of guidance for them as well.
Often times when I listen to binaural beats and when I am making them I go with what feels right to me. I know that when others use them they are going to be attracted to what feels right to them.
My personal experience is that meditation to delta or theta range defiantly helps me sleep at night. I favor 4Hz beat with 136.1Hz base. That gets me into the "zone"

I have not been able to lucid dream, astral project or any of those things just yet. So I can only go along with what others report.
When I first started making binaural beats, I tried various base frequencies in the range of 100 - 200Hz, because that was recommended in some of what I read. I tried the frequency of OM 136.1Hz and found that most pleasant. And, it is still my favorite base frequency. I make videos with other base frequencies as well because these frequencies are believed to have different effects and because different people might have different favorites just as I do.
I personally feel that all specific abilities attributed to BWE arise from the actual expectation of the end result.
This is simply an observation - not negative. If a BWE tracks was not labelled as "Astral Projection" for example, then the listener would not be "programmed" to expect Astral Projection.
In broad terms, there is no specific frequency range for Astral projection of anything else, but rather that at a specific frequency and combination of sounds - and everyone is unique - an expansion of consciousness can take place at which point a wide range of metaphysical abilities become available.
I rarely use Binaurals, except as "spatially dynamic" binaural beats which are in harmony with the base sound track. Actually, Isochronic and even Monaural beats are much more effective, but they can also be intrusive so while invoking an expanded state of consciousness, they can also disrupt that consciousness as well - but not necessarily. So when I use Isochronic and/or Monaural beats I integrate them at a Subliminal Frequency - that is to say - the Subconscious Mind can detect them but not the Conscious Mind.
In terms of frequencies - it depends on the nature of the track. I agree with you that 4Hz is a good frequency for e.g. Meditation because it is on the borderline between Alpha and Theta. For metaphysical and psychic abilities nearer 8Hz is better, being in the mid-Alpha sort of range. This is the range of for example drumming and other sounds in "native" cultures for this reason.
Another fallacy, in my opinion, is the tendency to "ramp" frequencies up or down, i.e. to start at a high frequency and ramp down to the "target frequency". The brain will entrain to any frequency very quickly and does not need, in my opinion, to be "guided" there.
Another tendency in BWE tracks that I personally disagree with is the use of multiple frequencies which serves no real purpose if there is a specific objective.
At the final analysis there are no hard and fast rules, and every listener responds differently. I have been making BWE tracks for at least a decade, and have yet to find the "holy grail" of BWE, although I know what I wish to accomplish and have the technology to achieve it.
Ultimately any BWE track should be "captivating" to listen to - so the user does not get bored quickly - harmonious, and integrate powerful BWE to entrain the Consciousness without BWE techniques that are actually intrusive and disruptive on the Consciousness. It is a very fine balance and nt an "exact science".
My starting point is the right ambient music and/or sound, and I build on that.
I have recently for example been experimenting with Buddhist Mantras, because they alone are designed to invoke BWE, and then I have integrated 4 or 5 different, subtle BWE technologies.
I have two tracks of the "Om Mantra" to which I have integrated 4 types of BWE which are largely unnoticeable.
In Love, Light and Service,
Adrian.