So my barrel chest doesn't factor in?
Well, I dont know how much it factors in, but its definitely not decisive. The main factor is definitely the vocal cords.
The human voice works much more like a string instrument than a flute or trumpet. If you give a violin a larger body, but keep the strings at the same length, tension, material and diameter/weight, it will sound different but it wont change much in regards to tone height.
For example, in ages of old eunuchs have been sopranos and altists. Despite having the body of grown men, they still had high pitched voices like a woman or child.
Likewise tall women still have female voices.
Likewise short asian men still have male voices. I know a guy who is about 1.50m and he still has a male voice. Heck the actor Peter Dinklage is 1.32m and he still has a male voice.
Likewise when people get more fat, or get more muscular, or get larger lungs from living at higher altitudes, their voices dont get lower.
Likewise, when you expand your range of possible tones you can sing, a big part is making your vocal cords stronger through constant training, so you can raise their tension. The possibilities to expand your range downwards are much more limited; you can really only learn technique, but not train any muscles for that.
So yes, the vocal cords are the main factor for this.