Spuzz Scott uses heft numbers, but it's still a number.
Yes it matters. You can test to find your best letter-number with said shaft. You start light and add lead tape until your impacts become centered. How many have seen a good fitter do this? If you have not then you are not getting a good fitting. Those kind of fitters and just using a letter-number that they think may fit a gamut of golfers like the golf industry does are worthless IMO.
My sweet spot target number for Moi heft match with metal woods ie. driver is 2780 kg-cm ^ 2 (toe down). Swing weight based off of a 14" fulcum swing weight scale changes as various weight in grips (grams) are used. MoI kg-cm ^ 2 does not. Does that tell you something? 😄