I guess it depends on the price of the set, it is sometimes nice to have the full range. End mill shanks are normally in 1/8'' increments when > 1/2 '' The most common sizes are 1/8, 3/16, 1/4, 5/16, 3/8, and 1/2. On a smaller mill like yours, 3./8 is about the largest I would use. With the exception of the occasional use of a face mill, I rarely use larger than 1/2'' on my machine, which is a 3500lb knee mill.
I would seriously look at ER tool holders. We have several full sets of ER collets in all the ER sizes we use, by 1/16's and then buy single collets for when we need multiples of a given size. This allows setting multiple cutters and changing them quickly.