After Margaret's comment, I went searching for the growth charts for breastfed babies. Here's the one for boys:
According to this, Samuel is actually a little above the curve that follows his birth weight. Whew, I must say I feel a little better now. If anyone is interested in seeing more, or in seeing the girl's chart, check on KellyMom.com.