The older you get the more you forget what lack of confidence feels like. You get set in your ways, build a group of people you interact with and generally your daily life is predictable. Now we can do some new things, try something different, but we have a life experience to fall back on and rely on. When you first started in life, you had none of that, and you just do not remember well the trepidation you felt as you tested your communication skills on strangers. What happens is that as you gain experience, your confidence builds. True, it gets shaken from time to time, but mostly experience builds confidence. Each time you do something different, or new, or uniquely challenging you gain more confidence.
The personal challenge is that if you stop pushing yourself into new experiences, then you will not build more confidence. You will grow stagnant. This can be dangerous if you ever need more confidence to tackle an especially challenging event in life. All the reason to find ways to push ourselves into new and different experiences.