Definitely yes, if you have a big or small penis is mostly defined by your genes. Sons of men with a large penis often have a large penis. Sons of men with a small penis often have a smaller penis. There are certainly exceptions, but in general genes are very dominant when it comes to erect penis size. From my own experience and research, genetics are the most dominant factor in size, environmental factors or lifestyle only have very limited to no effects.
However, after more than a century of generally dubious and sometimes lunatic penis enlargement attempts, there's still not much you can do. Sure, there are lots and lots of supposed options out there -- penis pills, creams, brutal stretching exercises, horrific-looking devices, and penis surgery. Almost none of it works. The few approaches that can work often have modest benefits and serious side effects. How serious? In some cases, erectile dysfunction-serious.