Assumptions:

Normal LAMP Web-server running web app. (Eg. AWS EC2+Apache2+MySQL+Php7)
Not directly targeted by some super-hacker or governmental organisation etc.
Related to point above, no social engineering and the web app itself is secure.

Targeted by whom?
Automated scans and exploits. Are there others?
Is running apt-get update && apt-get upgrade every so often enough to keep a web-server secure?

If no
What else should the 'average' web app programmer, who is also taking care of the server, do to keep the web server reasonably secure for a startup company.
It depends...
Yes, it always depends on many things. Please include assumptions for the most common cases (pareto principle) that the common web app programmer may or may not be aware of.

You've removed a lot of problems that normally get you in trouble (namely, assuming that the app you're hosting is completely secure). From a practical perspective, you absolutely have to consider those.

But presumably since you're aware of them, you have some protective measures in place. Let's talk about the rest, then.

As a start, you probably shouldn't run an update "every so often". Most distros operate security announcement mailing lists, and as soon as a vulnerability is announced there, it's rather public (well, it often is before that, but in your situation you can't really monitor all the security lists in the world). These are low-traffic lists, so you should really subscribe to your distro's and upgrade when you get notifications from it.

Often, a casually-maintained server can be brute-forced or dictionary attacked over a long period of time, since the maintainer isn't really looking for the signs. It's a good idea then to apply the usual counter-measures - no ssh password authentication, fail2ban on ssh and apache - and ideally to set up monitoring alerts when suspicious activity occurs. If that's out of your maintenance (time) budget, make a habit of logging in regularly to check those things manually.

While not traditionally thought of as a part of security, you want to make sure you can bring up a new server quickly. This means server configuration scripts (tools like Ansible, Chef, etc. are useful in system administration anyways) and an automatic backup system that you've tested. If your server's been breached, you've got to assume it's compromised forever and just wipe it, and that sucks if you haven't been taking regular backups of your data.
