The BGP & OSPF protocols are IETF standards, like TCP/IP. There are both closed and open source implementations of those standards.
For example the *nix routing software Quagga is open source, and implements both BGP and OSPF, as well as other routing protocols. Another open source routing software is Bird.
Going further, VyOS is an open source network operating system based on Linux, and uses Quagga as its routing engine. DD-WRT, a Linux-based network operating system for embedded devices, uses both Quagga and Bird depending on the build.
The routing protocols I mentioned are are just for basic Layer 3 (IP) connectivity. You'd want to use a higher order application to encrypt your traffic over the network. I suggested IPsec or OpenVPN as these are the most commonly used VPN applications in network operating systems.
This is all assuming that you want to build your own wireless mesh network, and not just encrypt your communications over existing ISP networks.