I would like to know if it is safe for the host system of a virtual machine (VM - VirtualBox OSE in my case) to execute malware. 

Can a virus break out and read or write data from the host system? Can it establish an Internet connection if I disable it in my VM?

Is a VM a safe environment to try to find out what a virus does?

Can a fork bomb "kill" the host system if I reduce the memory to about 1/4 of my total real memory? How much CPU-time/resources can it use?

Theoretically, the guest system is totally isolated by the VM and cannot even "see" the host, let alone attack it; so the guest cannot break out of the VM. Of course, in practice, it has occasionally happened (web archive link). An attack requires exploiting a security issue (i.e. a programming bug which turns out to have nasty consequences) in the VM implementation or, possibly, the hardware features on which the VM builds on. There are few exit routes for data out of the VM; e.g., for Internet access, the VM is emulating a virtual network card, which deals only with the lowest level packets, not full TCP/IP -- thus, most IP-stack issues remain confined within the VM itself. So bugs leading to breakout from VM tend to remain rare occurrences.

There are some kinds of attacks against which VM are very effective, e.g. fork bombs. From the point of view of the host system, the VM is a single process. A fork bomb in the guest will bring to its knees the scheduler in the guest OS, but for the host this will be totally harmless. Similarly for memory: the VM emulates a physical machine with a given amount of RAM, and will need about that amount of "real" RAM to back it up efficiently. Regardless of what the guest does, the VM will never monopolize more RAM than that. (You still want to limit VM RAM size to, say, at most 1/2 of your physical RAM size, because the extra "real" RAM is handy for disk caching; and the host OS will want to use some, too.)
