I want to make a little programming puzzle on my website. There's going to be a task. The user will be asked to upload a C++ source file with their solution. The file should be compiled, run with some input and checked if it produces right output. What are the security risks? How do I make sure that the uploaded file won't be able to do anything malicious?

Alternatively, there are sites like Tutorials Point that let you compile and run C++ code. How do they manage the security?

It is impossible to analyze a program to find out if it will do anything malicious. That is true regardless of whether you are attempting to analyze the source or compiled code.

The way to do what you are asking for is done by compiling and running the code in a sandbox. Once the program has terminated (or after a timeout you have decided upon) you destroy the sandbox.

The security of such a construction is as secure as the sandbox you are using. Depending on the requirements of the code you need to run the sandbox could be either something simple like Linux secure computing mode, or something complicated like a full blown virtual machine - ideally without network connectivity.

The more complicated the sandbox you need the larger risk of a security vulnerability in the sandbox undermining an otherwise good design.

Some languages can safely be compiled outside a sandbox. But there are languages where even compiling them can consume unpredictable amount of resources. This question on a sister site shows some examples of how a small source code can blow up to a large output.

If the compiler itself is free from vulnerabilities it may be sufficient to set limits on the amount of CPU, memory, and disk space it is allowed to consume. For better security you can run the compiler inside a virtual machine.

Obviously these methods can be combined for an additional layer of security. If I were to construct such a system I would probably start a virtual machine and inside the virtual machine use ulimit to limit the resource usage of the compiler. Then I would link the compiled code in a wrapper to run it in secure computing mode. Finally still inside the virtual machine I would run the linked executable.
