What is the difference between CSPRNG and PRNG? 

Is there performance differential between them? For example: We use PRNG for key generation which is very expensive and CSPRNG for IV/nonce in block ciphers which is fast?

What is the difference between the two types of RNGs?

"PRNG" means "Pseudorandom Number Generator" which means that a sequence of numbers (bits, bytes...) is produced from an algorithm which looks random, but is in fact deterministic (the sequence is generated from some unknown internal state), hence pseudorandom.
Such pseudorandomness can be cryptographically secure, or not. It is cryptographically secure if nobody can reliably distinguish the output from true randomness, even if the PRNG algorithm is perfectly known (but not its internal state). A non-cryptographically secure PRNG would fool basic statistical tests but can be distinguished from true randomness by an intelligent attacker.
For instance, consider the following generator:

There is an internal state s which is a sequence of 20 bytes.
The generator produces a long sequence of bytes by 20-byte chunks.
To produce the next chunk, the algorithm is: output s, then set s to SHA-1(s).

This PRNG will be very good statistically, but it is trivial to distinguish from true randomness: just take two consecutive 20-byte chunks in the output, and see if the second is the result of SHA-1 over the first. This is not a cryptographically secure PRNG.
Of course, every CSPRNG is a PRNG, but not every PRNG is a CSPRNG. Some non-CS PRNG like Xoshiro can achieve quite high a performance and be adequate in non-cryptographic situations where there is no intelligent attacker to defeat (e.g. physics simulations). Although there also are some known high-performance CSPRNG (e.g. these stream ciphers), a non-CS PRNG may give an edge in contexts where the lack of cryptographic security is not an issue.
