I know absolutely nothing in low-level stuff, so this will be a very newbie question. Please excuse my ignorance.

Is machine language - the series of numbers to that tell the physical computer exactly what to do - always binary? I.e. always composed of only zeros one ones? Or could it also be composed of numbers such as 101, 242, 4 etc.?

Everything in a computer (to be precise, in any typical contemporary computer) is binary, at a certain level.  "1s and 0s" is an abstraction, an idea we use to represent a way of distinguishing between two values.  In RAM, that means higher and lower voltage.  On the hard drive, that means distinct magnetic states, and so on.  Using Boolean logic and a base 2 number system, a combination of 1s and 0s can represent any number, and other things (such as letters, images, sounds, etc) can be represented as numbers.

But that's not what people mean when they say "binary code."  That has a specific meaning to programmers:  "Binary" code is code that is not in text form.  Source code exists as text; it looks like a highly formalized system of English and mathematical symbols.  But the CPU doesn't understand English or mathematical notation; it understands numbers.  So the compiler translates source code into a stream of numbers that represent CPU instructions that have the same underlying meaning as the source code.  This is properly known as "machine code," but a lot of people call it "binary".
