Or does the mantle and crust above you counteract the increase at one point and it actually decreases?
The below figure, taken from Wikipedia shows a model of the free fall acceleration, i.e., 'gravity'. The left-most point corresponds to the center of the Earth; then further right at $6.3\cdot1000$ km you are at the Earth's surface; and then further out you move into space. You can follow the blue line for PREM to get an idea of the average (expected) gravity. As you see, the gravity actually increases slightly within the Earth (reaching a maximum at the core-mantle boundary), but tapers down within the core.

To make this kind of calculations, you must think of the Earth like an onion: made up of many concentric spheres. Whenever you move a bit deeper into the Earth, you strip off all the layers you've crossed. As you get closer to the center of the Earth, there are fewer and fewer layers, and eventually, there's nothing left at the center! The reason why gravity goes up ever so slightly within the Earth is that you get close to the much denser core material. If the density of the Earth were constant (per the green 'constant density' line), the gravity would just decrease linearly.



See the other answer and the discussion below for some more details on the math and procedures required to make these calculations.
