FYI, the buffer tool always use the input layer's Coordinate Reference System (CRS) units. In your case, to buffer your points, it will always use WGS84 in degrees (Since WGS84 is a geographic coordinate system, and not a projected one).
Notice, that changing the CRS by using "set layer CRS" does not change their actual values, It only says to QGIS "from now on, read this values as if they are on this different CRS". Wish would tell for instance that "from now on read this degrees as if they are meter", and that won't work well! :-P
Therefore, it's very important that all your layers are set with their correct CRS. If on, QGIS "On the fly transformation" will read layers from different CRSs and display them in the Project chosen CRS. (thats why your layers work together)
The solution is very simple tho, all you need is to reproject you point data to a suitable projected CRS. For what I can see, the RGF93 / Lambert-93 is suitable for your working area.
- Right-click "points" layer, and choose "Save as...";
- Choose "Selected CRS" instead of "Layer CRS";
- Browse for the desired CRS. (typing 'RGF93' on the filter, it will help you to find and select the "RGF93 / Lambert-93" (EPSG:2154) CRS);
- Choose your output shapefile location, select the "add saved file to map" an click Ok.
The resulting layer can now be used as input to buffer tool, and You can now use meters in the buffers distance.
Hope this helps!