  We perform a joint analysis of the abundance, the clustering and the
galaxy-galaxy lensing signal of galaxies measured from Data Release 11 of the
Sloan Digital Sky Survey III Baryon Oscillation Spectroscopic Survey (SDSS
III-BOSS) in our companion paper, Miyatake et al. (2014). The lensing signal
was obtained by using the shape catalog of background galaxies from the Canada
France Hawaii Telescope Legacy Survey, which was made publicly available by the
CFHTLenS collaboration, with an area overlap of about 105 deg$^2$. We analyse
the data in the framework of the halo model in order to fit halo occupation
parameters and cosmological parameters ($\Omega_{\rm m}$ and $\sigma_8$) to
these observables simultaneously, and thus break the degeneracy between galaxy
bias and cosmology. Adopting a flat $\Lambda$CDM cosmology with priors on
$\Omega_b h^2$, $n_{\rm s}$ and $h$ from the analysis of WMAP 9-year data, we
obtain constraints on the stellar mass-halo mass relation of galaxies in our
sample. Marginalizing over the halo occupation distribution parameters and a
number of other nuisance parameters in our model, we obtain $\Omega_{\rm
m}=0.310^{+0.019}_{-0.020}$ and $\sigma_8=0.785^{+0.044}_{-0.044}$ (68%
confidence). We demonstrate the robustness of our results with respect to
sample selection and a variety of systematics such as the halo off-centering
effect and possible incompleteness in our sample. Our constraints are
consistent, complementary and competitive with those obtained using other
independent probes of these cosmological parameters. The cosmological analysis
is the first of its kind to be performed at a redshift as high as $0.53$.
