A new proton conductor, RbMg1−xH2x(PO3)3·yH2O, was synthesized by the coprecipitation method followed by sintering at 540 K. The range of solid solution for RbMg1−xH2x(PO3)3·yH2O was 0.00 < x < 0.18 and the highest conductivity of 5.5 × 10−3 S cm−1 was observed at 443 K for the composition of x = 0.11. The structure of RbMg1−xH2x(PO3)3·yH2O was determined from 298 to 553 K by Rietveld refinement with multi-profile analysis using X-ray and neutron diffraction data. The framework structure is composed of PO4 tetrahedra connected with each other by corner-sharing to form spiral PO4 chains along the c-direction. One-dimensional tunnels are formed between these spiral chains, in which water molecules are located. The water molecules form one-dimensional spiral chains and are connected to the spiral PO4 chain by hydrogen bonding. The introduction of acidic, hydrophilic head groups (–PO3H) into the PO4 framework by the formation of a solid solution provides binding sites for water and an environment for the efficient diffusion of protons. One-dimensional proton diffusion, similar to proton channels in biological systems, could be explained by the vehicle mechanism of H3O+ along the one-dimensional water chain.