Radiofrequency ablation (RFA) represents a convenient, minimally invasive, and cost-effective approach for the treatment of small liver cancers measuring less than 3 cm in diameter. Nonetheless, the existing RFA techniques encounter challenges in precisely controlling the extent of ablation and the risk of overheating, which can lead to damage on the surrounding tissues. The ability to control the ablation area also plays a crucial role in the success of RFA procedures. To address these issues, we introduce a novel method that utilizes tumor boundary temperature monitoring to achieve precise control over the RFA process. Through the utilization of COMSOL Multiphysics simulation software, the proposed method was verified by comprehensive simulation and modeling and its efficacy in achieving regional control was testified. Subsequently, ex vivo experiments were conducted employing a custom-designed cooling RFA instrument. The experimental results demonstrated that the RFA controlled by boundary temperature yielded an ablation area with a diameter approaching 30 mm. In comparison to the standard spherical solidification zone (characterized by both the ellipticity and regularity indices of 1), the cooling RFA discoloration zone, under boundary temperature control, exhibited a maximum deviation of 7% in the ellipticity index and 17% in the regularity index. The average ellipticity index was determined to be 0.96, while the average regularity index was 0.86. Collectively, these findings underscore the capability of the proposed temperature-controlled cooling RFA method to attain precise control over the ablation area, contributing to comprehensive ablation of tumor tissue within the target region.