Research on Real-time E-commerce Price Comparison System Using Python Web Scraping Technology - Chen, Fan (International Journal of Computer Science and Information Technology, 2024)
This paper presents a study on the development of a real-time price comparison system for e-commerce platforms using Python-based web scraping technologies. The objective is to analyze and compare product prices across major e-commerce platforms, including Taobao, JD.com, and Amazon, by leveraging different web scraping libraries such as BeautifulSoup, Scrapy, and Selenium. The study outlines the technical challenges associated with dynamic content extraction, anti-scraping mechanisms, and the effectiveness of various scraping tools. Through experimental evaluation, we compare the scraping efficiency, success rate, and data accuracy across the three platforms. The results indicate that JD.com, with its mostly static content, offers the highest scraping efficiency when using BeautifulSoup, while Amazon, despite slower page loading, achieves a high success rate using Scrapy. Taobao, on the other hand, presents significant challenges due to its dynamic loading and strict anti-scraping measures, making Selenium the most appropriate tool, albeit with slower processing times. This research provides valuable insights into the selection of optimal web scraping techniques for real-time price comparison systems and highlights the impact of platform-specific challenges on the accuracy and efficiency of data extraction.