How to extract product reviews automatically with AI to analyze pros and cons

This content is generated from the audio voiceover so it may contain errors.

Hello everyone! Welcome to a new video from iSocialWeb. I’m Luis Fernández, and we continue with our series of videos on Artificial Intelligence applied to SEO and websites. In this video, which I believe will be very useful for both agency webmasters and the regular viewers of this channel, as well as for affiliate marketers who may have their own affiliate niches or AdSense sites, we’re going to explore a rather simple script. My aim is to give you an idea that this is doable, it’s easy to accomplish, and you can adapt it to your projects. As always, it’s quite straightforward to program everything, and if you’re not familiar with coding, you can always rely on Chat GPT to assist you in adapting the code to your specific use cases.

What we’re going to do in this case is to scrape reviews from the competition or from a website that isn’t our own. If it’s our own website, we simply can’t download and use the CSV, uploading it directly, and saving ourselves the scraping part. We’re going to use GPT in this case, version 3.5, because it performs perfectly and is much more cost-effective and faster. However, you could use GPT-4 or a future version coming out soon because it’s progressing rapidly. We’ll ask it to summarize the reviews for us. With this, we’ll obtain a list of pros and cons based on real user reviews. This is perfect for creating affiliate articles with product reviews, as well as for expanding product descriptions in our eCommerce store based on user reviews. Often, these reviews are paginated, hidden, or not displayed directly.

It’s also very useful for populating eCommerce stores where we want to add a product that our competitors have, and we need a product description. In this case, we can review our competitors, extract their reviews, and create a description based on real opinions. All of these scenarios can be applied in various ways; there’s always room for customization. I’ll show you the code, and then it’s in your hands to create some really cool stuff. As always, we work with Collage, and the link will be in the video description. I won’t go into detail about the code, but essentially, we’re going to use two libraries: “tiktoken” for counting tokens and setting limits, and “OpenAI.”

Here, we import everything. You need to input your data here; you put your API key here, select the model here, and set the maximum number of tokens. Remember that GPT-3.5, for example, has around 4,048 maximum tokens, if I recall correctly, and you need to leave some room for the response and system input, as mentioned earlier. Next, you enter the URL directly. I chose an eCommerce website that’s fantastic and selected products from guitar sales. As you can see, we scroll down, and we have reviews on Zoman. There are some really detailed reviews; musicians really put a lot of effort into the content. Here, you can choose Zoman for this case. The scraping is set up for Zoman, but you can easily adapt it to Amazon or any other eCommerce platform if you wish, like PCCOMPONENTES or others.

If you’re using your own store, you can skip all this scraping part. It’s marked as scraping here; you can simply upload a CSV or the reviews directly and work with your product’s constraints. Once you have the URL, we’ll set some headers. If you want to do this more professionally, I recommend using proxies and a header generator, or even a library like “Apio” to manage all of this. Another important thing is that if you’re scraping Amazon, it can quickly block you, especially in this case where we’re using BeautifulSoup, which is not the most complex crawler and doesn’t mimic any search engine. If you use Selenium with a couple of proxies, you can scrape whatever you want. Here, we have a couple of utility functions: getting the HTML of the product, the reviews. We get the HTML of the page, which you can see here. It’s just code lines. Keep in mind that this is a crawler and doesn’t generate any kind of browser. If you want to scrape pages with JavaScript, you’ll need a crawler that can render JavaScript. Once you have the rendered or downloaded HTML, you simply scrape it, extract the product and the reviews. As you can see, getting the product name is as simple as grabbing the h1. This will work on most well-made websites because they typically have an h1. If there are multiple h1 tags, we’ll take the first one. As for the reviews, they depend on the specific webpage, and they can vary.

You can use Chat GPT to inquire, especially if you’re not well-versed in programming. Here, you can see all the reviews we’re targeting. I can press F12 or inspect, and I can look through here to find the reviews. We’re capturing this, so there’s no need to dig into the code. We can see that the reviews will come out like this. “Review test” is the text we’re capturing. So, if you notice in the code, we’re looking for a div with the class “reviewtest.” In your example webpage, instead of “review test,” it might be “review” or something similar. You can adapt it. There are different scenarios, but I believe that with a bit of dedication and Chat GPT, you can do this easily. After running all of this, I’ve left some print statements here to clarify what you’re doing. You can see the reviews like these. We’ve extracted four reviews, and this is the product name. Next, we make the typical call. In this case, we’re going to count the number of tokens. These are slightly complex functions, taken from the “tiktoken” documentation. You don’t need to understand them, just know that they return the number of tokens for a string and for messages. What we’re doing is limiting them. I set the default limit to 1,000, but we’re going to use the 2,000 tokens we marked above. We’re limiting it here and generating the prompt. We format the reviews, and this generates a prompt like this: “Summarize the following list of reviews for [product name], indicating the pros and cons based on the following reviews:” Colon. And here’s the list of reviews.

I left this here because often, when scraping a large website like Amazon, you’ll have reviews in multiple languages. You can manually filter them in the code or simply add this and trust that 90% of the time, it will do it correctly and leave it in Spanish. Then we make the call. It’s quite simple. It’s just these five lines of code. We input the model, which we selected above, in this case, GPT-3.5 Turbo. This is the system message, which you can customize to your liking. And a prompt, which I recommend playing around with a bit. I’ve left you with the most basic one here, but you can personalize it by product type, store type, etc. Then we capture the content and the response. This is the response it provides. You can see that for this “Hard Livento in HB 10g,” it says that the price is very affordable, it’s robust and durable, small and lightweight, easy to handle. You can see that it even mentions good service for beginners in the world of guitar. These responses are very real pros and cons. You could generate a list of questions and answers or request recommendations for a buying guide. Once you have all the groundwork, just know that you have to create with those reviews, change the token, not the prompt, and you can do whatever you want. And that’s it. I hope you like it. It’s a pretty powerful tool, and it works very well. If you have any questions, you know, the whole playlist is available on the iSocialWeb YouTube channel, and we’re in the comments to assist you. Greetings, and thank you very much.