Browser Automation with Google Gemini Model to run UI Tests

Browser Automation Using Google Gemini to run UI Test

🔧 Prerequisites

• Python 3.8 or above
• Visual Studio Code or any Python IDE
• Internet connection to download models and dependencies

📦 Installation Steps

1. Install browser-use

pip install browser-use

GitHub: https://github.com/browser-use/browser-use

2. Install Chromium browser with Playwright

playwright install chromium --with-deps --no-shell

3. Add your API keys to a .env file

OPENAI_API_KEY=your_openai_key_here

(Gemini/Gemma public model doesn't require a token.)

🤖 Example: AI Agent Running a Selenium Script

This example uses Gemini model to generate a Selenium script that runs a test in headful mode and interacts with your blog.

from smolagents import CodeAgent, DuckDuckGoSearchTool, InferenceClientModel
agent = CodeAgent(
    tools=[DuckDuckGoSearchTool()],
    model=InferenceClientModel(model_id="google/gemma-2b-it"),
    additional_authorized_imports=[
        "selenium",
        "selenium.webdriver",
        "selenium.webdriver.common.by",
        "selenium.webdriver.common.keys",
        "selenium.webdriver.support.ui",
        "selenium.webdriver.support.expected_conditions",
        "webdriver_manager.chrome"
    ]
)
agent.run("write selenium code to test to run in headful mode for 

https://softwaretesting-guideline.blogspot.com/ and click on different posts")

The resulting execution demonstrates how the AI agent parses the site's DOM, 

detects all hyperlink elements, and programmatically simulates user interactions 

by clicking on various blog post links.

✅ Benefits

• No manual code writing — just describe your intent
• Perfect for test engineers and QA automation
• Supports flexible tools and models

⚠️ Tips

• Make sure Playwright is installed correctly and compatible with your OS
• If using OpenAI or HuggingFace models, store API keys securely

🎯 Conclusion

By combining SmolAgents, Gemini model, and Selenium, you can create powerful browser-based test automation flows with minimal effort. AI-powered agents are the future of QA automation!

Happy Testing 🚀

Build an AI Agent with SmolAgents to Control Selenium Browser Automation

AI Agent Controlling Browser Using Selenium with SmolAgents

Artificial Intelligence is rapidly transforming how we automate web tasks. In this tutorial, we will explore how to control a browser using Selenium through a Python AI agent created using SmolAgents. We'll also discuss how to handle import restrictions by allowing additional modules explicitly.

🚀 What Is SmolAgents?

As I describe in my previous post, SmolAgents is a lightweight, open-source Python framework created by Hugging Face. It enables you to build smart autonomous agents that can interact with external tools, models, and libraries like selenium, duckduckgo, and more.

🧰 Prerequisites

1. Python Installed (Recommended: 3.11 or 3.12) 
2. VS Code or any Python IDE
3. Install SmolAgents:

   python -m pip install "smolagents[openai]"

4. Install Selenium & ChromeDriver Manager:

   pip install selenium webdriver-manager

⚠️ Issue with Unauthorized Imports

By default, SmolAgents restricts importing some external libraries such as selenium. This is for safety reasons. But in our use case, we can safely allow it using the additional_authorised_import parameter when initializing the agent.

💡 AI Agent Code Using Selenium

In the below code, we explicitly authorize the necessary Selenium imports and ask the agent to write code to open our blog https://softwaretesting-guideline.blogspot.com and click on different posts in headful mode.

from smolagents import CodeAgent, DuckDuckGoSearchTool, HfApiModel

agent = CodeAgent(
    tools=[DuckDuckGoSearchTool()],
    model=HfApiModel(),
    additional_authorised_import=[
        "selenium",
        "selenium.webdriver",
        "selenium.webdriver.common.keys",
        "webdriver_manager.chrome"
    ]
)

agent.run("write selenium code to test to run in headful mode for 

https://softwaretesting-guideline.blogspot.com/ and click on different posts")

📝 What This Code Does

• Creates a CodeAgent with authorized imports
• Uses a search tool for reasoning
• Requests the AI to generate a Selenium script to interact with your blog

🧪 Sample Output

The agent typically returns code like the following:

from selenium import webdriver
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import time

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://softwaretesting-guideline.blogspot.com/")
time.sleep(5)

# Click on all post links
links = driver.find_elements(By.CSS_SELECTOR, "h3.post-title a")
for link in links:
    print("Opening post:", link.text)
    link.click()
    time.sleep(3)
    driver.back()
    time.sleep(2)

driver.quit()

📌 Key Points

• Running in headful mode lets you visually see the browser actions
• You can modify the logic to open specific blog posts or interact with elements like labels, buttons, or images
• Use headless mode for automation pipelines by setting Chrome options accordingly

🔗 Resources

• SmolAgents GitHub
• Selenium Official Website

Conclusion: Using AI agents to control browsers opens up exciting new possibilities for intelligent test automation and web scraping. With SmolAgents and Selenium, you can offload even complex test tasks to an autonomous Python agent. Learn AI Agents with SmolAgents in Python for Web Automation

Integrating Selenium with Java and OpenAI API – Intelligent Automation for QA Engineers

🧠 Integrate Selenium with Java and OpenAI API – Full Guide

This guide shows how to use Selenium for web automation with OpenAI’s Java SDK. Learn how to retrieve your API key, manage it securely, and write a test script that interacts with OpenAI and a web page.

🔑 How to Get Your OpenAI API Key

1. Go to https://platform.openai.com/signup and sign up or log in.
2. Once logged in, click on your profile (top-right) > API keys.
3. Click Create new secret key and copy the key securely.

⚠️ Do not share your API key publicly. Store it safely using a .env file as explained below.

📦 Maven Project Setup

In your Java Maven project’s pom.xml, add these dependencies:

<dependencies>
  <!-- Selenium WebDriver -->
  <dependency>
    <groupId>org.seleniumhq.selenium</groupId>
    <artifactId>selenium-java</artifactId>
    <version>4.8.0</version>
  </dependency>

  <!-- OpenAI Java Client (Theo Kanning's library) -->
  <dependency>
    <groupId>com.theokanning.openai-gpt3-java</groupId>
    <artifactId>client</artifactId>
    <version>0.18.2</version>
  </dependency>

  <!-- dotenv-java to manage environment variables -->
  <dependency>
    <groupId>io.github.cdimascio</groupId>
    <artifactId>dotenv-java</artifactId>
    <version>5.2.2</version>
  </dependency>
</dependencies>

🔐 Step: Setup .env File

Create a file named .env in your project’s root folder with this content:

OPENAI_API_KEY=your_openai_api_key_here

Also add .env to your .gitignore file to keep your API key private.

💻 Real-World Java Example with OpenAI + Selenium

This example uses Selenium to open a flight search website and OpenAI to generate a user tip for filling out the form.

import io.github.cdimascio.dotenv.Dotenv;
import com.theokanning.openai.service.OpenAiService;
import com.theokanning.openai.completion.CompletionRequest;
import com.theokanning.openai.completion.CompletionChoice;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class FlightAutomationExample {
    public static void main(String[] args) {
        // Load environment variables
        Dotenv dotenv = Dotenv.load();
        String apiKey = dotenv.get("OPENAI_API_KEY");

        // Initialize OpenAI
        OpenAiService service = new OpenAiService(apiKey);

        // Use OpenAI to generate a flight booking tip
        CompletionRequest request = CompletionRequest.builder()
            .prompt("Give one important tip for booking flights online.")
            .model("text-davinci-003")
            .maxTokens(100)
            .build();

        CompletionChoice choice = service.createCompletion(request).getChoices().get(0);
        System.out.println("OpenAI Tip: " + choice.getText());

        // Launch browser with Selenium
        WebDriver driver = new ChromeDriver();
        driver.get("https://www.example.com/flights");

        // You can automate actions here, e.g., filling a form, clicking search, etc.

        driver.quit();
    }
}

✅ Summary

• 💡 Use OpenAI to generate smart content or logic in your automation tests.
• 🧪 Use Selenium for browser testing or automation steps.
• 🔐 Secure API keys using dotenv-java in your Maven project.

By combining these technologies, you can bring intelligence to your automation framework, such as generating test data, writing user prompts, or analyzing UI interactions.

---

If you found this helpful, share it with fellow QA engineers or developers!