All posts

Polite Public Data Collection with Broma

Polite Public Data Collection with Broma

Why polite public data collection matters

When you need to gather publicly available information — price lists, product reviews, public statistics — doing it manually is time‑consuming and error‑prone. Automating the process with Broma frees you from repetitive clicks, but you must stay within the website’s terms of service and avoid overwhelming the server. A polite approach protects your access and keeps the data flow sustainable.

Understanding site terms and ethical scraping

Before you open Broma, read the site’s Terms of Service or robots.txt. Look for clauses that prohibit automated scraping, required request rates, or prohibited data uses. If the site allows scraping but asks for a delay between requests, honor that limit. This ensures you won’t be blocked and maintains a good relationship with the site owner.

Setting up a Broma workflow for data collection

Step 1: Define the target page and data points

  1. Open Broma and create a new workflow.
  2. In the Start action, enter the URL of the page you want to scrape.
  3. List the specific elements you need (e.g., product title, price, rating). Writing them down helps you stay focused during selector creation.

Step 2: Choose robust selectors

Broma lets you pick a selector that survives minor layout changes. Use a combination of CSS class and text content when possible.

  • Primary selector: .product-item .price
  • Fallback selector: span[data-price]

If the class name changes, the fallback will still capture the price.

Step 3: Add polite delays and rate limiting

To respect the site’s rate limits:

  1. After each Click or Read Text action, add a Wait step.
  2. Set the wait time to a value that matches the site’s guidance (e.g., 2‑5 seconds).
  3. For high‑traffic sites, consider a Loop that iterates over a list of product URLs, inserting a longer wait (5‑10 s) between each iteration.

Step 4: Use AI assistance to draft the workflow (optional)

You can describe the task in plain language to an AI agent (your own ChatGPT, Claude, or any provider you subscribe to). The agent can output a step‑by‑step list that you paste into Broma’s Import feature. Review the generated steps, adjust the wait times, and verify the selectors.

Real‑world example: monitoring product reviews on a retail site

Imagine you need the average rating for a set of products on example‑store.com.

  1. Start → URL: https://example-store.com/category/shoes.
  2. Loop → Selector: .product-card (each card contains a product link).
  3. Inside the loop:
    • Click on the product link (selector: a.product-title).
    • Wait 3 seconds.
    • Read Text → Selector: .product-rating (e.g., 4.5).
    • Read Text → Selector: .product-price (e.g., $49.99).
    • Wait 4 seconds.
  4. End Loop.

The workflow now collects rating and price for every shoe listed on the category page.

Code block: short JavaScript expression for price extraction

If you prefer a custom expression, you can use Broma’s Run JavaScript action:

// Extract price, remove currency symbol, convert to number
const priceElem = document.querySelector('.price-class');
const priceText = priceElem ? priceElem.innerText : '';
const numericPrice = priceText.replace(/[^0-9.]/g, '') || 0;
return numericPrice;

Running and monitoring the workflow

Scheduling batch runs

Broma’s built‑in scheduler lets you run the workflow at a specific time or on a recurring basis (e.g., every morning at 08:00).

  1. Open the Scheduler tab.
  2. Choose Recurring → Daily → set the hour and minute.
  3. Select the workflow you created.
  4. Save. The app will launch the browser, execute the steps, and close the session automatically.

Reviewing logs and handling failures

After each run, Broma writes a Run Log that shows:

  • Start/end timestamps
  • Each action with success/failure status
  • Any error messages (e.g., element not found)

To troubleshoot:

  • Open the log, locate the failed step, and verify the selector.
  • If the page layout changed, update the selector or add a more flexible XPath.
  • Re‑run the workflow in a Debug mode to see live highlights.

Maintaining the workflow when pages change

Websites evolve; a selector that worked last week may break tomorrow. Follow these habits:

  • Use multiple selectors (primary + fallback) as shown earlier.
  • Add a Wait for Element before reading text, ensuring the element is present.
  • Run a quick test after any major site update; Broma’s UI highlights missing elements.
  • Version your workflow: duplicate the workflow before making big changes, so you can revert if needed.

Reusing the workflow across multiple accounts (if needed)

If you need to scrape data from several seller accounts on the same site, keep each account in a separate Browser Profile within Broma. Each profile stores its own cookies and login state, preventing cross‑account interference. Create a new workflow for each profile or parameterize the login step.

Troubleshooting common issues

  • Element not found: Check the selector in the browser’s dev tools. Ensure the class name isn’t generated dynamically (use text or attribute instead).
  • Rate‑limit errors: Increase the wait time or add a random jitter (wait 3-6 seconds).
  • Session expiration: Re‑login at the start of the workflow using the Login action, or use a profile that already contains a valid session.
  • Unexpected page layout: Add a Scroll action to bring the target element into view before reading.

Summary & next steps

A polite public data collection workflow in Broma combines three core ideas:

  1. Respect the site’s terms – read policies, honor rate limits, and avoid prohibited actions.
  2. Build resilient selectors – mix class, attribute, and text cues, and add fallback options.
  3. Automate with scheduled runs – use Broma’s scheduler, monitor logs, and adjust waits as the site evolves.

Start by defining a simple scrape (e.g., product titles on a catalog page). Apply the steps above, test, and iterate. As you gain confidence, expand to multi‑page loops, CSV exports, or integrate the data into your analytics dashboard. With Broma’s no‑code interface, you can keep the process reliable without writing a single line of code.