---
title: Ingest data from a Worker, and analyze using MotherDuck · Cloudflare Pipelines Docs
description: In this tutorial, you will learn how to ingest clickstream data to a R2 bucket using Pipelines. You will use the Pipeline binding to send the clickstream data to the R2 bucket from your Worker. You will also learn how to connect the bucket to MotherDuck. You will then query the data using MotherDuck.
source_url:
  html: https://6604c8ce.preview.developers.cloudflare.com/pipelines/tutorials/query-data-with-motherduck/
  md: https://6604c8ce.preview.developers.cloudflare.com/pipelines/tutorials/query-data-with-motherduck/index.md
---

In this tutorial, you will learn how to ingest clickstream data to a [R2 bucket](https://developers.cloudflare.com/r2) using Pipelines. You will use the Pipeline binding to send the clickstream data to the R2 bucket from your Worker. You will also learn how to connect the bucket to MotherDuck. You will then query the data using MotherDuck.

For this tutorial, you will build a landing page of an e-commerce website. A user can click on the view button to view the product details or click on the add to cart button to add the product to their cart.

## Prerequisites

1. A [MotherDuck](https://motherduck.com/) account.
2. Install [`Node.js`](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm).

## 1. Create a new project

You will create a new Worker project that will use [Static Assets](https://developers.cloudflare.com/workers/static-assets/) to serve the HTML file.

Create a new Worker project by running the following commands:

* npm

  ```sh
  npm create cloudflare@latest -- e-commerce-pipelines
  ```

* pnpm

  ```sh
  pnpm create cloudflare@latest e-commerce-pipelines
  ```

* yarn

  ```sh
  yarn create cloudflare e-commerce-pipelines
  ```

For setup, select the following options:

* For *What would you like to start with?*, choose `Hello World Starter`.
* For *Which template would you like to use?*, choose `SSR / full-stack app`.
* For *Which language do you want to use?*, choose `TypeScript`.
* For *Do you want to use git for version control?*, choose `Yes`.
* For *Do you want to deploy your application?*, choose `No` (we will be making some changes before deploying).

Navigate to the `e-commerce-pipelines` directory:

```sh
cd e-commerce-pipelines
```

## 2. Update the frontend

Using Static Assets, you can serve the frontend of your application from your Worker. The above step creates a new Worker project with a default `public/index.html` file. Update the `public/index.html` file with the following HTML code:

The above code does the following:

* Uses Tailwind CSS to style the page.
* Renders a list of products.
* Adds a button to view the details of a product.
* Adds a button to add a product to the cart.
* Contains a `handleClick` function to handle the click events. This function logs the action and the product ID. In the next steps, you will add the logic to send the click events to your pipeline.

## 3. Generate clickstream data

You need to send clickstream data like the `timestamp`, `user_id`, `session_id`, and `device_info` to your pipeline. You can generate this data on the client side. Add the following function in the `<script>` tag in your `public/index.html`. This function gets the device information:

```js
function extractDeviceInfo(userAgent) {
  let browser = "Unknown";
  let os = "Unknown";
  let device = "Unknown";


  // Extract browser
  if (userAgent.includes("Firefox")) {
    browser = "Firefox";
  } else if (userAgent.includes("Chrome")) {
    browser = "Chrome";
  } else if (userAgent.includes("Safari")) {
    browser = "Safari";
  } else if (userAgent.includes("Opera") || userAgent.includes("OPR")) {
    browser = "Opera";
  } else if (userAgent.includes("Edge")) {
    browser = "Edge";
  } else if (userAgent.includes("MSIE") || userAgent.includes("Trident/")) {
    browser = "Internet Explorer";
  }


  // Extract OS
  if (userAgent.includes("Win")) {
    os = "Windows";
  } else if (userAgent.includes("Mac")) {
    os = "MacOS";
  } else if (userAgent.includes("Linux")) {
    os = "Linux";
  } else if (userAgent.includes("Android")) {
    os = "Android";
  } else if (userAgent.includes("iOS")) {
    os = "iOS";
  }


  // Extract device
  const mobileKeywords = [
    "Android",
    "webOS",
    "iPhone",
    "iPad",
    "iPod",
    "BlackBerry",
    "Windows Phone",
  ];
  device = mobileKeywords.some((keyword) => userAgent.includes(keyword))
    ? "Mobile"
    : "Desktop";


  return { browser, os, device };
}
```

Next, update the `handleClick` function to make a `POST` request to the `/api/clickstream` endpoint with the data:

```js
async function handleClick(action, id) {
  console.log(`Clicked ${action} for product with id ${id}`);


  const userAgent = window.navigator.userAgent;
  const timestamp = new Date().toISOString();
  const { browser, os, device } = extractDeviceInfo(userAgent);


  const data = {
    timestamp,
    session_id: "1234567890abcdef", // For production use a unique session ID
    user_id: "user" + Math.floor(Math.random() * 1000), // For production use a unique user ID
    event_data: {
      event_id: Math.floor(Math.random() * 1000),
      event_type: action,
      page_url: window.location.href,
      timestamp,
      product_id: id,
    },
    device_info: {
      browser,
      os,
      device,
      userAgent,
    },
    referrer: document.referrer,
  };
  try {
    const response = await fetch("/api/clickstream", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ data }),
    });
    if (!response.ok) {
      throw new Error("Failed to send data to pipeline");
    }
  } catch (error) {
    console.error("Error sending data to pipeline:", error);
  }
}
```

The `handleClick` function does the following:

* Gets the device information using the `extractDeviceInfo` function.
* Creates a `POST` request to the `/api/clickstream` endpoint with the data.
* Logs any errors that occur.

## 4. Create an R2 bucket

You will create a new R2 bucket to use as the sink for our pipeline. Create a new r2 bucket `clickstream-data` using the [Wrangler CLI](https://developers.cloudflare.com/workers/wrangler/):

```sh
npx wrangler r2 bucket create clickstream-data
```

## 5. Create a pipeline

You need to create a new pipeline and connect it to the R2 bucket you created in the previous step.

Create a new pipeline `clickstream-pipeline` using the [Wrangler CLI](https://developers.cloudflare.com/workers/wrangler/):

```sh
npx wrangler pipelines create clickstream-pipeline --r2-bucket clickstream-data --compression none --batch-max-seconds 5
```

When you run the command, you will be prompted to authorize Cloudflare Workers Pipelines to create R2 API tokens on your behalf. These tokens are required by your Pipeline. Your Pipeline uses these tokens when loading data into your bucket. You can approve the request through the browser link which will open automatically.

Note

The above command creates a pipeline using two optional flags: `--compression none`, and `--batch-max-seconds 5`.

With these flags, your pipeline will deliver an uncompressed file of data to your R2 bucket every 5 seconds.

These flags are useful for testing, but we recommend keeping the default settings in a production environment.

```output
✅ Successfully created Pipeline "clickstream-pipeline" with ID <PIPELINE_ID>


Id:    <PIPELINE_ID>
Name:  clickstream-pipeline
Sources:
  HTTP:
    Endpoint:        https://<PIPELINE_ID>.pipelines.cloudflare.com
    Authentication:  off
    Format:          JSON
  Worker:
    Format:  JSON
Destination:
  Type:         R2
  Bucket:       clickstream-data
  Format:       newline-delimited JSON
  Compression:  NONE
  Batch hints:
    Max bytes:     100 MB
    Max duration:  300 seconds
    Max records:   10,000,000


🎉 You can now send data to your Pipeline!


Send data to your Pipeline's HTTP endpoint:


curl "https://<PIPELINE_ID>.pipelines.cloudflare.com" -d '[{"foo": "bar"}]'
```

## 5. Send clickstream data to your pipeline

You have setup the frontend of your application to make a call to the `/api/clickstream` route, everytime the user clicks on one of the buttons. The application makes a call to the `/api/clickstream` endpoint to send the clickstream data to your pipeline. The `/api/clickstream` endpoint is handled by a Worker in the `src/index.ts` file.

You will use the pipelines binding to send the clickstream data to your pipeline. In your `wrangler` file, add the following bindings:

* wrangler.jsonc

  ```jsonc
  {
    "pipelines": [
      {
        "binding": "PIPELINE",
        "pipeline": "clickstream-pipeline"
      }
    ]
  }
  ```

* wrangler.toml

  ```toml
  [[pipelines]]
  binding = "PIPELINE"
  pipeline = "clickstream-pipeline"
  ```

Next, update the type in the `worker-configuration.d.ts` file. Add the following type:

```ts
interface Env {
  PIPELINE: Pipeline;
}
```

Update the `src/index.ts` file to handle the `POST` request:

```ts
export default {
  async fetch(request, env, ctx): Promise<Response> {
    const pathname = new URL(request.url).pathname;
    const method = request.method;
    if (pathname === "/api/clickstream" && method === "POST") {
      const body = (await request.json()) as { data: any };
      try {
        await env.PIPELINE.send([body.data]);
        return new Response("OK", { status: 200 });
      } catch (error) {
        console.error(error);
        return new Response("Internal Server Error", { status: 500 });
      }
    }
    return new Response("Hello World!");
  },
} satisfies ExportedHandler<Env>;
```

The `src/index.ts` file does the following:

* Checks if the request is a `POST` request to the `/api/clickstream` endpoint.
* Extracts the data from the request body.
* Sends the data to your Pipeline.
* Returns a response to the client.

## 6. Deploy the application

You can run the local server to the test the application. However, the data will be not be sent to the production pipeline. You will need to deploy the application to send the data to the pipeline.

To start the development server execute the below command and naviagte to `http://localhost:8787`:

```sh
npm run dev
```

When you click on the `View Details` or the `Add to Cart` button, the `handleClick` function calls the `/api/clickstream` endpoint. This endpoint uses the pipelines bindinding and sends the clickstream data. Note that no actual data is sent to the pipeline when running the application locally.

To deploy the application, run the following command:

```sh
npm run deploy
```

This will deploy the application to the Cloudflare Workers platform.

```output
🌀 Building list of assets...
🌀 Starting asset upload...
🌀 Found 1 new or modified static asset to upload. Proceeding with upload...
+ /index.html
Uploaded 1 of 1 assets
✨ Success! Uploaded 1 file (2.37 sec)


Total Upload: 25.73 KiB / gzip: 6.17 KiB
Worker Startup Time: 15 ms
Uploaded e-commerce-clickstream (11.79 sec)
Deployed e-commerce-clickstream triggers (7.60 sec)
  https://<URL>.workers.dev
Current Version ID: <VERSION_ID>
```

You can access the application at the URL provided in the output of the command. Now when you click on the `View Details` or `Add to Cart` button, the clickstream data will be sent to your pipeline.

## 7. Connect the R2 bucket to MotherDuck

Your application sends clickstream data to the R2 bucket using pipelines. In this step, you will connect the R2 bucket to MotherDuck.

You can connect the bucket to MotherDuck in several ways, which you can learn about from the [MotherDuck documentation](https://motherduck.com/docs/integrations/cloud-storage/cloudflare-r2/). In this tutorial, you will connect the bucket to MotherDuck using the MotherDuck dashboard.

Before connecting the bucket to MotherDuck, you need to obtain the Access Key ID and Secret Access Key for the R2 bucket. You can find the instructions to obtain the keys in the [R2 API documentation](https://developers.cloudflare.com/r2/api/tokens/).

1. Log in to the MotherDuck dashboard and select your profile.

2. Navigate to the **Secrets** page.

3. Select the **Add Secret** button and enter the following information:

   * **Secret Name**: `Clickstream pipeline`
   * **Secret Type**: `Cloudflare R2`
   * **Access Key ID**: `ACCESS_KEY_ID` (replace with the Access Key ID)
   * **Secret Access Key**: `SECRET_ACCESS_KEY` (replace with the Secret Access Key)

4. Select the **Add Secret** button to save the secret.

## 8. Query the data

In this step, you will query the data stored in the R2 bucket using MotherDuck.

1. Navigate back to the MotherDuck dashboard and select the **+** icon to add a new Notebook.

2. Select the **Add Cell** button to add a new cell to the notebook.

3. In the cell, enter the following query and select the **Run** button to execute the query:

```sql
SELECT * FROM read_json_auto('r2://<BUCKET_NAME>/**/*');
```

Replace the `<BUCKET_NAME>` placeholder with the name of the R2 bucket.

The query will return the data stored in the R2 bucket.

## Conclusion

You have successfully built an e-commerce application that leverages Cloudflare's Pipelines, R2, and Workers. Through this tutorial, you've gained hands-on experience in:

1. Creating a Workers project with a static frontend
2. Generating and capturing clickstream data
3. Setting up a Cloudflare Pipelines to ingest data into R2
4. Connecting your R2 bucket to MotherDuck for advanced querying capabilities

This project serves as a foundation for building scalable, data-driven applications. You can now expand on this knowledge to create more complex e-commerce features, implement advanced analytics, or explore other Cloudflare products to enhance your application further.

For your next steps, consider exploring more advanced querying techniques with MotherDuck, implementing real-time analytics, or integrating additional Cloudflare services to further optimize your application's performance and security.

You can find the source code of the application in the [GitHub repository](https://github.com/harshil1712/cf-pipelines-bindings-demo).
