Custom Log Integration with Website Analytics Setup Guide

Overview

The Custom Log Integration allows you to send application and server logs directly to Search AI Website Analytics using a structured JSON format. This setup helps Search AI Website Analytics process and analyze traffic data from your infrastructure while maintaining consistent log validation and ingestion.

Prerequisites

Before setting up the integration, ensure you have:

Your Search AI Website Analytics Key and Endpoint from the Birdeye dashboard
The ability to send HTTP POST requests from your application, server, or logging infrastructure
Access to modify your application or server logging configuration

Configuration

Before configuring custom log integration, make sure Website Analytics is already set up in Birdeye.

To learn how to configure Website Analytics, refer to the article: How to Configure Website Analytics and View Setup Guides.

You can then use the setup details provided in the Birdeye dashboard to complete the custom log integration.

API Specification

Use the Search AI Website Analytics ingestion endpoint to send custom application or server logs to Birdeye.

Endpoint

Send logs using an HTTP POST request to the Search AI Website Analytics ingestion endpoint.

Example:

</> Bash
POST http://searchai-analytics-ingestor.birdeye.internal/api/v1/ingest/custom

Authentication

Include your Search AI Website Analytics Key in the request headers for authentication.

Example:

{
"X-API-Key: bot_xxxxxxxxxxxxxxxxxxxxx"
}

All requests sent to the ingestion endpoint must include a valid API key to ensure secure log delivery.

Request Format

The request body must be a JSON object representing a single log entry.

Request Body

The request body should include the required fields listed below. Optional fields can be included to provide additional context for reporting and analysis.

Required Fields

Field	Type	Description	Example
method	String	HTTP request method	GET, POST, PUT, DELETE
host	String	Hostname that received the request	api.example.com
path	String	Request path	/v1/users/123
status_code	Integer	HTTP response status code	200, 404, 500

Optional Fields

Field	Type	Description	Example
timestamp	Long	Event timestamp in milliseconds since Unix epoch	1748550600000
ip	String	Client IP address	192.168.1.10
user_agent	String	User-Agent header value	Mozilla/5.0
referer	String	Referring URL	https://google.com
bytes_sent	Long	Number of response bytes sent	1024
duration_ms	Long	Request processing duration in milliseconds	150

Field Validation Rules

Field	Validation
method	Required, cannot be null, empty, or blank
host	Required, cannot be null, empty, or blank
path	Required, cannot be null, empty, or blank
status_code	Required, cannot be null
bytes_sent	Optional, but must be ≥ 0 when provided
duration_ms	Optional, but must be ≥ 0 when provided

NOTE:

Optional fields such as referer, user_agent, bytes_sent, and duration_ms provide additional context that can improve reporting and analysis within Search AI Website Analytics.

Each request can include up to 5 MB of log data. For larger datasets, split the logs across multiple requests to maintain optimal ingestion performance.

Implementation

Step 1: Format Your Log Data

Prepare your log data in the JSON structure supported by Search AI Website Analytics. Each request can include one or multiple log entries in an array format.

Example:

JSON
[
 {
   "timestamp": 1748550600000,
    "method": "GET",
    "host": "example.com",
    "path": "/products",
    "status_code": 200,
    "ip": "192.168.1.1",
    "user_agent": "Mozilla/5.0...",
     "query_params": {
      "category": "electronics",
       "page": "1"
   },
   "referer": "https://example.com",
   "bytes_sent": 1024,
   "duration_ms": 150
  }
]

Make sure the request includes all required fields, such as the request method, hostname, request path, and response status code. When provided, the timestamp field must use Unix epoch time in milliseconds (UTC). Optional fields such as client IP, user agent, referer, response size, and request duration can be included for additional analytics insights.

Step 2: Send the Log Data to the Endpoint

After formatting the logs, send the data to the Website Analytics ingestion endpoint using an HTTP POST request.

The following Python example demonstrates how to send log data using your Website Analytics endpoint and API key:

Python
import requests
  
url = "http://searchai-analytics-ingestor.birdeye.internal/api/v1/ingest/custom"  
headers = {     
  "X-API-Key": "bot_xxxxxxxxxxxxxxxxxxxxx",     
  "Content-Type": "application/json" 
}  

payload = [     
  {         
    "method": "GET",         
    "host": "example.com",         
    "path": "/insights-team-id-28May/1",         
    "status_code": 200,         
    "ip": "192.168.1.10",         
    "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 26_4_2 like Mac OS
                  X)",             
    "timestamp": 1748550600000,         
    "bytes_sent": 1024     
   } 
]  

response = requests.post(     
           url,     
           headers=headers,     
           json=payload,     
           timeout=30 
)  

print("Status Code:", response.status_code)

This request sends the log payload directly to Search AI Website Analytics for processing and analysis.

You can also send log data using cURL:

curl --location 'http://searchai-analytics-ingestor.birdeye.internal/api/v1/ingest/custom' \ 
--header 'X-API-Key: bot_xxxxxxxxxxxxxxxxxxxxx' \ 
--header 'Content-Type: application/json' \ 
--data '[   
 {     
  "method": "GET",     
  "host": "example.com",     
  "path": "/insights-team-id-28May/1",     
  "status_code": 200,     
  "ip": "192.168.1.10",     
  "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 26_4_2 like Mac OS         
                 X)",    
  "timestamp": 1748550600000,     
  "bytes_sent": 1024   
 } 
]'

Step 3: API Response

When a request is successfully accepted for processing, the API returns:

HTTP/1.1 202 Accepted

The request has been successfully received and queued for asynchronous processing.

Response Codes

Status Code	Description
202	Request accepted for processing
400	Invalid request payload
401	Invalid or missing API key
403	Access denied
429	Rate limit exceeded
500	Internal server error

Error Handling

Search AI Website Analytics validates incoming log data and returns detailed error information when issues are detected.

Validation Errors

For validation-related issues such as invalid timestamps, malformed payloads, or unsupported status codes, the request may be rejected and require correction before resubmission.

Unexpected Errors

For unexpected issues such as server failures or temporary processing interruptions, the API may reject the request.

In such cases, the API will:

Return a 500 status code
Reject the current batch request until the issue is resolved

Best Practices

To improve reliability and performance when sending log data to the Search AI Website Analytics, Birdeye recommends the following best practices:

Send logs in batches of up to 5 MB per request
Implement retry handling for failed requests
Use asynchronous or background log processing when possible
Minimize performance impact by sending logs outside the main request flow
Consider temporary buffering for high-volume traffic
Enable compression for larger payloads

Security Considerations

When configuring your custom integration, follow these security recommendations:

Store Website Analytics Keys securely
Rotate API keys periodically
Monitor request activity for unusual behavior
Use HTTPS endpoints for production API requests