Contents x
    Enabling Data View from BigQuery
    • 10 Nov 2023
    • Print
    • Dark
      Light
    Contents

    Enabling Data View from BigQuery

    • Updated on 10 Nov 2023
    • Print
    • Dark
      Light
    Article Summary

    The following section describes a way to get access to traffic data as a view in BigQuery. For more information please contact support or your CS representative.

    Prerequisite

    To gain access to the Malicious Traffic Data View you would need the following:

    • A Google Cloud Platform account with access to BigQuery
    • A list of user emails with Google Cloud access

    View Types

    We currently support the following data types as views:

    • Requests - All the requests that are intercepted to the customer’s website (including risk)
    • PageViews - All requests made to the same page by the same user in a single browsing session count as one page view.

    Requests Fields

    Field NameDescription

    timestamp

    Time of the request - UTC

    app_id

    HUMAN app ID assigned per application

    account_name

    Customer account name

    application_name

    HUMAN app name

    ip

    IP the request came from

    is_mobile_sdk

    If the request originated in mobile device = True

    vid

    HUMAN visitor unique identifier

    human_client_uuid

    Page view identifier designated by HUMAN 

    location_href

    Full URL of the request (including domain, request params etc.)

    header_referer

    The previous page the request came from (the page that led to this request)

    domain

    Parent domain for the request as derived from location href

    path

    Path of the request (where was the request to within the domain)

    http_method

    HTTP Method (POST, GET, HEAD, etc)

    asn_org_name

    ISP provider for the request original IP

    asn_number

    ISP provider number for the request original IP

    ua

    User Agent string the request came from

    os_version

    Version of operating system used in the request

    os_family

    Type of operating system used in the request

    browser_version

    Version of browser used

    browser_family

    Type of browser used

    country

    Country the request came from

    filter_type

    Indicating if the request is classified as "always deny" or "always allow"

    filter_origin_reason

    The reason the origin was filtered (e.g. Spot.IM Crawler)

    risk_score

    Score given to request estimating likelihood of the request originating from bot traffic
     Range 0 (most likely human) to 100 (most likely bot)

    known_bot

    If the request came from a known bot = True

    socket_ip_classification

    Any known classifications/ characteristics we might have for the original IP. e.g: [{"name":"Google Cloud","class":"Cloud"}]

    traffic_source

    returns web or mobile

    incident_types

    incident types

    blocked

    True if the request was blocked

    simulated_block

    True if the request was blocked in monitor mode

    custom_param1-9

    Custom parameters as defined by the customer

    Pageviews Fields

    Field NameDescription

    timestamp

    Time of the request - UTC

    app_id

    HUMAN app id assigned per application

    account_name

    Customer account name

    application_name

    Customer application name

    ip

    IP the request came from

    is_mobile_sdk

    True if the request originated in mobile device

    human_client_uuid

    Page view identifier designated by HUMAN 

    http_method

    HTTP Method (POST, GET, HEAD, etc)

    location_href

    Full URL of the request (including domain, request params etc.)

    domain

    Parent domain for the request as derived from location href

    path

    Path of the request (where was the request to within the domain)

    ip_classification

    Any known classifications/ characteristics we might have for the original ip

    ua

    User agent string the request came from

    os_version

    Version of operating system used in the request

    os_family

    Type of operating system used in the request

    browser_version

    Version of the browser used

    browser_family

    Type of the browser used

    country

    Country the request came from

    incident_types

    incident types

    filter_type

    Indicating if the request is classified as "always deny" or "always allow"

    captchas_presented

    Number of Captchas shown to the user

    captchas_solved

    Number of Captchas solved by the user

    known_bot

    If the request came from a known bot = True

    blocked

    If the request was blocked = True

    score

    Score given to request estimating likelihood of the request originating from bot traffic
     Range 0 (most likely human) to 100 (most likely bot)

    custom_param1-9

    Custom parameters as defined by the customer

    Data Retention

    The data retrieved within the Big Query view is from the last 14 days

    Setting up the Big Query View

    Once HUMAN has created the view, and your Google account has the necessary permissions, you will be able to use it as follows:

    1. First time only
      1. Add our project to your view.
      2. Press ADD DATA (next to Resources)
      3. Click Pin a project and enter the project name.
      4. Click enter\pin to confirm.
    2. Once the project is pinned, you will be able to click on it and see the dataset (your company name) and inside the view (named requests or pageviews)
    3. Now click QUERY VIEW to open a base query.
      We recommend limiting your search by adding a condition on the query (e.g. WHERE Date(timestamp) > 'XXXX-XX-XX').
      If you receive a quota limit message for a user in your organization, try querying fewer days.
    Was this article helpful?
    What's Next