Method: projects.locations.setDedicatedCrawlRate

Sets the dedicated crawl rate for a crawlRateScope. If the dedicated crawl rate was not set, this will enable vertex AI's crawl bot to use the new dedicated crawl rate for crawling. If the dedicated crawl rate was set, vertex AI's crawl bot will try to update the rate to the new value. If the new value is too high, the crawl bot may crawl at a lower rate to avoid overloading the user's website.

HTTP request

POST https://discoveryengine.googleapis.com/v1beta/{location=projects/*/locations/*}:setDedicatedCrawlRate

The URL uses gRPC Transcoding syntax.

Path parameters

Parameters
location

string

Required. The location resource where crawl rate management will be performed. Format: projects/{project}/locations/{location}

Request body

The request body contains data with the following structure:

JSON representation 
{
  "crawlRateScope": string,
  "crawlRate": integer,
  "crawlType": enum (CrawlType),
  "mode": enum (Mode)
}
Fields
crawlRateScope

string

Required. The scope of the crawl rate that the user wants to config. Currently, only domain and host name are supported. A domain name example: example.com. A host name example: www.example.com. Please do not include / in the domain or host name.
crawlRate

integer

Optional. The crawl QPS set by the user. It is not guaranteed that Vertex crawl bot will crawl at this QPS. If the crawl rate is too high, the real QPS may be lower than the value set by the user to avoid overloading the user's website.
crawlType

enum (CrawlType)

Optional. Whether it's the crawl rate of user-triggered or auto-refresh.
mode

enum (Mode)

Optional. Whether the rate is explicitly set by users, or set by vertex AI.

Response body

If successful, the response body contains an instance of Operation.

Authorization scopes

Requires the following OAuth scope:

  • https://www.googleapis.com/auth/cloud-platform

For more information, see the Authentication Overview.

CrawlType

Crawl type is used to distinguish user-triggered and auto-refresh. User-triggered crawl rate is for deterministic use cases like crawling urls or sitemaps specified by users. Auto-refresh crawl rate is for best effort use cases like refreshing urls periodically.

Enums
CRAWL_TYPE_UNSPECIFIED The crawl type is unspecified.
USER_TRIGGERED The crawl type is user-triggered.
AUTO_REFRESH The crawl type is auto-refresh.

Mode

Whether the rate is explicitly set by users, or set by vertex AI.

Enums
MODE_UNSPECIFIED The mode is unspecified. If the user does not specify the mode, default to AUTOMATIC.
AUTOMATIC Vertex AI automatically splits crawlRate into user-triggered and auto-refresh. Users don't need to specify SetDedicatedCrawlRateRequest.CrawlType if the mode is AUTOMATIC.
EXPLICIT Users explicitly set user-triggered or auto-refresh crawl rate.