[MongoDB Atlas] Disk data stream #10555
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🚀 Benchmarks reportTo see the full report comment with |
|
efd6
reviewed
Jul 24, 2024
packages/mongodb_atlas/data_stream/disk/agent/stream/input.yml.hbs
Outdated
Show resolved
Hide resolved
packages/mongodb_atlas/data_stream/disk/agent/stream/input.yml.hbs
Outdated
Show resolved
Hide resolved
packages/mongodb_atlas/data_stream/disk/agent/stream/input.yml.hbs
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Add a processor to asses the error state from the CEL program. We use something like this
- fail:
if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null
message: error message set and no data to process
|
Review:
diff --git a/packages/mongodb_atlas/data_stream/disk/agent/stream/input.yml.hbs b/packages/mongodb_atlas/data_stream/disk/agent/stream/input.yml.hbs
index be868f1b1..1a37b8579 100644
--- a/packages/mongodb_atlas/data_stream/disk/agent/stream/input.yml.hbs
+++ b/packages/mongodb_atlas/data_stream/disk/agent/stream/input.yml.hbs
@@ -41,12 +41,12 @@ program: |
has(state.host_list) && size(state.host_list) > 0
?
state
- :
+ :
state.with(request("GET", state.url + "/api/atlas/v2/groups/" + state.group_id + "/processes?pageNum=" + string(state.page_num) + "&itemsPerPage=100").with({
"Header": {
"Accept": ["application/vnd.atlas." + string(timestamp(now).getFullYear()) + "-01-01+json"]
}
- }).do_request().as(resp,
+ }).do_request().as(resp,
bytes(resp.Body).decode_json().as(body, {
"host_list": body.results.map(e, state.url + "/api/atlas/v2/groups/" + state.group_id + "/processes/" + e.id + "/disks/"),
"next": 0,
@@ -54,16 +54,16 @@ program: |
})))
).as(state, state.next >= size(state.host_list) ? {} :
(
- has(state.disk_list) && size(state.disk_list) > 0
+ has(state.disk_list) && size(state.disk_list) > 0
?
state
- :
+ :
state.with(request("GET", string(state.host_list[state.next] + "?pageNum=" + string(state.disk_page_num) + "&itemsPerPage=100" ))
.with({
"Header": {
"Accept": ["application/vnd.atlas." + string(timestamp(now).getFullYear()) + "-01-01+json"]
}
- }).do_request().as(resp,
+ }).do_request().as(resp,
bytes(resp.Body).decode_json().as(body, {
"disk_list": body.results.map(e, e.partitionName),
"disk_next": 0,
diff --git a/packages/mongodb_atlas/data_stream/disk/elasticsearch/ingest_pipeline/default.yml b/packages/mongodb_atlas/data_stream/disk/elasticsearch/ingest_pipeline/default.yml
index b7c04ff46..b8e11c28b 100644
--- a/packages/mongodb_atlas/data_stream/disk/elasticsearch/ingest_pipeline/default.yml
+++ b/packages/mongodb_atlas/data_stream/disk/elasticsearch/ingest_pipeline/default.yml
@@ -20,6 +20,18 @@ processors:
field: groupId
target_field: group.id
ignore_missing: true
+ - rename:
+ field: processId
+ target_field: mongodb_atlas.process_id
+ ignore_missing: true
+ - rename:
+ field: partitionName
+ target_field: mongodb_atlas.partition_name
+ ignore_missing: true
+ - rename:
+ field: hostId
+ target_field: mongodb_atlas.host_id
+ ignore_missing: true
- rename:
field: response.MAX_DISK_PARTITION_IOPS_READ
target_field: mongodb_atlas.disk.read.iops.max.throughput
@@ -36,6 +48,22 @@ processors:
field: response.DISK_PARTITION_LATENCY_READ
target_field: mongodb_atlas.disk.read.latency.ms
ignore_missing: true
+ - rename:
+ field: response.MAX_DISK_PARTITION_IOPS_WRITE
+ target_field: mongodb_atlas.disk.write.iops.max.throughput
+ ignore_missing: true
+ - rename:
+ field: response.DISK_PARTITION_IOPS_WRITE
+ target_field: mongodb_atlas.disk.write.iops.throughput
+ ignore_missing: true
+ - rename:
+ field: response.MAX_DISK_PARTITION_LATENCY_WRITE
+ target_field: mongodb_atlas.disk.write.latency.max.ms
+ ignore_missing: true
+ - rename:
+ field: response.DISK_PARTITION_LATENCY_WRITE
+ target_field: mongodb_atlas.disk.write.latency.ms
+ ignore_missing: true
- rename:
field: response.DISK_PARTITION_SPACE_FREE
target_field: mongodb_atlas.disk.space.free.bytes
@@ -76,34 +104,6 @@ processors:
field: response.DISK_PARTITION_IOPS_TOTAL
target_field: mongodb_atlas.disk.total.iops.throughput
ignore_missing: true
- - rename:
- field: response.MAX_DISK_PARTITION_IOPS_WRITE
- target_field: mongodb_atlas.disk.write.iops.max.throughput
- ignore_missing: true
- - rename:
- field: response.DISK_PARTITION_IOPS_WRITE
- target_field: mongodb_atlas.disk.write.iops.throughput
- ignore_missing: true
- - rename:
- field: response.MAX_DISK_PARTITION_LATENCY_WRITE
- target_field: mongodb_atlas.disk.write.latency.max.ms
- ignore_missing: true
- - rename:
- field: response.DISK_PARTITION_LATENCY_WRITE
- target_field: mongodb_atlas.disk.write.latency.ms
- ignore_missing: true
- - rename:
- field: processId
- target_field: mongodb_atlas.process_id
- ignore_missing: true
- - rename:
- field: partitionName
- target_field: mongodb_atlas.partition_name
- ignore_missing: true
- - rename:
- field: hostId
- target_field: mongodb_atlas.host_id
- ignore_missing: true
- remove:
field:
- response
@@ -140,4 +140,3 @@ on_failure:
field: event.kind
value: pipeline_error
allow_duplicates: false
-
\ No newline at end of file
diff --git a/packages/mongodb_atlas/data_stream/disk/fields/fields.yml b/packages/mongodb_atlas/data_stream/disk/fields/fields.yml
index 1d86a9ef3..55ebc92a2 100644
--- a/packages/mongodb_atlas/data_stream/disk/fields/fields.yml
+++ b/packages/mongodb_atlas/data_stream/disk/fields/fields.yml
@@ -13,11 +13,11 @@
- name: max.throughput
type: double
metric_type: gauge
- description: Max throughput of I/O read operations for the disk partition used for MongoDB Atlas.
+ description: Maximum throughput of I/O read operations for the disk partition used by MongoDB Atlas.
- name: throughput
type: double
metric_type: gauge
- description: Throughput of I/O read operations for the disk partition used for MongoDB Atlas.
+ description: Current throughput of I/O read operations for the disk partition used by MongoDB Atlas.
- name: latency
type: group
fields:
@@ -25,12 +25,12 @@
type: double
metric_type: gauge
unit: ms
- description: Max read latency of the disk partition that hosts MongoDB Atlas.
+ description: Maximum read latency in milliseconds for the disk partition used by MongoDB Atlas.
- name: ms
type: double
metric_type: gauge
unit: ms
- description: Read latency of the disk partition that hosts MongoDB Atlas.
+ description: Current read latency in milliseconds for the disk partition used by MongoDB Atlas.
- name: space
type: group
fields:
@@ -41,7 +41,7 @@
type: double
metric_type: gauge
unit: byte
- description: Amount of disk space free in Bytes, on the disk partition used by MongoDB Atlas.
+ description: Current amount of free disk space in bytes on the disk partition used by MongoDB Atlas.
- name: max
type: group
fields:
@@ -49,17 +49,17 @@
type: double
metric_type: gauge
unit: byte
- description: Max amount of disk space free in Bytes, on the disk partition used by MongoDB Atlas.
+ description: Maximum amount of free disk space in bytes on the disk partition used by MongoDB Atlas.
- name: pct
type: double
metric_type: gauge
unit: percent
- description: Max percentage of disk free on the disk partition used by MongoDB Atlas.
+ description: Maximum percentage of free disk space on the disk partition used by MongoDB Atlas.
- name: pct
type: double
metric_type: gauge
unit: percent
- description: The percentage of disk free on the disk partition used by MongoDB Atlas.
+ description: Current percentage of free disk space on the disk partition used by MongoDB Atlas.
- name: used
type: group
fields:
@@ -67,7 +67,7 @@
type: double
metric_type: gauge
unit: byte
- description: Amount of disk space used in Bytes, on the disk partition used by MongoDB Atlas.
+ description: Current amount of used disk space in bytes on the disk partition used by MongoDB Atlas.
- name: max
type: group
fields:
@@ -75,17 +75,17 @@
type: double
metric_type: gauge
unit: byte
- description: Max amount of disk space used in Bytes, on the disk partition used by MongoDB Atlas.
+ description: Maximum amount of used disk space in bytes on the disk partition used by MongoDB Atlas.
- name: pct
type: double
metric_type: gauge
unit: percent
- description: Max percentage of disk used on the disk partition used by MongoDB Atlas.
+ description: Maximum percentage of used disk space on the disk partition used by MongoDB Atlas.
- name: pct
type: double
metric_type: gauge
unit: percent
- description: The percentage of disk used on the disk partition used by MongoDB Atlas.
+ description: Current percentage of used disk space on the disk partition used by MongoDB Atlas.
- name: total
type: group
fields:
@@ -95,11 +95,11 @@
- name: max.throughput
type: double
metric_type: gauge
- description: Max throughput of total I/O operations for the disk partition used for MongoDB Atlas.
+ description: Maximum throughput of total I/O operations for the disk partition used by MongoDB Atlas.
- name: throughput
type: double
metric_type: gauge
- description: Throughput of total I/O operations for the disk partition used for MongoDB Atlas.
+ description: Current throughput of total I/O operations for the disk partition used by MongoDB Atlas.
- name: write
type: group
fields:
@@ -109,11 +109,11 @@
- name: max.throughput
type: double
metric_type: gauge
- description: Max throughput of I/O write operations for the disk partition used for MongoDB Atlas.
+ description: Maximum throughput of I/O write operations for the disk partition used by MongoDB Atlas.
- name: throughput
type: double
metric_type: gauge
- description: Throughput of I/O write operations for the disk partition used for MongoDB Atlas.
+ description: Current throughput of I/O write operations for the disk partition used by MongoDB Atlas.
- name: latency
type: group
fields:
@@ -121,12 +121,12 @@
type: double
metric_type: gauge
unit: ms
- description: Max write latency of the disk partition that hosts MongoDB Atlas.
+ description: Maximum write latency in milliseconds for the disk partition used by MongoDB Atlas.
- name: ms
type: double
metric_type: gauge
unit: ms
- description: Write latency of the disk partition that hosts MongoDB Atlas.
+ description: Current write latency in milliseconds for the disk partition used by MongoDB Atlas.
- name: host_id
description: Unique identifier of the host for the MongoDB process.
type: keyword
diff --git a/packages/mongodb_atlas/data_stream/disk/manifest.yml b/packages/mongodb_atlas/data_stream/disk/manifest.yml
index 38bca0406..6c277eb76 100644
--- a/packages/mongodb_atlas/data_stream/disk/manifest.yml
+++ b/packages/mongodb_atlas/data_stream/disk/manifest.yml
@@ -10,7 +10,8 @@ streams:
- name: period
type: text
title: Period
- description: Period of fetching metrics. Value of Granularity and Period will be same (check readme for more details). Supported units for this parameter are h (hours) and m (minutes).
+ description: |
+ Interval for fetching metrics. The Granularity and Period values will be identical. Supported units are h (hours) and m (minutes). Example: 10m for 10 minutes.
default: 10m
multi: false
required: true
@@ -18,7 +19,8 @@ streams:
- name: http_client_timeout
type: text
title: HTTP Client Timeout
- description: Duration before declaring that the HTTP client connection has timed out. Valid time units are h (hours), m (minutes) and s (seconds). If the data collection period is extensive, ensure that this parameter's value is appropriately long.
+ description: |
+ Maximum duration for HTTP client connection before timing out. Valid units are h (hours), m (minutes), and s (seconds). For longer data collection periods, set a correspondingly longer timeout.
multi: false
required: false
show_user: false
@@ -26,7 +28,8 @@ streams:
- name: groupId
type: text
title: Group ID
- description: Identifier string that identifies your project. Groups and projects are synonymous terms. Your group ID is the same as your project ID. Ex. 32b6e34b3d91647abb20e7b8
+ description: |
+ Unique identifier for your project. Note that Group ID and Project ID are interchangeable terms. Example: `32b6e34b3d91647abb20e7b8`
multi: false
required: true
show_user: true
@@ -36,7 +39,8 @@ streams:
multi: false
required: false
show_user: false
- description: The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_filename) for details.
+ description: |
+ The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_filename) for details.
- name: tags
type: text
title: Tags
@@ -51,5 +55,5 @@ streams:
multi: false
required: false
show_user: false
- description: >-
+ description: |
Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html) for details. |
shmsr
requested changes
Jul 25, 2024
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
|
Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as |
…into package_mongodb_atlas_disk
efd6
reviewed
Oct 17, 2024
shmsr
requested changes
Oct 29, 2024
| link: https://github.com/elastic/integrations/pull/9754 | ||
| - version: "0.0.4" | ||
| changes: | ||
| - description: Add "hardware" data stream to MongoDB Atlas package. |
There was a problem hiding this comment.
Since this changelog belongs to MongoDB Atlas itself, we can streamline the description by removing the MongoDB Atlas reference. I see we've kept this format in previous entries, so let's maintain consistency for now and implement this improvement in future updates.
| type: text | ||
| title: Group ID | ||
| description: | | ||
| Unique identifier for your project. Note that Group ID and Project ID are interchangeable terms. Example: `32b6e34b3d91647abb20e7b8` |
There was a problem hiding this comment.
Suggested change
| Unique identifier for your project. Note that Group ID and Project ID are interchangeable terms. Example: `32b6e34b3d91647abb20e7b8` | |
| Unique identifier for your project. `Group ID` and `Project ID` are interchangeable terms. For example: `32b6e34b3d91647abb20e7b8` |
| type: text | ||
| title: Period | ||
| description: | | ||
| Interval for fetching metrics. The Granularity and Period values will be identical. Supported units are h (hours) and m (minutes). Example: 10m for 10 minutes. |
There was a problem hiding this comment.
Suggested change
| Interval for fetching metrics. The Granularity and Period values will be identical. Supported units are h (hours) and m (minutes). Example: 10m for 10 minutes. | |
| Interval for fetching metrics. The `Granularity` and `Period` values are identical. Supported units are 'h' (hours) and 'm' (minutes). For example: '10m' represents 10 minutes. |
For http_client_timeout we are even considering s (seconds) unit but not here. Why?
shmsr
approved these changes
Oct 29, 2024
💚 Build Succeeded
History
|
|
|
Package mongodb_atlas - 0.0.9 containing this change is available at https://epr.elastic.co/search?package=mongodb_atlas |
harnish-crest-data
pushed a commit
to chavdaharnish/integrations
that referenced
this pull request
Feb 4, 2025
* add disk data stream * update readme * update readme * address review coments * address review comments * address review comments * address review comments
harnish-crest-data
pushed a commit
to chavdaharnish/integrations
that referenced
this pull request
Feb 5, 2025
* add disk data stream * update readme * update readme * address review coments * address review comments * address review comments * address review comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Checklist
changelog.ymlfile.How to test this PR locally
Screenshots
Related issue