From 11c3a52c08f15a941cc5086628be67aac0549acf Mon Sep 17 00:00:00 2001 From: Eric Davis Date: Wed, 10 Jun 2020 19:07:32 -0400 Subject: [PATCH 1/6] first pass at trouble shooting additions --- .../troubleshooting.asciidoc | 69 ++++++++++++++++++- 1 file changed, 66 insertions(+), 3 deletions(-) diff --git a/docs/en/ingest-management/troubleshooting.asciidoc b/docs/en/ingest-management/troubleshooting.asciidoc index 06a6cce55..c60f23617 100644 --- a/docs/en/ingest-management/troubleshooting.asciidoc +++ b/docs/en/ingest-management/troubleshooting.asciidoc @@ -4,12 +4,75 @@ This release of Ingest Management is an experimental release. With this early release we explicitly invite you to install and test it **in a test environment**, but it is absolutely possible that you run into problems and need to tweak your setup to get it running. -We have collected the most common known problems below. If your question isn't answered here, please contact us https://ela.st/ingest-manager-feedback{here}. Your feedback is very valuable to us. +We have collected the most common known problems below. If your question isn't answered here, please review open issues in the https://github.com/elastic/kibana/issues, https://github.com/elastic/beats/issuess, https://github.com/elastic/package-registry/issues, and contact us https://ela.st/ingest-manager-feedback{here}. Your feedback is very valuable to us. +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +[[ingest-manager-not-in-kibana]] +=== The {ingest-manager} app is not listed in my Kibana cloud navigation list. + +In 7.8, the {ingest-manager} app is in Alpha state and as such is only shown when specifically enabled in the start up or cloud 'override' options. To enable {ingest-manager} on cloud, go to your deployment in the user console. Click on the 'edit' link directly beneath the deployment name. Then click on 'user setting overrides' underneath the Kibana heading. Here you must enter the following x-pack option and click the save button at the bottom: +xpack.ingestManager.enabled: true + +This will restart Kibana automtically and when done, you can refresh the browser and see the app in the left hand navigation + +[[ingest-management-setup-fails]] +=== The endpoint `/api/ingest_management/setup` returns an error that the package registry can't be reached. + +In order to install {integrations}, the {ingest-manager} app needs to connect to an external service, the {package-registry}. For this to work, Kibana server must be able to connect to http://epr-experimental.elastic.co on port 80. + +[[ingest-manager-app-crashes]] === The {ingest-manager} app in Kibana crashes. To find more about the error, open the development console of your browser, navigate to the **Network** tab and refresh the page. One of the requests to the Ingest Manager API will most likely have returned with an error. If the error message doesn't give you enough information to fix the problem on your own, please contact us https://ela.st/ingest-manager-feedback{here}. -=== The endpoint `/api/ingest_management/setup` returns an error that the package registry can't be reached. -In order to install {integrations}, the {ingest-manager} app needs to connect to an external service, the {package-registry}. For this to work, Kibana server must be able to connect to http://epr-experimental.elastic.co on port 80. \ No newline at end of file +[[agent-enrollment-timeout]] +=== I get a time-out when I try to run the agent enrollment on my host +This can look like the below console output: +fail to enroll: fail to execute request to Kibana: Post http://kibana:5601/api/ingest_manager/fleet/agents/enroll?: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) + +This is possibly due to a connectivity problem with the host talking to the Kibana instance. It may be a networking problem, attempt a 'ping' command from the host to confirm it can reach the Kibana instance. If this works, then it could be a problem with the specified enrollment command executed. If 'ping' works, check the URL and port, both are correct for your environment. If those are correct and working, check the enrollment key in the enrollment command you are trying to execute is valid. You can do this by viewing the Fleet ui in {ingest-manager} and clicking on the Enrollment Tokens tab. If you click the 'eyeball' for each listed secret, one listed should match the string being used in your host's enrollment command. If it is not listed, create a new enrollment token and replace it in the command, this can be done in that same UI. + +[[enrolled-agent-not-showing-up]] +I ran the Agent enrollment step but I don’t see the Agent in the UI +If Elastic Agent was successfully enrolled, but doesn't show up in the Fleet list a common problem is that it was not 'started'. Starting the Agent is a separate step which can be done as follows as the below. + +On linux & macos hosts: +./elastic-agent run + +On Windows hosts: +elastic-agent.exe run + +[[where-are-the-agent-logs]] +I need to find the logs when the Agent starts up + +When started successfully, the logs are in the folder where the Agent was started, in a sub-directory 'data/logs/metricbeat' +If that path of logs does not exist, it indicates Agent did not successfully run metricbeat, which is a higher level problem to triage. + + +[[ingest-manager-not-in-on-prem-kibana]] +=== The {ingest-manager} app is not listed in my Kibana on-premises navigation list. + +In 7.8, the {ingest-manager} app is in Alpha state and as such is only shown when specifically enabled in the start up config. To do this you must modify both the Elasticsearch and Kibana configs as well as adhering to all 'security enabled' requirements, like modifying the default elastic user password, see below for details: + +For Elasticsearch, file config/elasticsearch.yml you must set these start up options: +xpack.security.enabled: true +xpack.security.authc.api_key.enabled: true + +For Kibana config/kibana.yml you must set these start up options (the exception is tlsCheckDisabled which is not required if configure TLS checking) +xpack.ingestManager.enabled: true +xpack.ingestManager.fleet.tlsCheckDisabled: true +xpack.security.enabled: true +elasticsearch.username: "elastic" +elasticsearch.password: "abc123iUnbRftkABC123" + +NOTE: your elasticsearch.password above will be different, of course. It can be set with the documented Elastic apis, or you may wish to use the password re-setting script that comes with Elasticsearch. It is in the /bin elasticsearch directory and can be used like: +./bin/elasticsearch-setup-passwords auto + - copy the elatic user name and put it to the kibana config file above after running the script. Then re-start Kibana. From 34cf10e4bae7b1d447c8b51cb0b43d1db25d392b Mon Sep 17 00:00:00 2001 From: Eric Davis Date: Wed, 10 Jun 2020 22:11:00 -0400 Subject: [PATCH 2/6] 2nd and final round of faq notes --- .../troubleshooting.asciidoc | 43 ++++++++++++++++++- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/docs/en/ingest-management/troubleshooting.asciidoc b/docs/en/ingest-management/troubleshooting.asciidoc index c60f23617..8b1e3ac34 100644 --- a/docs/en/ingest-management/troubleshooting.asciidoc +++ b/docs/en/ingest-management/troubleshooting.asciidoc @@ -12,6 +12,13 @@ We have collected the most common known problems below. If your question isn't a * <> * <> * <> +* <> +* <> +* <> +* <> +* <> +* <> + * <> [[ingest-manager-not-in-kibana]] @@ -41,7 +48,7 @@ fail to enroll: fail to execute request to Kibana: Post http://kibana:5601/api/i This is possibly due to a connectivity problem with the host talking to the Kibana instance. It may be a networking problem, attempt a 'ping' command from the host to confirm it can reach the Kibana instance. If this works, then it could be a problem with the specified enrollment command executed. If 'ping' works, check the URL and port, both are correct for your environment. If those are correct and working, check the enrollment key in the enrollment command you are trying to execute is valid. You can do this by viewing the Fleet ui in {ingest-manager} and clicking on the Enrollment Tokens tab. If you click the 'eyeball' for each listed secret, one listed should match the string being used in your host's enrollment command. If it is not listed, create a new enrollment token and replace it in the command, this can be done in that same UI. [[enrolled-agent-not-showing-up]] -I ran the Agent enrollment step but I don’t see the Agent in the UI +=== I ran the Agent enrollment step but I don’t see the Agent in the UI If Elastic Agent was successfully enrolled, but doesn't show up in the Fleet list a common problem is that it was not 'started'. Starting the Agent is a separate step which can be done as follows as the below. On linux & macos hosts: @@ -51,11 +58,43 @@ On Windows hosts: elastic-agent.exe run [[where-are-the-agent-logs]] -I need to find the logs when the Agent starts up +=== I need to find the logs when the Agent starts up When started successfully, the logs are in the folder where the Agent was started, in a sub-directory 'data/logs/metricbeat' If that path of logs does not exist, it indicates Agent did not successfully run metricbeat, which is a higher level problem to triage. +[[what-is-my-agent-config]] +=== I need to know the configuration used in the running Agent. + +You can inspect the files in the folder from where you ran the Agent. The elastic-agent.yml file will have an uncommented citation if it is in 'Fleet' mode: +Management: mode: "fleet" + +The action_store.yml has the whole configuration, unencrypted. Look for 'outputs:hosts' section with elasticsearch location. This file will also have the current versions of the packages in use by the current configuration. The version of the Agent is in the download folder / zip. + +[[where-is-the-data-agent-is-sending]] +=== I'm not sure {ingest-manager} app is showing the data my Agent is sending. + +I think I have Agent set up and running and there are no errors on the Agent logs and I still don't see data in Kibana. The recommended way to assess this is to go straight to Elasticsearch data queries, or maybe just use Kibana Discover app. Look for any data in the index your Agent is sending to (metrics-* for example) and see if the name.host of your host machine shows. If not, then perhaps the data is getting trapped / blocked in the network or isn't sent off of the Agent host correctly in the first place due to some Firewall or security reason. While it should be redundant, it is all the same potentially helpful to install the stand-alone Metricbeat (see the Elastic docs for information) to see if that works to send data into ES, which would indicate a bug or problem in the Agent side. + +[[i-deleted-my-agent]] +=== I deleted my agent in the Fleet {ingest-manager} app so now what? + +Its ok, be cool: the data is still in Elasticsearch. But to get the Agent back in the UI, you will need to stop, then re-enroll the Agent on the host and run it again. + +[[i-need-to-stop-agent]] +=== I need to stop the Agent and all processes running on my host + +To stop the Agent and relating executables running on my hosts is as easy as stopping the Agent process. If you installed it as a service on Windows you will need to manage that, as well. After that, if needed, on Windows you can use Task Manager and shut down ‘Elastic Agent’ which will kill it and the sub-processes it created (beats). On Linux / macOS you can kill the elastic-agent process running, look with a ps | grep elastic-agent call to get the id to kill + +[[i-rebooted-my-host]] +=== I needed to reboot my host, I'm not sure if the Agent is still running. + +If you installed Windows Agent as a service with powershell, the Agent should still be running after a reboot of the host. On macOS and Linux you will need to start the Agent again from the command line each time. The ability to enhance support for those operating systems is known, as a temporary fix you could add that start command to a start up profile. + +[[what-is-the-endpoint-package]] +=== The {ingest-manager} app shows an Integration called 'Endpoint' and I want to use it. + +We all want to use it! But we can't in 7.8 - It is not yet even 'Alpha' state, as the {ingest-manager} app is. For some context, Endpoint is the integration that will allow the Elastic Security app to have a dedicated ‘Endpoint’ executable running like a beat and will be managed by the Agent as other beats are. It cannot yet be used in 7.8 so it is best left alone. It won't hurt anything sitting there, promise. [[ingest-manager-not-in-on-prem-kibana]] === The {ingest-manager} app is not listed in my Kibana on-premises navigation list. From 1058768fe2093f3e2ca88cade6849f59b715bb96 Mon Sep 17 00:00:00 2001 From: Eric Davis Date: Wed, 10 Jun 2020 22:14:47 -0400 Subject: [PATCH 3/6] minor typo fixes for troubleshooting docs --- docs/en/ingest-management/troubleshooting.asciidoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/en/ingest-management/troubleshooting.asciidoc b/docs/en/ingest-management/troubleshooting.asciidoc index 8b1e3ac34..960d22145 100644 --- a/docs/en/ingest-management/troubleshooting.asciidoc +++ b/docs/en/ingest-management/troubleshooting.asciidoc @@ -18,10 +18,9 @@ We have collected the most common known problems below. If your question isn't a * <> * <> * <> - * <> -[[ingest-manager-not-in-kibana]] +[[ingest-manager-not-in-cloud-kibana]] === The {ingest-manager} app is not listed in my Kibana cloud navigation list. In 7.8, the {ingest-manager} app is in Alpha state and as such is only shown when specifically enabled in the start up or cloud 'override' options. To enable {ingest-manager} on cloud, go to your deployment in the user console. Click on the 'edit' link directly beneath the deployment name. Then click on 'user setting overrides' underneath the Kibana heading. Here you must enter the following x-pack option and click the save button at the bottom: From 6d6ea895412cf85394c3109695a294b58df4fa78 Mon Sep 17 00:00:00 2001 From: Eric Davis Date: Thu, 11 Jun 2020 14:13:05 -0400 Subject: [PATCH 4/6] Update docs/en/ingest-management/troubleshooting.asciidoc Co-authored-by: Pier-Hugues Pellerin --- docs/en/ingest-management/troubleshooting.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/troubleshooting.asciidoc b/docs/en/ingest-management/troubleshooting.asciidoc index 960d22145..2c63fe130 100644 --- a/docs/en/ingest-management/troubleshooting.asciidoc +++ b/docs/en/ingest-management/troubleshooting.asciidoc @@ -26,7 +26,7 @@ We have collected the most common known problems below. If your question isn't a In 7.8, the {ingest-manager} app is in Alpha state and as such is only shown when specifically enabled in the start up or cloud 'override' options. To enable {ingest-manager} on cloud, go to your deployment in the user console. Click on the 'edit' link directly beneath the deployment name. Then click on 'user setting overrides' underneath the Kibana heading. Here you must enter the following x-pack option and click the save button at the bottom: xpack.ingestManager.enabled: true -This will restart Kibana automtically and when done, you can refresh the browser and see the app in the left hand navigation +This will restart Kibana automatically and when done, you can refresh the browser and see the app in the left hand navigation [[ingest-management-setup-fails]] === The endpoint `/api/ingest_management/setup` returns an error that the package registry can't be reached. From 8de423cd9602b3ae1767dcb56fab8332c092c29e Mon Sep 17 00:00:00 2001 From: Eric Davis Date: Thu, 11 Jun 2020 14:13:26 -0400 Subject: [PATCH 5/6] Update docs/en/ingest-management/troubleshooting.asciidoc Co-authored-by: DeDe Morton --- docs/en/ingest-management/troubleshooting.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/ingest-management/troubleshooting.asciidoc b/docs/en/ingest-management/troubleshooting.asciidoc index 2c63fe130..c7441aead 100644 --- a/docs/en/ingest-management/troubleshooting.asciidoc +++ b/docs/en/ingest-management/troubleshooting.asciidoc @@ -59,7 +59,7 @@ elastic-agent.exe run [[where-are-the-agent-logs]] === I need to find the logs when the Agent starts up -When started successfully, the logs are in the folder where the Agent was started, in a sub-directory 'data/logs/metricbeat' +When started successfully, the logs are in the folder where the Agent was started, in a sub-directory `data/logs/metricbeat`. If that path of logs does not exist, it indicates Agent did not successfully run metricbeat, which is a higher level problem to triage. [[what-is-my-agent-config]] From d37b2dfea7571702f386470735a4a139dcb69078 Mon Sep 17 00:00:00 2001 From: Eric Davis Date: Thu, 11 Jun 2020 14:22:22 -0400 Subject: [PATCH 6/6] updates to attempt accommodating pr feedback --- docs/en/ingest-management/troubleshooting.asciidoc | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/en/ingest-management/troubleshooting.asciidoc b/docs/en/ingest-management/troubleshooting.asciidoc index c7441aead..98f01e795 100644 --- a/docs/en/ingest-management/troubleshooting.asciidoc +++ b/docs/en/ingest-management/troubleshooting.asciidoc @@ -41,6 +41,7 @@ To find more about the error, open the development console of your browser, navi [[agent-enrollment-timeout]] === I get a time-out when I try to run the agent enrollment on my host + This can look like the below console output: fail to enroll: fail to execute request to Kibana: Post http://kibana:5601/api/ingest_manager/fleet/agents/enroll?: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) @@ -48,6 +49,7 @@ This is possibly due to a connectivity problem with the host talking to the Kiba [[enrolled-agent-not-showing-up]] === I ran the Agent enrollment step but I don’t see the Agent in the UI + If Elastic Agent was successfully enrolled, but doesn't show up in the Fleet list a common problem is that it was not 'started'. Starting the Agent is a separate step which can be done as follows as the below. On linux & macos hosts: @@ -76,9 +78,9 @@ The action_store.yml has the whole configuration, unencrypted. Look for 'output I think I have Agent set up and running and there are no errors on the Agent logs and I still don't see data in Kibana. The recommended way to assess this is to go straight to Elasticsearch data queries, or maybe just use Kibana Discover app. Look for any data in the index your Agent is sending to (metrics-* for example) and see if the name.host of your host machine shows. If not, then perhaps the data is getting trapped / blocked in the network or isn't sent off of the Agent host correctly in the first place due to some Firewall or security reason. While it should be redundant, it is all the same potentially helpful to install the stand-alone Metricbeat (see the Elastic docs for information) to see if that works to send data into ES, which would indicate a bug or problem in the Agent side. [[i-deleted-my-agent]] -=== I deleted my agent in the Fleet {ingest-manager} app so now what? +=== I deleted my agent in the Fleet {ingest-manager} app how can I get it back? -Its ok, be cool: the data is still in Elasticsearch. But to get the Agent back in the UI, you will need to stop, then re-enroll the Agent on the host and run it again. +Its ok, we got your back! The data is still in Elasticsearch. And get the Agent back in the UI, you will need only to stop, then re-enroll the Agent on the host and run it again. [[i-need-to-stop-agent]] === I need to stop the Agent and all processes running on my host @@ -86,14 +88,14 @@ Its ok, be cool: the data is still in Elasticsearch. But to get the Agent back To stop the Agent and relating executables running on my hosts is as easy as stopping the Agent process. If you installed it as a service on Windows you will need to manage that, as well. After that, if needed, on Windows you can use Task Manager and shut down ‘Elastic Agent’ which will kill it and the sub-processes it created (beats). On Linux / macOS you can kill the elastic-agent process running, look with a ps | grep elastic-agent call to get the id to kill [[i-rebooted-my-host]] -=== I needed to reboot my host, I'm not sure if the Agent is still running. +=== I needed to reboot my host, how do I restart the Agent?. -If you installed Windows Agent as a service with powershell, the Agent should still be running after a reboot of the host. On macOS and Linux you will need to start the Agent again from the command line each time. The ability to enhance support for those operating systems is known, as a temporary fix you could add that start command to a start up profile. +If you installed Windows Agent as a service with powershell, the Agent should still be running after a reboot of the host. On macOS and Linux you will need to start the Agent again from the command line each time as noted in the prior Agent start up steps. Adding support to install as a service on all supported systems will be done in a future release. To achieve this in the meantime, you can add the start command to a user's start up profile. [[what-is-the-endpoint-package]] -=== The {ingest-manager} app shows an Integration called 'Endpoint' and I want to use it. +=== The {ingest-manager} app shows an Integration called 'Endpoint', what is it? -We all want to use it! But we can't in 7.8 - It is not yet even 'Alpha' state, as the {ingest-manager} app is. For some context, Endpoint is the integration that will allow the Elastic Security app to have a dedicated ‘Endpoint’ executable running like a beat and will be managed by the Agent as other beats are. It cannot yet be used in 7.8 so it is best left alone. It won't hurt anything sitting there, promise. +In 7.8, the Endpoint Integration is non-functional. It cannot be used yet. It exists as an artifact of the current feature development, please watch for announcements during coming release cycles. As a teaser, Endpoint is the integration that will allow the Elastic Security app to have a dedicated executable running like a beat to protect the host and allow responses to security concerns that are found. It will be managed by the Agent as other beats are. [[ingest-manager-not-in-on-prem-kibana]] === The {ingest-manager} app is not listed in my Kibana on-premises navigation list.