Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Cloud Application Platform 1.5.2

24 Logging

There are two types of logs in a deployment of SUSE Cloud Application Platform, applications logs and component logs. The following provides a brief overview of each log type and how to retrieve them for monitoring and debugging use.

  • Application logs provide information specific to a given application that has been deployed to your Cloud Application Platform cluster and can be accessed through:

    • The cf CLI using the cf logs command

    • The application's log stream within the Stratos console

  • Access to logs for a given component of your Cloud Application Platform deployment can be obtained by:

    • The kubectl logs command

      The following example retrieves the logs of the router container of router-0 pod in the scf namespace

      tux > kubectl logs --namespace scf router-0 router
    • Direct access to the log files using the following:

      1. Open a shell to the container of the component using the kubectl exec command

      2. Navigate to the logs directory at /var/vcap/sys/logs, at which point there will be subdirectories containing the log files for access.

        tux > kubectl exec --stdin --tty --namespace scf router-0 /bin/bash
        
        router/0:/# cd /var/vcap/sys/log
        
        router/0:/var/vcap/sys/log# ls -R
        .:
        gorouter  loggregator_agent
        
        ./gorouter:
        access.log  gorouter.err.log  gorouter.log  post-start.err.log  post-start.log
        
        ./loggregator_agent:
        agent.log

24.1 Logging to an External Syslog Server

Cloud Application Platform supports sending the cluster's log data to external logging services where additional processing and analysis can be performed.

24.1.1 Configuring Cloud Application Platform

In your scf-config-values.yaml file add the following configuration values to the env: section. The example values below are configured for an external ELK stack.

env:
  SCF_LOG_HOST: elk.example.com
  SCF_LOG_PORT: 5001
  SCF_LOG_PROTOCOL: "tcp"

24.1.2 Example using the ELK Stack

The ELK stack is an example of an external syslog server where log data can be sent to for log management. The ELK stack consists of:

Elasticsearch

A tool for search and analytics. For more information, refer to https://www.elastic.co/products/elasticsearch.

Logstash

A tool for data processing. For more information, refer to https://www.elastic.co/products/logstash.

Kibana

A tool for data visualization. For more information, refer to https://www.elastic.co/products/kibana.

24.1.2.1 Prerequisites

Java 8 is required by:

24.1.2.2 Installing and Configuring Elasticsearch

For methods of installing Elasticsearch, refer to https://www.elastic.co/guide/en/elasticsearch/reference/7.1/install-elasticsearch.html.

After installation, modify the config file /etc/elasticsearch/elasticsearch.yml to set the following value.

network.host: localhost

24.1.2.3 Installing and Configuring Logstash

For methods of installing Logstash, refer to http://www.elastic.co/guide/en/logstash/7.1/installing-logstash.html.

After installation, create the configuration file /etc/logstash/conf.d/00-scf.conf. In this example, we will name it 00-scf.conf. Add the following into the file. Take note of the port used in the input section. This value will need to match the value of the SCF_LOG_PORT property in your scf-config-values.yaml file.

input {
  tcp {
    port => 5001
  }
}
output {
  stdout {}
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "scf-%{+YYYY.MM.dd}"
  }
}

Additional input plug-ins can be found at https://www.elastic.co/guide/en/logstash/current/input-plugins.html and output plug-ins can be found at https://www.elastic.co/guide/en/logstash/current/output-plugins.html. For this example, we will demonstrate the flow of data through the stack, but filter plugins can also be specified to perform processing of the log data. For more details about filter plug-ins, refer to https://www.elastic.co/guide/en/logstash/current/filter-plugins.html.

24.1.2.4 Installing and Configuring Kibana

For methods of installing Kibana, refer to https://www.elastic.co/guide/en/kibana/7.1/install.html.

No configuration changes are required at this point. Refer to https://www.elastic.co/guide/en/kibana/current/settings.html for additonal properties that can configured through the kibana.yml file.

24.2 Log Levels

The log level is configured through the scf-config-values.yaml file by using the LOG_LEVEL property found in the env: section. The LOG_LEVEL property is mapped to component-specific levels. Components have differing technology compositions (for example languages, frameworks) and results in each component determining for itself what content to provide at each level, which may vary between components.

The following are the log levels available along with examples of log entries at the given level.

  • off: disable log messages

  • fatal: fatal conditions

  • error: error conditions

    <11>1 2018-08-21T17:59:48.321059+00:00 api-group-0 vcap.cloud_controller_ng
    - - -
    {"timestamp":1534874388.3206334,"message":"Mysql2::Error: MySQL
    server has gone away: SELECT count(*) AS `count` FROM `tasks` WHERE
    (`state` = 'RUNNING') LIMIT 1","log_level":"error","source":"cc.db","data":
    {},"thread_id":47367387197280,"fiber_id":47367404488760,"process_id":3400,"file":"/
    var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.4.0/
    gems/sequel-4.49.0/lib/sequel/database/logging.rb","lineno":88,"method":"block in
    log_each"}
  • warn: warning conditions

    <12>1 2018-08-21T18:49:37.651186+00:00 api-group-0 vcap.cloud_controller_ng
    - - -
    {"timestamp":1534877377.6507676,"message":"Invalid bearer token:
    #<CF::UAA::InvalidSignature: Signature verification failed> [\"/var/vcap/
    packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.4.0/gems/
    cf-uaa-lib-3.14.3/lib/uaa/token_coder.rb:118:in `decode'\", \"/var/vcap/packages/
    cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.4.0/gems/cf-uaa-
    lib-3.14.3/lib/uaa/token_coder.rb:212:in `decode_at_reference_time'\", \"/var/
    vcap/packages-src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/cloud_controller_ng/
    lib/cloud_controller/uaa/uaa_token_decoder.rb:70:in `decode_token_with_key'\",
    \"/var/vcap/packages-src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/
    cloud_controller_ng/lib/cloud_controller/uaa/uaa_token_decoder.rb:58:in
    `block in decode_token_with_asymmetric_key'\", \"/var/vcap/packages-
    src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/cloud_controller_ng/
    lib/cloud_controller/uaa/uaa_token_decoder.rb:56:in `each'\", \"/
    var/vcap/packages-src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/
    cloud_controller_ng/lib/cloud_controller/uaa/uaa_token_decoder.rb:56:in
    `decode_token_with_asymmetric_key'\", \"/var/vcap/packages-
    src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/cloud_controller_ng/lib/
    cloud_controller/uaa/uaa_token_decoder.rb:29:in `decode_token'\", \"/var/vcap/
    packages-src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/cloud_controller_ng/lib/
    cloud_controller/security/security_context_configurer.rb:22:in `decode_token'\", \"/
    var/vcap/packages-src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/cloud_controller_ng/
    lib/cloud_controller/security/security_context_configurer.rb:10:in `configure'\",
    \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/middleware/
    security_context_setter.rb:12:in `call'\", \"/var/vcap/packages/cloud_controller_ng/
    cloud_controller_ng/middleware/vcap_request_id.rb:15:in `call'\", \"/var/vcap/
    packages/cloud_controller_ng/cloud_controller_ng/middleware/cors.rb:49:in
    `call_app'\", \"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/
    middleware/cors.rb:14:in `call'\", \"/var/vcap/packages/cloud_controller_ng/
    cloud_controller_ng/middleware/request_metrics.rb:12:in `call'\", \"/
    var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/
    ruby/2.4.0/gems/rack-1.6.9/lib/rack/builder.rb:153:in `call'\", \"/var/vcap/
    packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.4.0/
    gems/thin-1.7.0/lib/thin/connection.rb:86:in `block in pre_process'\", \"/
    var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/
    ruby/2.4.0/gems/thin-1.7.0/lib/thin/connection.rb:84:in `catch'\", \"/var/
    vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.4.0/
    gems/thin-1.7.0/lib/thin/connection.rb:84:in `pre_process'\", \"/var/vcap/
    packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.4.0/
    gems/thin-1.7.0/lib/thin/connection.rb:50:in `block in process'\", \"/
    var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/
    ruby/2.4.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `block in
    spawn_threadpool'\"]","log_level":"warn","source":"cc.uaa_token_decoder","data":
    {"request_guid":"f3e25c45-a94a-4748-7ccf-5a72600fbb17::774bdb79-5d6a-4ccb-a9b8-
    f4022afa3bdd"},"thread_id":47339751566100,"fiber_id":47339769104800,"process_id":3245,"file":"/
    var/vcap/packages-src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/cloud_controller_ng/
    lib/cloud_controller/uaa/uaa_token_decoder.rb","lineno":35,"method":"rescue in
    decode_token"}
  • info: informational messages

    <14>1 2018-08-21T22:42:54.324023+00:00 api-group-0 vcap.cloud_controller_ng
    - - -
    {"timestamp":1534891374.3237739,"message":"Started GET
    \"/v2/info\" for user: , ip: 127.0.0.1 with vcap-request-id:
    45e00b66-e0b7-4b10-b1e0-2657f43284e7 at 2018-08-21 22:42:54
    UTC","log_level":"info","source":"cc.api","data":{"request_guid":"45e00b66-
    e0b7-4b10-
    b1e0-2657f43284e7"},"thread_id":47420077354840,"fiber_id":47420124921300,"process_id":3200,"file":
    var/vcap/packages/cloud_controller_ng/cloud_controller_ng/middleware/
    request_logs.rb","lineno":12,"method":"call"}
  • debug: debugging messages

    <15>1 2018-08-21T22:45:15.146838+00:00 api-group-0 vcap.cloud_controller_ng
    - - -
    {"timestamp":1534891515.1463814,"message":"dispatch
    VCAP::CloudController::InfoController get /v2/
    info","log_level":"debug","source":"cc.api","data":{"request_guid":"b228ef6d-
    af5e-4808-
    af0b-791a37f51154"},"thread_id":47420125585200,"fiber_id":47420098783620,"process_id":3200,"file":
    var/vcap/packages-src/8d7a6cd54ff4180c0094fc9aefbe3e5f43169e13/cloud_controller_ng/
    lib/cloud_controller/rest_controller/routes.rb","lineno":12,"method":"block in
    define_route"}
  • debug1: lower-level debugging messages

  • debug2: lowest-level debugging message

    <15>1 2018-08-21T22:46:02.173445+00:00 api-group-0 vcap.cloud_controller_ng - - -
    {"timestamp":1534891562.1731355,"message":"(0.006130s) SELECT * FROM `delayed_jobs`
    WHERE ((((`run_at` <= '2018-08-21 22:46:02') AND (`locked_at` IS NULL)) OR
    (`locked_at` < '2018-08-21 18:46:02') OR (`locked_by` = 'cc_api_worker.api.0.1'))
    AND (`failed_at` IS NULL) AND (`queue` IN ('cc-api-0'))) ORDER BY `priority`
    ASC, `run_at` ASC LIMIT 5","log_level":"debug2","source":"cc.background","data":
    {},"thread_id":47194852110160,"fiber_id":47194886034680,"process_id":3296,"file":"/
    var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.4.0/
    gems/sequel-4.49.0/lib/sequel/database/logging.rb","lineno":88,"method":"block in
    log_each"}
Print this page