OTLP gRPC exporter not routing logs to AWS load balancer

[g3kr]

Describe the bug
I am doing a basic set up of open telemetry collector and agent. I am using a AWS Network loadbalancer endpoint for the otlp exporter. I was hoping it would route traffic to the target ips behind the load balancer, however it does not seen to work. I suspect the issue is with gRPC/TLS protocol, but not sure how to resolve. I have also tried to replace the NLB with ALB, it didn't work.

Steps to reproduce

1. Run otel-agent scraping for hostmetrics and export them to otel-collector fronted with AWS NLB

LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Type: network
      Scheme: internal
      Subnets: !Ref Subnets
      LoadBalancerAttributes:
        - Key: access_logs.s3.enabled
          Value: true
        - Key: access_logs.s3.bucket
          Value: alb-bucket
  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    DependsOn: LoadBalancer
    Properties:
      VpcId: vpc-0cd88c97e6ea97f0
      Port: 4317
      Protocol: TCP
      TargetType: ip
  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    DependsOn: LoadBalancer
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref TargetGroup
      LoadBalancerArn: !Ref LoadBalancer
      Port: 4317
      Protocol: TLS
      SslPolicy: ELBSecurityPolicy-TLS-1-2-2017-01	
      Certificates:
        - CertificateArn: !Ref Arn

What did you expect to see?
Host metrics being routed to the otel-collector with error

What did you see instead?
The error I see with Application LoadBalancer

{
    "kind": "exporter",
    "name": "otlp",
    "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded",
    "interval": "42.326371045s"
}```

The error I see with Network LoadBalancer 

```2022-04-20T18:03:41.384Z	warn	zapgrpc/zapgrpc.go:191	[core] grpc: addrConn.createTransport failed to connect to {10.127.158.110:4317 otel-LoadB-CN90JA7YF0DC-65470f93d920ee47.elb.us-west-2.amazonaws.com:4317 <nil> <nil> 0 <nil>}: failed to receive server preface within timeout	
{
    "grpc_log": true
}

What version did you use?
Version: opentelemetry-collector-contrib:latest

What config did you use?

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  hostmetrics:
    scrapers:
      cpu:
      disk:
      filesystem:
      load:
      memory:
      network:
      processes:

processors:
  batch: null

exporters:
  logging:
    loglevel: debug
  otlp:
    endpoint: dns:internal-otel-LoadB-UIL979Q3D802-382813316.us-west-2.elb.amazonaws.com:4317 #ALB
    tls:
      insecure: true
  otlp/2:
    endpoint: dns:otel-LoadB-CN90JA7YF0DC-65470f93d920ee47.elb.us-west-2.amazonaws.com:4317  #NLB
    tls:
      insecure: true

extensions:
  health_check:
    port: 13133

service:
  extensions: [health_check]
  pipelines:
    metrics:
      receivers:
        - otlp
        - hostmetrics
      processors:
        - batch
      exporters:
        - otlp
        - otlp/2

Environment
OS: AWS Linux 2

I would appreciate any help or directions to get past this. TIA

[jangaraj]

Your ALB target group needs better config with GRPC support, e.g.:

 TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId: !Ref VpcId
      Port: 4317
      Protocol: HTTP
      ProtocolVersion: GRPC
      HealthCheckIntervalSeconds: 5
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      Matcher:
        GrpcCode: 0-99
      HealthCheckTimeoutSeconds: 4
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2
      TargetType: ip

[bogdandrutu]

This issue seems to be resolved,

[gjshao44]

Your ALB target group needs better config with GRPC support, e.g.:

 TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId: !Ref VpcId
      Port: 4317
      Protocol: HTTP
      ProtocolVersion: GRPC
      HealthCheckIntervalSeconds: 5
      HealthCheckPath: /
      HealthCheckProtocol: HTTP
      Matcher:
        GrpcCode: 0-99
      HealthCheckTimeoutSeconds: 4
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 2
      TargetType: ip

Just to note, this suggested setup will fail as port 4317 does not support GRPC healthcheck. Putting it on port 13133 will also fail as it only supports http healthcheck, not GRPC. To resolve this issue, we need #8397, which is code reviewed, but not merged