Azure Container App via Terraform - platform_reserved_dns_ip_address is empty, not able to dynamically use NGINX as reverse prox

admin管理员组

文章数量:1127999

I'm deploying nginx as a reverse proxy in an Azure Container App environment.

As I'm proxy-passing dynamically, I need to explicitly set a resolver/dns ip address in the NGINX config.

I was on consumption plan until now, and had the resolver hardcoded to 10.0.0.2, which worked fine.

I now switched to a dedicated workload profile, with which that address no longer worked. It now seems to be changing dynamically upon environment creation.

Thus, I tried to dynamically alter the config file via terraform by using the app environment's platform_reserved_dns_ip_address. However, this value is just empty and does not pass a validation like below.

As per terraform documentation, infrastructure_subnet_id is configured so that should not be the issue.

Is there any way to either retrieve that resolver ip or workaround differently to be able to dynamically proxy in an ACA environment using NGINX?

Some code snippets:

# terraform excerpts from different modules

resource "azurerm_container_app_environment" "main" {
  name                           = "foo"
  location                       = var.location
  resource_group_name            = var.resource_group_name
  log_analytics_workspace_id     = azurerm_log_analytics_workspace.main.id
  infrastructure_subnet_id       = azurerm_subnet.containerEnv.id
  internal_load_balancer_enabled = var.ingress_vnet_only

  # note I'm only having the issue in dedicated, ie not consumption plan
  workload_profile {
    name                  = local.workload_profile.name
    workload_profile_type = local.workload_profile.workload_profile_type
    minimum_count         = local.workload_profile.minimum_count
    maximum_count         = local.workload_profile.maximum_count
  }
}

# output app environment
output "dns_ip_address" {
  value = azurerm_container_app_environment.main.platform_reserved_dns_ip_address
}

# input nginx
variable "dns_ip_address" {
  description = "IP address of the DNS server"
  type        = string

  validation {
    condition     = length(var.dns_ip_address) > 0
    error_message = "dns_ip_address must not be empty"
  }
}

# dynamic nginx config adaptation
resource "local_file" "nginx_config" {
  content = templatefile("${path.module}/config/nginx-template.conf", {
    dns_ip_address = var.dns_ip_address
  })
  filename = "${path.module}/auto-generated/nginx.conf"
}

# nginx conf excerpt
    resolver ${dns_ip_address}; 
    server {
        listen 80;
        server_name ~^(.*)\.foo\.bar$;

        location / {
            set $upstream_url http://$dynamic_address

            proxy_pass $upstream_url;
            proxy_http_version 1.1;
        }
    }

Also the JSON view of the app environment will say the IP address is null: "platformReservedDnsIP": null

I'm deploying nginx as a reverse proxy in an Azure Container App environment.

As I'm proxy-passing dynamically, I need to explicitly set a resolver/dns ip address in the NGINX config.

I was on consumption plan until now, and had the resolver hardcoded to 10.0.0.2, which worked fine.

I now switched to a dedicated workload profile, with which that address no longer worked. It now seems to be changing dynamically upon environment creation.

Thus, I tried to dynamically alter the config file via terraform by using the app environment's platform_reserved_dns_ip_address. However, this value is just empty and does not pass a validation like below.

As per terraform documentation, infrastructure_subnet_id is configured so that should not be the issue.

Is there any way to either retrieve that resolver ip or workaround differently to be able to dynamically proxy in an ACA environment using NGINX?

Some code snippets:

# terraform excerpts from different modules

resource "azurerm_container_app_environment" "main" {
  name                           = "foo"
  location                       = var.location
  resource_group_name            = var.resource_group_name
  log_analytics_workspace_id     = azurerm_log_analytics_workspace.main.id
  infrastructure_subnet_id       = azurerm_subnet.containerEnv.id
  internal_load_balancer_enabled = var.ingress_vnet_only

  # note I'm only having the issue in dedicated, ie not consumption plan
  workload_profile {
    name                  = local.workload_profile.name
    workload_profile_type = local.workload_profile.workload_profile_type
    minimum_count         = local.workload_profile.minimum_count
    maximum_count         = local.workload_profile.maximum_count
  }
}

# output app environment
output "dns_ip_address" {
  value = azurerm_container_app_environment.main.platform_reserved_dns_ip_address
}

# input nginx
variable "dns_ip_address" {
  description = "IP address of the DNS server"
  type        = string

  validation {
    condition     = length(var.dns_ip_address) > 0
    error_message = "dns_ip_address must not be empty"
  }
}

# dynamic nginx config adaptation
resource "local_file" "nginx_config" {
  content = templatefile("${path.module}/config/nginx-template.conf", {
    dns_ip_address = var.dns_ip_address
  })
  filename = "${path.module}/auto-generated/nginx.conf"
}

# nginx conf excerpt
    resolver ${dns_ip_address}; 
    server {
        listen 80;
        server_name ~^(.*)\.foo\.bar$;

        location / {
            set $upstream_url http://$dynamic_address

            proxy_pass $upstream_url;
            proxy_http_version 1.1;
        }
    }

Also the JSON view of the app environment will say the IP address is null: "platformReservedDnsIP": null

• azure
• nginx
• azure-devops
• terraform

Share Improve this question Follow edited Jan 9 at 13:25 honest_annie asked Jan 8 at 16:15 honest_anniehonest_annie 2177 bronze badges 11

• You could just use the same implicit reference as you are using in the output: dns_ip_address = azurerm_container_app_environment.main.platform_reserved_dns_ip_address. The variable is then not needed. – Marko E Commented Jan 8 at 17:26
• Are you executing the above one in Devops? @honest_annie – Jahnavi Commented Jan 9 at 5:28
• @MarkoE I just put different modules/files here into one code snippet for the sake of overview. Problem is as described that platform_reserved_dns_ip_address stays empty. – honest_annie Commented Jan 9 at 7:30
• 1 @Jahnavi No - running that locally and/or GItlab CI. – honest_annie Commented Jan 9 at 7:31
• Have you tried to retrieve the default domain by any chance? @honest_annie – Jahnavi Commented Jan 9 at 7:50

 |  Show 6 more comments

2 Answers 2

Sorted by: Reset to default 0

Platform_reserved_dns_ip_address property only visible when infrastructure_subnet_id is configured and it will be a in a range within the CIDR of the Subnet.

Refer terraform registry attribute reference for the relevant information.

Saying that, I have deployed a container app environment with infrastructure_subnet enabled and also within the range of CIDR under vnet integration and was able to view the reserved dns Ip address successfully.

terraform {
  required_providers {
    azapi = {
      source = "azure/azapi"
    }
  }
}

provider "azapi" {
}
provider "azurerm"{
features{}
subscription_id = "fxxxxb014"
}

data "azurerm_resource_group" "example" {
  name     = "createnewrg"
}

data "azurerm_log_analytics_workspace" "example"{
  name = "newmachine6367582528"
  resource_group_name = data.azurerm_resource_group.example.name
}
resource "azurerm_virtual_network" "network" {
  name                = "xxx"
  location            = data.azurerm_resource_group.example.location
  resource_group_name = data.azurerm_resource_group.example.name
  address_space       = ["10.0.0.0/16"]
}
 
resource "azurerm_subnet" "subnet" {
  name                 = "xxxx"
  resource_group_name  = data.azurerm_resource_group.example.name
  virtual_network_name = azurerm_virtual_network.network.name
  address_prefixes     = ["10.0.0.0/23"]
}
resource "azapi_resource" "containerapp_environment" {
  type      = "Microsoft.App/managedEnvironments@2022-03-01"
  name      = "acaacae"
  parent_id = data.azurerm_resource_group.example.id
  location  = data.azurerm_resource_group.example.location
 
  body = {
    properties = {
      appLogsConfiguration = {
        destination = "log-analytics"
        logAnalyticsConfiguration = {
          customerId = data.azurerm_log_analytics_workspace.example.workspace_id
          #sharedKey  = data.azurerm_log_analytics_workspace.example.primary_shared_key
        }
      }
      vnetConfiguration = {
        internal               = true
        infrastructureSubnetId = azurerm_subnet.subnet.id
        dockerBridgeCidr       = "10.2.0.1/16"
        platformReservedCidr   = "10.1.0.0/16"
        platformReservedDnsIP  = "10.1.0.2"
      }
    }
  }
  depends_on = [
    azurerm_virtual_network.network
  ]
  response_export_values  = ["properties.vnetConfiguration.platformReservedDnsIP"]
  ignore_missing_property = true
}
output "platform_reserved_dns_ip" {
  value = azapi_resource.containerapp_environment.output.properties.vnetConfiguration.platformReservedDnsIP
}

Output block:

output  "platform_reserved_dns_ip"  {
 value = azapi_resource.containerapp_environment.output.properties.vnetConfiguration.platformReservedDnsIP
}

Deployment succeeded:

Reference Blog Thomas Thornton for the above azapi terraform code.

After deployment, I have verified the same by using data block for container app environment and was also able to retrieve it as expected.

provider "azurerm"{
  features{}
subscription_id = "f7bxxxx4"
}

data "azurerm_resource_group" "example" {
  name     = "createnewrg"
}
data "azurerm_log_analytics_workspace" "example"{
  name = "newmachine6367582528"
  resource_group_name = data.azurerm_resource_group.example.name
}

data "azurerm_container_app_environment" "example"{
  name = "acaacae"
  resource_group_name = data.azurerm_resource_group.example.name
}
output "platform_reserved_dns_ip" {
  value = data.azurerm_container_app_environment.example.platform_reserved_dns_ip_address
}

I have now opened an issue here and have figured out a workaround, which uses the fact that the name server will still be present in a container's /etc/resolv.conf. It will just replace a variable in the config before NGINX is started:

# terraform excerpt
    container {
      name   = local.nginx
      image  = "${local.nginx}:${local.nginx_version}"
      cpu    = "0.5"
      memory = "1Gi"
      # As retrieving the dns ip directly via azurerm_container_app_environment.platform_reserved_dns_ip_address does not work in dedicated plan, we replace it manually.
      command = [
        "/bin/sh", "-c",
        <<-EOT
        echo "Starting entrypoint script...";
        DNS_IP=$(grep -m 1 'nameserver' /etc/resolv.conf | awk '{print $2}');
        [ -z "$DNS_IP" ] && echo 'No nameserver found' && exit 1;
        echo "Found DNS IP: $DNS_IP";
        cp /etc/nginx/nginx.conf /tmp/nginx.conf; \
        sed -i "s/PLACEHOLDER_DNS_IP/$DNS_IP/" /tmp/nginx.conf;
        echo "Nginx config:";
        cat /tmp/nginx.conf;
        exec nginx -c /tmp/nginx.conf -g 'daemon off;';
        EOT
      ]

      volume_mounts {
        name = "nginx-config"
        path = "/etc/nginx/"
      }
    }

本文标签：