Unable to perform azds up due to KeyNotFoundException in StreamBuildContainerLogsAsync about dev-spaces HOT 11 CLOSED

Thanks for reporting this. We're looking into it.

Out of curiousity, what do you see in Visual Studio? We try to surface as much error information as possible in VS itself -- so, if you're in a situation where the CLI's error message is clearer than VS's, we'll want to look into that, too.

from dev-spaces.

I think the same information might have been there, but it was a little harder to associate cause and effect because so many things happened at once. The project I was working in has three azds-enabled projects and all of them were attempting to do the same thing in parallel. Super cool that it works that way, but it made sorting out what order things were happening in sort of difficult. Additionally, it was a little difficult to separate the initial error from the subsequent exceptions thrown as a result of an unexpected outcome earlier in the process. Doing it via the CLI sort of forced a one-step-at-a-time approach which made it clearer to me -- might have been just as obvious to someone with more experience with azds than I have.

If you'd like to see what actually happens via VS, I can certainly provide that. Just let me know.

from dev-spaces.

@mprigge Can you find the log file for azds and share the complete exception. Unfortunately, due to a bug, we were not logging telemetry for some time.

You can find the azds log in %TEMP%/Azure Dev Spaces find the log file closest to the time stamp and get the complete exception message.

from dev-spaces.

This what you're looking for?

"2018-09-11T16:55:39.7566766Z" : Exception invoking GetServiceAsync {"ClassName":"Microsoft.AspNetCore.SignalR.HubException","Message":"An unexpected error occurred invoking 'GetServiceStatusAsync' on the server. ExecInternalServerErrorException: The given key 'spec' was not present in the dictionary.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.AspNetCore.SignalR.Client.HubConnection.<InvokeCoreAsyncCore>d__35.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.AspNetCore.SignalR.Client.HubConnection.<InvokeCoreAsync>d__29.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.AspNetCore.SignalR.Client.HubConnectionExtensions.<InvokeCoreAsync>d__34`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Mindaro.Client.ServiceClients.ExecSignalRServerProxy.<_RunAsync>d__17`1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Mindaro.Client.ManagementClients.ServiceManagementClientImplementation.<>c__DisplayClass26_1.<<GetServiceStatusAsync>b__0>d.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Mindaro.Client.ServiceClients.ExecSignalRServiceClient.<RunAsync>d__16.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at Microsoft.Mindaro.Client.ManagementClients.ServiceManagementClientImplementation.<GetServiceStatusAsync>d__26.MoveNext()","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":"8\nThrowForNonSuccess\nmscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\nSystem.Runtime.CompilerServices.TaskAwaiter\nVoid ThrowForNonSuccess(System.Threading.Tasks.Task)","HResult":-2146233088,"Source":"mscorlib","WatsonBuckets":null}(subscriptionId=6ebda517-4b6f-4e67-bbd9-f605a680eb7f)

Note that this is the exception thrown by the Visual Studio tooling. The message of the exception is the same either way, though.

from dev-spaces.

Thanks, @mprigge for reporting the issue, we have identified the root cause. This issue will happen if there is no spec node in the service specified in the helm chart. Can you please confirm that this is the case with your service also.

apiVersion: v1 kind: Service metadata: name: {{ template "webfrontend.fullname" . }} labels: app: {{ template "webfrontend.name" . }} chart: {{ template "webfrontend.chart" . }} release: {{ .Release.Name }} heritage: {{ .Release.Service }} **spec:** type: {{ .Values.service.type }} ports: - port: {{ .Values.service.port }} targetPort: http protocol: TCP name: http selector: app: {{ template "webfrontend.name" . }} release: {{ .Release.Name }}

Is this spec node missing from the chart's service.yaml. Will it be possible for you to share your service.yaml, to verify the fix.

from dev-spaces.

Hi @jikuma - My helm charts do indeed have spec nodes for both a Deployment and Service. However, if you're processing the helm chart outside of helm, it may be worth noting that I am using multi-line annotations in the Service (ingress configuration for Ambassador).

So, for instance, one of the service definitions looks something like this:

apiVersion: v1
kind: Service
metadata:
  name: {{ template "chartname.fullname" . }}
  labels:
    app: {{ template "chartname.name" . }}
    chart: {{ template "chartname.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
    environment: {{ .Values.global.environment }}
  annotations:
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v0
      kind:  Mapping
      name:  {{ include "chartname.fullname" . }}
      ambassador_id: {{ .Values.global.environment }}
      prefix: {{ template "chartname.ingressPrefix" . }}
      service: {{ include "chartname.fullname" . }}
      host: {{ template "chartname.ingressHostname" . }}
spec:
  type: ClusterIP
  ports:
    - port: 80
      targetPort: http
      protocol: TCP
      name: http
  selector:
    app: {{ template "chartname.name" . }}
    release: {{ .Release.Name }}

So, there definitely is a spec node, but I'm not sure if the "unusual" annotation that precedes it might be causing a problem with whatever parsing you're doing. It is worth noting that this is a valid configuration from Helm/Kubernetes' perspective.

from dev-spaces.

Thanks @mprigge the issue has been fixed and will be deployed to production in a couple of weeks. Thanks for reporting the issue.

Unfortunately, you won't be able to do azds up on the same service/azds workload till the fix reaches your environment, but you should be able to create a new folder/service and use dev space.

from dev-spaces.

Thank you! Glad I was able to help.

So, "fixing" the chart in some way will not work as a workaround today? Was it the annotation that was causing the problem?

Also, is there a good way to tell whether the environment has been patched? (not sure if I need to recreate the devspaces controller or not)

from dev-spaces.

No, it was not the annotation, it was a parsing code which we wrote had a bug. We parse the currently deployed release before deploying the chart. This is used to update the release as per azds requirement. If the parsing fails in the previously deployed release there is no way we can work around it by changing the current chart. Hence we need to wait for the fix. I will get back to you on, how to tell when env is updated.

from dev-spaces.

@lisaguthrie Is there any blog, we publish after rollout which I can point here.

from dev-spaces.

@mprigge the fix should have reached your controller. Can you please verify.

from dev-spaces.