Issue

When using .csx, we can bind a TextWriter log and make calls like log.Verbose("Foobar") and the streaming log does not write the records to the console.

Expected behavior

It should write the logs to the streaming logs (i.e. "Verbose: Foobar"), as does the Node experience.

Mitigation attempted

I attempted to add the following to host.json, but it did not address the issue:

    "tracing": {
      "consoleLevel": "Verbose"
    }

We need to define the set of allowed characters, length, etc. for function names, since they are used for file names / directories, HTTP routes, etc.

GitHub WebHook always returns returns the following error when invoked via the "Run Now" endpoint:

{
   "_body":"{\"Message\":\"An error has occurred.\"}",
   "status":500,
   "statusText":"Ok",
   "headers":{
     "Pragma":["no-cache"],
     "Date":["Thu"," 10 Mar 2016 05:16:34 GMT"],
     "Server":["Microsoft-IIS/8.0"],
     "X-AspNet-Version":["4.0.30319"],
     "instance-name":["functions-bay"],
     "X-Powered-By":["ASP.NET"],
     "Content-Type":["application/json; charset=utf-8"],
     "Cache-Control":["no-cache"],
     "Content-Length":["36"],
     "Expires":["-1"]
  },
  "type":2,
  "url":"**URL HERE**"
}

Ideally this would work like the Generic WebHook where it will pass the content through as JSON.

When using a queue as input for a "normal" WebJobs SDK function, the queue message will not be dequeued if the function fails, but will be retried and eventually added to a poisoned queue (https://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-webjobs-sdk-storage-queues-how-to/#poison).

WebJobs.Script currently requires failures to be handled manually: throwing an error in a Node.JS function will simply crash the whole process. And the message will be removed from the queue nonetheless.

It would be great if this actually worked similar to the WebJobs SDK.

Need to allow the binding to specify allowed HTTP verbs, etc.

At first I understood that we could pass somme parameters to the bindings out in function.json ({name} is used in your samples) by adding them in the object passed to the done function but it seems that these parameters are taken from the original input message. Do you think it would be possible to use the object passed to done as parameters for output binding? (could be used to define the blob name, the content, etc)

This line cause multiple instances to use the same log file and failed in multiple instances. Suggestion would be to (similar to Kudu) add machine name in file name and time stamp to help with gc old log files.

2016-02-25T01:49:42 System.IO.IOException: The process cannot access the file 'D:\home\LogFiles\Application\Functions\Host\561a87bf-dc59-49d7-a7d4-7c3fc023fe19.log' because it is being used by another process.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.StreamWriter.CreateFile(String path, Boolean append, Boolean checkHost)
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize, Boolean checkHost)
   at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding)
   at System.IO.File.InternalAppendAllText(String path, String contents, Encoding encoding)
   at System.IO.File.AppendAllText(String path, String contents)
   at Microsoft.Azure.WebJobs.Script.FileTraceWriter.AppendLine(String line)
   at Microsoft.Azure.WebJobs.Script.FileTraceWriter.Trace(TraceEvent traceEvent)
   at Microsoft.Azure.WebJobs.Host.CompositeTraceWriter.InvokeTraceWriters(TraceEvent traceEvent)
   at Microsoft.Azure.WebJobs.Host.CompositeTraceWriter.Trace(TraceEvent traceEvent)
   at Microsoft.Azure.WebJobs.Host.Loggers.TraceWriterFunctionInstanceLogger.LogFunctionStartedAsync(FunctionStartedMessage message, CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.Host.Loggers.CompositeFunctionInstanceLogger.<LogFunctionStartedAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<ExecuteWithLoggingAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.<TryExecuteAsync>d__1.MoveNext()

I used an invalid parameter in my C# queue binding and got an NPE in the script host. There should be a separate step of metadata validation that provides a nicer error message.

I used path instead of queueName, so I expected an error in the script output window.

In Function.json:

{
    "bindings": [
        {
            "type": "queueTrigger",
            "direction": "in",
            "path": "resizerequest" 
        }
}

Log output:

2016-03-04T02:09:51 System.ArgumentNullException: Value cannot be null.
Parameter name: input
   at Microsoft.Azure.WebJobs.Host.Bindings.Path.BindingTemplateParser.ParseTemplate(String input)
   at Microsoft.Azure.WebJobs.Script.Binding.FunctionBinding.GetBindings(ScriptHostConfiguration config, IEnumerable`1 bindingMetadatas, FileAccess fileAccess)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptorProvider.TryCreate(FunctionMetadata functionMetadata, FunctionDescriptor& functionDescriptor)
   at Microsoft.Azure.WebJobs.Script.Description.CSharpFunctionDescriptionProvider.TryCreate(FunctionMetadata functionMetadata, FunctionDescriptor& functionDescriptor)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.ReadFunctions(List`1 metadatas, IEnumerable`1 descriptorProviders)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.ReadFunctions(ScriptHostConfiguration config, IEnumerable`1 descriptionProviders)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize()
   at Microsoft.Azure.WebJobs.Script.ScriptHost.Create(ScriptHostConfiguration scriptConfig)
   at Microsoft.Azure.WebJobs.Script.ScriptHost

Testing GitHub WebHooks :)

I noticed that you reference a couple of libs that are far more common in the F# world (suave, FsPickler, fsharp.core - see https://github.com/Azure/azure-webjobs-sdk-script/blob/master/src/WebJobs.Script.Host/packages.config), but you don't seem to use these yet.

Are you planning to create a F# specific version?

I have written a node.js WebJob with WebHook trigger. It's not documented how to trigger a node.js function.
Using WebJob Extensions SDK we could do that using the following URL scheme

https://{site}.scm.azurewebsites.net/api/continuouswebjobs/{job}/passthrough/{class}/{function}

After playing with a function for a while, it can get into a state where each Save causes multiple compilations. e.g. all at once:

2016-03-03T20:44:08 Script for function 'cs' changed. Reloading.
2016-03-03T20:44:08 Script for function 'cs' changed. Reloading.
2016-03-03T20:44:08 Script for function 'cs' changed. Reloading.
2016-03-03T20:44:08 Script for function 'cs' changed. Reloading.
2016-03-03T20:44:08 Script for function 'cs' changed. Reloading.
2016-03-03T20:44:08 Script for function 'cs' changed. Reloading.
2016-03-03T20:44:08 Script for function 'cs' changed. Reloading.
2016-03-03T20:44:08 Compiling function script.
2016-03-03T20:44:08 Compiling function script.
2016-03-03T20:44:08 Compiling function script.
2016-03-03T20:44:08 Compiling function script.
2016-03-03T20:44:08 Compiling function script.
2016-03-03T20:44:08 Compiling function script.
2016-03-03T20:44:08 Compiling function script.
2016-03-03T20:44:08 Compiling function script.
2016-03-03T20:44:08 Compiling function script.

Need to get unit test coverage in place for all classes / features. Test coverage is really sparse right now.

There are challenges currently to supporting in/out binding in scripts for types other than C#. The reason is that for all the other script types, values have to get marshalled (e.g. input bindings write to a temp file, output bindings read from temp file). That pipeline of course loses any object identity, which is required by the SDK read/write bindings.

Need to figure out the story for Function execution timeouts, both to ensure that functions/processes that never terminate (e.g. while(true) cases) don't run forever. This might take the form of a default timeout that can be user configured per function via config. Note that the core WebJobs SDK has a TimeoutAttribute which we could leverage.

E.g. we might set a safety default timeout of something like 5 minutes or so to prevent runaway executions, but allow people to raise that for long running batch scenarios.

Similarly, we need to handle process timeouts for functions that are spawned exes (e.g. BAT functions)

As we saw, this no longer seems to work. e.g. with

{
  "bindings": [
    {
      "type": "blobTrigger",
      "name": "original",
      "direction": "in",
      "path": "images-original/{name}"
    },
    {
      "type": "blob",
      "name": "resized",
      "direction": "out",
      "path": "images-resized/{name}"
    }
  ]
}

and

echo %original%

original is empty.

I shouldn't need to do a private binary drop of Microsoft.Azure.WebJobs.dll and Newtonsoft.Json.dll to be able to use these from a function. Can we include these two by default?

Hi,

I'm working on getting this deployed and having some issues, so just wanted to clarify a few things from the Deployment section of the wiki:

• the wiki says "You deploy all this to a Continuous WebJob as you would normally for a regular WebJobs SDK job." I'm following Amit Apple's instructions for where to store jobs. That said, would these jobs go in the continuous folder even though they are triggered (queue for example)?
• is run.cmd supposed to live in wwwroot/? and if so, do the only contents need to be .\bin\WebJobs.Script.Host.exe?

I copied the simple QueueTrigger example, but it keeps saying the status is Pending Restart, hence my questions above. Thinking something may be wrongly configured.

Thanks in advance!
James

Invalid binding parameters cause the host to crash.

This should only impact C#, since it passes the bindings to the WebJobs SDK in the target function.

Example: blob output binding with path: "some-container-name/{invalidparam}"

e.g. function (workItem, context) should become function (context, workItem).

Right now it's text/plain. We can always let people override via response object, but application/json should be the default.

Currently, WebHooks use code=... (coming from WebHook SDK), while plain http use key=... (coming from our own logic).

One way or another, let's find a way to make them be the same.

D:\home\SiteExtensions\Kudu\bin\Scripts\NuGet.exe won't work with pre-installed site extension.

One option is to look for it under D:\Program Files (x86)\SiteExtensions\Kudu\{first folder in there. They all have it}\bin\Scripts

If there are errors in the function.json file, these need to be captured and written to the function log so the config errors are visible to the user.

I.e., in log streaming scenarios, these need to pop up. In general we need to solidify the error handling/reporting story to ensure that all config errors (including host config) are made visible / actionable.

I am trying to run webjob python script and unable to import packages. All the packages are installed in wwwroot/env (the web site virtual env). How can I activate the virtualenv from the webjob?

The ScriptHost registers an INameResolver which supports useful things for bindings like %rand-guid% %rand-int%. We should also modify it to also pull from app settings, to support scenarios where users want to bind based on setting values. Also round out the list of auto resolved values.

Whenever I try to deploy a WebJob using WebJobs.script as described on the wiki (bin, run.cmd, host.json, function folder) on either a Basic or Free web app, deployment fails with " Failed to add WebJob". There is no further information as to the cause. After upgrading to a Standard tier deployment works as expected.

Is it impossible to deploy on free and basic instances?

Need to clean up the whole binding pipeline for incoming HttpRequests, including:

• support for body types other than json (e.g. form url encoding, etc.)
• proper content type support + content negotiation

Call fails with:

The WebHook verification request must contain a 'code' query parameter

Currently the temp files used for the binding process are written to the LogFiles\Application\Functions\Binding directory. They don't belong in the Logs directory, they belong in TEMP storage.

Need to do proper error handling to ensure that if an exception causes the manager loop to fail (and therefor it's managed instance remains in a dead state) we can recover. We see this happen from time to time, and it results in the WebHost EnsureHostRunningHandler to return 503 until the site is restarted.

We support project.json package management for C# already. We should do something similar for Node.

Don't delete this issue. It is used to host Wiki images as per the technique described here

The ScriptHost is not disposed if an exception occurs in the run loop in the ScriptHostManager.

I had to resort to private reflection in #75 to get access to the binding data we need. Need to figure out the right long term approach, whether to expose this info publically from the core SDK.

To make it possible for the Functions to perform authorization, we should pass the various auth claims into the context object.

Idea is to allow users to specify a new "user" auth level value to indicate that a function requires an EasyAuth validated identity.

The suggestion is to return 204 (NoContent) or 200 with hello-world kind of content.

The declarative way to define webjob script is amazing.

However, I don't think it is a good idea to depend on C# code when developing a Node.js app. It could be better to provide a NPM package to natively support Node.js.

When the WebJobs SDK is running under Kudu SCM, it handles this via a signal file that the Kudu process manager “touches”, and the SDK is watching that to perform graceful shutdown.

Since we’re not running under Kudu we’ll have to handle this ourselves. The SDK already has graceful shutdown plumbed properly, so we’ll just need to signal it a the right time.

Currently the operations performed when processing input/output bindings involve unnecessary conversions to/from Stream. Something workable was put in place, but it needs to be refactored/simplified for performance and code reusability.

We should also see if we can simply the type hierarchy - currently we need a separate binding class for each binding type.

Also, investigate removing IResultProcessingBinding - this was added as a quick thing to get C# working, but it should be rationalized with the rest of the binding pipeline.

Create a default binding if not specified explicitly.

And use json.net schema validation to validate

For host/function secret keys, we need to encrypt them at rest on the file system

To allow bindings to target different storage accounts

Inspect function changes to prevent unnecessary host restarts if the function signature does not change.

I don't have clear repro steps, but somehow my host got into this bad state, where it fails to start with this error:

2016-03-04T00:25:33 System.UnauthorizedAccessException: Access to the path is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.MemoryMappedFiles.MemoryMappedView.CreateView(SafeMemoryMappedFileHandle memMappedFileHandle, MemoryMappedFileAccess access, Int64 offset, Int64 size)
   at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateViewAccessor(Int64 offset, Int64 size, MemoryMappedFileAccess access)
--- End of stack trace from previous location where exception was thrown ---
   at System.Reflection.Internal.MemoryMapLightUp.CreateViewAccessor(Object memoryMap, Int64 start, Int32 size)
   at System.Reflection.Internal.StreamMemoryBlockProvider.TryCreateMemoryMappedFileBlock(Int64 start, Int32 size, MemoryMappedFileBlock& block)
   at System.Reflection.Internal.StreamMemoryBlockProvider.GetMemoryBlockImpl(Int32 start, Int32 size)
   at System.Reflection.Internal.MemoryBlockProvider.GetMemoryBlock(Int32 start, Int32 size)
   at System.Reflection.PortableExecutable.PEReader.GetMetadataBlock()
   at System.Reflection.Metadata.PEReaderExtensions.GetMetadataReader(PEReader peReader, MetadataReaderOptions options, MetadataStringDecoder utf8Decoder)
   at Microsoft.CodeAnalysis.PEModule.InitializeMetadataReader()
   at Microsoft.CodeAnalysis.PEModule.get_MetadataReader()
   at Microsoft.CodeAnalysis.PEAssembly..ctor(AssemblyMetadata owner, ImmutableArray`1 modules)
   at Microsoft.CodeAnalysis.AssemblyMetadata.GetOrCreateData()
   at Microsoft.CodeAnalysis.AssemblyMetadata.GetModules()
   at Microsoft.CodeAnalysis.AssemblyMetadata.IsValidAssembly()
   at Microsoft.CodeAnalysis.CommonReferenceManager`2.GetMetadata(PortableExecutableReference peReference, CommonMessageProvider messageProvider, Location location, DiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CommonReferenceManager`2.ResolveMetadataReferences(TCompilation compilation, Dictionary`2 assemblyReferencesBySimpleName, ImmutableArray`1& references, IDictionary`2& boundReferenceDirectiveMap, ImmutableArray`1& boundReferenceDirectives, ImmutableArray`1& assemblies, ImmutableArray`1& modules, DiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.ReferenceManager.CreateAndSetSourceAssemblyFullBind(CSharpCompilation compilation)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.ReferenceManager.CreateSourceAssemblyForCompilation(CSharpCompilation compilation)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.GetRuntimeMetadataVersion(EmitOptions emitOptions)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.GetRuntimeMetadataVersion(EmitOptions emitOptions, DiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CreateModuleBuilder(EmitOptions emitOptions, IMethodSymbol debugEntryPoint, IEnumerable`1 manifestResources, CompilationTestData testData, DiagnosticBag diagnostics, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(EmitStreamProvider peStreamProvider, EmitStreamProvider pdbStreamProvider, EmitStreamProvider xmlDocumentationStreamProvider, EmitStreamProvider win32ResourcesStreamProvider, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, CompilationTestData testData, Func`1 getHostDiagnostics, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, CompilationTestData testData, Func`1 getHostDiagnostics, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, CancellationToken cancellationToken)
   at Microsoft.Azure.WebJobs.Script.Description.CSharpFunctionInvoker.GetFunctionTarget()
   at Microsoft.Azure.WebJobs.Script.Description.CSharpFunctionDescriptionProvider.GetFunctionParameters(IFunctionInvoker functionInvoker, FunctionMetadata functionMetadata, BindingMetadata triggerMetadata, Collection`1 methodAttributes, Collection`1 inputBindings, Collection`1 outputBindings)
   at Microsoft.Azure.WebJobs.Script.Description.FunctionDescriptorProvider.TryCreate(FunctionMetadata functionMetadata, FunctionDescriptor& functionDescriptor)
   at Microsoft.Azure.WebJobs.Script.Description.CSharpFunctionDescriptionProvider.TryCreate(FunctionMetadata functionMetadata, FunctionDescriptor& functionDescriptor)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.ReadFunctions(List`1 metadatas, IEnumerable`1 descriptorProviders)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.ReadFunctions(ScriptHostConfiguration config, IEnumerable`1 descriptionProviders)
   at Microsoft.Azure.WebJobs.Script.ScriptHost.Initialize()
   at Microsoft.Azure.WebJobs.Script.ScriptHost.Create(ScriptHostConfiguration scriptConfig)
   at Microsoft.Azure.WebJobs.Script.ScriptHostManager.RunAndBlock(CancellationToken cancellationToken)

While host is running, if I make a bad save to host.json (e.g. forget closing brace), the whole WebHost dies and doesn't recover after fixing the file. Same goes for function.json. At that point, the whole process needs a restart.

I think problem is that the exception ends up exiting the do/while loop in RunAndBlock.

We need to make this more resilient.

A queue message containing nested JSON get the nested properties turned into arrays.

The following queue message:

{"method":"foo","attr":{"templateName":"verify-user","templateContent":[],"detail":{"tags":["one","two"]},"check":false}}

Will output the following:

Node.js queue trigger function processed work item {"method":"foo","attr":[[[]],[[]],[[[[[],[]]]]],[[]]]}

When using the following function:

module.exports = function (context) {
context.log('Node.js queue trigger function processed work item ' + JSON.stringify(context.email))
}

The following function runs fine:

using System;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs.Host;

public static Task<HttpResponseMessage> Run(HttpRequestMessage input, TraceWriter log)
{
    HttpResponseMessage res = new HttpResponseMessage(HttpStatusCode.OK)
    {
        Content = new StringContent("Please pass a name on the query string")
    };

    return Task.FromResult(res);
}

Now replace HttpRequestMessage input with HttpRequestMessage req, and it blows up at runtime with a generic 500 error.

So really two issues:

1. Why does the parameter name matter?
2. Can we improve error handling?