Platform for Situated Intelligence (or in short, \psi, pronounced like the greek letter) is an open, extensible framework for development and research of multimodal, integrative-AI systems. These are systems that process various types of streaming sensor data (such as audio, video, depth, etc.) and that need to leverage and coordinate a variety of component technologies. Examples range from social robots or embodied agents that interact with people, to smart spaces such as instrumented meeting rooms, all the way to applications based on small devices that process streaming sensor data.

The framework alleviates the engineering challenges that arise when building such systems by providing:

• a modern, performant infrastructure for working with multimodal, temporally streaming data, and a programming paradigm for concurrent, coordinated computation that simplifies application development.
• a set of tools for multimodal data visualization, annotation, and processing, which support and accelerate debugging and maintenance.
• an ecosystem of components for various sensors, processing technologies, and effectors, enabling rapid prototyping and reuse.

The core infrastructure in Platform for Situated Intelligence is built on .NET Standard and therefore runs both on Windows and Linux. Some components and tools are more specific and are available only on one or the other operating system.

You can get started building \psi applications in two ways:

• leveraging \psi NuGet packages, or
• cloning and building the source code.

To learn more about \psi and how to build applications with it, we recommend you start with the Brief Introduction tutorial, which will walk you through for some of the main concepts. It shows how to create a simple program, describes the core concept of a stream, and explains how to transform, synchronize, visualize, persist and replay streams from disk.

The documentation for \psi is available in the github project wiki. It contains various informational resources, including tutorials, samples, and other specialized topics that can help you learn more about the framework.

If you find a bug or if you would like to request a new feature or additional documentation, please file an issue in github. Use the bug label when filing issues that represent code defects, and provide enough information to reproduce the bug. Use the feature request label to request new features, and use the documentation label to request additional documentation.

We are looking forward to engaging with the community to improve and evolve Platform for Situated Intelligence! We welcome contributions in many forms: from simply using it and filing issues and bugs, to writing and releasing your own new components, to creating pull requests for bug fixes or new features. The Contributing Guidelines page in the wiki describes many ways in which you can get involved, and some useful things to know before contributing to the code base.

To find more information about our future plans, please see the Roadmap document.

Platform for Situated Intelligence is currently being used in several industry and academic research labs, including (but not limited to):

• the Situated Interaction project, as well as other research projects at Microsoft Research.
• the MultiComp Lab at Carnegie Mellon University.
• the Speech Language and Interactive Machines research group at Boise State University.
• the Qualitative Reasoning Group, Northwestern University.
• the Intelligent Human Perception Lab, at USC Institute for Creative Technologies.
• the Teledia research group, at Carnegie Mellon University.
• the F&M Computational, Affective, Robotic, and Ethical Sciences (F&M CARES) lab, at Franklin and Marshall College.

If you would like to be added to this list, just file a GitHub issue and label it with the whoisusing label. Add a url for your research lab, website or project that you would like us to link to.

The codebase is currently in beta and various aspects of the framework are under active development. There are probably still bugs in the code and we may make breaking API changes.

Platform for Situated Intelligence is available under an MIT License. See also Third Party Notices.

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

We would like to thank our internal collaborators and external early adopters, including (but not limited to): Daniel McDuff, Kael Rowan, Lev Nachmanson and Mike Barnett at MSR, Chirag Raman and Louis-Phillipe Morency in the MultiComp Lab at CMU, as well as researchers in the SLIM research group at Boise State and the Qualitative Reasoning Group at Northwestern University.