This repository provides the sample code designed to interpret human demonstration videos and convert them into high-level tasks for robots.
MIT License
This repository provides the sample code designed to interpret human demonstration videos and convert them into high-level tasks for robots. Our Applied Robotics Research Group believes that this mechanism will serve as an effective interface for humans to instruct humanoid and industrial robots, for example, through the Learning-from-Observation framework. The prompts are designed for easy customization and seamless integration with existing robot control and visual recognition systems. For more information, please visit our project page and our paper, GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration. Please note that this repository only contains the sample code for task recognition using GPT-4V. It does not include mechanisms for incorporating user feedback or classical vision systems to recognize affordances from videos. For those components, please explore other repositories we have provided.
We have confirmed that the sample codes work with python 3.9.12 and 3.12.2
> pip install -r requirements.txt
python example.py sample_video/sample.mp4 --use_azure
We assume that the video path refers to a mp4 file of human demonstrating something. Add the --use_azure option if you use Azure OpenAI instead onf OpenAI
@article{wake2023gpt,
title={GPT-4V (ision) for robotics: Multimodal task planning from human demonstration},
author={Wake, Naoki and Kanehira, Atsushi and Sasabuchi, Kazuhiro and Takamatsu, Jun and Ikeuchi, Katsushi},
journal={arXiv preprint arXiv:2311.12015},
year={2023}
}
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.