diff --git a/doc/Tutorials.rst b/doc/Tutorials.rst index f22047c3..eadcdb35 100644 --- a/doc/Tutorials.rst +++ b/doc/Tutorials.rst @@ -11,5 +11,6 @@ Tutorials tutorials/Multi-Container tutorials/Multi-PC tutorials/Running-Packages + tutorials/Create-New-Package tutorials/Setting-Up-a-New-Robot tutorials/Usage diff --git a/doc/tutorials/Create-New-Package.md b/doc/tutorials/Create-New-Package.md new file mode 100644 index 00000000..f7172bd6 --- /dev/null +++ b/doc/tutorials/Create-New-Package.md @@ -0,0 +1,157 @@ +# How to create an HARMONI package +Harmoni follows the standard ROS conventions for package structure. +Below you can find some info about how to create a package that follows the HARMONI Unit conventions. + +[General info on how to create a ROS package](http://wiki.ros.org/ROS/Tutorials/CreatingPackage). +Notice that HARMONI uses *harmoni_catkin_ws* as workspace instead of *catkin_ws* described in the tutorial. + +## *CMakeLists.txt* and *setup.py* and *package.xml* +[MANDATORY] + +These files store all the information needed to correctly build a package with its dependencies. +These files are very similar among packages, so it is suggested to copy them from another package and modify the package name/path. +To build a package use +> catkin build [your_package] + +For more info see: [CMakeLists.txt documentation](http://wiki.ros.org/catkin/CMakeLists.txt) + +## *launch* folder +[MANDATORY] + +The *launch* folder stores the *.launch* file or files. +The structure of a *[your_package].launch* file is very similar to the *[your_package].test* file. The only difference is that the *[your_package].launch * file does NOT have the line that starts with . + +## *src* folder and *nodes* folder +[MANDATORY] + +Here is the actual implementation of the service. Some packages store the code related to service class in a *src* folder while some have a *nodes* folder. +It is suggested to use the template provided (*harmoni_core/harmoni_common_lib/src/harmoni_common_lib/service.py.template*). + +*[your_package]_service.py* contains the service implementation. + +Some services act on a per request basis, meaning that they receive some optional data, they do something with it and they return a response. +For example, the TTS service may receive some text in input and its job is to produce an audio file from the written text. +Services that act on a per request basis must implement the *request* method. + +The *request* method return this type of message : {"response": self.state, "message": self.result_msg} , where self.state describes the state of the service (see *harmoni_core/harmoni_common_lib/src/harmoni_common_lib/constants.py* ) while self.result_msg stores the output result. + +There are other services that, once started, keep on running. +For example, the microphone service, once started, keeps sending audio data. +These kind of services implement the *start* and *stop* methods. + +## Data +The general policy is that you use the HARMONI service system to send small commands and info messages among services. + +However, if you have high density data (e.g. images and audio), do [publishing and subscription as normal ROS nodes](http://wiki.ros.org/ROS/Tutorials/WritingPublisherSubscriber%28python%29) +In this case, be sure to follow the HARMONI namespace guidelines so that multiple packages can publish/subscribe to known interfaces (see [Namespaces and the *constants.py* file section](#constants)). This approach is especially useful when there are multiple packages that provide the same data, as is the case with different types of STT services. + + +## README file +[STRONGLY SUGGESTED] +There should be a brief description of what the package does. +There should be an entry for each parameter set in configuration.yaml with a brief description and value. +You should write what type of messages your package uses, if your package uses non-standard messages. + +``` +| Parameters | Definition | Values | +|----------------------|------------|--------| +|parameter_1 | | | +|parameter_2 | | | +|parameter_3 | | | +``` + + +## Tests +[STRONGLY SUGGESTED] + +The *test* folder stores the tests created for the service. +There are three types of files in this folder: a *[your_package].test* file, a *rostest-[your_package].py* file and a *unittest-[your_package].py* file. + +The *.test* file is very similar to the *.launch* file that is stored in the *launch* folder. +*[your_package].test* is a file that connects the configuration parameters with the actual implementation of the service, which is stored in the *src* or *nodes* folders. +The file *[your_package].test* contains also the link to the actual test file, which is usually called *rostest_[your_package].py*. +The file *[your_package].test* tries to complete the task written in the *rostest_[your_package].py* file, which can succed or fail. + +The structure of the *[your_package].test* file is usually like: + +``` + + + + + + + + +``` +Parameters can be specified in the *configuration.yaml* file or directly in the *.test* file. For example you could add a line to have an input parameter with the value "Hello". +``` + +``` + +Test files and launch files make use of namespaces when including config files. For this reason, it is important to follow the HARMONI namespace guidelines (see [Namespaces and the *constants.py* file section](#constants)). +If these are not added, it is possible for concurrently running services to overwrite eachother's params (e.g. in *harmoni_detectors/harmoni_face_detect/launch/face_detect_service.launch*). + + + + +## *config* folder and configuration.yaml file +[OPTIONAL] + +The *configuration.yaml* file is a file where the user stores information useful to run the service correctly. For example, to run the harmoni_microphone service Harmoni must know what device is the correct one. So, the *configuration.yaml* file for the harmoni_microphone service stores the name of the device you want to use. + +The *configuration.yaml* file stores the default_param, that is the default configuration to run the service. If you want you can create multiple parameters that have the same structure as the default_param. + +The parameters' values are then retrieved in service class implementation. + + +## *msg* folder +[OPTIONAL] + +This folder should be inside the package if the package doesn't use the standard messages. The description of the newly created message should be written in the README. + + +## Other folders (e.g. the *web* folder in harmoni_web or the *temp_data* folder in harmoni_camera) +[OPTIONAL] + +If needed, you may create additional folders. + + +## HARMONI conventions to follow + +### Where should you place your package? +It depends on what does your package do. + +Your package should be put in the folder corresponding to the type of service you have created (e.g. *actuators* "do" something, *sensors* retrieve data...). + +- Actuators -> *harmoni_actutators* + +- Detectors -> *harmoni_detectors* + +- Sensors -> *harmoni_sensors* + +- Dialogues -> *harmoni_dialogues* + +### Namespaces and the *constants.py* file +The HARMONI namespace guidelines require you to add the name of the service in the *constants.py* file that is in *harmoni_core/harmoni_common_lib/src/harmoni_common_lib/*. + +A new line should be put in the Enum corresponding to the type of service you have decided for your package. + +If your service is an actuator a line with the name of your service should be added in the ActuatorNameSpace(Enum), if it is a detector in the DetectorNameSpace(Enum) and so on... + +For example, if your service is of type *sensor*: + +``` +class SensorNameSpace(Enum): + microphone = "/harmoni/sensing/microphone/" + camera = "/harmoni/sensing/camera/" + [your_service] = "/harmoni/sensing/[your_service]" +``` + +## Troubleshooting +- Check that your package has built or do: +> catkin build [your_package] +- Check that the *.py* files (in *src* or *nodes*) in your package are executable diff --git a/docker-compose.yml b/docker-compose.yml index cf201143..a23277fa 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -19,7 +19,7 @@ services: CATKIN_WS: harmoni_catkin_ws privileged: true networks: - ros_net: + ros_net: ipv4_address: 172.18.3.4 hostname: harmoni_full ports: @@ -35,6 +35,7 @@ services: # Configuration - ~/.aws:/root/.aws/ - ~/.gcp/private-keys.json:/root/.gcp/private-keys.json + - ~/.hass:/root/.hass/ # Other - /tmp/.X11-unix:/tmp/.X11-unix - /etc/timezone:/etc/timezone:ro diff --git a/harmoni_actuators/harmoni_hass/CMakeLists.txt b/harmoni_actuators/harmoni_hass/CMakeLists.txt new file mode 100644 index 00000000..ecc9ecde --- /dev/null +++ b/harmoni_actuators/harmoni_hass/CMakeLists.txt @@ -0,0 +1,197 @@ +cmake_minimum_required(VERSION 2.8.3) +project(harmoni_hass) + +## Compile as C++11, supported in ROS Kinetic and newer +# add_compile_options(-std=c++11) + +## Find catkin macros and libraries +## if COMPONENTS list like find_package(catkin REQUIRED COMPONENTS xyz) +## is used, also find other catkin packages +find_package(catkin REQUIRED COMPONENTS + roscpp + rospy +) + +## System dependencies are found with CMake's conventions +# find_package(Boost REQUIRED COMPONENTS system) + + +## Uncomment this if the package has a setup.py. This macro ensures +## modules and global scripts declared therein get installed +## See http://ros.org/doc/api/catkin/html/user_guide/setup_dot_py.html +catkin_python_setup() +################################################ +## Declare ROS messages, services and actions ## +################################################ + +## To declare and build messages, services or actions from within this +## package, follow these steps: +## * Let MSG_DEP_SET be the set of packages whose message types you use in +## your messages/services/actions (e.g. std_msgs, actionlib_msgs, ...). +## * In the file package.xml: +## * add a build_depend tag for "message_generation" +## * add a build_depend and a exec_depend tag for each package in MSG_DEP_SET +## * If MSG_DEP_SET isn't empty the following dependency has been pulled in +## but can be declared for certainty nonetheless: +## * add a exec_depend tag for "message_runtime" +## * In this file (CMakeLists.txt): +## * add "message_generation" and every package in MSG_DEP_SET to +## find_package(catkin REQUIRED COMPONENTS ...) +## * add "message_runtime" and every package in MSG_DEP_SET to +## catkin_package(CATKIN_DEPENDS ...) +## * uncomment the add_*_files sections below as needed +## and list every .msg/.srv/.action file to be processed +## * uncomment the generate_messages entry below +## * add every package in MSG_DEP_SET to generate_messages(DEPENDENCIES ...) + +## Generate messages in the 'msg' folder +# add_message_files( +# FILES +# Message1.msg +# Message2.msg +# ) + +## Generate services in the 'srv' folder +# add_service_files( +# FILES +# Service1.srv +# Service2.srv +# ) + +## Generate actions in the 'action' folder +# add_action_files( +# FILES +# Action1.action +# Action2.action +# ) + +## Generate added messages and services with any dependencies listed here +# generate_messages( +# DEPENDENCIES +# std_msgs # Or other packages containing msgs +# ) + +################################################ +## Declare ROS dynamic reconfigure parameters ## +################################################ + +## To declare and build dynamic reconfigure parameters within this +## package, follow these steps: +## * In the file package.xml: +## * add a build_depend and a exec_depend tag for "dynamic_reconfigure" +## * In this file (CMakeLists.txt): +## * add "dynamic_reconfigure" to +## find_package(catkin REQUIRED COMPONENTS ...) +## * uncomment the "generate_dynamic_reconfigure_options" section below +## and list every .cfg file to be processed + +## Generate dynamic reconfigure parameters in the 'cfg' folder +# generate_dynamic_reconfigure_options( +# cfg/DynReconf1.cfg +# cfg/DynReconf2.cfg +# ) + +################################### +## catkin specific configuration ## +################################### +## The catkin_package macro generates cmake config files for your package +## Declare things to be passed to dependent projects +## INCLUDE_DIRS: uncomment this if your package contains header files +## LIBRARIES: libraries you create in this project that dependent projects also need +## CATKIN_DEPENDS: catkin_packages dependent projects also need +## DEPENDS: system dependencies of this project that dependent projects also need +catkin_package( +# INCLUDE_DIRS include +# LIBRARIES harmoni_hass +# CATKIN_DEPENDS roscpp rospy +# DEPENDS system_lib +) + +########### +## Build ## +########### + +## Specify additional locations of header files +## Your package locations should be listed before other locations +include_directories( +# include + ${catkin_INCLUDE_DIRS} +) + +## Declare a C++ library +# add_library(${PROJECT_NAME} +# src/${PROJECT_NAME}/harmoni_hass.cpp +# ) + +## Add cmake target dependencies of the library +## as an example, code may need to be generated before libraries +## either from message generation or dynamic reconfigure +# add_dependencies(${PROJECT_NAME} ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS}) + +## Declare a C++ executable +## With catkin_make all packages are built within a single CMake context +## The recommended prefix ensures that target names across packages don't collide +# add_executable(${PROJECT_NAME}_node src/harmoni_hass_node.cpp) + +## Rename C++ executable without prefix +## The above recommended prefix causes long target names, the following renames the +## target back to the shorter version for ease of user use +## e.g. "rosrun someones_pkg node" instead of "rosrun someones_pkg someones_pkg_node" +# set_target_properties(${PROJECT_NAME}_node PROPERTIES OUTPUT_NAME node PREFIX "") + +## Add cmake target dependencies of the executable +## same as for the library above +# add_dependencies(${PROJECT_NAME}_node ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS}) + +## Specify libraries to link a library or executable target against +# target_link_libraries(${PROJECT_NAME}_node +# ${catkin_LIBRARIES} +# ) + +############# +## Install ## +############# + +# all install targets should use catkin DESTINATION variables +# See http://ros.org/doc/api/catkin/html/adv_user_guide/variables.html + +## Mark executable scripts (Python etc.) for installation +## in contrast to setup.py, you can choose the destination +# install(PROGRAMS +# scripts/my_python_script +# DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION} +# ) + +## Mark executables and/or libraries for installation +# install(TARGETS ${PROJECT_NAME} ${PROJECT_NAME}_node +# ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION} +# LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION} +# RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION} +# ) + +## Mark cpp header files for installation +# install(DIRECTORY include/${PROJECT_NAME}/ +# DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION} +# FILES_MATCHING PATTERN "*.h" +# PATTERN ".svn" EXCLUDE +# ) + +## Mark other files for installation (e.g. launch and bag files, etc.) +# install(FILES +# # myfile1 +# # myfile2 +# DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION} +# ) + +############# +## Testing ## +############# + +## Add gtest based cpp test target and link libraries +# catkin_add_gtest(${PROJECT_NAME}-test test/test_harmoni_hass.cpp) +# if(TARGET ${PROJECT_NAME}-test) +# target_link_libraries(${PROJECT_NAME}-test ${PROJECT_NAME}) +# endif() + +## Add folders to be run by python nosetests +catkin_add_nosetests(test/unittest_hass.py) diff --git a/harmoni_actuators/harmoni_hass/README.md b/harmoni_actuators/harmoni_hass/README.md new file mode 100644 index 00000000..f579d1d1 --- /dev/null +++ b/harmoni_actuators/harmoni_hass/README.md @@ -0,0 +1,84 @@ + +## Home Assistant Service Parameters: +Parameters input for the home assistant service: + +| Parameters | Definition | Values | +|----------------------|------------|--------| +|hass_uri | URL on which Home Assistant is available | e.g. "http://10.254.254.50:8123/" | +|credential_path | path to the credentials file (see section below) | e.g. "$(env HOME)/.hass/token.json" | +|simulation | pretend that a device has been on for too long (see "check_log" section) | True/False | + + +INFO: We often refer to Home Assistant as "hass" + +## Setting up your Home Assistant + +Follow this guide to make your [Home Assistant setup](https://www.home-assistant.io/installation/) + +Only Home Assistant Container has been tested (using Docker). To run Home Assistant using Docker: +``` +docker start homeassistant +``` +Make sure that Home Assistant is running before making requests to harmoni_hass. To check if Home Assistant is running or not try to access it using the URL. + +### Set up credentials +The parameter *credential_path* refers to the file where the home assistant authentication token is stored. + +The structure of this json file is: +``` +{ + "token" : "YOUR_HOME_ASSISTANT_TOKEN" +} +``` + +## REST API +[Home Assistant's REST API documentation](https://developers.home-assistant.io/docs/api/rest/) + +IMPORTANT: entity_id is formed by the device domain and its name (e.g. entity_id for google home is media_player.google_home) + +### Turn on/off a device +| Parameters |Values | +|----------------------|--------| +|action | turn_on/turn_off | +|entity_id | \ | +|answer (optional) | yes/no | + +Example: {"action":"turn_off", "entity_id":"switch.oven_power"} + +Example with optional parameter: {"answer":"yes", "action":"turn_off", "entity_id":"oven_power", "type":"switch"} +If the answer parameter is set to "no", the action won't be executed. + +This command uses a POST API call to: /api/services/\/\turn_on or /api/services/\/\turn_off + + +### Play music from device +| Parameters |Values | +|----------------------|--------| +|action | play_media | +|entity_id | \ | +|media_content_id | \ | +|media_content_type | \ | +|answer (optional) | yes/no | + +Example: {"action":"play_media", "entity_id": "media_player.googlehome8554", "media_content_id": "media-source://media_source/local/relax.wav", "media_content_type": "audio/wav", "answer":"yes"} + +This command uses a POST API call to: /api/services/\/\play_media + +To set the media source for Home Assistant see [here](https://www.home-assistant.io/integrations/media_source/) and [here](https://www.home-assistant.io/more-info/local-media/setup-media/). + + + +### Check if an appliance is on for longer than X (default is 3 hours) +| Parameters |Values | +|----------------------|--------| +|action | check_log | +|entity | \ | +|answer (optional) | yes/no | +|hours (optional) | \ | +|minutes (optional) | \ | +|seconds (optional) | \ | +|days (optional) | \ | + +Example: "{ "action":"check_log", "entity":"switch.oven_power", "hours" : "4", "answer":"yes"}" + +This command uses a POST API call to: /api/logbook/\ (formatted as "2021-05-24T10:00:00+00:00") \ No newline at end of file diff --git a/harmoni_actuators/harmoni_hass/config/configuration.yaml b/harmoni_actuators/harmoni_hass/config/configuration.yaml new file mode 100644 index 00000000..b241115c --- /dev/null +++ b/harmoni_actuators/harmoni_hass/config/configuration.yaml @@ -0,0 +1,6 @@ +# Configuration file for home assistant +hass: + default_param: + hass_uri: "https://10.254.254.50:8123/" + credential_path: "$(env HOME)/.hass/token.json" #path where the home assistant token is mounted + simulation: True \ No newline at end of file diff --git a/harmoni_actuators/harmoni_hass/launch/hass_service.launch b/harmoni_actuators/harmoni_hass/launch/hass_service.launch new file mode 100755 index 00000000..53c4563c --- /dev/null +++ b/harmoni_actuators/harmoni_hass/launch/hass_service.launch @@ -0,0 +1,7 @@ + + + + + + + diff --git a/harmoni_actuators/harmoni_hass/package.xml b/harmoni_actuators/harmoni_hass/package.xml new file mode 100644 index 00000000..9cd79397 --- /dev/null +++ b/harmoni_actuators/harmoni_hass/package.xml @@ -0,0 +1,31 @@ + + + harmoni_hass + 0.0.0 + The harmoni_hass package + Eleonora Toscano + + + MIT + + + + Eleonora Toscano + + + catkin + rospy + python-catkin-pkg + rospy + rospy + rosunit + rostest + + + + + + + diff --git a/harmoni_actuators/harmoni_hass/setup.py b/harmoni_actuators/harmoni_hass/setup.py new file mode 100644 index 00000000..8e9b1dbc --- /dev/null +++ b/harmoni_actuators/harmoni_hass/setup.py @@ -0,0 +1,12 @@ +# ! DO NOT MANUALLY INVOKE THIS setup.py, USE CATKIN INSTEAD + +from distutils.core import setup +from catkin_pkg.python_setup import generate_distutils_setup + +# fetch values from package.xml +setup_args = generate_distutils_setup( + packages=['harmoni_hass'], + package_dir={'': 'src'}, +) + +setup(**setup_args) diff --git a/harmoni_actuators/harmoni_hass/src/hass_service.py b/harmoni_actuators/harmoni_hass/src/hass_service.py new file mode 100755 index 00000000..bf5f8680 --- /dev/null +++ b/harmoni_actuators/harmoni_hass/src/hass_service.py @@ -0,0 +1,292 @@ +#!/usr/bin/env python3 + +# Common Imports +from typing_extensions import OrderedDict +import rospy +import roslib + +from harmoni_common_lib.constants import State, ActuatorNameSpace +from harmoni_common_lib.service_server import HarmoniServiceServer +from harmoni_common_lib.service_manager import HarmoniServiceManager +import harmoni_common_lib.helper_functions as hf + +# Specific Imports +from std_msgs.msg import String, Bool +import numpy as np +import json +import requests +from datetime import datetime, timedelta +import pytz +#import ast + +class HassService(HarmoniServiceManager): + """ + Hass service + """ + + def __init__(self, name, param): + """ Initialization of variables and home assistant parameters """ + super().__init__(name) + self.name = name + self.service_id = hf.get_child_id(self.name) + + # The Home Assistant uri set in the configuration file + self.hass_uri = param["hass_uri"] + + # The path where the authorization token (from Home Assistant's settings) is + self.credential_path = param["credential_path"] + with open(self.credential_path) as f: + d = json.load(f) + self.token = d["token"] + + # POST ACTIONS + self.post_actions = { "turn_on", "turn_off", "play_media" } + + # Pretend that an appliance has been on for a few hours + self.simulation = param["simulation"] + + self.state = State.INIT + return + + def request(self, data): + """Completes the home assistant request if the input data is json, otherwise forwards the input data. + + Args: + data (str): if it contains "{", the string is interpreted as json. + This string of json which contains 3 items: {"action": str, "entity": str, "type": str} .... + action: the action you want to send to home assistant (e.g. turn_on, turn_off or check_log) + type: the entity type of the device (e.g., media_player, switch, light) + entity: the device on which to do the action (e.g. googlehome8554) + + Returns: + object: It containes information about the response received (bool) and response message (str) + response: bool + message: str + """ + + rospy.loginfo("Start the %s request" % self.name) + self.state = State.REQUEST + + try: + + if "{" not in data: + rospy.loginfo("No { in data, forwarding message") + self.result_msg = data + self.state = State.SUCCESS + self.response_received = True + + else: + rospy.loginfo("{ in data") + data_list = data.split("{") + message_to_forward = data_list[0] + data_list[1] = "{"+ data_list[1] + rospy.loginfo(data_list[0]) + rospy.loginfo(data_list[1]) + # TODO MANAGE MULTIPLE COMMANDS or ONLY COMMANDS NO TEXT + + rospy.loginfo("Request: %s " % data_list[1]) + json_data= json.loads(data_list[1]) + + # Responses from bot may have json commands in them but they can also be ignored using this parameter + if("answer" in json_data and json_data["answer"] == "no"): + self.state = State.SUCCESS + self.response_received = True + self.result_msg = message_to_forward # No action done + + else: + rospy.loginfo("Action: %s " % json_data["action"]) + + + if(json_data["action"] == "check_log"): + hass_response = self.check_log(json_data) + + elif(json_data["action"] in self.post_actions): + hass_response = self.post(json_data) + self.result_msg = message_to_forward + self.result_msg + + + rospy.loginfo(f"The status code for Home Assistant's response is {hass_response.status_code}") + rospy.loginfo(f"Home assistant request text: {hass_response.text}") + rospy.loginfo(f"Home assistant request url: {hass_response.request.url}") + rospy.loginfo(f"Home assistant request headers: {hass_response.request.headers}") + rospy.loginfo(f"Home assistant request body: {hass_response.request.body}") + + if hass_response is not None and hass_response.status_code == 200: + self.state = State.SUCCESS + self.response_received = True + + else: + self.start = State.FAILED + rospy.logerr("Service call failed") + rospy.logerr(f"Home Assistant's response is {hass_response.text}, with status code {hass_response.status_code}") + rospy.logerr("Did you put the correct uri and token in the configuration file?") + self.response_received = True + + except rospy.ServiceException as e: + self.start = State.FAILED + rospy.logerr("Service call failed") + rospy.logerr(e) + self.response_received = True + self.result_msg = e + + return {"response": self.state, "message": self.result_msg} + + + def check_log(self, json_data): + """Check if an appliance has been on for some time + + Args: + data (str): string of json which contains 3 items: {"action": str, "entity": str, "type": str} .... + action: "check_log" + entity_id: the entity type of the device (e.g., media_player, switch, light) and the device name (e.g. googlehome8554) + + Returns: + hass_response (str): It containes the response to the API request api/logbook + """ + + rospy.loginfo("Entity id: %s " + json_data["entity_id"]) + myHeaders = {"Authorization": "Bearer "+ self.token} + + # Home Assistant returns all info in UTC + dateTimeObj = datetime.now(pytz.utc) + rospy.loginfo("Current time: %s " % str(dateTimeObj)) + + # How much time before the current time I want to check for events (default is 3 hours) + m = s = d = 0 + h = 3 + + if("hours" in json_data): + h = int(json_data["hours"]) + if("days" in json_data): + d = int(json_data["days"]) + if("minutes" in json_data): + m = int(json_data["minutes"]) + if("seconds" in json_data): + s = int(json_data["seconds"]) + + delta = timedelta( + days = d, + hours = h, + minutes = m, + seconds = s + ) + + rospy.loginfo("Timespan to check: %s " % str(delta)) + timeToCheck = dateTimeObj - delta + + # formatting "2021-05-24T10:00:00+00:00" + timeToCheckFormatted = str(timeToCheck.year) + "-" + str(timeToCheck.month) + "-" + str(timeToCheck.day) + "T" + str(timeToCheck.hour) +":"+ str(timeToCheck.minute) + ":" + str(timeToCheck.second) + "+00:00" + rospy.loginfo("Time to check: %s " % timeToCheckFormatted) + + url = self.hass_uri + 'api/logbook/' + timeToCheckFormatted +'?' + "entity:"+ json_data["entity_id"] + + hass_response = requests.get( + url, + headers=myHeaders + ) + + json_array = hass_response.json() + eventTime = "" + + for item in json_array: + if "context_service" in item: + + if item["context_service"] in self.post_actions: + eventTime = item["when"] + + elif item["context_service"] in self.post_actions: + eventTime = "" + + # TODO ALSO CHECK STATE = "OFF" IF ENTITY_ID IS THE CORRECT ONE + + alertUser = False + + if eventTime is not "": + dateEventTime = datetime.strptime(eventTime, '%Y-%m-%dT%H:%M:%S.%f+00:00') + dateEventTime = dateEventTime.replace(tzinfo=pytz.utc) + delta = dateTimeObj - dateEventTime + rospy.loginfo(f"Timespan appliance on: {str(delta)}") + + # Check if the home appliance has been on for a few hours + if delta > timedelta(hours=2): + alertUser = True + +# TODO custom return msg, saved outside of this code + + if alertUser == True: + self.result_msg = "LOG: oven still on" + else: + self.result_msg = "NONE" + rospy.loginfo(f"Is appliance on? {alertUser}") + + # Check if this is a simulation + if self.simulation == True: + self.result_msg = "LOG: oven still on" + rospy.loginfo(f"Is simulation on? {self.simulation}") + + return hass_response + + + def post(self, json_data): + """ Do an API POST call + + Args: + data (str): string of json which contains 2 or 3 items: {"action": str, "entity_id": str} .... + action: "turn_on" or "turn_off" or other "post" actions + entity_id: the entity type of the device (e.g., media_player, switch, light) + with, separated by a dot, the name of the device on which to do the action (e.g. googlehome8554) + EXAMPLE "entity_id" : "mediaplayer.googlehome8554" + + Returns: + hass_response (str): It containes the response to the API request api/services + """ + + if "entity_id" in json_data: + rospy.loginfo("Entity: %s " % json_data["entity_id"]) + + parameters = {} + for x in json_data: + if x != "answer" and x != "action": + parameters[x] = json_data[x] + rospy.loginfo("json " + str(parameters)) + + myHeaders = {"Authorization": "Bearer "+ self.token} + + type = json_data["entity_id"].split(".")[0] + + url = self.hass_uri + 'api/services/' + type +'/' + json_data["action"] + + hass_response = requests.post( + url, + json=parameters, + headers=myHeaders, + + # SELF-SIGNED CERTIFICATE FOR A LOCAL CONNECTION + verify=False + ) + + # self.result_msg = hass_response.text + self.result_msg = " Fatto" + + return hass_response + + +def main(): + """Set names, collect params, and give service to server""" + + service_name = ActuatorNameSpace.hass.name + instance_id = rospy.get_param("/instance_id") + service_id = f"{service_name}_{instance_id}" + try: + rospy.init_node(service_name) + params = rospy.get_param(service_name + "/" + instance_id + "_param/") + s = HassService(service_name, params) + service_server = HarmoniServiceServer(service_id, s) + service_server.start_sending_feedback() + rospy.spin() + except rospy.ROSInterruptException: + pass + + +if __name__ == "__main__": + main() diff --git a/harmoni_actuators/harmoni_hass/test/hass.test b/harmoni_actuators/harmoni_hass/test/hass.test new file mode 100644 index 00000000..632e8396 --- /dev/null +++ b/harmoni_actuators/harmoni_hass/test/hass.test @@ -0,0 +1,8 @@ + + + + + + + + diff --git a/harmoni_actuators/harmoni_hass/test/rostest_hass.py b/harmoni_actuators/harmoni_hass/test/rostest_hass.py new file mode 100755 index 00000000..29b45b0a --- /dev/null +++ b/harmoni_actuators/harmoni_hass/test/rostest_hass.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 + + +PKG = "test_harmoni_hass" +# Common Imports +import unittest, rospy, roslib, sys + +# Specific Imports +from actionlib_msgs.msg import GoalStatus +from harmoni_common_msgs.msg import harmoniAction, harmoniFeedback, harmoniResult +from std_msgs.msg import String +from harmoni_common_lib.action_client import HarmoniActionClient +from std_msgs.msg import String +from harmoni_common_lib.constants import ActuatorNameSpace, ActionType +from collections import deque +import os, io +import ast + + +class TestHass(unittest.TestCase): + + def setUp(self): + """ + Set up the client for requesting to harmoni_hass + """ + rospy.init_node("test_hass", log_level=rospy.INFO) + self.text = rospy.get_param("test_hass_input") # "{ 'action':'turn_on', 'entity':'googlehome8554', 'type':'media_player'}" + self.instance_id = rospy.get_param("instance_id") + self.result = False + self.name = ActuatorNameSpace.hass.name + "_" + self.instance_id + self.service_client = HarmoniActionClient(self.name) + self.client_result = deque() + self.service_client.setup_client(self.name, self.result_cb, self.feedback_cb) + # NOTE currently no feedback, status, or result is received. + rospy.loginfo("TestHass: Started up. waiting for home assistant startup") + rospy.loginfo("TestHass: Started") + + def feedback_cb(self, data): + rospy.loginfo(f"Feedback: {data}") + self.result = False + + def status_cb(self, data): + rospy.loginfo(f"Status: {data}") + self.result = False + + def result_cb(self, data): + rospy.loginfo(f"Result: {data}") + self.result = True + + def test_request_response(self): + rospy.loginfo(f"The input text is {self.text}") + self.service_client.send_goal( + action_goal=ActionType.REQUEST.value, + optional_data=self.text, + wait=True, + ) + assert self.result == True + + + +def main(): + import rostest + + rospy.loginfo("test_hass started") + rospy.loginfo("TestHass: sys.argv: %s" % str(sys.argv)) + rostest.rosrun(PKG, "test_hass", TestHass, sys.argv) + + +if __name__ == "__main__": + main() diff --git a/harmoni_actuators/harmoni_hass/test/unittest_hass.py b/harmoni_actuators/harmoni_hass/test/unittest_hass.py new file mode 100755 index 00000000..e234100f --- /dev/null +++ b/harmoni_actuators/harmoni_hass/test/unittest_hass.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 + + +PKG = 'test_harmoni_hass' +# Common Imports +import unittest, rospy, roslib, sys +#from unittest.mock import Mock, patch +# Specific Imports +from actionlib_msgs.msg import GoalStatus +from harmoni_common_lib.constants import State +from harmoni_common_msgs.msg import harmoniAction, harmoniFeedback, harmoniResult +from std_msgs.msg import String +import os, io +import ast +from harmoni_hass.hass_service import HassService +import json + + +class TestHass(unittest.TestCase): + + def __init__(self, *args): + super(TestHass, self).__init__(*args) + + def setUp(self): + self.test_hass_input = { "action":"turn_on", "entity":"googlehome8554", "type":"media_player"} + self.result = False + rospy.loginfo("TestHass: Started up. waiting for hass startup") + self.hass_service = HassService("test_hass") + rospy.loginfo("TestHass: Started") + + + def test_request_response(self): + # Send a request to the real API server and store the response. + response = self.hass_service.request(self.test_hass_input) + # Confirm that the request-response cycle completed successfully. + rospy.loginfo(response) + if response["response"]==State.SUCCESS: + rospy.loginfo("The response succeed") + self.result = True #set the response to true if the request succeeded + assert(self.result == True) + +def main(): + #TODO convert to a test suite so that setup doesn't have to run over and over. + import rosunit + rospy.loginfo("test_hass started") + rospy.loginfo("TestHass: sys.argv: %s" % str(sys.argv)) + rosunit.unitrun(PKG, 'test_hass', TestHass, sys.argv) + +if __name__ == "__main__": + main() diff --git a/harmoni_actuators/harmoni_tts/src/harmoni_tts/__init__.py b/harmoni_actuators/harmoni_tts/__init__.py similarity index 100% rename from harmoni_actuators/harmoni_tts/src/harmoni_tts/__init__.py rename to harmoni_actuators/harmoni_tts/__init__.py diff --git a/harmoni_actuators/harmoni_tts/aws_tts_service.py b/harmoni_actuators/harmoni_tts/aws_tts_service.py new file mode 100755 index 00000000..550b3be3 --- /dev/null +++ b/harmoni_actuators/harmoni_tts/aws_tts_service.py @@ -0,0 +1,326 @@ +#!/usr/bin/env python3 + +# Common Imports +import rospy +import roslib + +from harmoni_common_lib.constants import State +from harmoni_common_lib.service_server import HarmoniServiceServer +from harmoni_common_lib.service_manager import HarmoniServiceManager +import harmoni_common_lib.helper_functions as hf + +# Specific Imports +from harmoni_common_lib.constants import ActuatorNameSpace +from botocore.exceptions import BotoCoreError, ClientError +from contextlib import closing +import soundfile as sf +import numpy as np +import boto3 +import re +import json +import ast +import sys + + +class AWSTtsService(HarmoniServiceManager): + """ + Amazon tts service + """ + + def __init__(self, name, param): + """Constructor method: Initialization of variables and polly parameters + setting up""" + super().__init__(name) + """ Initialization of variables and tts parameters """ + self.region_name = param["region_name"] + self.voice = param["voice"] + self.language = param["language"] + self.outdir = param["outdir"] + self.wav_header_length = param["wav_header_length"] + """ Setup the tts request """ + self._setup_aws_tts() + """Setup the tts service as server """ + self.state = State.INIT + return + + def _setup_aws_tts(self): + """[summary] Setup the tts polly request, connecting to AWS services""" + self.tts = boto3.client("polly", region_name=self.region_name) + self.vis_transl = { + "p": "BILABIAL", + "f": "LABIODENTAL", + "T": "INTERDENTAL", + "s": "DENTAL_ALVEOLAR", + "t": "DENTAL_ALVEOLAR", + "S": "POSTALVEOLAR", + "r": "POSTALVEOLAR", + "k": "VELAR_GLOTTAL", + "i": "CLOSE_FRONT_VOWEL", + "u": "CLOSE_BACK_VOWEL", + "@": "MID_CENTRAL_VOWEL", + "a": "OPEN_FRONT_VOWEL", + "e": "OPEN_FRONT_VOWEL", + "E": "OPEN_FRONT_VOWEL", + "o": "OPEN_BACK_VOWEL", + "O": "OPEN_BACK_VOWEL", + "sil": "IDLE", + "J": "PALATAL", + } + return + + def _split_text(self, text): + """[summary] + Split long sentences + Args: + text (str): Sentence to be synthetised + + Returns: + text_array (list): array which containes the part of the text splitted + """ + if "." in text: + text_array = text.split(".") + else: + text_array = [] + text_array.append(text) + return text_array + + def _split_behaviors(self, s): + """[summary] + Split the text from the behaviors + Args: + s (str): input text + + Returns: + list: list of splitted text and actions + """ + if len(s) >= 2 and s[-1] == "*" and s[0] == "*": + return [s] + else: + return re.split("\s+", s) + + def _get_text_and_actions(self, sentence): + """[summary] + Get text and actions from the sentence + Args: + sentence (str): Input text before requesting + + Returns: + (phrase, actions): Get the text and the action + """ + tokens = re.split("(\*[^\*\*]*\*)", sentence) + phrase = "".join(list(filter(lambda s: "*" not in s, tokens))) + rospy.loginfo("Processing the phrase: %s" % phrase) + tokens = list(map(lambda s: self._split_behaviors(s), tokens)) + words = [] + for t in tokens: + words += list(filter(lambda s: len(s) > 0, t)) + actions = [] + i = 0 + for w in words: + if re.match("\*.*\*", w): + args = w.strip("*").split() + name = args.pop(0) + actions.append([i, name, args]) + else: + i += 1 + return (phrase, actions) + + def _get_behaviors(self, response, actions): + """[summary] + Processing the response from AWS Polly and get the behaviors + Args: + response (json): Response from Polly service which contains information about word timing, duration, and visemes + actions (json): Collects the actions in the text (words into stars **: gesture and facial expressions) + + Returns: + data (json): It contains all the data including words and actions information + """ + + xSheet = [] + if "AudioStream" in response: + with closing(response["AudioStream"]) as stream: + data = stream.read() + xSheet = data.split(b"\n") + xSheet = [line.decode("utf-8") for line in xSheet if line != ""] + xSheet = [json.loads(line) for line in xSheet if line != ""] + else: + print("Could not stream audio") + word_times = list(filter(lambda l: l["type"] == "word", xSheet)) + data = [] + for w in word_times: + data.append( + { + "character": float(w["start"]) / 1000.0, # convert ms to seconds + "type": "word", + "start": float(w["time"]) / 1000.0, + "value": str(w["value"]), + } + ) + for a in actions: + if a[0] > len(word_times) - 1: + a[0] = xSheet[-1]["time"] / 1000.0 # convert ms to seconds + else: + a[0] = (word_times[a[0]]["time"]) / 1000.0 # convert ms to seconds + for a in actions: + args = a[2] + if a[1] == "web": + data.append( + { + "start": float(a[0]) + + 0.01, # prevent visemes and actions from being at exactly the same time + "type": "web", + "args": args, + "id": a[1], + } + ) # End edits + else: + data.append( + { + "start": float(a[0]) + + 0.01, # prevent visemes and actions from being at exactly the same time + "type": "action", + "args": args, + "id": a[1], + } + ) # End edits + visemes = list( + map( + lambda l: [l["time"], self.vis_transl[l["value"]]], + filter(lambda l: l["type"] == "viseme", xSheet), + ) + ) + for v in visemes: + data.append( + { + "start": float(v[0]) / 1000.0, # convert ms to seconds + "type": "viseme", + "id": v[1], + } + ) + return data + + def _get_audio(self, response): + """[summary] + This function writes the audio file getting data from Polly + Args: + response (obj): response from amazon Polly for getting audio data + + Returns: + data: audio data + """ + data = {} + data["file"] = self.outdir + "/tts.ogg" + if "AudioStream" in response: + with closing(response["AudioStream"]) as stream: + output = data["file"] + try: + with open(output, "wb") as file: + file.write(stream.read()) + except IOError as error: + print(error) + else: + print("Could not stream audio") + return data + + def _get_response(self, behavior_data): + """[summary] + + Args: + behavior_data (json): json containing behavior data (visemes, facial expressions, and gestures) and audio data (audio_frame, and audio_data) + + Returns: + response (str): it is a object stringified which contained information about + audio_frame (int) + audio_data (str): string of audio data array + behavior_data (str): string of behaviors + """ + behaviours = list(sorted(behavior_data, key=lambda i: i["start"])) + data, samplerate = sf.read(self.outdir + "/tts.ogg") + sf.write(self.outdir + "/tts.wav", data, samplerate) + file_handle = self.outdir + "/tts.wav" + data = np.fromfile(file_handle, np.uint8)[ + self.wav_header_length : + ] # Loading wav file + data = data.astype(np.uint8).tostring() + data_array = data + audio_frame = samplerate + response = { + "audio_frame": audio_frame, + "audio_data": data_array, + "behavior_data": str(behaviours), + } + return str(response) + + def request(self, input_text): + """[summary] + + Args: + input_text (str): Input string to synthetize + Returns: + object: It containes information about the response received (bool) and response message (str) + response: bool + message: str + """ + rospy.loginfo("Start the %s request" % self.name) + self.state = State.REQUEST + text = input_text + [text, actions] = self._get_text_and_actions(text) + try: + text = ( + '' + + text + + "" + ) + json_response = self.tts.synthesize_speech( + Text=text, + TextType="ssml", + OutputFormat="json", + VoiceId=self.voice, + SpeechMarkTypes=["viseme", "word"], + ) + behavior_data = self._get_behaviors(json_response, actions) + ogg_response = self.tts.synthesize_speech( + Text=text, + TextType="ssml", + OutputFormat="ogg_vorbis", + VoiceId=self.voice, + ) + audio_data = self._get_audio(ogg_response) + tts_response = self._get_response(behavior_data) + self.state = State.SUCCESS + self.response_received = True + self.result_msg = tts_response + rospy.loginfo("Request successfully completed") + except (BotoCoreError, ClientError) as error: + rospy.logerr("The erros is " + str(error)) + self.state = State.FAILED + self.response_received = True + self.result_msg = "" + return {"response": self.state, "message": self.result_msg} + + +def main(): + """[summary] + Main function for starting HarmoniPolly service + """ + service_name = ActuatorNameSpace.tts.name + instance_id = rospy.get_param("instance_id") + service_id = f"{service_name}_{instance_id}" + try: + rospy.init_node(service_name) + + param = rospy.get_param(service_name + "/" + instance_id + "_param/") + + s = AWSTtsService(service_id, param) + + service_server = HarmoniServiceServer(service_id, s) + + service_server.start_sending_feedback() + rospy.spin() + except rospy.ROSInterruptException: + pass + + +if __name__ == "__main__": + main() diff --git a/harmoni_actuators/harmoni_tts/config/configuration.yaml b/harmoni_actuators/harmoni_tts/config/configuration.yaml index d86c8964..332622fb 100644 --- a/harmoni_actuators/harmoni_tts/config/configuration.yaml +++ b/harmoni_actuators/harmoni_tts/config/configuration.yaml @@ -1,4 +1,3 @@ -# Configuration file for the microphone tts: default_param: region_name: "us-west-2" diff --git a/harmoni_actuators/harmoni_tts/nodes/aws_tts_service.py b/harmoni_actuators/harmoni_tts/nodes/aws_tts_service.py index 4f3077fc..5242e182 100755 --- a/harmoni_actuators/harmoni_tts/nodes/aws_tts_service.py +++ b/harmoni_actuators/harmoni_tts/nodes/aws_tts_service.py @@ -22,7 +22,7 @@ import sys -class AWSTtsService(HarmoniServiceManager): +class AWS_TTS_Service(HarmoniServiceManager): """ Amazon tts service """ @@ -63,6 +63,7 @@ def _setup_aws_tts(self): "o": "OPEN_BACK_VOWEL", "O": "OPEN_BACK_VOWEL", "sil": "IDLE", + "J": "PALATAL", } return @@ -311,7 +312,7 @@ def main(): param = rospy.get_param(service_name + "/" + instance_id + "_param/") - s = AWSTtsService(service_id, param) + s = AWS_TTS_Service(service_id, param) service_server = HarmoniServiceServer(service_id, s) diff --git a/harmoni_actuators/harmoni_tts/temp_data/tts.wav b/harmoni_actuators/harmoni_tts/temp_data/tts.wav index ff722f9b..91ce207b 100644 Binary files a/harmoni_actuators/harmoni_tts/temp_data/tts.wav and b/harmoni_actuators/harmoni_tts/temp_data/tts.wav differ diff --git a/harmoni_actuators/harmoni_web/web/assets/imgs/test_1.jpg b/harmoni_actuators/harmoni_web/web/assets/imgs/test_1.jpg new file mode 100755 index 00000000..9d9aab0a Binary files /dev/null and b/harmoni_actuators/harmoni_web/web/assets/imgs/test_1.jpg differ diff --git a/harmoni_actuators/harmoni_web/web/assets/imgs/test_2.jpg b/harmoni_actuators/harmoni_web/web/assets/imgs/test_2.jpg new file mode 100755 index 00000000..9d9aab0a Binary files /dev/null and b/harmoni_actuators/harmoni_web/web/assets/imgs/test_2.jpg differ diff --git a/harmoni_actuators/harmoni_web/web/assets/imgs/test_3.jpg b/harmoni_actuators/harmoni_web/web/assets/imgs/test_3.jpg new file mode 100755 index 00000000..9d9aab0a Binary files /dev/null and b/harmoni_actuators/harmoni_web/web/assets/imgs/test_3.jpg differ diff --git a/harmoni_actuators/harmoni_web/web/src/config/config.json b/harmoni_actuators/harmoni_web/web/src/config/config.json index a7f00eef..5817fa5c 100644 --- a/harmoni_actuators/harmoni_web/web/src/config/config.json +++ b/harmoni_actuators/harmoni_web/web/src/config/config.json @@ -1,5 +1,32 @@ { "pageContent": [ + { + "component": "container", + "id": "questions_container", + "children": [ + { + "component": "title", + "children": "Domanda", + "id": "title" + }, + { + "component": "row", + "id": "row", + "children": [ + { + "component": "img", + "children": "../assets/imgs/test.jpg", + "id": "img_1" + }, + { + "component": "img", + "children": "../assets/imgs/test.jpg", + "id": "img_2" + } + ] + } + ] + }, { "component": "container", "id": "test_container", diff --git a/harmoni_actuators/harmoni_web/web/src/css/style.css b/harmoni_actuators/harmoni_web/web/src/css/style.css index db0077a4..a242446f 100755 --- a/harmoni_actuators/harmoni_web/web/src/css/style.css +++ b/harmoni_actuators/harmoni_web/web/src/css/style.css @@ -8,7 +8,7 @@ body { font-family: "Palatino Linotype", "Book Antiqua", Palatino, serif; text-align: center; font-size: 2vmin; - background-color: green; + background-color: rgb(255, 240, 173); } .container{ diff --git a/harmoni_core/harmoni_common_lib/src/harmoni_common_lib/constants.py b/harmoni_core/harmoni_common_lib/src/harmoni_common_lib/constants.py index c915fa7a..0bc8afdd 100755 --- a/harmoni_core/harmoni_common_lib/src/harmoni_common_lib/constants.py +++ b/harmoni_core/harmoni_common_lib/src/harmoni_common_lib/constants.py @@ -44,7 +44,7 @@ class ActuatorNameSpace(Enum): tts = "/harmoni/actuating/tts/" web = "/harmoni/actuating/web/" gesture = "/harmoni/actuating/gesture/" - + hass = "/harmoni/actuating/hass/" class DialogueNameSpace(Enum): bot = "/harmoni/dialoging/bot/" diff --git a/harmoni_core/harmoni_decision/config/configuration.yaml b/harmoni_core/harmoni_decision/config/configuration.yaml index 069a083b..dc192df4 100644 --- a/harmoni_core/harmoni_decision/config/configuration.yaml +++ b/harmoni_core/harmoni_decision/config/configuration.yaml @@ -5,8 +5,9 @@ harmoni: web: ["default"] stt: ["default"] #face_detect: ["default"] + hass: ["default"] hardware: - microphone: ["default"] + #microphone: ["default"] #camera: ["default"] face: ["default"] speaker: ["default"] diff --git a/harmoni_core/harmoni_pattern/CMakeLists.txt b/harmoni_core/harmoni_pattern/CMakeLists.txt index 08779c06..20b3352d 100644 --- a/harmoni_core/harmoni_pattern/CMakeLists.txt +++ b/harmoni_core/harmoni_pattern/CMakeLists.txt @@ -19,7 +19,7 @@ find_package(catkin REQUIRED COMPONENTS ## Uncomment this if the package has a setup.py. This macro ensures ## modules and global scripts declared therein get installed ## See http://ros.org/doc/api/catkin/html/user_guide/setup_dot_py.html -#catkin_python_setup() +catkin_python_setup() ################################################ ## Declare ROS messages, services and actions ## @@ -158,9 +158,8 @@ include_directories( ## Mark executable scripts (Python etc.) for installation ## in contrast to setup.py, you can choose the destination -# install(PROGRAMS -# scripts/my_python_script -# DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION} +# install(PROGRAMS nodes/sequential_pattern +# DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION} # ) ## Mark executables and/or libraries for installation @@ -184,6 +183,10 @@ include_directories( # DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION} # ) +install(DIRECTORY nodes + DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION} +) + ############# ## Testing ## ############# diff --git a/harmoni_core/harmoni_pattern/config/configuration.yaml b/harmoni_core/harmoni_pattern/config/configuration.yaml index 52b6069a..897e1260 100644 --- a/harmoni_core/harmoni_pattern/config/configuration.yaml +++ b/harmoni_core/harmoni_pattern/config/configuration.yaml @@ -28,3 +28,19 @@ speak_test: default_param: trigger_intent: "Hey" pattern_scripting: $(find harmoni_pattern)/pattern_scripting/speak_test.json + + +system_initiative: + default_param: + trigger_intent: "{ \"action\":\"check_log\", \"entity\":\"googlehome8554\", \"type\":\"media_player\"}" + pattern_scripting: $(find harmoni_pattern)/pattern_scripting/system_initiative.json + +simple_dialogue: + default_param: + trigger_intent: "hey" + pattern_scripting: $(find harmoni_pattern)/pattern_scripting/simple_dialogue.json + +hass: + default_param: + trigger_intent: "{ \"action\":\"check_log\", \"entity\":\"googlehome8554\", \"type\":\"media_player\"}" + pattern_scripting: $(find harmoni_pattern)/pattern_scripting/hass.json diff --git a/harmoni_core/harmoni_pattern/launch/sequence_pattern.launch b/harmoni_core/harmoni_pattern/launch/sequence_pattern.launch index ecfb247e..e3d43977 100644 --- a/harmoni_core/harmoni_pattern/launch/sequence_pattern.launch +++ b/harmoni_core/harmoni_pattern/launch/sequence_pattern.launch @@ -3,7 +3,7 @@ - + diff --git a/harmoni_core/harmoni_pattern/nodes/__init__.py b/harmoni_core/harmoni_pattern/nodes/__init__.py new file mode 100755 index 00000000..5f7ce86a --- /dev/null +++ b/harmoni_core/harmoni_pattern/nodes/__init__.py @@ -0,0 +1 @@ +#!/usr/bin/env python3 \ No newline at end of file diff --git a/harmoni_core/harmoni_pattern/nodes/sequential_pattern.py b/harmoni_core/harmoni_pattern/nodes/sequential_pattern.py index 5de157f6..d4f71672 100755 --- a/harmoni_core/harmoni_pattern/nodes/sequential_pattern.py +++ b/harmoni_core/harmoni_pattern/nodes/sequential_pattern.py @@ -19,6 +19,7 @@ from collections import deque from time import time import threading +# import ast class SequentialPattern(HarmoniServiceManager): @@ -142,6 +143,9 @@ def _feedback_callback(self, feedback): def _detecting_callback(self, data, service_name): """Store data from detection to client_results dictionary""" data = data.data + + rospy.logdebug("Received data from detector " + data) + self.client_results[service_name].append({"time": time(), "data": data}) return @@ -171,6 +175,61 @@ def start(self): r.sleep() return + def request(self, data): + """Send goal request to appropriate child + + Returns: + list: names of all the services + """ + rospy.loginfo("Start the %s request" % self.name) + rospy.loginfo(data) + # if isinstance(data, str): + # data = ast.literal_eval(data) + # data = json.loads(data) + self.state = State.REQUEST + r = rospy.Rate(10) + + # Resetting the index so that the script can run again + self.script_set_index = 0 + # rospy.loginfo(self.script_set_index) + + while self.script_set_index < len(self.script) and not rospy.is_shutdown(): + if self.script[self.script_set_index]["set"] == "setup": + self.setup_services(self.script[self.script_set_index]["steps"]) + + elif self.script[self.script_set_index]["set"] == "sequence": + #self.count = -1 + data = self.do_steps(self.script[self.script_set_index]["steps"], data=data) + + elif self.script[self.script_set_index]["set"] == "loop": + #self.count = -1 + data = self.do_steps( + self.script[self.script_set_index]["steps"], looping=True, data=data + ) + # elif self.end_pattern: + ##TODO + # break + self.script_set_index += 1 + r.sleep() + rospy.loginfo("_________SEQUENCE PATTERN END__________") + prepared = [dict(zip(cl, self.client_results[cl])) for cl in self.client_results] + + + # rospy.loginfo("____SHOW_ALL_THE_SEQUENCE_PATTERN_MESSAGES___") + # for cl in self.client_results: + # rospy.loginfo("_________________") + # rospy.loginfo(cl) + # rospy.loginfo(self.client_results[cl]) + # rospy.loginfo("_____________________________________________") + + # if data is not None and len(data) < 500: + # rospy.loginfo("Data received from bot_default -> "+ data) + + j = json.dumps(prepared) + result_msg = str(j) + self.state = State.SUCCESS + return result_msg + def stop(self): """Stop the Pattern Player """ try: @@ -229,7 +288,7 @@ def setup_services(self, setup_steps): return - def do_steps(self, sequence, looping=False): + def do_steps(self, sequence, looping=False, data=None): """Directs the services to do each of the steps scripted in the sequence Args: @@ -237,19 +296,29 @@ def do_steps(self, sequence, looping=False): looping (bool, optional): If true will loop the sequence indefinitely. Defaults to False. """ - passthrough_result = None for cnt, step in enumerate(sequence, start=1): if rospy.is_shutdown(): return rospy.loginfo(f"------------- Starting sequence step: {cnt}-------------") - if passthrough_result: - rospy.loginfo(f"with prior result length ({len(passthrough_result)})") + if data: + rospy.loginfo(f"with prior result length ({len(data)})") else: rospy.loginfo("no prior result") - passthrough_result = self.handle_step(step, passthrough_result) + data_to_save = "" + + data = self.handle_step(step, data) + + # Save data and return it, if it comes from a bot or from hass + # TODO save for all data from services (maybe if it has a flag) + if next(iter(step)) == "bot_default" or next(iter(step)) == "hass_default": + data_to_save = data + # rospy.loginfo("Data from bot_default "+ data) + self.client_results[next(iter(step))].append( + {"time": time(), "data": data} + ) rospy.loginfo(f"************* End of sequence step: {cnt} *************") @@ -258,7 +327,8 @@ def do_steps(self, sequence, looping=False): if not rospy.is_shutdown(): self.do_steps(sequence, looping=True) - return + ## This function now returns data + return data_to_save def handle_step(self, step, optional_data=None): """Handle cases for different types of steps diff --git a/harmoni_core/harmoni_pattern/pattern_scripting/hass.json b/harmoni_core/harmoni_pattern/pattern_scripting/hass.json new file mode 100644 index 00000000..dc844fb4 --- /dev/null +++ b/harmoni_core/harmoni_pattern/pattern_scripting/hass.json @@ -0,0 +1,15 @@ +[ + + { + "set": "sequence", + "steps": [ + { + "hass_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new" + } + } + ] + } +] \ No newline at end of file diff --git a/harmoni_core/harmoni_pattern/pattern_scripting/simple_dialogue.json b/harmoni_core/harmoni_pattern/pattern_scripting/simple_dialogue.json new file mode 100644 index 00000000..5c971f16 --- /dev/null +++ b/harmoni_core/harmoni_pattern/pattern_scripting/simple_dialogue.json @@ -0,0 +1,63 @@ +[ + { + "set": "setup", + "steps": [ + { + "microphone_default": { + "action_goal": "ON", + "resource_type": "sensor", + "wait_for": "" + } + }, + { + "stt_default": { + "action_goal": "ON", + "resource_type": "detector", + "wait_for": "" + } + } + ] + }, + { + "set": "sequence", + "steps": [ + { + "tts_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new" + } + }, + [ + { + "speaker_default": { + "action_goal": "DO", + "resource_type": "actuator", + "wait_for": "new" + } + }, + { + "face_mouth_default": { + "action_goal": "DO", + "resource_type": "actuator", + "wait_for": "new" + } + } + ], + { + "stt_default": { + "resource_type": "detector", + "wait_for": "new" + } + }, + { + "bot_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new" + } + } + + ] + } +] \ No newline at end of file diff --git a/harmoni_core/harmoni_pattern/pattern_scripting/system_initiative.json b/harmoni_core/harmoni_pattern/pattern_scripting/system_initiative.json new file mode 100644 index 00000000..86bb4e07 --- /dev/null +++ b/harmoni_core/harmoni_pattern/pattern_scripting/system_initiative.json @@ -0,0 +1,85 @@ +[ + + { + "set": "sequence", + "steps": [ + { + "hass_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new", + "trigger": "{ \"action\":\"check_log\", \"entity\":\"oven_power\", \"type\":\"switch\"}" + } + }, + { + "bot_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new" + } + }, + { + "tts_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new" + } + }, + [ + { + "speaker_default": { + "action_goal": "DO", + "resource_type": "actuator", + "wait_for": "" + } + }, + { + "face_mouth_default": { + "action_goal": "DO", + "resource_type": "actuator", + "wait_for": "new" + } + } + ], + { + "bot_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new", + "trigger": "sì, grazie" + } + }, + { + "tts_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new" + } + }, + [ + { + "speaker_default": { + "action_goal": "DO", + "resource_type": "actuator", + "wait_for": "" + } + }, + { + "face_mouth_default": { + "action_goal": "DO", + "resource_type": "actuator", + "wait_for": "new" + } + } + ], + { + "hass_default": { + "action_goal": "REQUEST", + "resource_type": "service", + "wait_for": "new", + "trigger": "{ \"action\":\"turn_off\", \"entity\":\"oven_power\", \"type\":\"switch\"}" + } + } + ] + } +] \ No newline at end of file diff --git a/harmoni_core/harmoni_pattern/setup.py b/harmoni_core/harmoni_pattern/setup.py index e1cd3c49..b5841715 100644 --- a/harmoni_core/harmoni_pattern/setup.py +++ b/harmoni_core/harmoni_pattern/setup.py @@ -7,7 +7,7 @@ setup_args = generate_distutils_setup( # scripts=[''], # packages=['harmoni_pattern'], - # package_dir={'': 'src'}, + # package_dir={'': 'nodes'}, ) setup(**setup_args) diff --git a/harmoni_detectors/harmoni_stt/config/google_configuration.yaml b/harmoni_detectors/harmoni_stt/config/google_configuration.yaml index 453daae5..61005443 100644 --- a/harmoni_detectors/harmoni_stt/config/google_configuration.yaml +++ b/harmoni_detectors/harmoni_stt/config/google_configuration.yaml @@ -10,4 +10,4 @@ stt: sample_rate: 16000 #44100 for wav audio file audio_channel: 1 subscriber_id: "default" - credential_path: "$(env HOME)/.gcp/private-keys.json" #path where private keys are mounted + credential_path: "$(env HOME)/.gcp/private-keys.json" #path where private keys are mounted \ No newline at end of file diff --git a/harmoni_detectors/harmoni_stt/nodes/google_service.py b/harmoni_detectors/harmoni_stt/nodes/google_service.py index 5a02ee21..527d87ba 100755 --- a/harmoni_detectors/harmoni_stt/nodes/google_service.py +++ b/harmoni_detectors/harmoni_stt/nodes/google_service.py @@ -214,4 +214,4 @@ def main(): if __name__ == "__main__": - main() + main() \ No newline at end of file diff --git a/harmoni_detectors/harmoni_stt/test/google_stt.test b/harmoni_detectors/harmoni_stt/test/google_stt.test new file mode 100644 index 00000000..b28c3edd --- /dev/null +++ b/harmoni_detectors/harmoni_stt/test/google_stt.test @@ -0,0 +1,11 @@ + + + + + + + + + + + diff --git a/harmoni_detectors/harmoni_stt/test/test_google_stt.py b/harmoni_detectors/harmoni_stt/test/test_google_stt.py new file mode 100755 index 00000000..bc9799c7 --- /dev/null +++ b/harmoni_detectors/harmoni_stt/test/test_google_stt.py @@ -0,0 +1,129 @@ +#!/usr/bin/env python3 + + +PKG = "test_harmoni_stt" +# Common Imports +import unittest, rospy, roslib, sys + +# Specific Imports +from actionlib_msgs.msg import GoalStatus +from harmoni_common_lib.action_client import HarmoniActionClient +from harmoni_common_lib.constants import ( + DetectorNameSpace, + SensorNameSpace, + ActionType, + State, +) +from harmoni_common_msgs.msg import harmoniAction, harmoniFeedback, harmoniResult +from audio_common_msgs.msg import AudioData +from std_msgs.msg import String +import time +import os, io +from google.cloud import speech + +class TestGoogleStt_Common(unittest.TestCase): + + + + def wav_to_data(self, path): + with io.open(path, "rb") as f: + content = f.read() + return content + + def setUp(self): + self.feedback = State.INIT + self.result = False + self.test_file = rospy.get_param("test_google_stt_input") + self.audio = self.wav_to_data(self.test_file) + rospy.init_node("test_google_stt", log_level=rospy.INFO) + self.rate = rospy.Rate(20) + self.output_sub = rospy.Subscriber( + "/harmoni/detecting/stt/default", String, self._detecting_callback + ) + + # provide mock microphone + self.audio_pub = rospy.Publisher( + SensorNameSpace.microphone.value + "default", + AudioData, + queue_size=10, + ) + rospy.Subscriber( + DetectorNameSpace.stt.value + "stt_default", + String, + self.text_received_callback, + ) + + # startup stt node + self.server = "stt_default" + self.client = HarmoniActionClient(self.server) + self.client.setup_client( + self.server, self._result_callback, self._feedback_callback, wait=True + ) + rospy.loginfo("TestGoogleStt: Turning ON stt server") + self.client.send_goal(action_goal=ActionType.ON, optional_data="Setup", wait=False) + + time.sleep(3) + # self.client.send_goal( + # action_goal=ActionType.REQUEST.value, + # wait=False + # ) + rospy.loginfo("TestGoogleStt: Started up. waiting for google stt startup") + + # wait for start state + #while not rospy.is_shutdown() and self.feedback != State.START: + # self.rate.sleep() + + rospy.loginfo("TestGoogleStt: publishing audio") + + self.audio_pub.publish(self.audio) + self.audio_pub.publish(self.audio[:14000]) + + rospy.loginfo( + f"TestGoogleStt: audio subscribed to by #{self.output_sub.get_num_connections()} connections." + ) + + time.sleep(5) + + def _feedback_callback(self, data): + rospy.loginfo(f"TestGoogleStt: Feedback: {data}") + self.feedback = data["state"] + + def _status_callback(self, data): + rospy.loginfo(f"TestGoogleStt: Status: {data}") + self.result = True + + def _result_callback(self, data): + rospy.loginfo(f"TestGoogleStt: Result: {data}") + self.result = True + + def text_received_callback(self, data): + rospy.loginfo(f"TestGoogleStt: Text back: {data}") + self.result = True + + def _detecting_callback(self, data): + rospy.loginfo(f"TestGoogleStt: Detecting: {data}") + self.result = True + + +class TestGoogleStt_Valid(TestGoogleStt_Common): + def test_IO(self): + rospy.loginfo( + "TestGoogleStt[TEST]: basic IO test to ensure data " + + "('hello' audio) is received and responded to. Waiting for transcription..." + ) + while not rospy.is_shutdown() and not self.result: + self.rate.sleep() + assert self.result == True + + +def main(): + # TODO combine validity tests into test suite so that setup doesn't have to run over and over. + import rostest + + rospy.loginfo("test_google_stt started") + rospy.loginfo("TestGoogleStt: sys.argv: %s" % str(sys.argv)) + rostest.rosrun(PKG, "test_google_stt", TestGoogleStt_Valid, sys.argv) + + +if __name__ == "__main__": + main() diff --git a/harmoni_sensors/harmoni_camera/temp_data/test_example.png b/harmoni_sensors/harmoni_camera/temp_data/test_example.png deleted file mode 100644 index 78c5edc9..00000000 Binary files a/harmoni_sensors/harmoni_camera/temp_data/test_example.png and /dev/null differ diff --git a/harmoni_sensors/harmoni_microphone/config/configuration.yaml b/harmoni_sensors/harmoni_microphone/config/configuration.yaml index f09160d7..2626ec5d 100755 --- a/harmoni_sensors/harmoni_microphone/config/configuration.yaml +++ b/harmoni_sensors/harmoni_microphone/config/configuration.yaml @@ -5,5 +5,5 @@ microphone: chunk_size: 1024 total_channels: 1 audio_rate: 16000 - device_name: default + device_name: "default" test_outdir: "$(find harmoni_microphone)/temp_data/test_example.wav" diff --git a/harmoni_sensors/harmoni_microphone/nodes/microphone_service.py b/harmoni_sensors/harmoni_microphone/nodes/microphone_service.py index f26c1093..274ae990 100755 --- a/harmoni_sensors/harmoni_microphone/nodes/microphone_service.py +++ b/harmoni_sensors/harmoni_microphone/nodes/microphone_service.py @@ -19,13 +19,10 @@ class MicrophoneService(HarmoniServiceManager): """Reads from a microphone and publishes audio data. - As a sensor service, the microphone is responsible for reading the audio data from a physical microphone and publishing it so that it can be recorded or transcribed by a detector. - The microphone has many parameters which are set in the configuration.yaml - The public functions exposed by the microphone include start(), stop(), and pause() """ @@ -124,7 +121,6 @@ def _close_stream(self): def _read_stream_and_publish(self): """Continously publish audio data from the microphone - While state is START publish audio """ r = rospy.Rate(10) @@ -166,7 +162,6 @@ def _get_device_index(self): def start_recording_data(self): """Init the subscriber to microphone/default for recording audio. - The callback in the subscriber will save the audio to a file specified in the configuration yaml. """ diff --git a/harmoni_sensors/harmoni_microphone/test/rostest_microphone.py b/harmoni_sensors/harmoni_microphone/test/rostest_microphone.py index ea622767..20880f56 100755 --- a/harmoni_sensors/harmoni_microphone/test/rostest_microphone.py +++ b/harmoni_sensors/harmoni_microphone/test/rostest_microphone.py @@ -76,7 +76,7 @@ def test_request_response(self): # optional_data=self.data, wait=False, ) - rospy.sleep(1) + rospy.sleep(4) assert self.result == True, "Mic should be publishing by now" # TODO: Fix preempt requests so stop can interrupt start rospy.loginfo("Next test by sending the 'off' goal")