Train robot - supervised vision

MyGym enables you to use pre-trained vision models to extend the versatility of your training scenarios. Each workspace has five virtual cameras, that observe the scene. You can use images from a camera and pretrained YOLACT model to do image segmentation. This way you retrieve information about position of robot and task objects using camera images instead of using ground truth. The reward for training will be calculated based on data returned by vision.

training yolact

To train a robot using image segmentation set in the config file:

"reward_type": "3dvs",

and specify the path to YOLACT preptrained model and model config:

"yolact_path":"trained_models/weights_yolact_mygym_23/crow_base_15_266666.pth", "yolact_config":"trained_models/weights_yolact_mygym_23/config_train_15.obj",

Note

If you do not have myGym’s pretrained vision model, download it first:

cd myGym

sh download_vision.sh

We recommed to turn the visgym parameter off:

"visgym": 0",

You can visualize how the vision performs:

"visualize":1,

Start the training with this modified config file:

python train.py --config my_config.json

Alternatively, you can specify the above parameters in the command line:

python train.py --reward_type=3dvs
--yolact_path=trained_models/weights_yolact_mygym_23/crow_base_15_266666.pth
--yolact_config=trained_models/weights_yolact_mygym_23/config_train_15.obj
--visgym=0 --visualize=1

After the training finishes, find your model in the logdir.

To learn more about myGym’s vision models, see Vision.