To open any entity form from custom button added in entity grid ribbon just add following code in action section.
when pass Entity Object Type code as a value to IntParameter, then CRM will open that entity form.
e.g. If wants to open account form from custom button shown on Activity grid ribbon, then use
IntParameter Value="1"